Ask the people in your office what they think of the ERP, and you will hear the same few words: clunky, not intuitive, "I don't want to look at it all day." Owners tend to file that under grumbling. It is useful information, because those same people are the ones who will check the AI's work.
The systems we build put a named person between the draft and the customer. The AI does the first pass, and a person verifies, decides, and signs. That arrangement is only as good as the check, and the most important safety step in the whole system gets done by a tired person late in the week.
1. The review is the safety step
A model writes by predicting what should come next. When the order record and the mill cert are in front of it, the prediction matches the record. When something is missing or misread, it still writes, and a wrong heat number looks exactly as confident as a right one.
So the AI flags what it is unsure about, and a person checks each draft against the source before it leaves the building. That check is where a swapped digit on a heat number gets caught. Take it away, or let it turn into a glance and a click, and the rest of the system is a faster way to ship a mistake.
2. People skip checks that are slow
Picture what checking one value takes in many offices today. The reviewer reads a heat number on the draft, opens the shared drive, searches for the mill test report, waits for the PDF to load, zooms in, finds the number, and switches back. Then the next value, and the next draft.
At 9 a.m. on a Monday, a careful person does that every time. At 3 p.m. on a Friday, with the pile still high and a truck waiting, they start to trust the draft. They skim, or they approve the ones that look right. Or they stop using the tool and go back to typing it themselves, because at least they know where they stand.
We call this the Friday afternoon test: would a tired person, late on a Friday with the pile still high, still do the review properly in this software? If the honest answer is no, the review will be skipped exactly when it matters most.
Our view: this is why design belongs in the safety conversation. Design is how the person using the software feels at 3 p.m. on a Friday. How it looks comes second.
3. Make the right check the easy one
Your shop floor already knows this idea under another name. Poka-yoke means mistake-proofing, like the fixture that only takes the part the right way round, so doing it right is also the easiest thing to do. Nobody relies on a machinist being extra careful at the end of a double shift. The fixture does part of the remembering.
Review software can work the same way. When the source document is one step away, checking a value takes less effort than skipping it. When the source is five clicks and a second program away, skipping it wins on a Friday, every time.
We did not start in manufacturing. We started in AI and software for healthcare and consumer apps, where people delete an app that annoys them. That habit came with us: we treat the reviewer's effort as a cost to cut, the same way you would treat setup time on a machine.
4. One example: a file preview on hover
A small example from a recent client shows the idea. Every time the AI works from a file, the draft shows a link to that file. The reviewer does not have to click it or open another program. They hover over the link, and a large preview of the document appears right there, so they can check the value against the source in a glance.
It looks like a minor feature. Go back to the Monday-morning routine from section 2 and count what it removes: the search, the second program, the switch back. The check that remains is reading one number against one page, which a tired person will still do on a Friday. Multiply that by every value on every draft, and it is the difference between a review people do and one they learn to skip.
5. How to run the test on any tool
You do not need to know anything about AI to judge review software. You need the person who will do the reviewing, ten real drafts, and a Friday afternoon. Sit next to them, do not help, and watch for five things:
- Can they see the source file for a value without leaving the screen? Count the clicks and the programs it takes.
- Can they tell which values the AI was unsure about, or does every field look equally confident?
- Can they fix a wrong value where they see it, or do they have to go and edit it somewhere else?
- Does the tool record who approved each draft, and can they see the steps the AI took to produce it?
- How long did the ten drafts take, compared with doing them by hand? If checking is slower than typing, checking will lose.
A tool that passes all five is one people will keep using properly on a bad week. A tool that fails the first one will be skimmed on a Friday, whatever the demo looked like on a Tuesday morning.
What to do this week
- Ask the person who would review AI drafts to show you how they check one value today, from the draft to the source and back. Count the clicks and the programs.
- Run the Friday afternoon test on the tool your team already uses most for checking work, even if that is the ERP.
- The next time a vendor demos AI to you, ask to see the review screen instead of the AI, and bring ten of your own documents.
- Set one rule for your reviewers: nobody approves a value they could not see the source for.
This essay comes from Manufacturing's ChatGPT Moment Hasn't Happened Yet: Why AI Fits Your Paperwork and How to Prepare. For how the reviewer's job changes and how to train for it, read Verify, Don't Retype: How Your Team's Job Changes When AI Drafts First.
Questions people ask
Why does software design matter for AI safety?
The safety step is a person checking the AI's draft against the source. If that check means opening other programs and hunting for files, people skim it or skip it, especially late in a busy week. A review screen that puts the source file one hover away keeps the check fast enough that people do it every time.
What should a good screen for reviewing AI work let a person do?
See the source file for a value without leaving the screen, tell which values the AI was unsure about, fix a wrong value in place, and see who approved the draft and what steps the AI took.
Terms in this piece
- Friday afternoon test
- whether a tired person, late on a Friday with the pile still high, would still review the AI's work properly in a piece of software.
- First pass
- the draft the AI prepares, such as a cert packet or a quote, before a person checks it.
- Reviewer
- the named person who checks the AI's work and approves it before it counts.
- Verification
- a person checking the AI's work against the source documents and correcting what is wrong before signing off.
- Activity feed
- the step-by-step record of everything the AI did, with times and sources, that a reviewer can read.
