Three years of AI features in factory software: how I think about agents

The experiences here are abstracted. I can’t go into the specific implementation, so this is only about approach and lessons.

For the past few years I worked at a company that makes software for factories, and built several AI-related features along the way. At the time I never thought about which “paradigm” they belonged to. Looking back, they happen to line up from simple to complex. This post first lays out the forms I tried, then compares them with the approaches people commonly talk about now, and finally talks about memory: how to make an AI tool more accurate the more it’s used.

Part 1: The forms I tried

1. A single call: turning a passage into structured data

The first thing I built turned a voice recording into a list of tasks and filled them into a task system automatically. At heart it’s structuring text: the model runs a prompt once, extracts and arranges the information, and outputs some JSON. No loop, no tools.

For many scenarios this one step is enough. There’s no need to jump straight to an agent.

2. Routing + converting to parameters: querying data in natural language

Later I built an assistant that let factory owners query production data in natural language. It was built on a low-code AI workflow platform, one step per node: first decide which business module the question belongs to, then extract the query conditions, and finally convert them into the parameters the system’s API expects. The model’s only job was to turn human language into a consistent structure; the query itself still went through the system’s API.

This kind of approach can only answer the few kinds of questions designed in advance. And the hard part isn’t AI, it’s “what does the user actually want to see?”

3. One extraction + human confirmation: recognising handwritten forms

Much of the information on the shop floor is still on paper. Take a photo, have the model extract it into a fixed JSON format, find the matching records in the system, and finally give the user a form to confirm.

A small trick: handwriting is often hard to read, so put this customer’s commonly used options in the prompt and let the model read against them. Accuracy goes up a lot.

4. A general agent: an assistant for implementation colleagues

Later I built an agent for implementation and customer service colleagues: upload a customer’s materials, or describe the need in a sentence, and it helps configure the system, write formulas, write spreadsheet macros, generate print templates, and even generate page plugins.

How it was built

I built a simple agent harness with the Vercel AI SDK: it could upload files, run code in a sandbox (E2B), load skills, and connect to a few of the system’s tools. The features weren’t written one by one; they were the same harness with different skills and tools.

For getting files in and out of the sandbox, I used two small tricks:

• Upload: images go straight to the model; attachments like spreadsheets only have their links appended to the end of the message, and when the model thinks it needs them, it uses a tool to move the file into the sandbox and read it. That way the sandbox doesn’t have to start for every conversation. Anthropic’s code execution tool takes another approach: referenced files are put into the sandbox automatically.¹ One puts them in by default; the other lets the model fetch them as needed.
• Download: generated files that need to reach the user were also a tool, called by the AI when needed. Anthropic’s approach is simpler: the sandbox has an agreed output directory, and files put there are collected and stored automatically.¹

Looking back, the difference is where the “contract” lives. They made “files in the output directory are for the user” a system rule; I told the AI in the prompt “call this tool to give the user a file”, and if the AI forgot, the file never reached the user. Important conventions and state are best remembered by the system, not left to the model.

Writing formulas: give it tools to look things up and validate

At first, formulas were written with just a prompt. It could write some, but accuracy couldn’t be guaranteed. Inside the agent the approach changed: a skill on how to write each kind of formula, plus two tools, one to look up which operators and parameters formulas support, and one to validate whether a formula is legal. If not, it returns an error, and the agent takes the error and fixes it itself until it passes.

But correct syntax doesn’t mean a correct result. To confirm the formula computes what the customer wants, it’s best to have the user provide a set of inputs, actually run it in the system, and compare the output.

Writing spreadsheet macros: spell out inputs and outputs in the skill

At first the AI fetched data by fixed column numbers, and all sorts of things went wrong. I first talked it through with the AI round after round until we had a version that worked, then had it turn the whole conversation into a skill. The key in the skill is to spell out two things: what customers’ spreadsheets usually look like and what the final output should look like; and which tables the data source has and which fixed fields their headers contain. Then the AI finds columns by field name instead of guessing column numbers.

Another lesson: often what held me up wasn’t technology, but rushing the AI to write before I’d worked out the requirements myself. If the requirements aren’t clear, it doesn’t matter how fast AI writes.

Print templates and page plugins: the parts that can’t be verified

Print templates were generated by the AI in the sandbox with ExcelJS, and page plugins were written by the AI too. Both have the same problem: the AI can’t see the final result. What the printout looks like, whether the plugin calls the API correctly, it can’t verify either itself; only a person can try them.

Later I thought: if print templates were defined in HTML and CSS instead, the AI could render them and see the result itself. That’s changing the system to be “AI-friendly”.

Two things I figured out

First: closing the loop matters most; the AI must be able to verify its own work. What it could check and fix itself within the harness (like formula syntax) worked well; what it couldn’t had to rely on people.

Second: technically it was already very much a general agent, but what’s really valuable isn’t the framework, it’s writing the industry’s business knowledge into skills. Anyone can build a framework; business knowledge only belongs to people in the trade.

5. Agent writes conversion code + rules block bad results: spreadsheet import

Factory spreadsheets come in every shape imaginable. Small factories’ spreadsheets are mainly made to look neat and be convenient: for example, something that’s three fields in the system, they write together joined by slashes. I hit this on my first visit to a customer: their spreadsheet was completely incompatible with our import format. The customer was being trained on one side while I sat next to them writing a macro to convert their spreadsheet into a format we could import. I was anxious the whole hour.

The AI import I built later followed this idea:

• The AI writes conversion code rather than converting the data directly. It reads the customer’s spreadsheet, runs the conversion logic, and outputs the JSON the system needs.
• Rules block it before submission. Some format rules were written in the skill many times, and the AI still kept getting them wrong. So I hard-coded the rules: when the AI submits a result, it’s checked, and if it’s wrong, it’s sent back with what’s wrong, and the AI fixes it until it passes.
• Conversion plans are kept as code. The same customer’s spreadsheet format rarely changes, so next time just run that code again, and it’s converted in seconds.
• Text has to map to data in the system. What the AI extracts is only text; first check whether the system already has a matching record, and only create one if it doesn’t.
• Which errors the system handles, and which go to the customer. How text maps to system data is an error for the system to handle; problems in the customer’s spreadsheet itself, like duplicate rows, should be shown clearly to the customer so they can fix them, rather than quietly merged for them. We can’t judge whether that would be right.

Part 2: The structures people talk about now

Workflows and agents

In “Building effective agents”, Anthropic divides these systems into two kinds:²

• Workflow: the steps are set in code, and the model works at fixed points.
• Agent: the model decides what to do next and which tool to call, looping until the task is done.

Their advice is to start with the simplest approach and only add complexity when it’s truly needed. All of these are built on the same building block: a model that can call tools and retrieve information, which the article calls the “augmented LLM”.² Form 1 above is the simplest case: just this building block, called once.

The article lists five common workflows, which map neatly onto what I tried:

In real scenarios the two are often mixed: the whole thing is a workflow, and the hardest part to hard-code in the middle goes to an agent. Spreadsheet import was like that: reading the sheet and writing conversion code in the middle went to the agent; when the agent finishes, it calls a “submit” tool, which is really the workflow’s next step. If the check passes, the agent leaves its loop and enters the hard-coded process that follows; if not, the reason for the error goes back to it to keep fixing.

That was my own approach. Later I looked at how others do it, and the thinking is similar:

• HumanLayer’s “12-Factor Agents” observes that many “AI agents” that run well in production are actually mostly hard-coded, with LLM steps sprinkled in at the right places; it recommends owning your control flow and using small, focused agents.³
• The Vercel AI SDK docs have a ready-made pattern: give the agent a tool like done; when the agent calls it, the loop ends, and the final result is taken from that call’s arguments.⁴
• Pydantic AI by default also has the model call an “output tool” to hand in the result; if it fails validation, the error goes back to the model to output again.⁵

So you can think of it like this: the workflow governs the overall order and rules, and the agent only handles the hardest part in the middle to hard-code. Make the workflow’s next step one of the agent’s tools, and when the agent calls it, you move from the flexible part back to the deterministic part.

What an agent is made of

What people now call an agent is basically these pieces put together:

• The model: does the thinking. This is up to the model companies; we can’t change it.
• Tools: do the doing. Querying data, writing files, running code, calling system APIs, including all kinds of search and retrieval. Unified connections like MCP are common now. The RAG you often hear about is also a kind of retrieval tool: split material into chunks, search semantically for similar pieces, and give them to the model. It’s good for getting started when knowledge isn’t yet finely organised, but the results aren’t necessarily great.
• The loop and the shell: let the model think a step, do a step, look at the result, think the next step; the shell also handles permissions, the sandbox, and what to do when things go wrong.
• Context: what to feed the model at each step. Instruction files, skills, history, results returned by tools all live here.

What people talk about most lately is “context engineering”:⁶ when models are about equally capable, what decides the outcome is what you show them at each step and what you don’t. More context isn’t better. Putting only the customer’s common options in the prompt when recognising forms, above, is a very small piece of context engineering.

First get clear: what the model knows and what it doesn’t

Before preparing context for a model, it helps to place it first. It’s like a highly educated new colleague on their first day: they have general knowledge and professional skills, but know nothing about your company. So what you need to tell it falls into two kinds.

What it knows, help it recall. Common sense, the general flow of manufacturing and ERP, the ability to write code: the model has these. But knowing doesn’t mean it will think of them during a task. Point out the part this task needs, like “this is a bill of materials; the levels of the numbering show parent and child parts”, and it can draw on what it already knows more easily.

What it doesn’t know, spell out. What your system can do, which modules it has, what the fields mean, what rules and limits there are, how a task should be done: the model knows none of this. Systems are designed by people, and every one is different. You can organise it as “what it can do, how to use it, what the rules are, what it can’t do”. How you hand it to the model, in the prompt, as a skill, or in RAG, all work.

Skills: SOPs ready to use

A skill is a written-up way of doing something: when to use it, how to do it, what the pitfalls are, and it can include ready-written scripts to run. Normally the agent only knows each skill’s name and description, and reads the full text when it meets a related task.⁷

It’s a lot like how people work: when a newcomer doesn’t know how, they look up the SOP. An industry’s business knowledge is, after all, a set of SOPs. Sort them into folders and let the agent drill down level by level, reading only the part it needs. In one sentence: a skill is an SOP; RAG is a search tool.

Subagents

When a task is too big, the main agent hands part of it to a subagent, which completes it in its own context and hands back only the result.

An example: fixing typos in recording transcripts. With 100 files, one agent working through from start to finish gets messier and messier context as it goes. Split them into 10 batches for 10 subagents working at once, and it’s much faster. Two things are key:

• Context isolation. The main agent only gets each batch’s results, so it isn’t stuffed full of intermediate steps or led astray by other batches.⁸
• The main agent checks the work. When results come back, review them first and fix problems.

Anthropic’s deep research feature works this way: a lead agent splits the task, several subagents research in parallel, then compress their findings and hand them back. In their internal tests it did about 90 percent better than a single agent, at the cost of roughly 15 times the tokens of a normal chat.⁹

But more subagents isn’t better. When subtasks depend on each other a lot and need shared context, they’re not suited to splitting, and most coding tasks are like that.⁹ Cognition puts it more bluntly: subagents can’t see each other’s decisions and easily produce contradictory results.¹⁰ So batches that are independent and can run in parallel suit splitting; tasks that depend on each other in sequence are safer with one agent working step by step.

How to get “deterministic output”

Industrial settings need it right every time. My view:

• Have AI output structured results (like JSON) rather than change the system directly;
• Validate everything that can be validated in code, and make validation a tool the AI can call so it can check and fix its own work;
• Put AI only at the step that’s hardest to standardise;
• AI won’t necessarily follow rules written in the prompt; hard-code key rules to block submission, and tell the AI what’s wrong;
• Whatever can be kept as code, keep as code, and just run it next time;
• Have a person confirm results before writing them into the system;
• Be able to roll back when something goes wrong.

Part 3: Memory: making it more accurate the more it’s used

Let me be clear first: the model itself doesn’t get smarter with use. I took one idea from OpenClaw: an agent can change its body, not its brain. The brain is decided by the model company; what the agent can change is its memory. “More accurate with use” really means what it’s shown becomes more accurate.

I haven’t fully worked this part out myself, so I’ll organise it around three questions.

What to remember

LangChain’s memory library LangMem divides agent memory into three kinds.¹¹ In a factory setting, roughly:

• Facts: knowledge about this factory. Which processes it commonly uses, what products it has, how it usually writes its spreadsheets.
• Experiences: what it has done before, what went wrong, and why.
• Procedures: how the agent should do things. Skills, prompt rules, and descriptions of tools and parameters.

Where to keep it, and how to use it

1. Inject it into the system prompt every time. Define a structure to store memory, let the agent write to it, and put it in context at the start of every conversation. ChatGPT’s memory works this way.¹² Good for small amounts of stable information.
2. Keep it in the file system and fetch it as needed. Memory is stored as individual files; before a task the agent looks at the directory, reads whichever it needs, and writes back when done. OpenClaw works this way; Anthropic’s memory tool for Claude follows the same idea.¹³ So does my own personal knowledge base: I let the agent record details itself and note down the pitfalls it hits.

Factory settings involve a lot of information, so I lean towards the second. Cram it all into one prompt and it fills up with “in this situation, do this”; letting the model take a few more steps and look up the part it needs may be better. I’m not sure about this yet.

Who writes it, and when

• People write it. Or AI writes first and people revise. For example, recognition results corrected by hand are written back to memory, for reference next time the same factory comes up.
• The agent writes it while working. Each time it hits a problem, it records the error and the reason, and next time searches first for whether it’s been recorded. Takes effect immediately, but gives the agent one more thing to do while working.¹¹
• Tidy it up afterwards. After a task ends, run another pass to organise, merge and overwrite old entries. Doesn’t slow things down, but updates more slowly.¹¹

A few things to watch out for:

• Memory goes stale. According to one paper’s abstract, when instructions conflict with reality, agents follow the old instructions 90 percent of the time.¹⁴ Clean up regularly, or set the rule “when it conflicts with reality, follow reality”.
• Skills the AI summarises itself are usually not as good as ones people have organised.¹⁵ Best to have AI write first and people revise.
• What to remember matters more than how to store it.¹⁶
• Most “self-evolution” is really memory, not learning.¹⁷ The model itself hasn’t changed; what’s changed is what it’s shown.

Part 4: A few views of my own

• First validate whether something is worth doing, then decide whether to make it a complex agent. For many problems, a good prompt plus good knowledge is enough.
• Technology changes fastest; business knowledge is scarcest. Frameworks go out of date; writing industry experience into skills is what accumulates.
• Give what’s certain to code and what’s uncertain to AI, and let a person confirm at the end.
• Be clear about which errors the system should handle and which should go to the customer.
• Let AI verify its own work. If the system can’t support that, consider making the system “AI-friendly”.
• Don’t only look at accuracy; also look at whether users know what it can do. Faced with an empty input box, many people don’t know what to ask; turn common scenarios into buttons and forms that work as soon as you open them.

AI wrote this post, and you can absolutely have AI read it. More importantly, I wanted to record some of my understanding and share it with everyone.