The person who knows how an AI feature's output should read is usually not the person who can change it. They review a draft, spot that the tone is wrong or the guidance is missing a step, and file a ticket. An engineer edits a string literal, opens a pull request, waits for CI, and deploys. The round trip is days for a change that took thirty seconds of thought.
Do that a few times and one of two things happens. Either the AI behaviour stops improving because the loop is too slow to be worth using, or someone gives the domain expert access to the codebase and you discover new failure modes.
We've just shipped a feature where the loop is about a minute and no engineer is involved. The mechanism isn't obvious from the outside.
The line we drew
Prompt text does not live in the codebase. Prompts, and the model each agent uses, live in a prompt management tool — we used Langfuse — where SimTutor's instructional design lead can edit them in a UI and promote one to production.
On its own that would be reckless, so two things stay in the codebase:
- The prompt tool owns the prose. Persona, tone, domain guidance, the model choice.
- Code owns the contracts. Output schemas, the tool catalogue, and the rules for calling tools — appended to the prompt at runtime, where an editor can't remove them.
The output of every content-generating agent is validated against a Zod schema. A prompt edit can change what the assistant says without being able to change the response format the application depends on.
What it buys beyond speed
We didn't fully anticipate two effects.
Swapping a model is a configuration change. The schema contract means the consuming code can't tell the difference, so "should we try the cheaper model" becomes something you can just try on a Tuesday.
Adding a capability is a tool definition and a line in a prompt. The agent loop, the memory, the auth path, and the UI already exist and don't move, so engineering time goes into what the assistant can do.
Tracing
Handing prompt editing to someone outside engineering is only defensible if you can see what happened afterwards. Prompt management on its own isn't enough — you need tracing in the same tool.
Production runs are traced end to end: the prompt version the run used, the model, every tool call, token counts, latency. The classic AI bug report — "it did something strange yesterday" — becomes a lookup: you pull the trace and read what the model was shown and what it did.
More usefully, because traces carry the prompt version, questions like did last week's persona edit make responses slower or more expensive stop requiring archaeology. You filter traces. One person changes behaviour without a deploy, and anyone can see exactly what the change did.
If prompt editing lives in one tool and observability in another with no shared version identifier, we'd think twice about handing out prompt access. You've removed the deploy gate without replacing it with anything.
Two other decisions that mattered
Evaluate feature flags server-side. The whole feature sits behind a flag checked on every request rather than in the UI. Turning it off removes access from already-open tabs within a minute. A flag checked in the browser only controls what's visible, so if you want the ability to actually stop a feature — which you do when it has a per-token cost and a new external dependency — the check has to happen where the work is done.
The flag client also needs to fail quietly: serving the last good payload on a failed refresh, and falling back to configured defaults on a cold start when the flag service is unreachable. During a flag outage, flags stop changing and the product keeps working.
Log experiment exposures from day one. Flags become experiments the moment you record which variation each user was actually served. It's one assignment callback writing a row, and without it there's no way to attribute a difference in outcomes to the variation you were testing.
Keeping the line
Every new capability forces a decision about which side an instruction belongs on, and both errors are real. Put too much in the prompt and someone can break the editor by rewording a sentence. Put too much in code and engineering is back in the loop for wording changes, which is the problem you were solving.
We got that wrong a couple of times and moved things. It stays an ongoing judgement, but it's a judgement about one clear line, which beats having no line at all.
This came out of a recent build of the AI assistant inside SimTutor Author, the authoring tool for SimTutor's dialogue simulations. If you're working through the same set of decisions, [start a discussion](https://www.pattern.co.nz/contact).