Can you replay a failure?
Pick last week's worst output. If nobody can reconstruct the inputs, the tool calls, and the model version that produced it, you do not have a system, you have an anecdote generator.
Decision framework
Ship, fix, or stop. Seven checks that decide it.
This is the review I run before telling anyone their AI system is ready. No pricing here, no booking form. Just the framework, so you can run it yourself tonight.
Thirteen years of enterprise IT taught me that pilots rarely fail on model quality. They fail on ownership, observability, and blast radius. Every check below exists because I watched its absence cost somebody a quarter.
The seven checks
Answer each one out loud, with evidence.
Do evals come from real failures?
Synthetic test sets pass forever. The only eval suite that catches regressions is one seeded from the outputs that actually embarrassed you, added the day they happened.
What is the blast radius of a wrong call?
List every irreversible action the system can take: sends, refunds, writes, deletes. Each one needs either a confirmation gate or a compensating action. Unbounded write access is not a launch state.
Can you attribute cost to a workflow?
A single monthly model bill tells you nothing. If spend is not tagged per workflow and per tenant, the first cost spike becomes an emergency instead of a line item.
Is untrusted text separated from instructions?
Retrieved documents, ticket bodies, and web pages are data, never directions. Any system that concatenates them into the instruction channel is one crafted paragraph away from a bad day.
Who owns it on Monday?
Name the person, not the team. If the answer is the consultant who built it, the handoff has not happened, and pilots die at handoff more often than they die at the demo.
What is the fallback when the model is down?
Providers degrade. A queue, a cached answer, a human path, or a clear failure message. Something must exist between the model and your customer.
Reading the result
Three outcomes. Only one of them is a rebuild.
Ship
Six or seven checks pass and the failures you have left are cosmetic. Launch narrow, watch the traces, expand scope weekly.
Fix
The concept holds but the scaffolding does not. Evals, tracing, permission gates, and cost attribution are a few weeks of unglamorous work, not a rebuild.
Stop
The workflow has no measurable outcome, or the data underneath it is not trustworthy. Fix the data and the process first. The model was never the problem.
FAQ
Common questions.
- Is this a paid offer?
- No. This page is the framework itself, written out, and you are welcome to run it yourself. The paid version is the Systems Decision Audit at TechTide AI, where scope and pricing are set.
- How long does a review like this take?
- Ninety minutes with the people who built the system is usually enough to reach a ship, fix, or stop call. Writing up the ranked plan takes another day.
- Who should run it?
- Anyone with access to the traces, the eval results, and the bill. If none of those exist, that is already the finding.
- What if every check fails?
- Then stop is the honest answer, and stopping early is the cheapest outcome on this page. Most systems fail two or three checks, not all seven.
- What if the problem is not the AI?
- Frequently it is not. When the real constraint is an operations layer running on Airtable bases, the AI work is downstream of a data problem. That is a different path.
Ran the checks. Now what?
If the failures sit in the AI system itself, the paid version of this framework is the Systems Decision Audit. If the checks kept pointing back at an operations layer living in Airtable, start with the Airtable Rescue Decision Sprint instead. Both decision engagements, including scope and pricing, live on TechTide AI.
Only one check failing? The Production Block Call is the smaller way in. One named blocker, one call, one recommendation.
Working code and technical write-ups live on alexcinovoj.dev. More of the reasoning behind these checks is in the field notes.