The Evaluation Framework: The Most Underbuilt Part Of An AI System

Hiren is CTO & co-founder of Simform, an AI-native digital engineering services company.

gettyWhen a production AI system starts to lose quality, spotting the decline is only part of the problem. The harder question is whether the team can explain what changed and judge, with enough confidence, whether the next fix will actually improve the system.

I was brought in to assess a production AI system that had begun to regress after months of stable performance. The production metrics showed the deterioration, but the team could not pinpoint when the behavior had shifted or tell whether the proposed fix would recover the quality they had lost.

My first question was whether we could take the same representative interactions and compare how the last-known-good version and the proposed fix handled them. We could not. The system was being monitored in production, but the measurement infrastructure needed to compare behavior across versions had never been built.

In that situation, engineering could make another change, but it could not show with confidence that the change improved the behavior that mattered. For the business, that meant additional engineering effort without consistent evidence of a better customer outcome, less manual intervention or a lower cost of completing the work.

What the team needed was a repeatable way to judge every proposed change against the same production-relevant expectations before it was released. Evaluation had to become part of the release process rather than a periodic quality check.

A production evaluation framework has three parts: a maintained set of representative examples, a defined way to judge how the system performs on them and clear thresholds for deciding whether a change is ready to release. Together, they answer a practical question: What evidence is enough to say that the next version is better?

It may begin with historical or curated examples, but it should incorporate real production interactions, known failure patterns and situations the team did not anticipate before launch. When a meaningful production failure occurs, that case should become part of the evaluation set so future versions have to prove they do not recreate it. This is especially important for low-frequency failures that carry a disproportionate cost, such as an incorrect action, unnecessary escalation or work that has to be corrected manually.

Some behaviors can be tested automatically, such as whether the system used the correct policy, extracted the expected information or passed the right inputs to another system. Others depend on context and require human review against defined criteria. The scoring method needs to stay consistent enough that changes in performance reflect the system, rather than changes in how it was assessed.

Teams should decide in advance which regressions are serious enough to block a change. A decline in a critical behavior may matter more than an improvement in the overall score, especially when that behavior affects customer experience, manual effort or the cost of completing the work.

Evaluation and observability thus serve different purposes in the release process. Observability helps engineers understand what is happening in the live system and investigate failures as they occur. Evaluation tells them whether the version they are about to release performs better on the behaviors that matter. The two should work together: production failures feed the evaluation set, and that evaluation set informs the next release decision.

The trade-off for CTOs is between investing engineering time in validation before it becomes urgent and prioritizing product work with a more visible near-term payoff. When the system is performing well, the value of stronger evaluation can be difficult to demonstrate compared with the next feature, integration or model improvement.

But deferring evaluation does not remove the work; it pushes it into future releases. Engineers may then have to reconstruct test inputs, inspect outputs manually and rebuild confidence around each significant change. What appeared to save time earlier becomes recurring validation work, slowing delivery when the team is already trying to move quickly.

The cost also shows up in the economics of the system. A change may reduce inference spend while increasing retries, escalations, human review or reopened work elsewhere in the workflow. For outcome-based applications, I care more about the fully loaded cost per resolved outcome than the cost of any one technical component. A cheaper configuration is only an improvement if the organization is also completing the work successfully at a lower overall cost. Evaluation should therefore reflect the outcome the system exists to produce, not only the quality of its output.

An evaluation framework should sit directly in the change process rather than be reviewed occasionally. Before release, the candidate version should run against the maintained evaluation set and be compared with the current version against agreed thresholds. After release, new production failures should feed back into that set, with clear ownership for keeping it current.

And before that work is cut from scope, I would ask one question: If we change this system next month, what evidence will tell us that the change helped?​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?