Making AI Responses Measurable: Quality Assurance for AI Systems

As part of an AI initiative, a major Swiss energy provider built a central AI platform. Unlike traditional software, language models do not always provide the same answer. It is therefore difficult to assess whether an adjustment improves or degrades quality without appropriate measurements.

Evaluation and monitoring were therefore key components of the platform from the very beginning. The goal was to detect errors early on, systematically test changes, and manage prompts in a traceable manner. To achieve this, the client needed a solution that would allow for continuous monitoring of the quality of the AI applications and enable data-driven further development of the platform.

• Tracing: We have integrated automated tracing that records every request including all steps of the Agentic workflow in MLflow. This allows for transparent tracking of latency and costs and enables errors to be traced back to specific processing steps.
• Operational Monitoring: We implemented a continuous quality monitoring system using predefined and custom scorers. With the help of LLM-as-a-Judge, AI responses can be evaluated for relevance, accuracy, and fidelity to the source, allowing potential quality deviations to be identified early on.
• Evaluation: We established a structured evaluation process in which individual AI components are tested against domain-specific test datasets before any changes are made. The results provide measurable quality metrics and enable a transparent comparison of different model and prompt versions.
• Prompt Management: We implemented centralized management and version control for prompts. This makes it possible to track changes and activate new prompt versions without having to redeploy them.

We fully integrated the entire solution into the existing Databricks infrastructure, eliminating the need for any additional tools.

By implementing MLflow, we enabled the client to specifically identify and resolve errors as early as the testing phase. Changes to prompts and models can now be systematically tested and compared against defined quality criteria.

In doing so, we established a central foundation for the continuous quality assurance of AI applications. The client can now make data-driven optimizations and further develop its AI platform in a targeted manner.