Mallika Rao Warns Evaluation Debt Can Break Production AI, Proposes Five-Layer Stack and Maturity Model

At QCon AI, Mallika Rao opened a 52:58 presentation by warning that "evaluation debt" is an invisible engineering liability that accumulates silently in production AI systems and can "explode" into reliability failures that erode user trust. She said the danger is not always visible on standard dashboards: semantic errors and coverage gaps can silently change user-facing behavior and downstream pipelines. The takeaway is immediate for teams running personalized or money — moving systems, where unseen evaluation gaps can create operational and compliance risk.

Rao anchored the talk in systems she helped build, describing personalized search and ranking at global scale and a commerce rewards pipeline she compared to "Walmart — scale." Her search setups touched trillions of documents, operated within sub-100 millisecond latency budgets, and depended on hundreds of internal microservices. The rewards pipeline served about 25 million users monthly, processed dollar — denominated transactions, and had to meet compliance across 50 states — illustrating how high scale and regulatory constraints magnify evaluation shortfalls.

To address these risks, Rao proposed a five-layer evaluation stack that runs from infrastructure through user experience and a diagnostic maturity model teams can use to measure coverage. She argued that single — number metrics and legacy test datasets fail to protect modern, distributed, personalized architectures. Effective evaluation, she said, must surface semantic errors across multiple operational layers rather than relying on aggregate metrics alone.

Rao used case studies from Twitter, Walmart, and Netflix to demonstrate repeatable failure modes at scale, showing how similar evaluation gaps recur across different architectures and domains. Those examples highlighted concrete consequences: silent semantic failures that evade monitoring yet degrade the user experience, and increased operational or compliance exposure in systems that move money or enforce rules across jurisdictions.

Her maturity model is explicitly diagnostic: it helps engineering leaders map where evaluation coverage is thin, identify which layer or layers are accruing debt, and prioritize fixes before issues reach customers. Rao recommended practical next steps for builders — map evaluation coverage across the five-layer stack, adopt the diagnostic model to quantify gaps, and evolve adoption patterns so evaluation practices keep pace with product and infrastructure changes.

Rao closed with engineering principles for integrating evaluation into release and observability workflows so debt is detected and paid down early rather than allowed to compound. Her message at the practitioner — led forum emphasized that evaluation frameworks — not just model architectures — often determine whether AI systems ship safely and sustain trust in production.