Developer Issues Shared Playbook for Trustworthy Third‑Party Evaluations of Frontier AI

Guidance urges robust, transparent third‑party evaluations for frontier AI models and lays out lessons and recommended approaches for assessing capabilities, safety mitigations and reporting validity to help shape emerging standards.

A developer has published a playbook for trusted third‑party evaluations of frontier AI models, offering concrete lessons and recommended approaches for designing assessments that produce additional evidence about model capabilities and safety mitigations. The guidance is intended to make third‑party testing more rigorous and comparable, which matters because consistent evaluation practices will shape how safety and readiness are judged across the field. The document details how to assess three core elements: model capabilities, implemented safeguards, and the validity of reported findings. It walks through design choices that help evaluations produce reliable, interpretable results rather than isolated performance snapshots, and stresses that assessment protocols should be explicit about scope, assumptions and limitations so results can be meaningfully compared or aggregated.

A central theme is the need to account for the model’s operational context — what the guidance calls the “harness.” Frontier models increasingly use external tools, maintain multi‑step internal state, and act inside larger automated workflows; observed behavior depends not only on the model itself but on the surrounding environment, tool access and specific setup that enable actions. The playbook therefore recommends that evaluators either include the harness when testing end‑to‑end behavior or clearly separate harness effects from core model capabilities when reporting results.