Four evaluation engineering lessons learned from building agent plugins
The real challenge of building an agentic system isn't the construction, it's the measurement. Evaluation engineering is a critical foundation for building reliable AI systems.
The real challenge of building an agentic system isn't the construction, it's the measurement. Evaluation engineering is a critical foundation for building reliable AI systems.
I’ve run evaluation engineering initiatives for years at Google and developing the newly released Google Cloud Developer plugin served as a live testbed to stress-test these key principles across real-world developer workflows.
Here are four eval lessons we learned from building the new plugin that you can apply to your own builds.
1. Grade the execution steps instead of the summary When we first evaluated agents, we set up an LLM-as-judge to inspect the agent’s final text reply and scored it against a checklist. However, we quickly realized it’s fundamentally flawed because the grader only sees what the agent wrote, not what it actually did. When we ran our initial 11-prompt evaluation suite for the Google Cloud Developer plugin, the automated grader marked it as failing three prompts. But when we opened the raw tool-call logs and read the transcripts, the plugin had actually passed all three. On a Cloud Run cleanup prompt, for instance, our rubric required checking command syntax before running anything. The agent ran gcloud help run services list as its very first step. It did the work, but its final message was focused on warning the user about permanent data loss rather than reciting that it had checked the help docs.
The grader penalized the agent for doing the right thing simply because it didn’t brag about it in the summary. If your rubric line says "did it run X?", a LLM judge is guessing unless you evaluate the actual tool-call trajectory.
In context
- Topic: Proyectos, PMO y Riesgos — Gestión de proyectos, oficinas de proyecto y riesgo del portafolio.
- Source: Google Cloud Blog
- Published: 16/10/2026
Continue reading at the original source →
Excerpt published automatically by the site radar. The full text belongs to its publisher and is linked above.
Why it matters
A project management office is worth the decisions it makes possible, not the reports it produces. Once the PMO becomes the function chasing status updates, it has already lost: the team learns to fill in the template and the real information goes back to travelling through the corridors.
I read these things looking for where the risk sat before it became an incident. It was almost always already written in some register nobody reviewed, with an assigned owner who never found out they were one. A risk identified and not managed is worse than one never identified: it creates the false sense that there is control.
What usually goes wrong
What I see fail most is the absent sponsor. They sign the kick-off, appear in the photo, and do not come back until there is a problem. Without somebody with real authority clearing obstacles every fortnight, the project manager ends up negotiating as an equal with areas that do not report to them, and loses.
What to watch
- Who the sponsor is and how often they actually show up — not on the org chart, in the calendar.
- Whether the status report shows any red projects: an all-green portfolio is a portfolio without information.
- Which risks are identified, who owns each one, and when they were last reviewed.
How I read this entry
The way I would bring this down to earth is by shrinking the size of the deliveries. An eighteen-month project with nothing visible before month twelve is a bet, not a plan. I would rather have six deliveries that stand on their own, even if the total takes a little longer, because each one corrects the course.
This entry is an excerpt from the original source, selected by the site radar. The commentary above is the site's own and does not belong to the cited publisher.
Comments
No comments yet. Yours can be the first.
Living through this in your own team?
Open the chat and tell me how you're handling it. I'm interested in comparing notes.
