Owned the evaluation and observability workstream for an AI agent that quizzes learners on training content. Built the harness and tracing that let the team test the product while they were still building it.
Role
AI Engineer, owning the evaluation and observability workstream.
Challenge
Microsoft was building a new AI agent to quiz learners on training content. They needed it evaluated and observable during the build, not after it shipped. That meant broad coverage stood up fast, with correctness anchored to subject-matter-expert judgment instead of guesswork.
Contributions
- Evaluation roadmap: Designed the workstream as a staged roadmap where each phase fed the next, so evaluation grew alongside the agent instead of getting bolted on at the end
- Prompt iteration: Built a tool that let users edit the agent’s prompts and see the results of each change right away, which tightened the loop between a change and its effect
- Synthetic data: Generated synthetic datasets to produce evaluation data before launch, then had subject-matter experts label it, which grounded the evals in expert judgment before the agent ever met a real user
- Automated dev-time evals: Moved evaluation into development itself, combining deterministic checks with LLM-as-a-judge scoring so problems surfaced before they shipped
- Observability: Closed the loop with anonymized production traces, feeding real usage back into the evaluation dataset so the evals kept getting better after launch
Results
The client came away with a product evaluated and tested throughout the build, with broad coverage stood up faster than an in-house team could have managed. Left behind the eval suite, the observability stack, and a written handoff, so the team could keep running and extending the work after the contract ended.
Technologies
- Python
- Synthetic data generation
- Deterministic and LLM-as-a-judge evaluation
- Distributed tracing and observability
