Built the evaluation and observability infrastructure for an experimental monitoring agent, so the team could iterate fast and see what the agent did in production.
Role
AI Engineer on a team improving LogicMonitor’s Edwin AI agent.
Challenge
LogicMonitor had built an experimental observability agent and brought in extra help to get it to production for their users.
Contributions
- Evaluation Framework: Designed and built an evaluation process that ran both at dev time and in production, on Promptfoo and LangFuse
- Evaluation Types: Wrote tool-choice, tool-correctness, and LLM-as-a-judge evaluations that covered agent performance from several angles
- Model Sweep Architecture: Built infrastructure to compare models systematically, which gave the client the performance numbers behind their model-selection decisions
- Real-time Observability: Let the client’s SMEs watch agent interactions in real time and catch problems before they turned into customer complaints
- Rapid Iteration: Gave the team an evaluation harness that let them test changes before any code shipped
Results
Delivered an evaluation and observability stack that changed how the team built and monitored the agent, with model recommendations backed by real performance numbers.
Technologies
- Promptfoo (evaluation framework)
- LangFuse (observability)
- OpenTelemetry
- Python
- OpenAI
