Shipped to OpenAI in 1 month
Connecting models to Federal Reserve data
Led a team of three that built a Python MCP server for OpenAI in one month. It gives AI assistants access to Federal Reserve economic data, with 389 evaluation cases checking tool use, accuracy, and how the assistant handles unclear questions and errors.
Problem
OpenAI needed its models to find and analyze data from FRED, the Federal Reserve economic database. The assistant had to choose the right data, recognize incomplete results, and explain what the numbers mean.
My part
I led the team and worked on tool descriptions, input limits, and evaluations. We built tools to find economic data series, compare them, and examine revisions. Our tests inspect which tools the assistant calls and what it asks for. Another model grades its explanations. We also test unclear questions, requests the server cannot answer, and attempts to bypass instructions.
Result
We shipped in one month with 389 evaluation cases, 299 of them testing the assistant itself, plus logs and dashboards for investigating failures. The evaluations caught the cases where the assistant could not decide whether to answer or ask a question, so we fixed those before delivery instead of after.
