Build diary: shipping a Copilot Studio agent to production in six weeks

A build diary is only useful if it includes the weeks that went badly. Here is the honest version.

Week 1: Evals before prompts#

We wrote nothing agent-shaped in the first week. Instead we collected 120 anonymised conversations and turned them into an eval set with expected outcomes: deflect, escalate, or refuse.

def score(run, expected):
    if run.action != expected.action:
        return 0.0
    return 1.0 if run.cited_sources else 0.5

Crude, but it made regressions visible from day one.

Week 2-3: Tools, not prompts#

Every deterministic operation became an Azure Function with a typed contract. The agent decides when to call; the function decides what happens.

Contract tests#

Each tool ships with a contract test that runs in the pipeline. When the ERP team renamed a field, the build failed instead of the agent.

Week 4-6: The boring parts#

Environment strategy, DLP policies, a handover format the human agents actually liked, and a rollback plan. None of it demos well. All of it is why the thing is still running.