Customer-service agent in Copilot Studio with custom Python tools

Problem#

A service desk handled roughly 1,200 tickets per week across three languages. Agents spent most of their time retrieving order, contract and shipping data from four systems before they could answer anything. Response quality varied by shift, and the existing rule-based bot deflected under 5% of contacts.

The constraint was organisational as much as technical: the CRM was frozen for the year, so nothing could be built inside it.

Architecture#

A Copilot Studio agent owns the conversation, knowledge grounding and escalation. Anything that needs deterministic behaviour (lookups, entitlement checks, refund eligibility) lives in Python tools hosted on Azure Functions and exposed to the agent as custom actions through a Power Platform custom connector.

Dataverse holds conversation metadata and the audit trail. Secrets live in Key Vault; the Functions app uses managed identity. Deployment runs through solution-aware ALM pipelines with separate dev, test and production environments.

Solution#

We started with the ten highest-volume intents and wrote an eval set of 120 real, anonymised conversations before building anything. Each tool was shipped with its own contract test, so a change in the ERP schema fails in the pipeline rather than in front of a customer.

Escalation is explicit: when confidence is low, or when money or personal data is involved, the agent hands over with a structured summary instead of guessing. Human agents rate every handover, and those ratings feed the weekly eval run.

Results#

• 38%: Contacts fully deflected
• 2.4 min: Median saving per handled contact
• 120: Conversations in the regression eval set
• 0: CRM changes required

Lessons learned#

The eval set was worth more than any prompt engineering. Once we could measure regressions, the team stopped arguing about wording and started shipping. The second lesson: keep money-touching operations deterministic: the agent decides when, code decides what.