Noida customer-support SaaS, 40 engineers, Python and Go on AWS, adding AI reply suggestions for 500 business customers
Weeks 1–2Baseline and evalsCollect 500 real tickets, holding 10% back as a blind test set; define output schema and quality rubricEval set and current scores
Weeks 2–3Gateway and schemaModel gateway, structured outputs, retries, timeouts, per-tenant keysReliable JSON outputs in staging
Weeks 3–5Latency and costPrompt caching with static instructions first, a smaller model for classification, streaming; check cached tokens in usage logsp95 latency and cost per ticket within targets
Weeks 4–6SecurityTreat ticket text as untrusted, separate instructions from data, filter outputs, no tool calls driven by email contentRed-team report with fixes
Weeks 6–8RolloutFeature flag, tenant opt-in, monitoring and alertsLive feature with evals in CI
Where it startsprototype works in a notebook, p95 latency 9 s, cost at scale unknown, no defence against prompt injection hidden in customer emails
Where it endsThe SaaS ships reply suggestions with controlled latency and cost. Weekly they watch suggestion acceptance rate, p95 latency and cost per ticket.