Work
Client names are withheld unless they've offered them. Everything else here is what I would tell you on a call, in the same order.
Inbound claims correspondence was routed by hand by a team of six. We classified and pre-summarised it instead, and left the humans deciding the edge cases rather than sorting the obvious ones.
Handling time down 31% · no change in complaint rate
Two previous attempts had been shelved because nobody believed the citations. We built the evaluation set first, with the lawyers, and only then chose the retrieval approach.
Six weeks to production · 94% citation accuracy on the held-out set
Assistants were already in use, quietly, with no policy. Rather than ban them we wrote rules people could follow and ran the training that made the rules make sense.
Programme now run in-house · policy signed off by security
The audit found the underlying product data was too inconsistent to answer the questions customers were actually asking. We stopped, and the budget went to fixing the catalogue.
Project cancelled at week 3 · CHF 400k reallocated
Adoption, not accuracy, was the hard part. The model was fine at week two; the remaining months went into fitting it to how planners already worked.
Weekly active use by 8 of 9 planners after six months
The first conversation is mostly me asking questions. Bring the messy version — the tidy version is usually wrong.