AI / ML engineering project · retrieval-augmented generation · evaluation

A support-ticket triage system that knows when not to answer.

It searches 48,178 past tickets, drafts a next-best-action with a local LLM, and escalates to a human whenever the retrieved evidence is too weak to trust or the message is outside what the knowledge base covers. This page is built from its real results on tickets it never saw, including the places it fails.

How a message flows through it

The gate is the new part. It first checks that the message resembles anything in the knowledge base, then predicts the chance that the triage is correct, and decides before the LLM is called.

Customer message "charged twice…" Embed + search FAISS · top 5 of 48,178 Two-stage gate 1. relevant to the KB? 2. P(correct) ≥ τ? Answer LLM + cited tickets Escalate human, LLM skipped ≥ τ < τ

Drag the threshold. See the trade-off.

Every point on the curve is a threshold. Answering more queries means more wrong answers. The gate picks a point on this curve instead of answering everything.

answered automatically
of those answers are wrong
escalated to a human

Click or drag on the chart to move the threshold.

Try it on real tickets it has never seen

These are held-out customer messages, excluded from the index. Pick one to see the retrieved tickets, the gate's decision, and whether it matched the true label. Green dot means answered, amber means escalated.

What I found

All numbers are measured against gold category labels, so no LLM judge is involved. The confidence model was fit on leave-one-out queries from the knowledge base and tested on the held-out split.

Does the learned gate beat simpler methods?

The data hides two different problems

Stress tests on text it was never built for

Real public datasets with no overlap with the knowledge base: CLINC150 out-of-scope and everyday-assistant queries, and Banking77 customer messages from a neighbouring domain.

Does it escalate what it should never answer?

Typos (about 8% of letters corrupted)

Failure analysis

Most confused category pairs

What if the overlapping labels were merged?

Confident misses the gate cannot catch

Where the knowledge base runs out

Clustering the queries the gate escalates shows which topics the system cannot handle. Low category purity means the same topic is filed under many categories, which points to a taxonomy problem as well as missing knowledge.

Limits I would name before you ask

Run it yourself

Everything runs locally with no API key and no cost.

python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
brew install ollama && ollama pull llama3.2:3b && ollama pull nomic-embed-text
python -m src.data --source both     # download and split the datasets
python -m src.ingest                 # embed 48K tickets into FAISS
python -m src.reliability            # fit the confidence gate, write the report
streamlit run app.py                 # live dashboard