🎓 The Customer#
The client is a Japanese university with roughly 40,000 students and staff. Every enrollment period, the student office gets buried — visa questions, course registration, tuition deadlines, scholarship forms, faculty transfer rules. The volume is seasonal and brutal: a handful of staff fielding thousands of nearly-identical questions in a compressed window, with real students blocked on real deadlines while they wait for a reply.
The barriers behind the ask:
• 📚 Document Sprawl: Answers live across handbooks, policy PDFs, forms, and per-faculty rules — inconsistently formatted, some scanned, some table-heavy
• ⏰ Seasonal Overload: Enrollment-period volume spikes far beyond what the office can staff for
• 🌐 Mixed Audience: Students at different enrollment years, faculties, and departments need different answers to the same question
• 🔒 Regulated Data: Student records and PII can't leak into logs, model providers, or unfiltered responses
🧨 Inheriting a Failed System#
The university had already paid another company to build this. It failed in production, and I was brought in afterward to figure out why and fix it.
The prior vendor shipped a simple RAG pipeline — embed documents, embed the query, cosine-similarity top-k, hand the chunks to an LLM. No metadata schema. No reranking. And the document processing was naive: plain text extraction that didn't handle tables, scanned pages, or complex layouts, so a meaningful fraction of the corpus turned into garbled, half-usable text before it ever reached the index.
The result was predictable: students asked a real question the corpus actually answered, and the system retrieved the wrong paragraph — or the right document with the wrong faculty's tuition schedule, or a table that had been flattened into unreadable text. Staff ended up re-answering the same questions the chatbot was supposed to handle, which defeated the entire point of building it.
🔍 Diagnosis#
Before touching any code, I traced the failures back to three root causes, all upstream of the LLM itself:
- No metadata — every chunk looked identical to the retriever regardless of which faculty, department, or enrollment year it applied to, so semantically similar-but-wrong documents routinely outranked the correct one.
- No reranking — raw vector similarity was the only signal. There was nothing to correct for near-duplicate chunks or to weight document authority and recency.
- Bad document processing — naive parsing meant tables (fee schedules, deadline grids) and scanned/complex-layout pages were silently corrupted at ingestion time. The retriever wasn't failing; the source chunks were already broken before indexing.
None of this was fixable by prompt-tuning the generation step. The system needed to be rebuilt from ingestion up.
🧭 The Recommendation — an Agent, Not a Bigger RAG#
The default expectation, based on how the first vendor had framed the project, was "fix the RAG pipeline." I advised something bigger: replace the single-shot retrieval call with a chatbot agent that could ask a clarifying question before answering, apply metadata filters once it had that context, and choose how much surrounding text to pull into the prompt based on the question — instead of always guessing from one static retrieval pass.
That meant handling the natural objection — isn't a bigger, better-tuned RAG pipeline the safer, faster fix, especially with enrollment season approaching? I laid out the failure modes a static pipeline couldn't solve on its own: a question like "what's the deadline for me" is unanswerable without knowing the student's faculty and enrollment year first, and no amount of reranking recovers information the retriever was never told to filter on. An agent that asks one clarifying question before retrieving closes that gap; a bigger RAG index doesn't. With that laid out against the enrollment-season deadline, the client agreed to the agent architecture, and we scoped the rebuild in stages so the office wasn't left without a tool while it shipped.
📦 The Result#
The rebuilt system shipped in phases ahead of the next enrollment cycle and now handles the majority of first-line student office inquiries: retrieval accuracy went from a system that routinely surfaced the wrong paragraph to one validated at ~90% on an LLM-as-judge offline benchmark, peak-season ticket volume to the office dropped by roughly half, and the rollout reached the full ~40,000-person student and staff population with zero critical incidents. The architecture is below.
💡 How It Was Built#
🏗️ System Design on Azure
The system splits into three loops: an offline document pipeline that turns source PDFs into a searchable, metadata-tagged index; a runtime agent loop that answers student queries; and an offline evaluation loop that gates every index update before it reaches production.
The runtime loop uses two models for two different jobs: Claude Sonnet 4.5 handles orchestration — deciding when to call the clarification tool, when to retrieve, and how to resolve parent vs. child context — while Claude Haiku 4.5 drafts the final answer once the right context is assembled. Splitting reasoning from generation this way keeps the expensive model on the step that actually needs judgment and the cheap, fast model on the step that's mostly assembly.
📄 Fixing Document Processing — pdfplumber + LLM Fallback
The inherited system's core defect was ingestion, so that's where the rebuild started. Every document now goes through a parser router: pages with clean, extractable text go through pdfplumber directly — fast and cheap. Pages that don't (scanned images, dense fee tables, multi-column layouts) fall back to a vision-capable LLM call that reads the page as an image and returns structured text, including table contents in a format the chunker can reason about. That single routing decision — text extraction where it's reliable, LLM fallback where it isn't — eliminated the silent corruption that had been quietly breaking the previous system's retrieval before a query ever ran.
🌳 Parent-Child Retrieval Structure
Rather than indexing flat, fixed-size chunks, documents are indexed as a parent-child hierarchy: a page or section is the parent, and each paragraph within it is a child chunk with its own embedding. Retrieval matches at the child level — a single paragraph is a precise semantic-match target — but the system can resolve up to the parent page or section when a question needs the surrounding context to be answered correctly (e.g., a deadline that only makes sense next to the eligibility rule two paragraphs above it). Which level gets fed to the LLM is a configurable retrieval parameter, not a fixed choice, so the same index serves both narrow factual lookups and broader "explain this policy" questions.
🛠️ Agent Tools for Better Grounding
The agent carries tools to ask the student for the context the retriever needs before it searches — enrollment year, faculty, department, campus. Once that's known, it's not just a courtesy: it becomes structured metadata that combines with the reranker to filter out documents that don't apply to that student and boost the ones that do. A vague question like "when do I need to submit my form" turns into a query scoped to the student's actual faculty and cohort, instead of a global similarity search across every faculty's paperwork.
🧪 Offline Evaluation on Every Index Update
I proposed — and the client agreed to build — a golden test dataset of real question/answer pairs drawn from historical student office inquiries. Every time the document index changes (new policy PDF, updated fee schedule, corrected form), the pipeline automatically runs the full test set through the current retrieval + generation stack and scores each answer with an LLM-as-judge, checking correctness and groundedness against the source document. An index update that regresses accuracy doesn't get promoted. This turned "did the last change make things better or worse" from a guess into a number, and it's the guardrail that lets the index keep improving over time without silently reintroducing the previous vendor's failure mode.
🔐 Guardrails
Student PII (student IDs, names, contact details) is detected and redacted before it reaches logs or gets forwarded to the model provider, and every response passes through a content-safety gate before it reaches the student. Additional gates were added to satisfy specific client requirements around data handling — the details are internal, but the pattern is a hard stop between "the model generated this" and "the student sees this."
🚀 Multi-Stage Deployment
The rollout went dev → staging → a single-department pilot → full production, timed so the highest-risk stage (pilot, with real students) landed in a lower-volume window well before peak enrollment — not during it. Each stage validated against the same offline eval suite used for index updates, so a deployment and a data update carry the same reliability bar.
Impact & Results#
Rebuilding the ingestion, retrieval, and agent layers — not just re-tuning the model — is what took this system from failed to production-ready:
- Retrieval accuracy: ~55% → ~90% on the LLM-as-judge offline benchmark after fixing document processing, metadata, and reranking
- Office workload: ~50% reduction in peak-enrollment ticket volume to student office staff
- Scale: ~40,000 students and staff served, phased rollout with zero critical incidents
- Trust: PII guardrails with zero reportable incidents since launch
The system now handles the first line of enrollment-season questions so the student office can focus on the cases that actually need a human.
Key Achievements
50% fewer support tickets to student office
90% RAG retrieval accuracy, up from 55%
0 reportable PII incidents since launch
Rolled out to ~40,000 students and staff via a phased, multi-stage deployment with zero critical incidents during enrollment