CX Consulting AI

The core innovation isn't a RAG pipeline — plenty of teams have those. It's the orchestration layer. Every /api/ask (app/api/routes_chat.py:306) request routes through an LLM-driven planner (app/core/intelligent_planner.py:68, IntelligentPlanner.plan_response at :75) that classifies user_intent across 14 categories and picks one of 13 next_action values (planner.py:39-52). The chosen plan then runs through three guard layers before execution: an IntentVerifier that re-asks the LLM when confidence is low, a ToolValidator that checks preconditions, and a DecisionVerifier (app/core/decision_verifier.py:32) that makes a second LLM call with an intentionally conservative prompt ("only reject if OBVIOUSLY inappropriate") to catch bad intent classification. If the DecisionVerifier flips the plan, it hands back a suggested_intent and suggested_action and orchestration continues with the corrected plan. Then execution dispatches to one of 13 dedicated executor classes: DirectAnswerer, StrategyCreator, ClarificationAsker, CasualResponder, DeliverableGenerator, IterativeResponseGenerator, DocumentAnalyzer, WebSearcher, QuestionnaireUpdater, MoMGenerator, KnowledgeCollector, ProgressiveContextBuilder, GenericDocumentGenerator. Deliverable output auto-saves via `_auto_save_deliverable` at orchestrator.py:966: writes markdown to disk (orchestrator.py:1023), inserts a database row (orchestrator.py:1050), and once a project accumulates 50+ deliverables the auto-save also re-indexes into ChromaDB (orchestrator.py:1084-1089) so future queries retrieve from prior work.

HybridRetriever at app/services/retrieval.py:17 runs BM25 (BM25Okapi, retrieval.py:45) and semantic search (ChromaDB with BAAI/bge-base-en-v1.5, downloaded in app/scripts/download_embeddings.py:26) in parallel, then fuses ranked lists via Reciprocal Rank Fusion at retrieval.py:82-96 with rrf_k=60 (RRF_K default in config.py:290, env-overridable). Two-stage retrieval hybrid_cross_encoder at retrieval.py:112 pulls RRF top-M then reranks to top-K with a CrossEncoder model (default cross-encoder/ms-marco-MiniLM-L-6-v2 from config.py:222; actually loaded in agents/retriever.py:110 when USE_RERANKING is on). Optional MMR diversification at services/mmr_diversifier.py runs after fusion, before the reranker. Collections are scoped in document_service.py:37-39: CX_GLOBAL_COLLECTION = 'global_kb' for shared frameworks, USER_COLLECTION_PREFIX = 'user_' for project docs, DELIVERABLE_COLLECTION_PREFIX = 'deliverable_' for generated outputs. A query cannot cross projects because it never asks for another project's collection.

p95 went from 2.3 seconds on the initial build to 620ms in production — a 73% cut, measured against the client's production traffic (not verifiable in the repo). The optimizations that got it there: async FastAPI streaming (SSE / chunked responses so the client sees tokens as they generate — the /ask endpoint returns StreamingResponse with media_type='text/event-stream' at routes_chat.py:1320), backend routing that sends simpler intents to a smaller/faster model, and making CrossEncoder reranking optional per-request (USE_RERANKING env flag at agents/retriever.py:89) because the client's eval set hit 91% accuracy without it. Prometheus histograms at app/observability/metrics.py:13 track LLM_LATENCY per backend for ongoing regression detection.

Users upload PDFs, DOCX, TXT, CSV, XLSX, or Markdown through app/api/routes_upload.py (upload_docs_api at :59) or the aliased app/api/routes_documents.py. Uploaded bytes are written to a NamedTemporaryFile matching the original extension, then handed to RagEngine.process_document at services/rag_engine.py:326. The engine dispatches by extension to the right loader, extracts text, chunks it (~512 tokens, 50 overlap), computes content-addressed chunk IDs, embeds with BGE-base-en-v1.5, upserts into the tenant-scoped ChromaDB collection (user_{project_id} for project docs, global_kb for shared frameworks), and rebuilds the BM25 sparse index in parallel so hybrid retrieval sees the new content on the next query.

Tenancy is column-based on top of SQLModel. Every Project has an owner and a shared_with list of user IDs; every route dependency verifies the current user is owner or shared. ChromaDB collection names embed the project ID (user_{project_id}, deliverable_{project_id}) so retrieval is naturally scoped — a query cannot cross projects because it never sees another project's collection. Cleaner than row-level security, easier to reason about than schema-per-tenant.

Config-driven backend selection via LLM_BACKEND=vllm|llama.cpp|azure|ollama. Local llama.cpp with Qwen3-4B-Q4_K_M runs on the AWS g4dn.2xlarge T4 GPU for cost control; Azure OpenAI is the drop-in for when a client needs enterprise-managed inference; vLLM is available for higher-throughput single-model serving; Ollama for local dev. Tenacity-wrapped generate_async retries with exponential backoff on transient failures. Startup loads exactly one backend at a time on the deployed node — no runtime failover, but the switch is a one-env-var deploy.

docker-compose.g4dn.yml runs two services: the FastAPI app container (with GPU reservation via nvidia device driver, LLM_THREAD_QOS=USER_INITIATED) and Redis 7-alpine for conversation memory. Models live on the host at /opt/models bind-mounted into the container so image size stays small; SQLite persists via a second bind mount at ./app/data:/app/data. Env comes from .env.prod.g4dn. Nginx as a TLS/buffering front is a documented recommendation in docs/DEPLOY_g4dn_2xlarge.md:59 for production but is not part of the compose stack. Alembic manages SQLModel migrations across 6 tables (User, ProjectUserLink, Project, Chat, ChatMessage, GeneratedDeliverable — declared in app/models/chat_models.py and user_models.py) across 3 migration files in alembic/versions/.

After the client project, I kept building. The personal continuation (github: LangChain) carries the same architecture forward and adds the pieces the client version doesn't need: NLI-verified citations via DeBERTa-v3-base at span level (87% recall on RAGTruth's gold hallucination spans), hash-chained SQLite audit log for tamper-evident provenance, LangGraph agent mode with tool binding (rag_search, web_search via Tavily, calculator, sql_query on DuckDB), Dagster scheduled ingest with hash-based delta detection, and a benchmark harness that runs against BRIGHT, FinanceBench, RAGTruth, BIPIA, CUAD, and Amazon ESCI. Same retrieval philosophy, deeper testing, cleaner observability (OpenTelemetry + Phoenix + Langfuse + LangSmith).

Real numbers from real evaluation runs on the LangChain framework, not vibes. Verifier catches 87% of RAGTruth's gold hallucination spans. Retrieval hits 40% on FinanceBench's partial corpus (limited by how many source PDFs were ingested, not by the retriever). ~13 nDCG@10 on BRIGHT's pony subset with FastEmbed 384-d + BM25 RRF (reference: SFR-Embedding-Mistral averages 18.3 nDCG@10 on full BRIGHT — so a small embedder in the same ballpark). 17% BIPIA adversarial attack success rate against a defensive system prompt (published unprotected GPT-3.5 numbers on BIPIA are >50%). BuyChat, a sibling product built on the same retrieval stack, ranks Amazon ESCI at 0.94 recall@100.

Consultants and product were the primary users, not other engineers, so most of the work was translating what they wanted into what a RAG system could actually deliver. Explaining why a reranker helps a fuzzy query but not a narrow one, why streaming feels faster than a lower absolute latency, why some deliverables need to draft-then-critique and others can be one-shot. Every ambiguous request became a backend milestone I could ship against. Honesty note on the metric strip: the 91% retrieval accuracy and 94% context retention are from the client's own evaluation set — they're not measured inside this repo. The 620ms p95 is production telemetry. And 500+ concurrent users is peak session count, not simultaneous in-flight requests — the ConcurrencyLimitMiddleware at main.py:192 caps in-flight at max(2, min(4, cpu_count // 2)), and a p95 of 620ms with request bursts gives comfortable headroom for hundreds of active sessions on a single g4dn.2xlarge node.