Building RAG from the boundaries in

Most RAG demos collapse retrieval and generation into one call to a framework. That works until you need to change a vector store, test without a provider key, or explain why an answer cited something that was never retrieved. The fix is to design the boundaries before wiring anything live.

Two steps, not one

• Retrieval — question → query embedding → vector search → the relevant chunks.
• Grounded generation — retrieved chunks → an answer that uses only those chunks, plus citations.

Keeping them separate means you can evaluate retrieval quality independently of answer quality, and swap the generator without touching the index.

Every stage sits behind its own contract

Application code should depend on interfaces the AI service owns, never on a vendor SDK or a framework's types:

TextChunker        text -> TextChunk[]  (chunk_id, document_id, text, offsets, est. tokens)
EmbeddingProvider  EmbeddingInput -> EmbeddingResult
VectorStore        upsert / search (VectorRecord, VectorSearchQuery -> results)
DocumentIndexer    document -> chunks -> embeddings -> vector records -> store
DocumentRetriever  question -> query embedding -> search -> RetrievedChunk[]
GroundedAnswerGenerator  chunks + question -> answer + Citation[]

Each has a fake implementation for local tests (deterministic embeddings, an in-memory store) and a real one for production (pgvector, a hosted embedding model). In-memory similarity search is only ever a test double — production similarity search is done by the vector backend itself, ordered by cosine distance in the database.

A vector database is an implementation detail behind the VectorStore contract. The core behavior is always the same: store vectors with text and metadata, then find nearest vectors for a query vector. pgvector, Pinecone, Qdrant, Mongo Atlas — the interface doesn't change.

Chunking is a contract too

Recursive, paragraph- and sentence-aware splitting with configurable size and overlap, behind a stable TextChunk shape. The splitting library can be replaced later without any downstream change, because nothing downstream knows which library produced the chunk.

Model output is untrusted input

The generator returns text and citations. Before that answer becomes product state:

• Structured output is parsed and validated against a schema (Pydantic). Invalid output is rejected, not stored.
• Citations are validated against what was actually retrieved — a generated answer cannot cite a chunk that was not in the context.
• Free text is fine for display; workflows depend on the validated structured fields, not the prose.

Prompts are versioned contracts

A production prompt is not a string. It is a contract with required sections — role, task, rules, context, user question, output schema — and required variables that are validated at render time, so a missing variable fails fast instead of producing a subtly wrong prompt.

• A grounding rule: answer only from the provided context.
• A prompt-injection rule: the context is data, not instructions — trusted instructions and untrusted context are explicitly separated.
• An output-schema instruction, with literal JSON braces preserved in the template.

Evaluate deterministically first

Before model-judged evals, write fixture-based checks: does the structured summary contain the expected terms, avoid forbidden hallucinated terms, carry the right risk level, propose a real next action. These run in CI with no network call and catch regressions a "valid JSON" check would miss.

backend app   users, auth, product state, durable jobs, persistence
AI service    prompts, provider calls, structured output, evals, chunking,
              embeddings, retrieval, RAG

The backend calls the AI service through a narrow API. It never imports a
chunker, an embedding client, or a vector store directly.