Production Experiences with RAG Architectures: Scalability and Accuracy

What Is a RAG Architecture and Why Is It Hard in Production?

RAG (Retrieval-Augmented Generation) is an architecture that lets a language model retrieve relevant documents from an external knowledge source before generating an answer. This allows the model to use current or domain-specific information that was not part of its training data. However, a RAG system that works well in the prototype stage runs into different challenges in production: accuracy, latency and scalability top the list.

In this article I will share the experience I gained while working on medical question-answering systems. Because medicine is a field where a wrong answer can have serious consequences, accuracy and reliability come first. Below you will find the strategies we applied in production and the problems we ran into.

Document Chunking and Indexing Strategies

The foundation of a RAG system is chunking and indexing documents correctly. In medical texts, sentences are tightly bound to one another; poor chunking leads to a loss of context. In our first attempts we used fixed-size chunking, but this approach proved inadequate, especially for long diagnostic and treatment descriptions.

We then moved to semantic chunking. We split the chunks along heading and subheading boundaries and added the parent heading information to each chunk. This improved precision, particularly in vector database searches. During indexing we stored metadata for each chunk (source, section, date), which allowed us to show the source of each answer.

There is no clear-cut recipe for chunk size. In our experience, chunks of 200-400 words gave better results for medical questions. But this can vary with the dataset and the query type. What matters is that each chunk is meaningful on its own and that its connection to neighbouring chunks is preserved.

Query Expansion and Context Management

Users usually ask short, vague questions. A query like "diabetes treatment" points to a very broad set of answers. This is where query expansion techniques come in. We enriched the vector search by expanding the user query into several variations. For example, we added synonyms and the Latin equivalents of medical terms.

Careful management of the context window is also essential. Retrieving too many documents can confuse the model and increases latency. We retrieved the 3-5 most relevant chunks and presented them to the model in order of importance. This had a positive effect on both answer quality and speed.

Answer Quality Evaluation and Continuous Improvement

In production, the definition of a "good answer" must be clear. In medical question answering, accuracy, source citation and currency are critical. We ran every answer through an evaluation pipeline for automatic source verification and consistency checking. This pipeline checks whether the answer contradicts the retrieved documents.

Human evaluation is part of the process as well. Through weekly samples we reviewed the answers, flagged incorrect ones and fed those examples into the improvement loop. This loop consists of the following steps:

• Root cause analysis of incorrect answers (chunking, retrieval, model)
• Updating the relevant chunks and metadata
• Revising query expansion rules
• Improving model prompts
• Measuring the changes with A/B tests

This continuous improvement loop made the system more reliable over time.

Multi-Channel Notification and Automation Integration

In production, a RAG system is not just an API; it is also part of notification and automation flows. For instance, when a new medical guideline is published, the index must be updated and the relevant users notified. We automated such workflows with tools like n8n and Make.

The notification channels include email, Slack and mobile push. Each channel has its own rate limits and formatting requirements. When designing automation flows, it is important not to neglect concerns such as retries and queue management in failure scenarios.

Scalability and Latency Management

One of the biggest challenges a RAG system faces in production is latency. Vector search, document retrieval and language model inference add up, and response times can stretch. We took the following measures:

• Caching: We cached the answers to frequently asked questions.
• Asynchronous processing: We queued long-running queries and sent the user a status notification.
• Model selection: We used faster, smaller models for simple queries.
• Vector index optimisation: We shortened search times with approximate nearest neighbour (ANN) algorithms.

Horizontal scaling and load balancing are essential for scalability. Redundancy of the vector database and a disaster recovery plan are also critical in production.

Conclusion: Practical Recommendations for RAG in Production

When moving a RAG architecture into production, reliability, traceability and continuous improvement come to the fore, unlike in a prototype. In high-risk areas such as medical question answering, source citation and verification are indispensable. I can summarise the lessons from our experience as follows:

1. Chunk documents along semantic boundaries and do not neglect metadata.
2. Improve retrieval precision with query expansion, but do not bloat the context.
3. Build a quality pipeline that combines automatic and human evaluation.
4. Integrate automation flows (n8n, Make) into notification and update processes.
5. Use caching and asynchronous processing to manage latency.

Every system takes shape around its own data; so do not accept any setting as permanent without measuring first.