Fine-tuning or RAG? How to give your data to an AI model

A generic AI model knows nothing about your company: it doesn't know your products, your procedures, your documents. If you want it to answer questions about what concerns you, your data has to reach it somehow, and the two main routes are fine-tuning and RAG. The choice between fine-tuning and RAG shapes costs, timelines and result quality, so it's worth understanding before signing a quote, whether ours or someone else's.

What RAG does, explained without jargon

RAG stands for retrieval-augmented generation, and the idea is simple: instead of teaching the model something, you hand it the documents it needs to answer, at the right moment. When the user asks a question, the system searches your content (manuals, product sheets, procedures, contracts) for the most relevant passages and attaches them to the question. The model reads and answers based on that material.

The practical advantages:

• the data stays updatable: change a document and the answers change immediately, without retraining anything;
• the answers are verifiable: the system can cite the source, and readers can check;
• permissions stay manageable: you can filter the retrieved documents based on who's asking, which is impossible with knowledge baked into the model;
• you can start small: the documents you already have are enough, cleaned up and organized.

The real work of RAG, what separates a prototype from a reliable system, lies in retrieval quality: how you split the documents, how you index them, how you pick the passages to hand to the model. If retrieval brings the wrong text, the model answers well about the wrong thing.

What fine-tuning does, and what it doesn't

Fine-tuning partially retrains a model on your examples: question-and-answer pairs, texts in your format, annotated dialogues. The result is a model that has absorbed a style or a behavior.

The key point, which overturns many expectations: fine-tuning teaches how to answer, not what to know. It works well for locking in a tone of voice, for getting output in a rigid, repetitive format, for specializing a model on a narrow, high-volume task. It works badly as a repository of company knowledge: facts learned in training grow stale, can't be updated document by document, and the model can still mix them up.

There are also the costs to consider: preparing a quality dataset requires significant human work, every substantial update requires a new training and evaluation cycle, and you need a process to measure whether the specialized model is better than the one you started with. It's a sensible investment when the task is stable and the volumes pay for it.

The practical rule for choosing

After several projects, the summary we give clients is this:

• Does the knowledge change over time? Price lists, procedures, living documentation: RAG.
• Do you need to cite sources or manage different permissions for different users? RAG.
• Is the problem the format or style of the output, always the same and high-volume? Fine-tuning.
• Is the budget limited and do you need a result quickly? RAG, because it gets to a usable system with less initial investment.
• Do you already have a working RAG but the prompts have grown enormous to force the behavior? That's the moment fine-tuning starts to make sense, on top of the RAG, not in its place.

The two techniques aren't in competition: mature systems often combine them, with RAG for knowledge and fine-tuning for behavior. But in our experience most business projects get what they need from well-executed RAG alone.

How to start with a concrete project

The path we follow when a company wants to give its data to a model always starts from the use case, not the technology: which questions the system must be able to handle, who will ask them, what happens when it gets one wrong. Then we look at the state of the data, because scattered, duplicated and outdated documents produce scattered, duplicated and outdated answers: cleaning up the sources is often half the project.

An example from our work: in CareCloud, the ERP management system we developed for healthcare and social care facilities, the integrated AI chat works alongside sites, patients, shifts, documents and analytics. In a context like that, AI has value only if it answers based on the facility's data, with the same attention to permissions as the rest of the system: it's the kind of constraint that shapes the architecture far more than the trends of the moment.

Want to figure out which route suits your data?

We develop custom software with integrated AI features, from RAG over company documents to assistants inside management systems. If you're weighing how to put your data to work with an LLM, book a free call: we'll analyze your use case and tell you, with concrete reasons, whether you need RAG, fine-tuning or neither.