There are three ways to make a general AI model useful with your own company's information. Retrieval-augmented generation (RAG) looks up the relevant passages at question time and hands them to the model. Fine-tuning adjusts the model itself with your examples. Long context skips both and simply places whole documents in the prompt. For most business knowledge questions, RAG is the right starting point, because it keeps answers tied to current, citable documents. Fine-tuning is for changing how a model behaves, not what it knows. Long context is for a small number of documents that must be read in full. Many working systems combine two of the three.
What does each approach actually change?
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| What changes | The prompt, with retrieved passages | The model's weights | The prompt, with whole documents |
| Best at | Facts from a large, changing document set | Style, format, terminology, narrow tasks | Reasoning across one or a few full documents |
| Keeping current | Re-index the changed documents | Retrain | Send the new document |
| Citations | Natural: the passages are known | Not available | Possible, less precise |
| Access control | Can filter by user at retrieval | Hard: knowledge is baked in | Controlled by what is sent |
| Up-front effort | Medium: pipeline and index | High: curated training data and evaluation | Low |
| Cost per question | Low | Lowest | Highest |
| Main failure | Retrieves the wrong passage | Outdated or over-fitted model | Slow, costly, detail lost in the middle |
When is RAG the right choice?
Choose RAG when the answer lives in documents: policies, contracts, manuals, product data, past tickets. The documents are split into passages, converted to embeddings and stored in a vector database. At question time the system retrieves the closest passages and the model answers from them. Update a document and the next answer reflects it, with no retraining.
RAG also suits organisations where different staff may see different documents, because retrieval can respect permissions. Its weakness is that answer quality is capped by retrieval quality. Poorly scanned PDFs, tables split across chunks and vague questions all hurt. We walk through a build in a private knowledge assistant with RAG and compare ready-made front ends in Open WebUI vs AnythingLLM.
When is fine-tuning the right choice?
Choose fine-tuning when the problem is behaviour. Examples: always answer in your house report format, classify tickets into your own categories, use your industry's terminology correctly, or make a small model perform one narrow task as well as a much larger one. Methods such as LoRA and QLoRA train a small set of added weights, which brings the job within reach of a desktop system. See our guides to LoRA and QLoRA on GB10 systems and on DGX Station GB300.
Fine-tuning is a poor way to teach facts. The model may blend them, cannot cite them, and has to be retrained when they change. It also needs a few hundred to a few thousand good examples and a test set to prove the result is better. A related technique, distillation, uses a large model to produce that training data.
When is long context the right choice?
Choose long context when the task needs the whole document at once: compare two contracts clause by clause, summarise a full report, review a codebase. It is also the fastest way to test an idea, since there is nothing to build.
The cost is paid on every request. Each token in the prompt has to be processed and held in the KV cache, so long prompts raise memory use and time to first token, and reduce how many users one machine can serve. Models can also miss details buried in the middle of very long inputs.
How do the running costs compare?
CALC. Take a 200-page staff handbook at roughly 500 tokens per page: about 100,000 tokens. The team asks 500 questions a day.
- Long context: 500 questions x 100,000 tokens = 50,000,000 input tokens a day.
- RAG: 5 retrieved passages x 500 tokens = 2,500 tokens per question. 500 x 2,500 = 1,250,000 input tokens a day.
RAG processes 40 times fewer input tokens for the same questions (50,000,000 / 1,250,000). On a cloud API that is a direct difference in the bill. On your own hardware it is the difference between a few users and many. Prompt caching narrows the gap for a document that never changes, but does not remove the memory it occupies. The page and passage sizes here are illustrative; measure your own documents.
RAG gives a model your knowledge. Fine-tuning gives it your habits. Long context gives it one document to read carefully. Most disappointments come from using one to do another's job.
Which should a business choose?
| If your need is | Start with |
|---|---|
| Answer staff questions from company documents | RAG |
| Answers must cite the source | RAG |
| Documents change weekly | RAG |
| Output must follow a strict format or tone | Fine-tuning, or a detailed prompt first |
| A small, fast model for one repetitive task | Fine-tuning |
| Review or compare a few long documents | Long context |
| A quick proof of concept this week | Long context |
| Domain assistant with correct terminology and current facts | Fine-tuning plus RAG |
A sensible order of work: try a good prompt first, then long context to prove the idea, then RAG when the document set grows, and fine-tuning only when a measured behaviour problem remains.
What hardware does each approach need?
- RAG: a system that runs the language model plus a small embedding model, and fast storage for the index. Memory needs are modest because prompts stay short.
- Fine-tuning: more memory than inference on the same model, for hours rather than seconds. Large unified memory helps; see why 128GB of unified memory matters.
- Long context: memory for the KV cache of every concurrent long prompt, and strong compute to keep time to first token acceptable.
Buying in Singapore
For RAG and LoRA fine-tuning on a desk, the NVIDIA DGX Spark 4TB (S$9,450) or HP ZGX Nano G1n 4TB (S$11,499), both with 128GB of unified memory. For faster long-context work and more concurrent users in a workstation, the NVIDIA RTX PRO 6000 Blackwell Workstation Edition with 96GB (S$29,100). To hold the source documents with snapshots, the Synology DS925+ (S$1,260) (listed 11 Nov 2026; prices change; ask for a current quotation). DGX Station GB300 systems such as the ASUS ExpertCenter Pro ET900N G3 are price on request. To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.
Frequently asked questions
What is the difference between RAG and fine-tuning? RAG retrieves relevant passages from your documents at question time and gives them to the model. Fine-tuning changes the model's weights using your examples. RAG adds knowledge; fine-tuning changes behaviour.
Does a long context window replace RAG? Not for large or frequently used document sets. Sending whole documents with every question costs far more compute and memory per request, and models can miss details in very long inputs. Long context suits a few documents that must be read in full.
Can RAG and fine-tuning be used together? Yes. A common design fine-tunes a model for the organisation's format and terminology, then uses RAG to supply current facts with citations.
Which approach is cheapest to start? Long context needs no build work, so it is cheapest to start but most expensive per question. RAG has a moderate set-up cost and a low running cost. Fine-tuning has the highest up-front effort.
