Most AI applications do not need fine-tuning to solve their first real problem. Many teams spend time training a model when a better prompt or a simple document search system would have worked.
The right choice depends on what you need to change: the model's instructions, the information it can access, or the way it consistently responds. Here is a practical decision tree you can use before spending money on an AI project.
1. Understand the Three Options
Prompting means giving an existing model instructions, examples, and context to guide its response. You can use system prompts, few-shot examples, and structured output instructions without changing the model's weights.
Use prompting when: the task is straightforward and you mainly need to improve instructions, response format, or behavior at a reasonable cost.
For example, you could instruct GPT-4.1 mini to return customer feedback as JSON with three fields: sentiment, issue, and priority. If the results are inconsistent, test clearer instructions and examples before considering training.
Retrieval-Augmented Generation (RAG) connects a model to external information when answering a question. The application retrieves relevant document chunks from a source such as PostgreSQL with pgvector, then supplies those chunks to the model as context.
Use RAG when: answers must reflect private, detailed, or frequently changing information that should not be built into the model itself.
For example, a company assistant can search updated product manuals, support documentation, and internal policies without retraining the model every time a document changes.
Fine-tuning trains a base model further on carefully prepared examples so it learns a more consistent pattern of behavior. Depending on the model and training method, this can improve task-specific formatting, classification, terminology, or response style.
Use fine-tuning when: you have enough representative training examples and need a behavior improvement that prompting alone cannot deliver reliably.
Fine-tuning is not the best way to keep a model updated with today's prices, new policies, or the latest customer records. Those facts are usually better supplied through retrieval or tools.
2. Follow This Decision Tree Before Choosing
Use these questions in order. Start with the simplest option that can meet your quality requirements.
Step 1: Does the answer depend on changing knowledge?
- Yes: Start with RAG or a tool/API that retrieves the latest information. Examples include current product prices, company policies, and support tickets.
- No: Continue to Step 2.
Step 2: Do you mainly need a specific style, format, or response pattern?
- Yes: Try prompting first, including a few good examples. If the required behavior remains inconsistent, evaluate fine-tuning.
- No: Continue to Step 3.
Step 3: Do you have labelled, high-quality examples?
- No: Collect examples and build an evaluation set before fine-tuning. Start with prompting or RAG if either can solve the task.
- Yes: Fine-tuning may be worth testing if you need repeatable behavior that prompting cannot achieve.
Step 4: Does your budget support the additional complexity?
- Limited budget: Begin with prompting and an existing model API. Add retrieval when access to external knowledge is necessary.
- More budget and a measurable behavior gap: Compare fine-tuning against a prompting baseline using the same test cases.
The key rule is simple: choose the method based on the problem, not on which technique sounds most advanced. You can also combine them—for example, a fine-tuned model can use RAG to access changing information.
3. Three Real Scenarios Worked Through the Tree
Scenario A: A Salesforce support assistant
A company wants an assistant that answers questions about product features and current troubleshooting procedures. Its documentation changes every month, and incorrect answers could create support problems.
The decision tree points to RAG because the knowledge changes. Store the documents in a searchable index, retrieve relevant passages for each question, and ask the model to answer using that evidence. You might use Python, an embedding model, and PostgreSQL with pgvector.
Start by testing whether the assistant retrieves the right document and answers accurately. Fine-tuning is not required just to make newly updated documentation available.
Scenario B: A support-ticket classifier
A QA team wants to classify incoming tickets into categories such as login_issue, payment_failure, and performance_bug. The categories are stable, and the team has 3,000 labelled historical tickets.
Start with prompting and a fixed evaluation set. If a smaller model consistently misses important categories or fails to follow the required JSON schema, test fine-tuning on a training split and evaluate on held-out tickets.
Compare accuracy, per-category precision and recall, latency, and total cost. The existence of 3,000 labelled examples makes fine-tuning possible to evaluate; it does not guarantee that fine-tuning will outperform a good prompt.
Scenario C: A marketing content assistant
A team wants short LinkedIn posts in a consistent brand voice. The facts are provided in each request, and the main requirement is tone, structure, and predictable length.
Begin with prompting. Supply a brand guide, two or three strong examples, and explicit instructions about sentence length and formatting. If the model still fails to reproduce the desired pattern across a large set of representative tasks, investigate fine-tuning.
RAG is unnecessary unless the assistant also needs to retrieve external brand documents or changing company information.
4. Why Teams Reach for Fine-Tuning Too Early
Fine-tuning sounds like building a custom model, so teams sometimes assume it must produce better results. But training introduces extra work: preparing examples, removing inconsistent labels, managing training runs, evaluating the result, and maintaining the model as requirements change.
It can also hide the real problem. If an assistant gives incorrect answers because it cannot access the latest policy, training it on yesterday's policy will not solve the underlying issue.
A better process is to establish a baseline, identify the specific failure, and test the least complicated fix. Improve the prompt first when instructions are unclear; improve retrieval when the wrong information is supplied; evaluate fine-tuning when the model's behavior remains the problem.
For a concrete starting point, create 100–200 representative test cases, record the expected outputs, and compare your alternatives against the same set. That number is a practical starting suggestion, not a universal minimum; complex or high-risk applications may need substantially more evaluation data.
5. Make the Choice Measurable
Before deploying any approach, define what success means. It might be classification accuracy, retrieval precision, correct JSON formatting, fewer unsupported claims, lower latency, or lower cost per successful task.
Track the baseline and compare changes rather than relying on a few impressive demonstrations. A technique is useful when it improves the metric that matters without introducing unacceptable cost or risk.
If you are learning AI engineering from a background in Salesforce, data engineering, QA, testing, or support, build one small project with prompting first. Add RAG when you need external knowledge, and explore fine-tuning when evaluation shows a clear need for more consistent behavior.
Which of the three would you pick for your own use case, and why?
Join the KKRGENAI community discussion to share your decision tree, compare project ideas, and learn from other AI builders.
