Lesson 1: Fine-Tune vs. RAG vs. Prompt Engineering

Fine-tuning is the right tool for fewer situations than most engineers think. Before training, you should be able to articulate exactly why prompt engineering or RAG will not solve the problem.

The Decision Framework

Ask these questions in order. Stop when you find a solution that works:

1. Does a careful system prompt solve it?

→ YES: Use prompt engineering. Cost: ~$0.

→ NO: Continue.

2. Is the problem lack of knowledge (facts the model doesn't have)?

→ YES: Use RAG. Cost: ~$0.01–$0.10/query.

→ NO: Continue.

3. Is the problem style, tone, or task format (not knowledge)?

→ YES: Fine-tuning is worth exploring.

4. Do you have >1,000 high-quality training examples?

→ YES: Proceed with fine-tuning.

→ NO: Collect data first — or use few-shot prompting.

Approach Selector

Select an approach to see when it fits, what it costs, and a working code example.

Prompt Engineering

When to use:

Style or tone can be described in a system prompt
Output format varies per request (few-shot examples cover it)
You need to ship today — no data collection time

Cost: ~$0 setup. Standard inference cost per call.

Quality: High for tasks within the model's capability. Inconsistent at ~85–90% format compliance at scale.

# Prompt engineering — zero setup cost
from openai import OpenAI
client = OpenAI()

system_prompt = """You are a product data extractor.
Always respond with valid JSON in this exact schema:
{"name": string, "price": string, "features": list[string]}
Never add commentary outside the JSON object."""

def extract_product(description: str) -> dict:
    import json
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": description},
        ],
        response_format={"type": "json_object"},   # JSON mode
    )
    return json.loads(response.choices[0].message.content)

result = extract_product("Sony WH-1000XM5 headphones, $349.99, 30hr battery.")
# → {"name": "Sony WH-1000XM5", "price": "$349.99", "features": ["30hr battery"]}

When Fine-Tuning Is the Right Choice

Consistent output format at scale

You need every response in a specific JSON schema and prompt-only reliability is 85%. Fine-tuning reaches 99%.

Domain-specific tone at high volume

Legal, medical, or brand tone that is too complex for a system prompt but needed on millions of requests.

Reduce prompt length for cost

Moving a 2,000-token few-shot prompt into model weights cuts inference cost by 70% at scale.

Teach the model new factual knowledge

Fine-tuning does not reliably implant facts — it changes behavior, not memory. Use RAG for knowledge.

Solve occasional hallucinations

Hallucinations are a probabilistic model property. Fine-tuning on examples may reduce them slightly but won't eliminate them.

You have fewer than 200 training examples

Below 200 examples, few-shot prompting usually outperforms fine-tuning. Collect more data first.

In the next lesson, we dive into how supervised fine-tuning works mechanically: what gradient updates happen, what the Alpaca data format looks like, and how the training loop differs from pre-training.

# Prompt engineering — zero setup cost from openai import OpenAI client = OpenAI() system_prompt = """You are a product data extractor. Always respond with valid JSON in this exact schema: {"name": string, "price": string, "features": list[string]} Never add commentary outside the JSON object.""" def extract_product(description: str) -> dict: import json response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": description}, ], response_format={"type": "json_object"}, # JSON mode ) return json.loads(response.choices[0].message.content) result = extract_product("Sony WH-1000XM5 headphones, $349.99, 30hr battery.") # → {"name": "Sony WH-1000XM5", "price": "$349.99", "features": ["30hr battery"]}