Training a model is 20% of the work. Serving it reliably at low cost with good latency is the other 80%. This lesson covers the serving stack for fine-tuned open-source models.
Select a serving option to see when it fits, cost tier, latency characteristics, and deployment code.
from huggingface_hub import InferenceClient
# After pushing your model to HF Hub:
hf_client = InferenceClient(model="your-username/llama-3b-product-extractor")
def extract_product(description: str) -> dict:
import json
prompt = (
"### Instruction:\nExtract product JSON.\n\n"
f"### Input:\n{description}\n\n"
"### Response:\n"
)
response = hf_client.text_generation(
prompt,
max_new_tokens=200,
do_sample=False,
return_full_text=False,
)
return json.loads(response.strip())
result = extract_product("The Sony WH-1000XM5 headphones, priced at $349.99, feature 30hr battery.")
print(result)
# → {"name": "Sony WH-1000XM5", "price": "$349.99", "features": ["30hr battery"]}| Option | Latency | Cost | When to Use |
|---|---|---|---|
| HF Inference API | Medium (shared) | Low (free tier) | Prototyping, low traffic |
| HF Dedicated Endpoint | Fast (dedicated) | Medium ($0.60–$6/hr) | Production, consistent traffic |
| vLLM on GPU VM | Fastest (batching) | GPU VM cost (~$1–3/hr) | High traffic, optimized throughput |
| Modal / Replicate | Medium | Per-token, no idle cost | Bursty, unpredictable traffic |
| Ollama (local) | Variable | Hardware cost only | Local dev, private data |
Choosing your serving stack: Start with HF Inference API during development. Move to vLLM on a dedicated GPU when you cross 50 requests/minute. Use Modal or Replicate for unpredictable traffic spikes. GGUF/CPU for privacy-critical deployments.
You are ready for the capstone: fine-tuning a domain expert model end-to-end, deploying it, and benchmarking it against a prompted frontier model.