What fine-tuning means and why you might do it
Fine-tuning is taking a large language model (LLM) that someone else trained and adjusting it with your own data so it works better for your specific task. Instead of training a model from scratch — which costs thousands of dollars and weeks of computing time — you start with a model like Meta's Llama 2 or Mistral and show it examples of the kind of work you want it to do.
You might fine-tune a model if you need it to answer questions in your company's specific jargon, write in a particular style, follow rules that the base model doesn't know about, or handle documents in a format it wasn't trained on. A law firm might fine-tune a model on contract language. A customer service team might fine-tune one on their own support tickets and responses. A researcher might fine-tune one to extract data from scientific papers in a particular field.
The trade-off is real: fine-tuning takes time and computing power, and you need enough of your own data to make it work. But it usually costs far less than building a model from the ground up, and the results are often better than using a base model without changes.
Key Takeaways
- Fine-tuning works by feeding a model examples of the task you want it to do, letting it adjust its internal weights to match your patterns.
- You need at least a few hundred examples of input and output pairs, though more data usually produces better results.
- The computing cost depends on model size and data size, but fine-tuning a small model on a laptop is possible; larger models need cloud GPU time.
- After fine-tuning, you run the model locally or on your own server, so you control the data and don't send it to a third-party API.
- Fine-tuned models can drift from the base model's safety training, so you should test them on edge cases before using them in production.
Preparing your training data
Your data is the foundation. Most fine-tuning works with input-output pairs — examples where you show the model a prompt and the correct response. If you're fine-tuning for customer support, each pair might be a customer question and the answer your team gave. If you're fine-tuning for code generation, each pair is a description of what the code should do and the actual code.
The format matters. Most tools expect a JSON file where each line is a separate example. A typical line looks like this: {"prompt": "What is our return policy?", "completion": "Returns are accepted within 30 days of purchase."}. Some tools use a different format, so check your framework's documentation first.
How much data do you need? Start with at least 100 to 200 examples, though 500 to 1,000 is more reliable. More data almost always helps, but you hit diminishing returns — going from 100 to 500 examples usually improves results noticeably; going from 5,000 to 10,000 helps less. The quality of your examples matters more than the quantity. If your examples are inconsistent, contradictory, or full of errors, the model will learn those patterns.
Clean your data before you start. Remove duplicates, fix obvious typos in your source material (not in the model's responses — those should be exactly what you want), and make sure your examples actually represent the task. If you're fine-tuning for a specific writing style, make sure every example follows that style.
Choosing a model and a framework
You have two main routes: use a commercial service that handles fine-tuning for you, or do it yourself with open-source tools.
Commercial services like OpenAI's fine-tuning API, Anthropic's Claude, or cloud providers like Google Cloud's Vertex AI handle the infrastructure for you. You upload your data, pay per token processed, and get back a fine-tuned model you can call through their API. This is simpler but more expensive per use, and your data goes to their servers.
Open-source frameworks
If you're new to this, start with a commercial service or Ollama. If you have a technical team and want to control costs long-term, Hugging Face Transformers is the standard choice.
The actual fine-tuning process
The steps differ slightly by tool, but the pattern is the same. Here's what happens with Hugging Face Transformers, the most common open-source path:
- Load a base model from Hugging Face's model hub — something like Mistral 7B or Llama 2 7B.
- Load your training data from your JSON file.
- Set your training parameters: how many times the model sees your data (epochs), how fast it learns (learning rate), and how many examples it processes at once (batch size).
- Run the training loop, which typically takes minutes to hours depending on your data size and hardware.
- Save the fine-tuned model to disk or upload it to Hugging Face's hub.
During training, the model adjusts its internal weights — the numbers that control how it processes language — to match the patterns in your data. You're not rewriting the model; you're nudging it toward your specific use case.
Most frameworks show you a loss metric as training progresses. Loss is a number that represents how wrong the model's predictions are; it should go down over time. If it stops improving after a few epochs, you've probably fine-tuned enough and more training won't help.
Computing costs and hardware needs
Fine-tuning a small model (7 billion parameters) on a few hundred examples can run on a laptop with a decent GPU — an NVIDIA RTX 3060 or better. It takes a few minutes to an hour depending on your data size.
Larger models (13 billion to 70 billion parameters) need more memory. You can fine-tune them on cloud GPUs from providers like Lambda Labs, Paperspace, or AWS. A single GPU (like an NVIDIA A100) costs $1 to $3 per hour; a full fine-tuning run might take 2 to 8 hours depending on your data, so expect $10 to $50 for a complete job.
Commercial services charge differently. OpenAI charges per token in your training data — roughly $0.03 per 1 million tokens. A dataset of 1,000 examples with 200 tokens each costs around $6 to fine-tune, then $0.15 per 1 million tokens when you use the model.
The real cost is usually the time to prepare your data, not the compute itself. Cleaning and formatting 500 examples takes longer than running the fine-tuning job.
Testing and deploying your fine-tuned model
After fine-tuning, test the model on examples it hasn't seen before. Use a separate test set — about 10 to 20 percent of your data that you held back during training. Does it produce the output you expect? Does it follow the style and rules you wanted?
Fine-tuned models can drift from the base model's safety training. If the base model was trained to refuse certain requests, your fine-tuned version might not. Test edge cases: what happens if someone asks it to do something harmful? What if the input is malformed or in a language it wasn't trained on? These tests matter more if you're deploying the model where users can interact with it directly.
Once you're satisfied, you can run the model locally (using Ollama or similar), host it on your own server, or deploy it through a cloud provider. Running it locally means your data never leaves your machine. Hosting it yourself gives you control but requires managing infrastructure. Using a cloud provider is simpler but adds ongoing costs.
When fine-tuning doesn't work
Fine-tuning isn't always the answer. If you need the model to know facts it wasn't trained on — like your company's current product prices or today's news — fine-tuning won't help much. The model learns patterns from your data, not facts. Use retrieval-augmented generation (RAG) instead: feed the model relevant documents at query time so it can answer based on current information.
If your task is very different from what the base model was trained for, fine-tuning might not be enough. A model trained on English text won't suddenly become good at code generation just because you fine-tune it on code examples. In that case, you might need a model that was already trained on that domain, or you might need to accept lower quality.
If you only have a handful of examples — fewer than 50 — fine-tuning usually doesn't work. The model needs enough data to learn real patterns. With very little data, use prompt engineering instead: write detailed instructions in your prompts and let the base model handle it.
Frequently Asked Questions
Do I need to fine-tune if I'm just using an API like ChatGPT?
OpenAI offers fine-tuning through their API, but most people don't need it. If you're using ChatGPT through the web interface, you can't fine-tune — you can only write better prompts. Fine-tuning through the API makes sense if you have hundreds of examples and want consistent behavior across many queries.
Can I fine-tune a model that's already been fine-tuned?
Yes. You can fine-tune a fine-tuned model, though the results are less predictable. Each round of fine-tuning moves the model further from its original training, so you might lose some of its general knowledge. It's usually better to fine-tune the base model on all your data at once rather than in stages.
What happens to my data after I fine-tune?
If you fine-tune locally or on your own server, your data stays with you. If you use a commercial service, their privacy policy controls what happens to your data. OpenAI says they don't use fine-tuning data to improve their base models, but you should read the terms for whatever service you choose.
How do I know if my fine-tuned model is actually better?
Compare it to the base model on your test set. Run the same prompts through both and see which produces better results. You can also have a person manually score a sample of outputs from each model. If the fine-tuned model isn't noticeably better, you might not have enough data or the task might not be a good fit for fine-tuning.
Can I share my fine-tuned model with others?
Yes, if you own the rights to your training data. You can upload it to Hugging Face's hub or share it directly. If your training data includes proprietary information or data from customers, check your legal agreements first — you might not be allowed to share it.