Fine-Tuning Techniques, Explained
An illustrated guide to fine-tuning: how CPT, SFT, DPO, GRPO, SDPO and model distillation work, where LoRA fits, and how to evaluate quality, latency and cost.
Your data can teach a model more than a prompt can.Follow six animated training flows, from domain textand expert demonstrations to preferences, rewardsand distillation. Choose the signal your task needs.
- 01Start with the learning signal
- 02CPT: learn the language of a domain
- 03SFT: demonstrate the behavior you want
- 04DPO: teach a preference between responses
- 05GRPO: practice against a reliable verifier
- 06SDPO: learn from hindsight
- 07Distillation: move a capability into a smaller model
- 08LoRA and QLoRA: how much of the model changes
- 09Bring the training loop into your real environment
Companies with demanding, repeatable AI use cases can build an advantage by improving the models behind their products. Prompts, retrieval and context engineering provide essential instructions and information. Training adds another lever: it changes the model’s learned behavior, so the capability can carry across many requests.
The opportunity is practical. A domain specialist can understand unfamiliar terminology more readily. A tool-using agent can learn the schemas and workflows of your environment. A smaller model can learn a focused capability from a larger teacher. Those benefits must be measured on your workload; fine-tuning is a way to pursue them, not an automatic guarantee.
Start with the learning signal
The most useful question is what evidence you can provide. A corpus teaches domain language. An excellent response demonstrates behavior. A preference pair expresses a judgment. A verifier rewards a successful outcome. Detailed feedback explains where an attempt went wrong. Teacher guidance transfers a capability to a student.
The flows below follow that evidence through a training step. They are conceptual illustrations, not recorded model runs or performance benchmarks. The Python documentation image and GSM8K question are real sources; preference pairs, code attempts and reward displays are constructed examples. Library names identify implementation options, not endorsements or customer deployments.
CPT: learn the language of a domain
Continued pretraining extends a pretrained model’s language-modeling objective on a new corpus. A manufacturer might curate technical manuals; a developer-tools team might use approved code and documentation. Text becomes tokens, the model predicts each next token, and the observed continuation supplies the target. You do not need to write an ideal response for every document.
Collect the corpus
Technical documentation, code and approved domain text become training material. No question–answer labels are required.
corpus = domain_docs + general_replay
Read the full flow
- Collect the corpus. Technical documentation, code and approved domain text become training material. No question–answer labels are required.
- Pack token sequences. Tokenize and pack the text. A replay mix helps preserve useful knowledge from the original model.
- Predict the next token. The model predicts each next token from the preceding text. The next token in the document supplies the target.
- Update the weights. Cross-entropy penalizes missed targets. Gradients update the model or its adapters across the corpus.
CPT is useful when the gap is domain fluency: terminology, recurring structures and relationships that appear throughout a body of text. It does not automatically produce a helpful assistant or a reliable tool user. Those behaviors may require later instruction training. Curate the corpus, manage duplication and mix in general data where appropriate; specializing too narrowly can erode existing capabilities.
Keep the distinction between stable knowledge and changing facts. A model may benefit from learning the language of inventory management. Today’s stock count should still come from the inventory system. Training is also a poor mechanism for making a single fact easy to remove later.
Implementation reference: Transformers causal language modeling
SFT: demonstrate the behavior you want
Supervised fine-tuning teaches from examples of the desired response. These can be expert-written answers, structured JSON, or successful tool interactions. For an agent, capture the real tool names, arguments, observations and next actions. A polished answer alone cannot demonstrate all the steps that made it reliable.
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
48 / 2 = 24 48 + 24 = 72 #### 72
Pair task and answer
Use reviewed responses, structured outputs or successful tool traces. This viewer shows a real GSM8K training example.
record = {"prompt": question, "completion": answer}Read the full flow
- Pair task and answer. Use reviewed responses, structured outputs or successful tool traces. This viewer shows a real GSM8K training example.
- Mask the prompt loss. In this completion-only setup, prompt tokens provide context but do not contribute to the training loss.
- Learn from the answer. Teacher forcing supplies the earlier gold answer tokens as context. The model predicts the next answer token, without seeing that target in advance.
- Save the new behavior. Backpropagate the answer-token loss, then evaluate a held-out task with the same tools and format used in production.
In the completion-only setup illustrated here, the prompt is context and its tokens are excluded from the loss. Earlier gold response tokens also provide context as the model predicts the next response token: this is teacher forcing. The current target is not visible ahead of its prediction. Other SFT configurations train on different token subsets, so inspect the actual loss mask and chat template.
SFT is often a useful starting point when you can show what success looks like. Include difficult and varied examples, including correct refusals and recovery from tool errors where relevant. Hold out tasks that test generalization; near-duplicates of training examples can make a weak model look strong.
Implementation reference: TRL SFTTrainer
Dataset shown in the walkthrough: GSM8K, released by OpenAI
DPO: teach a preference between responses
Direct preference optimization starts with two responses to the same prompt: one chosen, one rejected. A pair can express a preference for grounded tool use, a clearer explanation or adherence to an internal policy. The labels need a coherent rubric. A longer answer is not necessarily a better answer.
Find shipment #1042 using the documented orders.get tool.
orders.get({order_id: "1042"})get_order_status({id: "1042"})Compare a response pair
A reviewer or validated rubric chooses the better answer to the same prompt. These two responses are constructed teaching examples.
batch = {"prompt": task, "chosen": a, "rejected": b}Read the full flow
- Compare a response pair. A reviewer or validated rubric chooses the better answer to the same prompt. These two responses are constructed teaching examples.
- Score with two policies. Both the trainable policy and a frozen reference score the chosen and rejected responses. The reference provides an anchor.
- Compute the preference loss. Compare chosen-versus-rejected log-probability margins relative to the reference. Beta controls the objective’s reference tradeoff.
- Update the policy. Increase the relative preference for chosen answers. Standard offline DPO needs neither fresh rollouts nor a separately trained reward model.
The trainable policy and a frozen reference score both responses. DPO optimizes the chosen-versus-rejected log-probability margin relative to that reference. Standard offline DPO works on existing pairs without generating new rollouts during each training step or fitting a separate reward model.
Use it when comparison is easier to collect than a single perfect answer. SFT can also teach tone and judgment; DPO is a different supervision format, not the exclusive route to those behaviors. Evaluate whether learned preferences carry over to unseen situations and whether the model preserves factual and task performance.
Research: Direct Preference Optimization
Implementation reference: TRL DPOTrainer
GRPO: practice against a reliable verifier
Group relative policy optimization lets the current model attempt the task several ways. A verifier or reward function scores the attempts. The update compares each score with its group baseline and reinforces relatively successful behavior. Unlike actor–critic approaches, GRPO does not require a separate learned critic to estimate the baseline.
48 clips in April. Half as many in May. How many in total?
Sample several attempts
The current policy generates a group of answers to one problem. This small example uses four candidates.
completions = policy.generate(prompt, n=4)
Read the full flow
- Sample several attempts. The current policy generates a group of answers to one problem. This small example uses four candidates.
- Run the verifier. Check the final result or run executable tests. Here the checker accepts 72; the four displayed rewards are illustrative.
- Compare within the group. Center rewards on the group baseline, commonly with standard-deviation scaling. Better-than-group attempts get positive advantage.
- Improve and sample again. Use a clipped policy objective to reinforce better attempts, with reference regularization when configured. No learned critic is needed.
This is attractive when success is checkable: a program passes tests, a calculation is correct, or an agent completes a task in a controlled environment. The quality of the reward becomes central. An incomplete test suite can reward a shortcut; a weak judge can prefer plausible prose over a correct result.
Generation is part of the training cost. Plan for rollout throughput, sequence length, environment execution and weight synchronization alongside the optimizer. Libraries such as TRL and verl can coordinate training, while vLLM can serve generation. A reliable evaluation set remains separate from the reward used for learning.
Research: DeepSeekMath, which introduced GRPO
Implementation reference: TRL GRPOTrainer
SDPO: learn from hindsight
A failed test often tells you more than “incorrect.” It can identify an exception, a missing edge case or a violated contract. Self-distillation policy optimization uses this richer information to form a teaching signal. The student attempts the task; a self-teacher conditions on useful feedback; the student then learns from the teacher’s feedback-informed token predictions.
def mean(xs):
return sum(xs) / len(xs)Return None for an empty list. Keep the numeric mean for non-empty lists.
Let the student try
The student generates an attempt from the original task. This teaching example asks it to implement a safe mean function.
def mean(xs):
return sum(xs) / len(xs)Read the full flow
- Let the student try. The student generates an attempt from the original task. This teaching example asks it to implement a safe mean function.
- Collect rich feedback. Run the attempt in its environment. A failing test explains more than a single success or failure score.
- Build the self-teacher. Condition the same model on useful feedback or a successful attempt. Its feedback-informed token predictions become teaching targets.
- Distill the correction. Train the student’s original-context distribution toward the feedback-informed teacher. Exact divergence and teacher updates depend on the implementation.
The important separation is between the teacher’s information and the student’s starting point. The teacher can see hindsight that the student did not originally have. Training tries to transfer the benefit of that hindsight into the student’s initial policy, so later attempts may need less correction.
SDPO is an emerging approach rather than a universal replacement for SFT or reinforcement learning. Feedback quality, the teacher construction and the distillation objective matter. TRL exposes it in an experimental module, with options for successful-rollout context, different teacher updates and blended losses. Pin versions and validate the specific implementation you use.
Research: Reinforcement Learning via Self-Distillation
Implementation reference: TRL experimental SDPO
Distillation: move a capability into a smaller model
Conventional teacher–student distillation can make a strong capability more practical to serve. Choose a teacher that performs well on your target tasks, collect its guidance, and train a student to reproduce the useful behavior. Accepted teacher responses can become SFT examples. When token distributions are available, a distillation loss can transfer a richer signal.
Accepted responses or token probabilities teach the student the target behavior.
Choose the target tasks
Collect representative prompts and select a teacher that performs well on them. Check data rights and model license terms.
tasks = held_in_training_distribution
Read the full flow
- Choose the target tasks. Collect representative prompts and select a teacher that performs well on them. Check data rights and model license terms.
- Capture teacher guidance. Generate and review teacher answers, or access its token probabilities. The available signal determines the distillation setup.
- Train the student. Use SFT on accepted responses or match teacher distributions with a distillation loss. On-policy approaches also learn on student-generated text.
- Measure the tradeoff. Benchmark the student against the teacher with held-out tasks. Ship when the smaller model meets the quality and serving budget.
The student need not match every capability of the teacher. It needs to meet your product’s requirements on its intended workload. That focus can create room to reduce serving resources or latency, but a smaller model can also require extra retries. Compare cost per successful task, including failures and fallbacks, rather than model size alone. Respect the teacher’s license and service terms when collecting training material.
Implementation reference: TRL Generalized Knowledge Distillation
LoRA and QLoRA: how much of the model changes
These techniques answer a different question. Full fine-tuning updates the model’s weights directly. LoRA keeps the original weights frozen and trains low-rank updates. QLoRA adds a quantized frozen base to reduce memory requirements while training adapters. They change the parameter and memory strategy; the learning signal can still come from demonstrations, preferences or another compatible objective.
An adapter approach can make experiments easier to run and store. It does not replace data preparation, evaluation or a deployment plan. Check the supported architecture, target modules and serving path, and measure the resulting behavior instead of assuming adapters or full fine-tuning will always win.
Bring the training loop into your real environment
For Agent RFT, the environment is part of the task. A general model may have learned tool use on different schemas and workflows. Training on your actual tool contracts and representative observations can improve how consistently it selects tools, supplies arguments and recovers from errors. The benefit to pursue is a better user experience: fewer avoidable retries, shorter completion time and lower cost for successful work.
Build the evaluation suite before scaling training. Measure task success, invalid calls, grounded answers, regressions outside the target domain, end-to-end latency and cost. Separate customer or task families across training and evaluation when leakage is possible. Run the base model and candidate through the same harness so you can attribute a difference to the model.
Use industry data and benchmarks to understand what your users need. TensorOps works with data partners to help teams identify relevant data and evaluate coverage. Combine that external perspective with your own reviewed examples and production failure analysis. A useful training set reflects the tasks you want to become good at, not simply the data that is easiest to export.
Start with the smallest experiment that can establish an advantage. You may need one method, or a sequence such as domain adaptation followed by SFT and preference or reward-based refinement. Budget for data work, training, rollouts, evaluation and ongoing serving. Expand when the improvement survives held-out tests and a realistic production trial.