Detailed Explanation of LLM Alignment Principles: Differences Between SFT, RLHF, and DPO

LLM alignment is a post-training process that makes models better conform to human intent. It builds instruction following and preference consistency through SFT, RLHF, and DPO, helping explain why SFT is often performed before DPO training.

What Is LLM Alignment?

LLM alignment refers to using methods such as post-training to make a model’s answers and behavior better conform to human intent, preferences, and safety requirements. It is not only about making the model “able to answer,” but also about making the model more useful and more aligned with expectations in specific scenarios, while reducing outputs that deviate from human requirements as much as possible.

As LLM capabilities continue to improve, alignment is regarded as a key link connecting model capability with practical usability. This article focuses on three common types of methods in the post-training stage: SFT, RLHF, and DPO. It explains what problems they solve respectively, how they relate to one another, and how to make a preliminary choice in practice.

LLM alignment can be understood as a form of “capability calibration”: pretraining enables the model to acquire broad language and knowledge capabilities, while alignment methods in post-training further tell the model how these capabilities should be used under instructions, preferences, and safety boundaries.

Key point: SFT, RLHF, and DPO all belong to LLM post-training or alignment-related methods, but their training signals, objective focuses, and practical requirements are not the same.

The Overall Relationship Among SFT, RLHF, and DPO

SFT, RLHF, and DPO are usually not completely mutually exclusive alternatives, but alignment methods that serve different stages and different objectives. A common order of understanding is: first use SFT to teach the model how to answer according to instructions, and then use preference optimization methods such as RLHF or DPO to bring the model closer to human preferences.

MethodCore objectiveMain training signalCommon roleKey concerns
SFTImitate correct answers and establish instruction-following abilityLabeled instruction and response dataTeach the model to answer according to task format and reference answersWhether the labeled data quality and task objectives are clear
RLHFAlign with human preferences, such as being helpful, harmless, and honestHuman feedback or preference signals, combined with reinforcement learningMake model behavior better conform to human preference orientationsPreference objectives, feedback quality, and resource conditions
DPOSimplify the alignment approach through preference optimizationThe relationship between preferred and non-preferred responses in preference dataUse preference data more directly for alignment trainingPreceding SFT, data distribution, and training stability

The differences among these three types of methods can be summarized as follows: SFT is more like “learning from demonstrations,” RLHF places more emphasis on “human preference feedback,” and DPO converts preference relationships into a more direct optimization objective. They can all serve human intent alignment, but their focuses differ.

SFT Builds Instruction-Following Ability by Imitating Correct Answers

SFT, or supervised fine-tuning, has the core objective of making the model imitate the correct answers in labeled data. For LLMs, the value of SFT lies in adjusting a general pretrained model into a model that is better at executing instructions, enabling it to learn how to generate answers according to user questions, task requirements, and expected formats.

What Problems Is SFT Suitable For Solving?

When task objectives are relatively clear and a relatively large amount of high-quality labeled data can be prepared, SFT is usually a natural starting point. For example, tasks such as question answering, summarization, classification explanations, and formatted output are suitable for using SFT to establish basic behavior if clear inputs and reference outputs can be provided.

The advantage of SFT is that its training objective is clear: given an input, the model learns to generate an output close to the labeled answer. Therefore, it is suitable for helping the model form stable instruction-following ability.

Where Are the Boundaries of SFT?

SFT is not equivalent to complete human preference alignment. It mainly learns “what kind of answer is correct or acceptable in the given samples,” but it does not necessarily fully cover complex preference scenarios. For example, when multiple answers are usable but differ in quality, when an answer needs to balance usefulness and safety, or when it needs to conform to human preferences at a finer granularity, relying on SFT alone is often not enough.

Practical tip: If the model is not yet stable in basic instruction formats, task boundaries, and response style, SFT should usually be considered first, rather than jumping directly to more complex preference optimization.

RLHF Uses Human Preference Signals to Push the Model Closer to Expectations

RLHF, or reinforcement learning from human feedback, aims to make the model better conform to human preferences. Unlike SFT, which focuses on imitating correct answers, RLHF pays more attention to whether model outputs conform to human preference judgments in directions such as “helpful, harmless, and honest.”

The Core Difference Between RLHF and SFT

The training signal for SFT usually comes from standard answers, emphasizing “learning how to answer”; the training signal for RLHF comes from human feedback or preference judgments, emphasizing “which kind of answer better meets human expectations.”

Comparison dimensionSFTRLHF
Main objectiveInstruction following and imitation of correct answersAlignment with human preferences
FocusWhether the output is close to the labeled answerWhether the output is more helpful, more harmless, and more honest
Typical training signalInput and reference responseHuman feedback or preference signal
Suitable stageEstablishing basic response capabilityFurther adjusting behavioral orientation on top of basic capability

In practical understanding, SFT can be viewed as “first teaching the model how to solve problems,” while RLHF can be viewed as “then adjusting the model’s answer tendencies according to human preferences.” The former provides basic behavior, and the latter further pushes model behavior to be consistent with preference objectives.

DPO Simplifies the Alignment Process Through Preference Optimization

DPO, or direct preference optimization, is usually described as a new type of alignment method that evolves from complexity toward simplicity. It also serves preference alignment, but its approach places more emphasis on directly using preference data for optimization.

The key idea of DPO is related to the correspondence between the optimal policy under a KL constraint and the reward function. In plain terms, it attempts to convert the information of “which answer better conforms to preference” more directly into a model training objective, rather than understanding alignment only as an additional complex process stacked on top.

DPO Does Not Mean Data Preparation Is Unnecessary

It should be noted that DPO is not a method for “skipping training preparation.” It still relies on preference data and also requires attention to whether the data distribution is suitable for the algorithm’s requirements. Relevant practical documentation recommends that before starting DPO training, SFT should first be performed using the preferred response portion of the preference dataset, in order to help the data satisfy the distribution requirements of the DPO algorithm.

Therefore, the “simplicity” of DPO is more appropriately understood as simplification in the optimization idea, rather than complete omission of engineering preparation.

Why Is SFT Often Performed Before DPO Training?

Performing SFT before DPO training is mainly intended to put the model and data into a state more suitable for preference optimization. Relevant documentation recommends using the preferred response portion of the preference dataset for SFT, because this helps ensure that the data satisfies the distribution requirements of the DPO algorithm.

Focus on Data Distribution, Not Mechanical Stacking of Methods

DPO relies on preference samples. If the model has not yet stably learned the basic distribution of preferred responses before entering DPO, directly performing preference optimization may increase training instability. The role of preceding SFT is to let the model first gain basic adaptation to the form, content, and task patterns of preferred responses.

Use SFT Loss to Help Stabilize Training

Relevant documentation also mentions that SFT loss can be mixed into the DPO loss to stabilize training. This indicates that in practice, DPO and SFT are not necessarily an either-or choice; under appropriate settings, SFT can serve as an auxiliary signal to stabilize the preference optimization process.

Key point: The core of preceding SFT before DPO is not that “one more training step is definitely better,” but that the design should revolve around data distribution matching and training stability.

Differences Among the Three Methods and Selection Criteria

Choosing SFT, RLHF, or DPO should not depend only on whether the method name is popular, but should be based on task requirements, data conditions, and resource conditions. Different methods solve different problems and have different suitable starting points.

Selection dimensionSFTRLHFDPO
More suitable objectiveEstablish instruction-following abilityFurther approach human preferencesUse preference data for more direct optimization
Data conditionsSuitable for tasks with more labeled dataRequires human feedback or preference signalsRequires preference data and attention to the distribution of preferred responses
Requirements for base model stateCan serve as a starting point for post-trainingUsually built on existing basic behaviorSFT is often recommended before training
Practical concernsQuality of labeled answers and task consistencyWhether preference objectives are clear and whether resource conditions matchData distribution, preceding SFT, and loss stability
Should not be misunderstood asComplete preference alignmentBetter as long as there is human feedbackCompletely not requiring preceding training or data preparation

The following framework can be used for preliminary judgment:

  1. If the goal is to first teach the model to answer stably according to instructions, and there is a relatively large amount of labeled data, prioritize SFT.
  2. If the model already has basic response capability but needs to better conform to human preferences, such as being more helpful, more harmless, and more honest, RLHF can be evaluated.
  3. If preference data is already available and a more direct preference optimization approach is desired, DPO can be evaluated.
  4. If using DPO, special attention should be paid to whether preceding SFT is needed and whether SFT loss needs to be mixed into the DPO loss to enhance training stability.

Common Misconceptions and Practical Checklist

Misconception 1: Equating SFT with Complete Alignment

SFT can help the model imitate correct answers and improve instruction-following ability, but it is not equivalent to complete human preference alignment. For preference ranking, safety orientation, and complex response quality judgments, further methods such as RLHF and DPO may still be needed.

Misconception 2: Believing That DPO Can Skip Basic Training

DPO is indeed often regarded as a more concise preference optimization method, but this does not mean preceding training can be ignored. Practical documentation recommends using preferred responses for SFT before DPO to help satisfy the data distribution requirements of the DPO algorithm.

Misconception 3: Choosing Methods Only by Popularity

Different teams have different data, tasks, and resource conditions. When choosing a method, one should return to the problem itself: is the lack in basic instruction-following ability, or is further preference alignment needed? Are labeled answers available, or is preference data available? Do training stability and resource conditions allow it?

Practical Checklist

Before starting alignment training, you can first check the following questions:

  • Is the goal clear: to improve instruction following, or to align with human preferences?
  • Does the data match: is there standard answer data, or paired or ranked preference data?
  • Is the basic behavior stable: can the model already generate usable answers according to instructions?
  • Is preceding SFT needed: especially before DPO training, is it necessary to first perform SFT using preferred responses?
  • Is a stability strategy ready: is it necessary to mix SFT loss into the DPO loss?
  • Are the resource conditions appropriate: does the method choice fit the current labeling, training, and evaluation resources?

Overall, SFT, RLHF, and DPO are not simply in a higher-or-lower-level relationship, but rather a combination of tools aimed at different training signals and alignment objectives. Only by understanding their objective differences, data requirements, and practical boundaries can one design a more robust LLM human intent alignment solution.