igopalakrishna/Qwen2.5-7B-DPO-Factuality-LoRA-MinChosen9-MinDelta6

Qwen-2.5-7B DPO LoRA Fine-tune for Factuality

This repository contains a version of Qwen/Qwen2.5-7B-Instruct that has been fine-tuned using Direct Preference Optimization (DPO) with a parameter-efficient (PEFT) LoRA approach.

Research Experiment

This model was trained as part of a research project investigating the effects of DPO on model factuality and the differences between full fine-tuning and PEFT methods.

• Base Model: Qwen/Qwen2.5-7B-Instruct
• Training Method: DPO with LoRA (r=32) and 8-bit quantization.
• Dataset: chardizard/dpo-mix5-Llama3-Factuality (filtered for high-quality pairs with minchosen=9, mindelta=6).
• Training Steps: 1000 steps.

Evaluation and Findings

The model was evaluated on the MMLU (general knowledge) and TruthfulQA (factuality) benchmarks and compared against the original baseline and a full DPO fine-tune.

Metric	Baseline (Qwen 7B)	DPO Full Fine-tune	This DPO LoRA Model
MMLU (5-shot acc)	0.7175	0.7189	0.7182
TruthfulQA (mc2)	0.6465	0.6455	0.0000

Key Finding: Format Overfitting in LoRA

A significant finding from this experiment is the model's 0% score on the TruthfulQA multiple-choice benchmark. The detailed logs confirmed the model still possessed the knowledge to answer MMLU questions correctly, but the DPO training on a purely conversational dataset caused format overfitting. The LoRA-tuned model learned the style of generating cautious, paragraph-style answers so strongly that it failed to produce the required single-letter format for TruthfulQA.

This is a valuable research result, suggesting that PEFT methods like LoRA may be more susceptible to this type of format overfitting than a full fine-tune, which did not exhibit the same catastrophic failure on this benchmark.

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "igopalakrishna/Qwen2.5-7B-DPO-Factuality-LoRA-MinChosen9-MinDelta6"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "user", "content": "What were the main causes of the American Revolutionary War?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to("cuda")

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)