DExperts¶
Paper: DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Authors: Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, Yejin Choi
DExperts steers a base model at decoding time by combining it with a small expert and anti-expert, both fine-tuned on the target attribute. At each step the next-token distribution is re-weighted by the difference between the expert and anti-expert, promoting tokens the expert favors and the anti-expert disfavors.
DExperts is a step-level control that composes a contrastive-mixture logits processor into the decoding stack, so it works alongside other output controls and with a decoding driver. Both auxiliaries run their own forward pass at every decoding step (two extra small-model forwards per generated token) and must share the base model's vocabulary. Proxy-tuning is the same control with the expert set to a tuned small model and the anti-expert to its untuned counterpart; that pairing is the main example below.
Method parameters¶
| parameter | type | description |
|---|---|---|
expert_name_or_path |
str |
Expert LM (steers toward the attribute) |
anti_expert_name_or_path |
str |
Anti-expert LM (steers away from the attribute) |
alpha |
float |
Contrast strength; the expert enters at +alpha, the anti-expert at -alpha |
hf_model_kwargs |
dict |
Extra kwargs for loading both auxiliary models |
The expert and anti-expert must share the base model's vocabulary.
Setup¶
If running this from a Google Colab notebook, uncomment the clone cell below. It is not necessary when running from a virtual environment where the package is already installed.
# !git clone https://github.com/generative-computing/steerability.git
# %cd steerability
The following authentication steps may be necessary to access any gated models (after being granted access by Hugging Face). Uncomment the following if you need to log in to the Hugging Face Hub:
# !pip install -q python-dotenv
# from dotenv import load_dotenv
# import os
# load_dotenv()
# token = os.getenv("HUGGINGFACE_TOKEN")
# from huggingface_hub import login
# login(token=token)
Example: instruction-following via proxy-tuning¶
The steering goal is to make an untuned base model follow instructions, with no training. The base is Qwen/Qwen2.5-1.5B, a pretrained-only model. The expert is Qwen/Qwen2.5-0.5B-Instruct and the anti-expert is Qwen/Qwen2.5-0.5B, a tuned/untuned pair one size down. Their per-step log-prob difference is a direct read-out of what instruction-tuning changes, and DExperts applies that difference to the larger base at decode time.
All prompts below are formatted with the Qwen chat template (the proxy-tuning convention is to format prompts for the tuned expert). The base model has seen the template's tokens in pretraining but has not been tuned to answer within them, which is the gap the contrast has to close. The auxiliaries load in reduced precision via hf_model_kwargs.
from transformers import AutoModelForCausalLM, AutoTokenizer
from steerability.algorithms.core.steering_pipeline import SteeringPipeline
from steerability.algorithms.output_control.dexperts.control import DExperts
MODEL_NAME = "Qwen/Qwen2.5-1.5B"
EXPERT_NAME = "Qwen/Qwen2.5-0.5B-Instruct"
ANTI_EXPERT_NAME = "Qwen/Qwen2.5-0.5B"
Baseline: the untuned base model¶
We give the base model the instruction in the chat format and decode greedily, exactly as we will for the steered runs.
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto", dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
prompt = "Give three tips for improving the readability of Python code. Answer with a numbered list."
chat = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(chat, return_tensors="pt").to(model.device)
baseline_outputs = model.generate(
**inputs,
do_sample=False,
max_new_tokens=120,
pad_token_id=tokenizer.eos_token_id,
)
baseline_text = tokenizer.decode(baseline_outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(baseline_text)
afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone
The base model does not answer at all; greedy decoding collapses into a degenerate token loop the moment it has to write in the assistant slot. This is what "not instruction-tuned" looks like in the deployment format, and it is the baseline failure the tuned/untuned contrast has to fix.
Sanity check: expert = anti-expert changes nothing¶
Before the real pairing, a wiring check. The processor mixes log p_base + alpha * (log p_expert - log p_anti_expert), so pointing both slots at the same model cancels the contrast term exactly, and greedy generation must reproduce the base model token for token. This confirms the plumbing, and it shows why a pair that actually differs is required for the method to do anything at all.
sanity_dexperts = DExperts(
expert_name_or_path=ANTI_EXPERT_NAME,
anti_expert_name_or_path=ANTI_EXPERT_NAME,
alpha=1.0,
hf_model_kwargs={"dtype": "auto"},
)
sanity_pipeline = SteeringPipeline(
model_name_or_path=MODEL_NAME,
controls=[sanity_dexperts],
device_map="auto",
hf_model_kwargs={"dtype": "auto"},
)
sanity_pipeline.steer()
output = sanity_pipeline.generate(
input_ids=inputs["input_ids"].to(sanity_pipeline.model.device),
max_new_tokens=120,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
sanity_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(sanity_text)
print("\nidentical to baseline:", sanity_text == baseline_text)
afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone afone identical to baseline: True
The outputs match token for token, degenerate loop included. Everything DExperts adds rides on the difference between the two auxiliaries, so identical auxiliaries provably show nothing; that is why this configuration is a correctness check and not a demo.
The shared-vocabulary constraint¶
The mixture adds log-probs over token ids, so both auxiliaries must use the base model's vocabulary (the control enforces this at steer() time). The Qwen2.5 family shares one tokenizer across sizes; the check below makes the constraint explicit.
expert_tokenizer = AutoTokenizer.from_pretrained(EXPERT_NAME)
anti_expert_tokenizer = AutoTokenizer.from_pretrained(ANTI_EXPERT_NAME)
probe = "Steering a 1.5B base with a 0.5B contrast."
assert tokenizer(probe)["input_ids"] == expert_tokenizer(probe)["input_ids"] == anti_expert_tokenizer(probe)["input_ids"]
print("base, expert, and anti-expert tokenize identically")
base, expert, and anti-expert tokenize identically
DExperts with the tuned/untuned pair¶
Now the real pairing. The expert and anti-expert differ only by instruction-tuning, so alpha * (log p_expert - log p_anti_expert) isolates the tuning direction and adds it to the base model's logits at every step.
dexperts = DExperts(
expert_name_or_path=EXPERT_NAME,
anti_expert_name_or_path=ANTI_EXPERT_NAME,
alpha=1.0,
hf_model_kwargs={"dtype": "auto"},
)
pipeline = SteeringPipeline(
model_name_or_path=MODEL_NAME,
controls=[dexperts],
device_map="auto",
hf_model_kwargs={"dtype": "auto"},
)
pipeline.steer()
Same prompt, same greedy decoding; only the logits mix has changed.
output = pipeline.generate(
input_ids=inputs["input_ids"].to(pipeline.model.device),
max_new_tokens=120,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
1. **Use meaningful variable and function names**: Choosing descriptive names for variables and functions can make the code more readable and easier to understand. This helps in reducing the cognitive load of the reader and improves the overall maintainability of the code. 2. **Follow PEP 8 style guide**: PEP 8 is a set of guidelines for writing Python code that is readable and consistent. Adhering to these guidelines can help in writing clean, well-organized and maintainable code. This includes using appropriate indentation, spacing, and naming conventions. 3. **Use comments judiciously**: Comments
The same untuned base, on the same prompt, now answers with the requested numbered list. The small pair contributed only the direction (what instruction-tuning changes); the fluency and content still come from the larger base model. This is proxy-tuning, and it needed no training because the tuned/untuned pair already existed.
Contrast strength alpha¶
alpha is a strength dial on the tuning direction. Small values apply the tuning direction more gently; large values let the small auxiliaries dominate the mix, so the output drifts toward what the 0.5B expert itself would write.
for alpha in [0.5, 1.0, 2.0]:
pipeline = SteeringPipeline(
model_name_or_path=MODEL_NAME,
controls=[
DExperts(
expert_name_or_path=EXPERT_NAME,
anti_expert_name_or_path=ANTI_EXPERT_NAME,
alpha=alpha,
hf_model_kwargs={"dtype": "auto"},
)
],
device_map="auto",
hf_model_kwargs={"dtype": "auto"},
)
pipeline.steer()
output = pipeline.generate(
input_ids=inputs["input_ids"].to(pipeline.model.device),
max_new_tokens=100,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
print(f"alpha={alpha}:")
print(tokenizer.decode(output[0], skip_special_tokens=True))
print()
alpha=0.5: 1. Use meaningful variable and function names to clearly convey their purpose. 2. Utilize whitespace and indentation to improve code structure and readability. 3. Employ consistent naming conventions and formatting throughout the codebase.
alpha=1.0: 1. **Use meaningful variable and function names**: Choosing descriptive names for variables and functions can make the code more readable and easier to understand. This helps in reducing the cognitive load of the reader and improves the overall maintainability of the code. 2. **Follow PEP 8 style guide**: PEP 8 is a set of guidelines for writing Python code that is readable and consistent. Adhering to these guidelines can help in writing clean, well-organized and maintainable code. This includes
alpha=2.0: - **Use Meaningful Variable and Function Names:** Choose names for variables and functions that clearly indicate their purpose, making the code easier to understand at a glance. - **Implement Consistent Formatting:** Adopting a consistent indentation style (usually 4 spaces) and line length (typically 79 characters) can significantly enhance readability by improving the separation of code structure from its content. - **Utilize Comments Wisely:** While not replacing the need for comments, adding comments to explain complex logic or
All three strengths produce a valid answer on this prompt, and the dial shows up as style. At 0.5 the list is terse, at 1.0 each point is developed, and at 2.0 the formatting and phrasing drift furthest toward the small expert's own register. The control's default of 1.0 is a reasonable middle here (the paper tunes alpha per task).
Reversal: swapping expert and anti-expert¶
Swapping the two slots negates the contrast. The same machinery now subtracts the instruction-tuning direction, steering the base model even further from answering than the unsteered baseline.
reversed_pipeline = SteeringPipeline(
model_name_or_path=MODEL_NAME,
controls=[
DExperts(
expert_name_or_path=ANTI_EXPERT_NAME,
anti_expert_name_or_path=EXPERT_NAME,
alpha=1.0,
hf_model_kwargs={"dtype": "auto"},
)
],
device_map="auto",
hf_model_kwargs={"dtype": "auto"},
)
reversed_pipeline.steer()
output = reversed_pipeline.generate(
input_ids=inputs["input_ids"].to(reversed_pipeline.model.device),
max_new_tokens=100,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
uação
The reversed pair does not answer at all; the output degenerates immediately, confirming that the direction of the contrast, not just its magnitude, is under your control.
Takeaway¶
DExperts buys attribute steering at decode time for the price of two small-model forwards per generated token, and proxy-tuning shows the recipe needs no attribute-specific training when a tuned/untuned pair already exists. Reach for it when the behavior you want can be named as a difference between two small models you already have.
The control shares its contrastive-mixture processor with contrastive_decoding.ipynb, where a single weaker amateur is contrasted against the base itself. Auxiliary load options (dtype, device) pass through hf_model_kwargs. See the output control section of the docs for the full family.