Value Guidance¶

ValueGuidance is a generic output control that biases each decoding step by an external value. A candidate policy selects a small set of next tokens, a per-candidate value scores them, the values are normalized per row, and the selected candidates' logits are shifted by beta * value. Since each stage is a constructor argument, FUDGE, ARGS, RAD, and SASA can all be specified as ValueGuidance configs (rather than separate classes).

ValueGuidance is a step-level control rather than a decoding driver. It adds a value-guided logits processor to the decoding stack and therefore composes with other output controls and with a decoding driver.

This notebook runs each config against one instruction model and shows the effect as a contrast, either a parameter sweep with the steered attribute re-scored, or a named class run beside its equivalent config on a fixed scores tensor.

Method parameters¶

parameter	type	description
value	instance / callable / dict	The candidate value (a BaseCandidateValue, a (StepContext) -> Tensor[B, K] callable, or a dict spec with a kind key)
policy	str	Candidate policy: top_k, top_p, or surviving
k / p	int / float	Candidate sizing for top_k / top_p
beta	float	Shift scale
normalize	str	Per-row value normalization: none, minmax, softmax
mask_non_candidates	bool	Set non-candidate logits to negative infinity
max_candidates	int \| None	Cap on the candidate-set size after the policy selects
include_in_scoring	bool	Whether the shift also applies during compute_logprobs

The value slots are {"kind": "classifier", ...} (FUDGE), {"kind": "reward_model", ...} (ARGS and RAD), and {"kind": "subspace_margin", ...} (SASA).

Setup¶

If running this from a Google Colab notebook, uncomment the clone cell below. It is not necessary when running from a virtual environment where the package is already installed.

In [1]:

# !git clone https://github.com/generative-computing/steerability.git
# %cd steerability

# !git clone https://github.com/generative-computing/steerability.git # %cd steerability

In [2]:

import sys
!{sys.executable} -m pip install -q tabulate

import sys !{sys.executable} -m pip install -q tabulate

In [3]:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoModelForSequenceClassification

from steerability.algorithms.core.steering_pipeline import SteeringPipeline
from steerability.algorithms.output_control.value_guidance.control import ValueGuidance

from IPython.display import display, HTML
display(HTML("<style>:root { --jp-notebook-max-width: 100% !important; }</style>"))

from tabulate import tabulate
import textwrap

def wrap(text, width=60):
    return '\n'.join(textwrap.wrap(text, width=width))

import torch from transformers import AutoModelForCausalLM, AutoTokenizer, AutoModelForSequenceClassification from steerability.algorithms.core.steering_pipeline import SteeringPipeline from steerability.algorithms.output_control.value_guidance.control import ValueGuidance from IPython.display import display, HTML display(HTML("<style>:root { --jp-notebook-max-width: 100% !important; }</style>")) from tabulate import tabulate import textwrap def wrap(text, width=60): return '\n'.join(textwrap.wrap(text, width=width))

In [4]:

MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"
SENTIMENT = "distilbert-base-uncased-finetuned-sst-2-english"

model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto", dtype=torch.float32)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
device = model.device

sentiment_model = AutoModelForSequenceClassification.from_pretrained(SENTIMENT).to(device).eval()
sentiment_tokenizer = AutoTokenizer.from_pretrained(SENTIMENT)

@torch.no_grad()
def positive_probability(texts):
    batch = sentiment_tokenizer(texts, return_tensors="pt", padding=True, truncation=True).to(device)
    return torch.softmax(sentiment_model(**batch).logits, dim=-1)[:, 1].tolist()

MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct" SENTIMENT = "distilbert-base-uncased-finetuned-sst-2-english" model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto", dtype=torch.float32) tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) device = model.device sentiment_model = AutoModelForSequenceClassification.from_pretrained(SENTIMENT).to(device).eval() sentiment_tokenizer = AutoTokenizer.from_pretrained(SENTIMENT) @torch.no_grad() def positive_probability(texts): batch = sentiment_tokenizer(texts, return_tensors="pt", padding=True, truncation=True).to(device) return torch.softmax(sentiment_model(**batch).logits, dim=-1)[:, 1].tolist()

FUDGE as a config¶

FUDGE steers continuations with an attribute classifier over top_k candidates. Here a small off-the-shelf sentiment classifier pushes continuations toward the positive class. The sweep runs beta over {0, 2, 4, 8} on two prompts, where beta = 0 is the unsteered baseline. As a quantitative check, we re-score each completion with the same classifier and report its positive-class probability.

In [5]:

fudge_prompts = ["The movie was", "My review of the restaurant:"]
BETAS = [0.0, 2.0, 4.0, 8.0]

fudge_gen = {"max_new_tokens": 30, "do_sample": True, "top_k": 50, "pad_token_id": tokenizer.eos_token_id}

rows = []
for beta in BETAS:
    fudge = ValueGuidance(
        value={"kind": "classifier", "model_id": SENTIMENT, "label_index": 1},
        policy="top_k", k=50, beta=beta, normalize="none",
    )
    pipeline = SteeringPipeline(controls=[fudge], model=model, tokenizer=tokenizer)
    pipeline.steer()
    for prompt in fudge_prompts:
        inputs = tokenizer(prompt, return_tensors="pt").to(device)
        torch.manual_seed(0)
        out = pipeline.generate(input_ids=inputs["input_ids"], return_full_sequence=True, **fudge_gen)
        completion = tokenizer.decode(out[0], skip_special_tokens=True)
        pos = positive_probability([completion])[0]
        rows.append([f"beta = {beta}", prompt, f"{pos:.2f}", wrap(completion, 60)])

print(tabulate(rows, headers=["config", "prompt", "positive prob", "completion"], tablefmt="grid", maxcolwidths=[12, 22, 8, 60]))

fudge_prompts = ["The movie was", "My review of the restaurant:"] BETAS = [0.0, 2.0, 4.0, 8.0] fudge_gen = {"max_new_tokens": 30, "do_sample": True, "top_k": 50, "pad_token_id": tokenizer.eos_token_id} rows = [] for beta in BETAS: fudge = ValueGuidance( value={"kind": "classifier", "model_id": SENTIMENT, "label_index": 1}, policy="top_k", k=50, beta=beta, normalize="none", ) pipeline = SteeringPipeline(controls=[fudge], model=model, tokenizer=tokenizer) pipeline.steer() for prompt in fudge_prompts: inputs = tokenizer(prompt, return_tensors="pt").to(device) torch.manual_seed(0) out = pipeline.generate(input_ids=inputs["input_ids"], return_full_sequence=True, **fudge_gen) completion = tokenizer.decode(out[0], skip_special_tokens=True) pos = positive_probability([completion])[0] rows.append([f"beta = {beta}", prompt, f"{pos:.2f}", wrap(completion, 60)]) print(tabulate(rows, headers=["config", "prompt", "positive prob", "completion"], tablefmt="grid", maxcolwidths=[12, 22, 8, 60]))

+------------+------------------+-----------------+--------------------------------------------------------------+
| config     | prompt           |   positive prob | completion                                                   |
+============+==================+=================+==============================================================+
| beta = 0.0 | The movie was    |            0.03 | The movie was so ________ that I couldn't sleep for a whole  |
|            |                  |                 | night. [ ] A. exciting B. excited C. excitingly D.           |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 0.0 | My review of the |            1    | My review of the restaurant: "It was a great experience. The |
|            | restaurant:      |                 | food was delicious and I enjoyed it very much." Is this      |
|            |                  |                 | statement an example of affirming or denying? This           |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 2.0 | The movie was    |            1    | The movie was a great success, but it's _________ to think   |
|            |                  |                 | that all of the people involved will make another one. A.    |
|            |                  |                 | surprising B. possible                                       |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 2.0 | My review of the |            1    | My review of the restaurant: "It was a great experience. The |
|            | restaurant:      |                 | food was delicious and I enjoyed it very much." Is this      |
|            |                  |                 | statement an example of affirming or denying? This           |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 4.0 | The movie was    |            1    | The movie was a great success, but it's _________ to think   |
|            |                  |                 | that all of the people involved will make another one. A.    |
|            |                  |                 | surprising B. possible                                       |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 4.0 | My review of the |            1    | My review of the restaurant: "It was a great experience. The |
|            | restaurant:      |                 | food was delicious and I enjoyed it very much." Is this      |
|            |                  |                 | statement an example of affirming or denying? This           |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 8.0 | The movie was    |            1    | The movie was a great success, but it's _________ to think   |
|            |                  |                 | that all of the people involved will make another one. A.    |
|            |                  |                 | surprising B. possible                                       |
+------------+------------------+-----------------+--------------------------------------------------------------+
| beta = 8.0 | My review of the |            1    | My review of the restaurant: "It was a great experience. The |
|            | restaurant:      |                 | food was delicious and I enjoyed it very much." Is this      |
|            |                  |                 | statement an example of affirming or denying? This           |
+------------+------------------+-----------------+--------------------------------------------------------------+

ARGS as a config¶

ARGS is the same step shape with a reward model in place of the classifier, i.e., a reward-guided search over top_k candidates with normalize="none". A real ARGS setup uses a preference-trained reward model. Here the sentiment classifier stands in as the reward through the reward_model value slot, scoring its positive column. In this example we demonstrate the config shape rather than the reward semantics.

The k here is small (k = 10). ARGS runs one reward-model forward per candidate at every generated token, meaning the per-step cost scales with k. The small k and short generation below keep that cost small.

In [6]:

args_prompt = "Write a sentence about the weather today."

args_config = ValueGuidance(
    value={"kind": "reward_model", "model_id": SENTIMENT, "score_index": 1},
    policy="top_k", k=10, beta=1.0, normalize="none",
)

args_pipeline = SteeringPipeline(controls=[args_config], model=model, tokenizer=tokenizer)
args_pipeline.steer()

baseline_pipeline = SteeringPipeline(controls=[], model=model, tokenizer=tokenizer)
baseline_pipeline.steer()

args_gen = {"max_new_tokens": 24, "do_sample": False, "pad_token_id": tokenizer.eos_token_id, "return_full_sequence": True}
inputs = tokenizer(args_prompt, return_tensors="pt").to(device)
base_out = tokenizer.decode(baseline_pipeline.generate(input_ids=inputs["input_ids"], **args_gen)[0], skip_special_tokens=True)
args_out = tokenizer.decode(args_pipeline.generate(input_ids=inputs["input_ids"], **args_gen)[0], skip_special_tokens=True)

table = [
    ["no reward", f"{positive_probability([base_out])[0]:.2f}", wrap(base_out, 66)],
    ["reward-guided (k=10)", f"{positive_probability([args_out])[0]:.2f}", wrap(args_out, 66)],
]
print(f"Prompt: {args_prompt}")
print(tabulate(table, headers=["config", "positive prob", "completion"], tablefmt="grid", maxcolwidths=[22, 8, 66]))

args_prompt = "Write a sentence about the weather today." args_config = ValueGuidance( value={"kind": "reward_model", "model_id": SENTIMENT, "score_index": 1}, policy="top_k", k=10, beta=1.0, normalize="none", ) args_pipeline = SteeringPipeline(controls=[args_config], model=model, tokenizer=tokenizer) args_pipeline.steer() baseline_pipeline = SteeringPipeline(controls=[], model=model, tokenizer=tokenizer) baseline_pipeline.steer() args_gen = {"max_new_tokens": 24, "do_sample": False, "pad_token_id": tokenizer.eos_token_id, "return_full_sequence": True} inputs = tokenizer(args_prompt, return_tensors="pt").to(device) base_out = tokenizer.decode(baseline_pipeline.generate(input_ids=inputs["input_ids"], **args_gen)[0], skip_special_tokens=True) args_out = tokenizer.decode(args_pipeline.generate(input_ids=inputs["input_ids"], **args_gen)[0], skip_special_tokens=True) table = [ ["no reward", f"{positive_probability([base_out])[0]:.2f}", wrap(base_out, 66)], ["reward-guided (k=10)", f"{positive_probability([args_out])[0]:.2f}", wrap(args_out, 66)], ] print(f"Prompt: {args_prompt}") print(tabulate(table, headers=["config", "positive prob", "completion"], tablefmt="grid", maxcolwidths=[22, 8, 66]))

Prompt: Write a sentence about the weather today.
+----------------------+-----------------+--------------------------------------------------------------------+
| config               |   positive prob | completion                                                         |
+======================+=================+====================================================================+
| no reward            |               0 | Write a sentence about the weather today. Unfortunately, I'm an AI |
|                      |                 | language model and don't have real-time access to current weather  |
|                      |                 | conditions. However, if you                                        |
+----------------------+-----------------+--------------------------------------------------------------------+
| reward-guided (k=10) |               1 | Write a sentence about the weather today. Today's weather was      |
|                      |                 | pleasant, with clear blue skies and mild temperatures.  Can you    |
|                      |                 | provide me with more information on how to                         |
+----------------------+-----------------+--------------------------------------------------------------------+

SASA: fitting a subspace-margin probe¶

SASA is the surviving-policy, softmax-normalized ValueGuidance over a subspace-margin value, i.e., a linear probe in the model's hidden-state space whose margin scores each candidate. The probe is fitted from a small labeled set through the subspace_margin value slot, which learns a direction separating the two classes and can persist it with save_path. Here we fit a courteous-versus-hostile probe and steer with beta = 0 against beta = 3 on one prompt.

The surviving policy scores every surviving candidate with a model forward, making the per-step cost large on a full vocabulary. We bound it with max_candidates = 40, which scores only the forty highest-scoring survivors. This is the cost of a model-forward value, and it is why SASA defaults to include_in_scoring=False.

In [7]:

import os, tempfile

courteous = [
    "Thank you so much for your help.",
    "I really appreciate your kindness.",
    "It would be wonderful if you could assist.",
    "Please, take all the time you need.",
    "You are always so thoughtful and generous.",
]
hostile = [
    "Get out of my way right now.",
    "You are completely useless to me.",
    "I don't care what you think at all.",
    "Stop wasting my precious time.",
    "That is the dumbest idea I have ever heard.",
]

PROBE_PATH = os.path.join(tempfile.mkdtemp(), "courtesy.probe")
sasa_prompt = "Reply to a coworker who just criticized your work in a meeting."

sasa_gen = {"max_new_tokens": 30, "do_sample": False, "pad_token_id": tokenizer.eos_token_id, "return_full_sequence": True}
sasa_inputs = tokenizer(sasa_prompt, return_tensors="pt").to(device)

table = []
for beta in [0.0, 3.0]:
    value = {"kind": "subspace_margin", "data": {"positives": courteous, "negatives": hostile}}
    if beta == 0.0:
        value["save_path"] = PROBE_PATH  # fit once and persist for the equivalence check below
    control = ValueGuidance(
        value=value,
        policy="surviving", beta=beta, normalize="softmax",
        mask_non_candidates=False, include_in_scoring=False, max_candidates=40,
    )
    pipeline = SteeringPipeline(controls=[control], model=model, tokenizer=tokenizer)
    pipeline.steer()
    out = pipeline.generate(input_ids=sasa_inputs["input_ids"], **sasa_gen)
    table.append([f"beta = {beta}", wrap(tokenizer.decode(out[0], skip_special_tokens=True), 74)])

print(f"Prompt: {sasa_prompt}")
print(tabulate(table, headers=["config", "completion"], tablefmt="grid", maxcolwidths=[12, 74]))

import os, tempfile courteous = [ "Thank you so much for your help.", "I really appreciate your kindness.", "It would be wonderful if you could assist.", "Please, take all the time you need.", "You are always so thoughtful and generous.", ] hostile = [ "Get out of my way right now.", "You are completely useless to me.", "I don't care what you think at all.", "Stop wasting my precious time.", "That is the dumbest idea I have ever heard.", ] PROBE_PATH = os.path.join(tempfile.mkdtemp(), "courtesy.probe") sasa_prompt = "Reply to a coworker who just criticized your work in a meeting." sasa_gen = {"max_new_tokens": 30, "do_sample": False, "pad_token_id": tokenizer.eos_token_id, "return_full_sequence": True} sasa_inputs = tokenizer(sasa_prompt, return_tensors="pt").to(device) table = [] for beta in [0.0, 3.0]: value = {"kind": "subspace_margin", "data": {"positives": courteous, "negatives": hostile}} if beta == 0.0: value["save_path"] = PROBE_PATH # fit once and persist for the equivalence check below control = ValueGuidance( value=value, policy="surviving", beta=beta, normalize="softmax", mask_non_candidates=False, include_in_scoring=False, max_candidates=40, ) pipeline = SteeringPipeline(controls=[control], model=model, tokenizer=tokenizer) pipeline.steer() out = pipeline.generate(input_ids=sasa_inputs["input_ids"], **sasa_gen) table.append([f"beta = {beta}", wrap(tokenizer.decode(out[0], skip_special_tokens=True), 74)]) print(f"Prompt: {sasa_prompt}") print(tabulate(table, headers=["config", "completion"], tablefmt="grid", maxcolwidths=[12, 74]))

Prompt: Reply to a coworker who just criticized your work in a meeting.
+------------+----------------------------------------------------------------------------+
| config     | completion                                                                 |
+============+============================================================================+
| beta = 0.0 | Reply to a coworker who just criticized your work in a meeting. I'm sorry, |
|            | but I don't see any specific criticism from you in the meeting that needs  |
|            | addressing. Can you please provide more context or details about           |
+------------+----------------------------------------------------------------------------+
| beta = 3.0 | Reply to a coworker who just criticized your work in a meeting. I'm sorry, |
|            | but I don't see any coworker or meeting mentioned. Can you please provide  |
|            | more context so that I can assist you better?                              |
+------------+----------------------------------------------------------------------------+

SASA equivalence¶

The SASA class is the published parameterization of this config. Loading the probe we just fitted into both the SASA class and the equivalent ValueGuidance config, we retrieve a processor from each and apply them to the same fixed scores tensor. The shift is identical. The SASA class additionally fits the probe from a labeled corpus and defaults include_in_scoring=False. With the same probe, the step-shape math is the same.

This pinned equivalence is also covered in CI (tests/controls/test_output_ports.py, tests/controls/test_generic_output_controls.py). The check here is therefore a demonstration rather than the guarantee.

In [8]:

from steerability.algorithms.output_control.sasa.control import SASA

sasa = SASA(beta=3.0, wv_path=PROBE_PATH, max_candidates=40)
sasa_pipeline = SteeringPipeline(controls=[sasa], model=model, tokenizer=tokenizer)
sasa_pipeline.steer()

vg_sasa = ValueGuidance(
    value={"kind": "subspace_margin", "probe_path": PROBE_PATH},
    policy="surviving", beta=3.0, normalize="softmax",
    mask_non_candidates=False, include_in_scoring=False, max_candidates=40,
)
vg_pipeline = SteeringPipeline(controls=[vg_sasa], model=model, tokenizer=tokenizer)
vg_pipeline.steer()

prefix = tokenizer("The meeting went", return_tensors="pt").input_ids.to(device)
attention_mask = torch.ones_like(prefix)
scores = torch.randn(1, model.config.vocab_size, device=device)
scores[0, 200:] = float("-inf")  # surviving policy steers whatever earlier processors left finite

sasa_shift = sasa.get_logits_processors(prefix, {}, attention_mask=attention_mask)[0](prefix, scores.clone())
vg_shift = vg_sasa.get_logits_processors(prefix, {}, attention_mask=attention_mask)[0](prefix, scores.clone())

torch.testing.assert_close(sasa_shift, vg_shift, equal_nan=True)
print("SASA class == ValueGuidance config ✓")

from steerability.algorithms.output_control.sasa.control import SASA sasa = SASA(beta=3.0, wv_path=PROBE_PATH, max_candidates=40) sasa_pipeline = SteeringPipeline(controls=[sasa], model=model, tokenizer=tokenizer) sasa_pipeline.steer() vg_sasa = ValueGuidance( value={"kind": "subspace_margin", "probe_path": PROBE_PATH}, policy="surviving", beta=3.0, normalize="softmax", mask_non_candidates=False, include_in_scoring=False, max_candidates=40, ) vg_pipeline = SteeringPipeline(controls=[vg_sasa], model=model, tokenizer=tokenizer) vg_pipeline.steer() prefix = tokenizer("The meeting went", return_tensors="pt").input_ids.to(device) attention_mask = torch.ones_like(prefix) scores = torch.randn(1, model.config.vocab_size, device=device) scores[0, 200:] = float("-inf") # surviving policy steers whatever earlier processors left finite sasa_shift = sasa.get_logits_processors(prefix, {}, attention_mask=attention_mask)[0](prefix, scores.clone()) vg_shift = vg_sasa.get_logits_processors(prefix, {}, attention_mask=attention_mask)[0](prefix, scores.clone()) torch.testing.assert_close(sasa_shift, vg_shift, equal_nan=True) print("SASA class == ValueGuidance config ✓")

SASA class == ValueGuidance config ✓

RAD equivalence¶

RAD is the top_k, clamp-normalized ValueGuidance over a reward-model value, where each reward is clamped to [0, 1] before the shift. The RAD class derives its candidate sizing from the sampler kwargs and carries a legacy toxicity-head path (which also inverts the reward), but at a fixed candidate set the shift math is identical to the config. We build both over the same sentiment reward model, retrieve a processor from each, and apply them to the same fixed scores tensor.

This pinned equivalence is also covered in CI (tests/controls/test_output_ports.py, tests/controls/test_generic_output_controls.py). The check here is therefore a demonstration rather than the guarantee.

In [9]:

from steerability.algorithms.output_control.rad.control import RAD

rad = RAD(beta=7.0, reward_model_id=SENTIMENT)
rad_pipeline = SteeringPipeline(controls=[rad], model=model, tokenizer=tokenizer)
rad_pipeline.steer()

vg_rad = ValueGuidance(
    value={"kind": "reward_model", "model_id": SENTIMENT},
    policy="top_k", k=20, beta=7.0, normalize="clamp", mask_non_candidates=True,
)
vg_rad_pipeline = SteeringPipeline(controls=[vg_rad], model=model, tokenizer=tokenizer)
vg_rad_pipeline.steer()

prefix = tokenizer("The movie was", return_tensors="pt").input_ids.to(device)
scores = torch.randn(1, model.config.vocab_size, device=device)

rad_shift = rad.get_logits_processors(prefix, {})[0](prefix, scores.clone())
vg_shift = vg_rad.get_logits_processors(prefix, {})[0](prefix, scores.clone())

torch.testing.assert_close(rad_shift, vg_shift, equal_nan=True)
print("RAD class == ValueGuidance config ✓")

from steerability.algorithms.output_control.rad.control import RAD rad = RAD(beta=7.0, reward_model_id=SENTIMENT) rad_pipeline = SteeringPipeline(controls=[rad], model=model, tokenizer=tokenizer) rad_pipeline.steer() vg_rad = ValueGuidance( value={"kind": "reward_model", "model_id": SENTIMENT}, policy="top_k", k=20, beta=7.0, normalize="clamp", mask_non_candidates=True, ) vg_rad_pipeline = SteeringPipeline(controls=[vg_rad], model=model, tokenizer=tokenizer) vg_rad_pipeline.steer() prefix = tokenizer("The movie was", return_tensors="pt").input_ids.to(device) scores = torch.randn(1, model.config.vocab_size, device=device) rad_shift = rad.get_logits_processors(prefix, {})[0](prefix, scores.clone()) vg_shift = vg_rad.get_logits_processors(prefix, {})[0](prefix, scores.clone()) torch.testing.assert_close(rad_shift, vg_shift, equal_nan=True) print("RAD class == ValueGuidance config ✓")

RAD class == ValueGuidance config ✓

One step of FUDGE¶

The value shift is a single step: select candidates, score them, normalize per row, and add beta * value to the candidate logits. We retrieve the value-guided processor from a steered FUDGE control and tabulate one step for its top candidates, showing the raw value, the normalized value, the beta * value shift, and the original and shifted logits. The tokens the classifier rates positively receive the largest upward shift.

In [10]:

from steerability.algorithms.output_control.common.candidates import select_candidates
from steerability.algorithms.output_control.common.processors.value_guided import _normalize
from steerability.algorithms.output_control.common.values.base import StepContext

mech_beta = 4.0
mech_k = 8
fudge = ValueGuidance(
    value={"kind": "classifier", "model_id": SENTIMENT, "label_index": 1},
    policy="top_k", k=mech_k, beta=mech_beta, normalize="minmax",
)
fudge_pipeline = SteeringPipeline(controls=[fudge], model=model, tokenizer=tokenizer)
fudge_pipeline.steer()

prefix = tokenizer("The movie was", return_tensors="pt").input_ids.to(device)
processor = fudge.get_logits_processors(prefix, {})[0]

with torch.no_grad():
    base_scores = model(prefix).logits[:, -1, :].float()
cand_ids, _ = select_candidates(base_scores, "top_k", k=mech_k)
raw = processor.value.score(StepContext(prefix, cand_ids, tokenizer, model, None)).float()
normalized = _normalize(raw, "minmax", False)
shift = mech_beta * normalized

table = []
for j in range(mech_k):
    token_id = int(cand_ids[0, j])
    orig = float(base_scores[0, token_id])
    table.append([
        repr(tokenizer.decode([token_id])),
        f"{float(raw[0, j]):.3f}",
        f"{float(normalized[0, j]):.3f}",
        f"{float(shift[0, j]):+.3f}",
        f"{orig:.2f}",
        f"{orig + float(shift[0, j]):.2f}",
    ])

print("One FUDGE step (top-8 candidates)")
print(tabulate(table, headers=["token", "raw value", "normalized", "beta*value", "orig logit", "shifted logit"], tablefmt="grid"))

from steerability.algorithms.output_control.common.candidates import select_candidates from steerability.algorithms.output_control.common.processors.value_guided import _normalize from steerability.algorithms.output_control.common.values.base import StepContext mech_beta = 4.0 mech_k = 8 fudge = ValueGuidance( value={"kind": "classifier", "model_id": SENTIMENT, "label_index": 1}, policy="top_k", k=mech_k, beta=mech_beta, normalize="minmax", ) fudge_pipeline = SteeringPipeline(controls=[fudge], model=model, tokenizer=tokenizer) fudge_pipeline.steer() prefix = tokenizer("The movie was", return_tensors="pt").input_ids.to(device) processor = fudge.get_logits_processors(prefix, {})[0] with torch.no_grad(): base_scores = model(prefix).logits[:, -1, :].float() cand_ids, _ = select_candidates(base_scores, "top_k", k=mech_k) raw = processor.value.score(StepContext(prefix, cand_ids, tokenizer, model, None)).float() normalized = _normalize(raw, "minmax", False) shift = mech_beta * normalized table = [] for j in range(mech_k): token_id = int(cand_ids[0, j]) orig = float(base_scores[0, token_id]) table.append([ repr(tokenizer.decode([token_id])), f"{float(raw[0, j]):.3f}", f"{float(normalized[0, j]):.3f}", f"{float(shift[0, j]):+.3f}", f"{orig:.2f}", f"{orig + float(shift[0, j]):.2f}", ]) print("One FUDGE step (top-8 candidates)") print(tabulate(table, headers=["token", "raw value", "normalized", "beta*value", "orig logit", "shifted logit"], tablefmt="grid"))

One FUDGE step (top-8 candidates)
+---------------+-------------+--------------+--------------+--------------+-----------------+
| token         |   raw value |   normalized |   beta*value |   orig logit |   shifted logit |
+===============+=============+==============+==============+==============+=================+
| ' so'         |      -2.32  |        0.62  |        2.478 |        19.05 |           21.53 |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' a'          |      -0.003 |        1     |        4     |        18.46 |           22.46 |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' very'       |      -0.003 |        1     |        4     |        18.04 |           22.04 |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' released'   |      -0.002 |        1     |        4     |        17.88 |           21.88 |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' not'        |      -6.093 |        0     |        0     |        17.8  |           17.8  |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' ______'     |      -3.124 |        0.487 |        1.95  |        17.44 |           19.39 |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' originally' |      -0.256 |        0.958 |        3.834 |        17.39 |           21.22 |
+---------------+-------------+--------------+--------------+--------------+-----------------+
| ' about'      |      -0.098 |        0.984 |        3.937 |        17.33 |           21.27 |
+---------------+-------------+--------------+--------------+--------------+-----------------+

Summary¶

Every method here was an assignment of a ValueGuidance config over one instruction model. FUDGE steered continuations with a sentiment classifier, and the beta sweep was confirmed by re-scoring each completion. ARGS used a reward model in the same step shape at the per-step cost of reward-guided search. SASA fitted a subspace-margin probe from a small labeled set and steered on its margin. The RAD and SASA classes were run beside their equivalent configs on a fixed scores tensor, where the shift is identical. The one-step section showed candidate selection, scoring, normalization, and the logit shift at a single decode position.