Skip to content

Scorers

steerability.evaluation.scorers

Adapt Inspect scorers into per-row SampleScorer rewards for steering controls.

The adapter lets an Inspect scorer (includes(), match(), model_graded_fact(...)) drive controls that consume a SampleScorer (prompt optimizers, sequence rerankers) while algorithms/ stays free of Inspect imports. A model-graded scorer used this way runs grader traffic from inside a control's steer() or decode loop; when the grader is a local model it shares the GPU with the pipeline, so prefer API graders or size headroom accordingly.

sample_scorer_from_inspect(scorer, *, target_key='reference', to_float=None, model_name='steerability/sample-scorer')

Adapt an Inspect scorer into a per-row SampleScorer.

Each call builds a standalone TaskState from the row (row["input"] as the user turn, the response as the assistant turn, row[target_key] as the target, the row as metadata), runs the scorer, and converts Score.value to a float.

Parameters:

Name Type Description Default
scorer Scorer

The Inspect scorer to adapt.

 required
target_key str

Row key holding the reference target; a missing or None value scores against an empty target.

 'reference'
to_float Callable | None

Converts Score.value to a float; defaults to inspect_ai.scorer.value_to_float() (maps "C"/"I"/"P"/"N" and numerics).

 None
model_name str

Model name recorded on the constructed state and output.

 'steerability/sample-scorer'

Returns:

Type Description
SampleScorer

A SampleScorer mapping (response, row) to a float.
Source code in steerability/evaluation/scorers.py 
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
def sample_scorer_from_inspect(
    scorer: Scorer,
    *,
    target_key: str = "reference",
    to_float: Callable | None = None,
    model_name: str = "steerability/sample-scorer",
) -> SampleScorer:
    """Adapt an Inspect scorer into a per-row `SampleScorer`.

    Each call builds a standalone `TaskState` from the row (`row["input"]` as the user turn, the
    response as the assistant turn, `row[target_key]` as the target, the row as metadata), runs the
    scorer, and converts `Score.value` to a float.

    Args:
        scorer: The Inspect scorer to adapt.
        target_key: Row key holding the reference target; a missing or None value scores against
            an empty target.
        to_float: Converts `Score.value` to a float; defaults to
            `inspect_ai.scorer.value_to_float()` (maps `"C"`/`"I"`/`"P"`/`"N"` and numerics).
        model_name: Model name recorded on the constructed state and output.

    Returns:
        A `SampleScorer` mapping `(response, row)` to a float.
    """
    convert = to_float if to_float is not None else value_to_float()
    state_model = ModelName(model_name)

    def score(response: str, row: Mapping[str, Any]) -> float:
        query = str(row.get("input", ""))
        target = Target(row.get(target_key) or "")
        state = TaskState(
            model=state_model,
            sample_id=0,
            epoch=0,
            input=query,
            messages=[ChatMessageUser(content=query), ChatMessageAssistant(content=response)],
            target=target,
            output=ModelOutput.from_content(model_name, response),
            metadata=dict(row),
        )

        async def run_scorer():
            return await scorer(state, target)

        result = _run_coroutine_fn(run_scorer)
        if result is None:
            raise ValueError(
                f"The Inspect scorer returned no Score for input {query!r}; a SampleScorer must "
                "produce a float for every row."
            )
        return float(convert(result.value))

    return score