⎈

THRESHOLD HELMSMAN In development

Confidence scoring and output stabilisation

Helmsman provides calibrated confidence signals for language model outputs. It detects when a model is guessing, when it has prematurely converged on an answer, and when its reasoning chain has collapsed into a false hollow of apparent certainty.

Where Keel controls what the agent does, Helmsman assesses how much the agent should trust its own outputs. Where Driftwatch measures drift over time, Helmsman scores confidence in the moment.

What it does

Confidence scoring. Per-output confidence estimate based on consistency, hedging language, and reasoning chain coherence. Not self-reported model confidence, which is notoriously uncalibrated.
Premature convergence detection. Identifies when the model has settled on an answer too quickly, without adequate exploration of alternatives. Flags outputs where the model confuses fluency with correctness.
Keel integration. Helmsman signals feed into Keel's risk assessment. A low-confidence output proposing an irreversible action automatically escalates the risk tier, requiring stronger approval.
Ambiguity scoring. Rates the ambiguity of inputs and instructions. High-ambiguity inputs trigger the Keel deadman: pause and ask for clarification rather than act on assumptions.

The Vessel

Keel, Driftwatch, and Helmsman compose into a single safety stack. Helmsman scores confidence. Keel enforces approval thresholds informed by that score. Driftwatch records everything for longitudinal analysis. Each works standalone; together they form a structural safety envelope for autonomous agents.

Helmsman is in active development. The theoretical framework draws on research into epistemic meta-layers and lagged evaluative functions. Integration points with Keel are defined and tested.

Research details at threshold.systems. ORCID 0009-0004-1442-1743.