JANTAREvaluation Scorecard: What Jantar Has Measured
Every figure on this page is read from the scorecard of the release serving it, the same evidence /api/system/scorecard answers with. Most root-cause figures are measured on JANTAR-REF-1, a synthetic reference spacecraft we built, so each section names what it was measured on. A section that did not run says so and names the command that produces it.
11/12 benchmarks available; 30/30 pass. Missing sections show the exact command that produces them — never a fabricated zero. Evidence grade: partial. From this release’s scorecard.
Golden — reference-scenario RCA
Measured on: synthetic · JANTAR-REF-1
| Metric | Value | Gate | Note |
|---|---|---|---|
| Root-cause top-1 | 1.000 | Gate = 1.000: passed | |
| Root-cause top-3 | 1.000 | Descriptive, not a gate | |
| Citation integrity | 1.000 | Gate = 1.000: passed | |
| Suppression accuracy | 1.000 | Gate = 1.000: passed | |
| Overall accuracy | 1.000 | Gate = 1.000: passed | |
| Drafts releasable without review | 0.250 | Descriptive, not a gate | Low is correct. Most drafts are deliberately HELD for human approval — holding is the release gate working, not a failure to produce one. The gated 'Releasability matches expectation' row below is the quality signal. |
| Releasability matches expectation | 1.000 | Gate = 1.000: passed | |
| Investigations | 16/20 | Descriptive, not a gate |
Measured on synthetic · JANTAR-REF-1. mode=airgap · model=deterministic · detector=annotation. Source: make bench. Measured at commit 2bc4474. From this release’s scorecard.
Moat — grounded RCA + anti-hallucination
Measured on: synthetic · JANTAR-REF-1
| Metric | Value | Gate | Note |
|---|---|---|---|
| Root-cause top-1 (grounded) | 1.000 | Gate = 1.000: passed | |
| Grounded honesty (no fabrication) | 1.000 | Gate = 1.000: passed | |
| Citation integrity under ablation | 1.000 | Gate = 1.000: passed | |
| Citation integrity (grounded) | 1.000 | Gate = 1.000: passed | |
| No unsafe release | 1.000 | Gate = 1.000: passed | |
| Ablation effect (teeth) | 1.000 | Descriptive, not a gate | |
| Releasability matches expectation | 0.750 | Descriptive, not a gate | Agreement with the frozen golden releasability, not a draft-quality score. Below 1.0 is expected: these faults are found by REAL ECOD detection, whose severity can take a higher escalation path than the annotation trigger the expectation was frozen against. A held (non-released) draft is the release gate working. |
| Ablation probes | 32 | Descriptive, not a gate |
Measured on synthetic · JANTAR-REF-1. mode=airgap · model=deterministic · grounded_faults=16 · moat_holds=True. Source: make bench-moat. Measured at commit 2bc4474. From this release’s scorecard.
Detection — OPSSAT-AD
Measured on: real · OPSSAT-AD (ESA OPS-SAT-1, CC BY 4.0)
Detection is not diagnosis: no root cause is claimed on this data. ESA-ADB detection has also been run, on the dataset’s first mission resampled to hourly, and is recorded in our evaluation baseline. It is not on this scorecard because its dataset is too large to re-score inside the release build, and this page shows only what the release measured. We will walk through that run on request.
| Metric | Value | Gate | Note |
|---|---|---|---|
| ecod (baseline) F-beta (degenerate) | 1.000 (degenerate: not a detection score) | Descriptive, not a gate | NOT a detection-quality number: a single predicted event covers 100% of the labelled timeline and is 48149x the median label duration (1 events for 113 labels) — it overlaps labels by construction, so the event-wise score measures coarseness, not detection quality |
| ecod (baseline) affiliation F-beta | 0.519 | Descriptive, not a gate | |
| ecod (baseline) false alarms per channel-day | 0.000 | Descriptive, not a gate | 0 false positives over 9 channels x 148.7948 days; at the threshold on this row, not a friendlier one |
| graph F-beta | 0.564 | Descriptive, not a gate | |
| graph affiliation F-beta | 0.434 | Descriptive, not a gate | |
| graph false alarms per channel-day | 0.002 | Descriptive, not a gate | 2 false positives over 9 channels x 148.7948 days; at the threshold on this row, not a friendlier one |
| shape F-beta | 0.573 | Descriptive, not a gate | |
| shape affiliation F-beta | 0.572 | Descriptive, not a gate | |
| shape false alarms per channel-day | 0.038 | Descriptive, not a gate | 51 false positives over 9 channels x 148.7948 days; at the threshold on this row, not a friendlier one |
| shape threshold | 3.871 | Descriptive, not a gate | chosen on the fit split by F-beta and reported on the held-out split — never tuned on the split it is scored on |
Measured on real · OPSSAT-AD (ESA OPS-SAT-1, CC BY 4.0). Reported on a held-out split, never tuned on. mission=OPS-SAT-1 · channels=9 · labels=113 · eval=test, fit=train | OPSSAT-AD data is CC BY 4.0 (Zenodo DOI 10.5281/zenodo.12588359; Ruszczak, Kotowski, Evans, Nalepa, Scientific Data 2025, doi:10.1038/s41597-025-05035-3); attribution required. Evaluated on the public dataset — attribution is not endorsement.. Source: python -m jantar.eval detect (needs `make data`). Measured at commit 2bc4474. From this release’s scorecard.
Detection vs static limits — matched false-alarm budgets
Measured on: real · OPSSAT-AD (ESA OPS-SAT-1, CC BY 4.0)
| Metric | Value | Gate | Note |
|---|---|---|---|
| static limits, MAD (baseline) F-beta, budget 1/channel-day (channel axis) | 0.000 | Descriptive, not a gate | P 0.000 · R 0.000 (0/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.80) · alarm occupancy 0.000 · met at k (sigma) 85.48, chosen on train |
| static limits, MAD (baseline) F-beta, budget 5/channel-day (channel axis) | 0.000 | Descriptive, not a gate | P 0.000 · R 0.000 (0/113 labels found) · 7 false alarms = 4.24 per observed channel-day on test (train 4.63) · alarm occupancy 0.002 · met at k (sigma) 61.63, chosen on train |
| static limits, MAD (baseline) F-beta, budget 25/channel-day (channel axis) | 0.050 | Descriptive, not a gate | P 0.064 · R 0.027 (3/113 labels found) · 44 false alarms = 26.63 per observed channel-day on test (train 24.95) · alarm occupancy 0.025 · met at k (sigma) 11.3, chosen on train |
| static limits, MAD (baseline) F-beta, budget 1/channel-day (subsystem axis) | 0.000 | Descriptive, not a gate | P 0.000 · R 0.000 (0/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.80) · alarm occupancy 0.000 · met at k (sigma) 78.31, chosen on train |
| static limits, MAD (baseline) F-beta, budget 5/channel-day (subsystem axis) | 0.034 | Descriptive, not a gate | P 0.125 · R 0.009 (1/113 labels found) · 7 false alarms = 4.24 per observed channel-day on test (train 4.83) · alarm occupancy 0.006 · met at k (sigma) 56.62, chosen on train |
| static limits, MAD (baseline) F-beta, budget 25/channel-day (subsystem axis) | 0.527 | Descriptive, not a gate | P 0.507 · R 0.628 (71/113 labels found) · 71 false alarms = 42.97 per observed channel-day on test (train 24.95) · alarm occupancy 0.221 · met at k (sigma) 1.87, chosen on train |
| static limits, p99 (baseline) F-beta, budget 1/channel-day (channel axis) | 0.338 | Descriptive, not a gate | P 0.889 · R 0.097 (11/113 labels found) · 2 false alarms = 1.21 per observed channel-day on test (train 0.80) · alarm occupancy 0.001 · met at k (p99 excursions) 1.31, chosen on train |
| static limits, p99 (baseline) F-beta, budget 5/channel-day (channel axis) | 0.393 | Descriptive, not a gate | P 0.769 · R 0.133 (15/113 labels found) · 6 false alarms = 3.63 per observed channel-day on test (train 4.83) · alarm occupancy 0.002 · met at k (p99 excursions) 1.141, chosen on train |
| static limits, p99 (baseline) F-beta, budget 25/channel-day (channel axis) | 0.385 | Descriptive, not a gate | P 0.508 · R 0.195 (22/113 labels found) · 29 false alarms = 17.55 per observed channel-day on test (train 24.95) · alarm occupancy 0.005 · met at k (p99 excursions) 1.032, chosen on train |
| static limits, p99 (baseline) F-beta, budget 1/channel-day (subsystem axis) | 0.398 | Descriptive, not a gate | P 0.889 · R 0.124 (14/113 labels found) · 2 false alarms = 1.21 per observed channel-day on test (train 0.80) · alarm occupancy 0.001 · met at k (p99 excursions) 1.31, chosen on train |
| static limits, p99 (baseline) F-beta, budget 5/channel-day (subsystem axis) | 0.428 | Descriptive, not a gate | P 0.741 · R 0.159 (18/113 labels found) · 7 false alarms = 4.24 per observed channel-day on test (train 4.83) · alarm occupancy 0.002 · met at k (p99 excursions) 1.132, chosen on train |
| static limits, p99 (baseline) F-beta, budget 25/channel-day (subsystem axis) | 0.407 | Descriptive, not a gate | P 0.426 · R 0.345 (39/113 labels found) · 54 false alarms = 32.68 per observed channel-day on test (train 24.95) · alarm occupancy 0.020 · met at k (p99 excursions) 0.9825, chosen on train |
| shape F-beta, budget 1/channel-day (channel axis) | 0.219 | Descriptive, not a gate | P 1.000 · R 0.053 (6/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.80) · alarm occupancy 0.009 · met at z_threshold 15.13, chosen on train |
| shape F-beta, budget 5/channel-day (channel axis) | 0.268 | Descriptive, not a gate | P 0.478 · R 0.097 (11/113 labels found) · 12 false alarms = 7.26 per observed channel-day on test (train 4.83) · alarm occupancy 0.023 · met at z_threshold 9.781, chosen on train |
| shape F-beta, budget 25/channel-day (channel axis) | 0.486 | Descriptive, not a gate | P 0.444 · R 0.788 (89/113 labels found) · 109 false alarms = 65.96 per observed channel-day on test (train 24.95) · alarm occupancy 0.385 · met at z_threshold 2.715, chosen on train |
| shape F-beta, budget 1/channel-day (subsystem axis) | 0.274 | Descriptive, not a gate | P 0.700 · R 0.080 (9/113 labels found) · 3 false alarms = 1.82 per observed channel-day on test (train 0.80) · alarm occupancy 0.017 · met at z_threshold 11.88, chosen on train |
| shape F-beta, budget 5/channel-day (subsystem axis) | 0.495 | Descriptive, not a gate | P 0.581 · R 0.310 (35/113 labels found) · 18 false alarms = 10.89 per observed channel-day on test (train 4.83) · alarm occupancy 0.092 · met at z_threshold 8.084, chosen on train |
| shape F-beta, budget 25/channel-day (subsystem axis) (saturated) | 0.572 | Descriptive, not a gate | P 0.541 · R 0.743 (84/113 labels found) · 45 false alarms = 27.23 per observed channel-day on test (train 8.85) · alarm occupancy 0.348 · saturated at z_threshold 4.213, chosen on train |
| graph F-beta, budget 1/channel-day (channel axis) | 0.140 | Descriptive, not a gate | P 0.308 · R 0.044 (5/113 labels found) · 9 false alarms = 5.45 per observed channel-day on test (train 0.80) · alarm occupancy 0.080 · met at z_threshold 1.435, chosen on train |
| graph F-beta, budget 5/channel-day (channel axis) (saturated) | 0.361 | Descriptive, not a gate | P 0.343 · R 0.460 (52/113 labels found) · 23 false alarms = 13.92 per observed channel-day on test (train 4.83) · alarm occupancy 0.485 · saturated at z_threshold 0.2764, chosen on train |
| graph F-beta, budget 25/channel-day (channel axis) (saturated) | 0.361 | Descriptive, not a gate | P 0.343 · R 0.460 (52/113 labels found) · 23 false alarms = 13.92 per observed channel-day on test (train 4.83) · alarm occupancy 0.485 · saturated at z_threshold 0.2764, chosen on train |
| graph F-beta, budget 1/channel-day (subsystem axis) (saturated) | 0.587 | Descriptive, not a gate | P 0.600 · R 0.540 (61/113 labels found) · 8 false alarms = 4.84 per observed channel-day on test (train 0.60) · alarm occupancy 0.490 · saturated at z_threshold 0.6745, chosen on train |
| graph F-beta, budget 5/channel-day (subsystem axis) (saturated) | 0.587 | Descriptive, not a gate | P 0.600 · R 0.540 (61/113 labels found) · 8 false alarms = 4.84 per observed channel-day on test (train 0.60) · alarm occupancy 0.490 · saturated at z_threshold 0.6745, chosen on train |
| graph F-beta, budget 25/channel-day (subsystem axis) (saturated) | 0.587 | Descriptive, not a gate | P 0.600 · R 0.540 (61/113 labels found) · 8 false alarms = 4.84 per observed channel-day on test (train 0.60) · alarm occupancy 0.490 · saturated at z_threshold 0.6745, chosen on train |
| ecod (baseline) F-beta, budget 1/channel-day (channel axis) (degenerate, saturated) | 0.644 | Descriptive, not a gate | P 1.000 · R 0.265 (30/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.00) · alarm occupancy 0.174 · saturated at -ln(alpha) 7.622, chosen on train · NOT a detection-quality number: a single predicted event covers 85% of the labelled timeline and is 40628x the median label duration (2 events for 113 labels) — it overlaps labels by construction, so the event-wise score measures coarseness, not detection quality |
| ecod (baseline) F-beta, budget 5/channel-day (channel axis) (degenerate, saturated) | 0.644 | Descriptive, not a gate | P 1.000 · R 0.265 (30/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.00) · alarm occupancy 0.174 · saturated at -ln(alpha) 7.622, chosen on train · NOT a detection-quality number: a single predicted event covers 85% of the labelled timeline and is 40628x the median label duration (2 events for 113 labels) — it overlaps labels by construction, so the event-wise score measures coarseness, not detection quality |
| ecod (baseline) F-beta, budget 25/channel-day (channel axis) (degenerate, saturated) | 0.644 | Descriptive, not a gate | P 1.000 · R 0.265 (30/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.00) · alarm occupancy 0.174 · saturated at -ln(alpha) 7.622, chosen on train · NOT a detection-quality number: a single predicted event covers 85% of the labelled timeline and is 40628x the median label duration (2 events for 113 labels) — it overlaps labels by construction, so the event-wise score measures coarseness, not detection quality |
| ecod (baseline) F-beta, budget 1/channel-day (subsystem axis) (saturated) | 0.000 | Descriptive, not a gate | P 0.000 · R 0.000 (0/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.00) · alarm occupancy 0.000 · saturated at -ln(alpha) 8.326, chosen on train |
| ecod (baseline) F-beta, budget 5/channel-day (subsystem axis) (saturated) | 0.000 | Descriptive, not a gate | P 0.000 · R 0.000 (0/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.00) · alarm occupancy 0.000 · saturated at -ln(alpha) 8.326, chosen on train |
| ecod (baseline) F-beta, budget 25/channel-day (subsystem axis) (saturated) | 0.000 | Descriptive, not a gate | P 0.000 · R 0.000 (0/113 labels found) · 0 false alarms = 0.00 per observed channel-day on test (train 0.00) · alarm occupancy 0.000 · saturated at -ln(alpha) 8.326, chosen on train |
Measured on real · OPSSAT-AD (ESA OPS-SAT-1, CC BY 4.0). Reported on a held-out split, never tuned on. mission=OPS-SAT-1 · fit=train, eval=test · budgets are false alarms per OBSERVED channel-day (1.65 observed vs 1339 calendar channel-days on test) · chance level: labels cover 0.270 of the telemetry on the channel axis, 0.439 on the subsystem axis · occupancy cap 0.5 · subsystems=1 | OPSSAT-AD data is CC BY 4.0 (Zenodo DOI 10.5281/zenodo.12588359; Ruszczak, Kotowski, Evans, Nalepa, Scientific Data 2025, doi:10.1038/s41597-025-05035-3); attribution required. Evaluated on the public dataset — attribution is not endorsement.. Source: python -m jantar.eval baselines (needs `make data-opssat`). Measured at commit 2bc4474. From this release’s scorecard.
Grounding — retrieval + faithfulness
Measured on: synthetic · JANTAR-REF-1
| Metric | Value | Gate | Note |
|---|---|---|---|
| Retrieval MRR (hashing) | 0.879 | Descriptive, not a gate | |
| Retrieval MRR (bge) | 1.000 | Descriptive, not a gate | |
| Faithfulness accuracy (lexical) | 0.773 | Descriptive, not a gate | |
| Hard-negative rejection (lexical) | 0.727 | Descriptive, not a gate | share of on-topic competing claims correctly refused — the lexical floor's documented blind spot; relevance passes them, entailment does not |
| Faithfulness accuracy (nli/onnx-nli) | 0.833 | Descriptive, not a gate | |
| Hard-negative rejection (nli/onnx-nli) | 1.000 | Descriptive, not a gate | share of on-topic competing claims correctly refused — the lexical floor's documented blind spot; relevance passes them, entailment does not |
| Faithfulness accuracy (nli/fastembed-reranker) | 0.758 | Descriptive, not a gate | |
| Hard-negative rejection (nli/fastembed-reranker) | 0.636 | Descriptive, not a gate | share of on-topic competing claims correctly refused — the lexical floor's documented blind spot; relevance passes them, entailment does not |
Measured on synthetic · JANTAR-REF-1. failure_modes=22 · doc_sections=22 · not run: faithfulness alignscore. Source: make bench-grounding. Measured at commit 2bc4474. From this release’s scorecard.
Calibration — confidence vs correctness
Measured on: synthetic · JANTAR-REF-1
| Metric | Value | Gate | Note |
|---|---|---|---|
| Calibration gap | 0.143 | Descriptive, not a gate | |
| Monotonic | yes | Descriptive, not a gate | |
| HIGH unlocked | yes | Descriptive, not a gate | 'no' is the honest default, not a failure: HIGH confidence is unlocked from measured calibration evidence (hazard H4), never claimed. It stays gated until the observed accuracy band earns it. |
| Observations | 43 | Descriptive, not a gate |
Measured on synthetic · JANTAR-REF-1. candidate=True · mode=airgap. Source: make bench-calibration. Measured at commit 2bc4474. From this release’s scorecard.
RUL — C-MAPSS prognostics
Measured on: synthetic · NASA C-MAPSS (simulated turbofan engines)
Not run for this release. It is produced by python -m jantar.eval rul (needs `make data`).
Ground faults — pass-quality fault matrix
Measured on: synthetic · injected station passes, JANTAR-REF-1 graph
| Metric | Value | Gate | Note |
|---|---|---|---|
| Detection precision (with controls) | 1.000 | Gate >= 0.900: passed | |
| Detection recall (injected faults) | 1.000 | Gate >= 0.900: passed | |
| Root-cause attribution (attributable classes) | 1.000 | Gate >= 0.800: passed | |
| Mean time to detect | — | Descriptive, not a gate | Not yet measured — needs the live injection loop against a collector. |
| Citation integrity (ground faults via pipeline) | 1.000 | Gate = 1.000: passed | |
| Injected fault passes | 11 | Descriptive, not a gate | |
| Nominal control passes | 2 | Descriptive, not a gate |
Measured on synthetic · injected station passes, JANTAR-REF-1 graph. attributable=4 · false_positives=0. Source: make bench-ground. Measured at commit 2bc4474. From this release’s scorecard.
Refusal — public missions, no grounding
Measured on: real · OPSSAT-AD (ESA OPS-SAT-1, CC BY 4.0)
No failure modes were supplied, so the only correct answer on each anomaly is to decline. These rows measure fail-closed behaviour, not diagnosis. The gated rows ran on the deterministic model, which makes no model calls, and took their anomalies from the operators’ labels rather than from Jantar’s detection.
| Metric | Value | Gate | Note |
|---|---|---|---|
| Declined, never a cause (OPS-SAT-1) | 1.000 | Gate = 1.000: passed | |
| Citation integrity (OPS-SAT-1) | 1.000 | Gate = 1.000: passed | |
| Fabricated causes (OPS-SAT-1) | 0 | Gate = 0: passed | |
| Commands staged (OPS-SAT-1) | 0 | Gate = 0: passed | |
| Honest refusals releasable (OPS-SAT-1) | 1.000 | Descriptive, not a gate | Not gated. An honest INSUFFICIENT should reach the operator — declining and withholding are different things; only an unsupported confident claim is blocked. |
| Anomalies investigated (OPS-SAT-1) | 434/434 | Descriptive, not a gate |
Measured on real · OPSSAT-AD (ESA OPS-SAT-1, CC BY 4.0). graph=manifest-only (zero failure modes) · corpora=OPS-SAT-1. Source: make bench-refusal (needs `make data-opssat`). Measured at commit 2bc4474. From this release’s scorecard.
Portability — same result under every model
Measured on: synthetic · JANTAR-REF-1
| Metric | Value | Gate | Note |
|---|---|---|---|
| Golden expectations met (deterministic) | 1.000 | Descriptive, not a gate | |
| Root-cause top-1 (deterministic) | 1.000 | Descriptive, not a gate | |
| Citation integrity (deterministic) | 1.000 | Descriptive, not a gate | |
| Releasability matches (deterministic) | 1.000 | Descriptive, not a gate | |
| Median seconds (deterministic) | 0.000 | Descriptive, not a gate | |
| Time to root cause, p95 seconds (deterministic) | 0.100 | Descriptive, not a gate | nearest-rank over 16 scenario timings: rank 16 of 16, the slowest run |
| Output tokens per investigation (deterministic) | — | Descriptive, not a gate | |
| Cross-model verdict agreement | — | Descriptive, not a gate | Not measured: 1 model measured (deterministic), and agreement needs at least two. A single backend agrees with itself on every scenario, so its 1.000 would say nothing about portability. |
Measured on synthetic · JANTAR-REF-1. 1 model measured: deterministic · scenarios_compared=16. Source: make bench-portability. Measured at commit 2bc4474. From this release’s scorecard.
RCA at scale — accuracy as the pool grows
Measured on: synthetic · generated confuser-family craft (ADR 0025)
| Metric | Value | Gate | Note |
|---|---|---|---|
| Citation integrity at scale | 1.000 | Gate = 1.000: passed | |
| Grounded honesty at scale | 1.000 | Gate = 1.000: passed | |
| Top-1 at 18 failure modes (production) | 1.000 | Descriptive, not a gate | |
| Top-1 at 405 failure modes (production) | 1.000 | Descriptive, not a gate | The floor, not a score: the deterministic scaffold ranks by channel overlap and cannot see a deviation's sign. A learned ranker is measured against it. An exhausted tool budget fails closed to INSUFFICIENT, and the exhausted count below says whether the envelope, not the ranking, set this number. |
| True cause survived the hypothesis cap (production) | 1.000 | Descriptive, not a gate | |
| Investigations that exhausted the envelope (production) | 0/405 | Descriptive, not a gate |
Measured on synthetic · generated confuser-family craft (ADR 0025). model=deterministic · top-1 by pool size — production: 18→1.0 · 54→1.0 · 108→1.0 · 207→1.0 · 405→1.0. Source: make bench-rca-scale. Measured at commit 2bc4474. From this release’s scorecard.
Reasoning baseline — what a redesign must beat
Measured on: synthetic · JANTAR-REF-1
| Metric | Value | Gate | Note |
|---|---|---|---|
| Unsupported claims released (airgap) | 0 | Gate = 0: passed | |
| Ungrounded steps released (airgap) | 0 | Gate = 0: passed | |
| Calls naming another customer's data (airgap) | 0 | Gate = 0: passed | |
| Restart-resume reaches the same outcome (airgap) | 1.000 | Gate = 1.000: passed | |
| Insufficient evidence: declined or honest (airgap) | 1.000 | Gate = 1.000: passed | |
| Insufficient evidence: declined outright (airgap) | 0.312 | Descriptive, not a gate | Below 1.0 is honest degradation: with a cause's documentation removed the scaffold names a documented sibling on the same component, never the removed cause. Only a cause alone on its component can only be declined. |
| Root-cause top-1 (airgap) | 1.000 | Descriptive, not a gate | |
| Correct abstention (airgap) | 1.000 | Descriptive, not a gate | |
| Grounding repair rounds (airgap) | 0 | Descriptive, not a gate | |
| Runs that exhausted their envelope (airgap) | 0/20 | Descriptive, not a gate | |
| Tool calls per investigation, mean (airgap) | 24.100 | Descriptive, not a gate | |
| Seconds per investigation, p95 (airgap) | 0.040 | Descriptive, not a gate | a measurement on the machine that ran it, never an SLA |
| Output tokens per investigation (airgap) | — | Descriptive, not a gate | |
| Unsupported claims released (cloud) | 0 | Gate = 0: passed | |
| Ungrounded steps released (cloud) | 0 | Gate = 0: passed | |
| Calls naming another customer's data (cloud) | 0 | Gate = 0: passed | |
| Restart-resume reaches the same outcome (cloud) | 1.000 | Gate = 1.000: passed | |
| Insufficient evidence: declined or honest (cloud) | 1.000 | Gate = 1.000: passed | |
| Insufficient evidence: declined outright (cloud) | 0.312 | Descriptive, not a gate | Below 1.0 is honest degradation: with a cause's documentation removed the scaffold names a documented sibling on the same component, never the removed cause. Only a cause alone on its component can only be declined. |
| Root-cause top-1 (cloud) | 1.000 | Descriptive, not a gate | |
| Correct abstention (cloud) | 1.000 | Descriptive, not a gate | |
| Grounding repair rounds (cloud) | 0 | Descriptive, not a gate | |
| Runs that exhausted their envelope (cloud) | 0/20 | Descriptive, not a gate | |
| Tool calls per investigation, mean (cloud) | 24.100 | Descriptive, not a gate | |
| Seconds per investigation, p95 (cloud) | 0.040 | Descriptive, not a gate | a measurement on the machine that ran it, never an SLA |
| Output tokens per investigation (cloud) | — | Descriptive, not a gate |
Measured on synthetic · JANTAR-REF-1. modes=airgap/deterministic, cloud/deterministic · scenarios=20. Source: make bench-reasoning. Measured at commit 2bc4474. From this release’s scorecard.
Where to go next
- The OPS-SAT-AD benchmark: detection on real CubeSat telemetry, with its protocol
- Try the interlock: one anomaly, investigated with and without its evidence
- How to investigate a satellite anomaly: the method these figures test