Public measurement scorecard

Held-out results, confidence intervals, frozen canaries, and the weak spots—generated from checked-in research outputs.

This page is generated from the checked-in backtest JSON. It reports the weak results with the same prominence as the stronger ones and does not turn a small or missing sample into a claim. These metrics measure agreement with a later bad price outcome (rug or dead under the recorder label), not whether a product is real, safe, or worth trading.

this is model evaluation, not trading performance. A grade is a risk flag, not a certificate. Nothing here predicts returns.

Frozen task-1401 evaluation#

Dataset: 1054 coins observed from 2026-10-10 through 2026-10-13, with a 3-day outcome horizon. The earlier 726 rows were the train split; the later 328 rows were held out, including 61 bad outcomes. Tuning used train rows only and the code refuses test rows. Rates use 95% Wilson intervals; fewer than 20 rows gets counts only.

Evidence was restricted to a point-in-time token record fetched no more than one hour after t0. Later records were excluded. Labels are keyless recorder proxies, not chain-adjudicated facts.

Held-out result	Before	Shipped	Reading
Links-only LARP MCC	0.039	0.054	Weak. Link shape alone barely separated bad coins from the rest.
Rug-rule MCC	0.043	0.159	Better, but still modest and dataset-specific.

Links-only LARP baseline: where it is weak#

The shipped held-out MCC is 0.054. The LARP grade appeared only 14 times and 4 later had a bad outcome. Because that is fewer than 20 rows, the harness withholds precision and the grade-level confidence interval; it is too small to rate. The raw confusion matrix is 4 true positives, 10 false positives, 57 false negatives and 257 true negatives.

The EARLY_BUT_REAL grade had 19/184 bad outcomes: 10.3% (95% CI 6.7%–15.6%). The whole held-out set had 61/328 bad outcomes: 18.6% (95% CI 14.8%–23.2%). This is a links-only prior. It cannot score the fetched-page, domain, GitHub or model-judgement stage.

Rug verdict rules#

For the shipped rules, warning grades had precision 28.6% (95% CI 20.3%–38.6%) and recall 42.6% (95% CI 31.0%–55.1%) on the held-out split. The confusion matrix is 26 true positives, 65 false positives, 35 false negatives and 202 true negatives.

The strongest measured split was low liquidity: SPECULATIVE had 23/42 bad outcomes, 54.8% (95% CI 39.9%–68.8%). WATCH had 34/236, 14.4% (95% CI 10.5%–19.5%). Dev percentage, organic score and top-10 concentration did not predict a worse 72-hour outcome in this dataset; they now lower confidence instead of setting the verdict.

Latest skill-care run#

This rolling check was generated 2026-10-16 from 1000 labelled coins: 670 train and 330 held out. It proposed no change. There were 0 new labelled coins since the last accepted tuning; the gate requires 150. The recorded reason also says no subject cleared noise on the held-out window.

Held-out check	Current	Candidate	What happened
Links-only LARP MCC	0.084	0.058	Candidate was worse. Candidate precision was 27.3% (95% CI 13.2%–48.2%); recall was 9.7% (95% CI 4.5%–19.5%).
Rug-rule MCC	0.142	0.142	No change. Precision was 28.2% (95% CI 19.8%–38.6%); recall was 38.7% (95% CI 27.6%–51.2%).

Frozen canary#

The fixed task-1401 test split is the canary. Shipped LARP MCC is 0.054 against a frozen baseline of 0.054; the candidate scored 0.053. Shipped rug-rule MCC is 0.159, the same as its frozen baseline and candidate. The gate reported that the canary held, but it still rejected the candidate because the held-out improvement test failed.

Not measured#

Full LARP judgement: fetched pages, domain history, GitHub evidence and the model stage cannot be reconstructed as they appeared in the past. Forward scoring waits for at least 50 recorded verdicts matched to later outcomes.
Rug model output: probability and bias are measured forward from recorded pack verdicts, not by this replay.
Trench scan: there is no per-token labelled outcome.
Dev-history backtest: No checked-in dev-history backtest result exists yet. This section deliberately shows no metric.

Source and rebuild#

Frozen data and report: packages/packs/eval/skills-eval.json and packages/packs/eval/skills-backtest.json.
Rolling skill care: packages/packs/eval/skill-care-latest.json.
Method details: LARP and skill evaluation.

dexterm/scripts/docs-sync.sh regenerates this page before rebuilding the analyst docs knowledge base. npm run docs:check compares this file byte-for-byte with the generator output, so edited claims or favorable rounding fail the check.