{ "schema": "rsi-v2", "updated_at": "2026-10-10T01:11:16.204509+00:00", "experiment_id": "chem-sonnet-v2-20261009-03", "status": "complete", "models": { "policy": { "adapter": "claude_code", "model": "claude-sonnet-5-5", "effort": "high", "version": "2.1.293" }, "roles": { "adapter": "claude_code", "model": "claude-sonnet-5-5", "effort": "high" } }, "protocol": { "tasks": [ "chemistry/sargent-lab/propylene-active-learning", "chemistry/pfizer/suzuki-condition-screen" ], "rounds": 5, "trials_per_task": 3, "screen_trials_per_task": 1, "edit_budget": [ 2, 2, 1, 1, 1 ], "max_repairs": 3, "selection": { "delta": 0.16666666666666666, "p_accept": 0.8, "token_saving": 0.15, "screen_stop_if_zero_and_incumbent_at_least": 0.5 }, "reporting": "Evolve-set scores only. No local H0 rerun, no OOD set, no generalization claim." }, "external_reference": { "reported_percent": 33.3, "source": "user-specified benchmark reference", "paired_local_baseline": false }, "billing": { "route": "existing_claude_max_subscription_only", "paid_api_usd": 0, "extra_usage_required": false, "on_quota_limit": "stop_and_preserve; no_automatic_resume" }, "selected_round": 3, "selected_sha256": "b232ad942c6f44e92710c1bf6a8c148a76d813fdc9a918bc1c68f88066815528", "notes": [ "Every score here is an evolve-set score on the same two tasks that guided the revisions. It is not evidence of performance on unseen tasks.", "No local native Claude Code baseline was run. The 33.3% benchmark figure is an external reference, not a paired comparison.", "With six binary trials per candidate, the 95% interval on a pass rate is roughly ±35–40 points. Read the posteriors, not just the rates.", "Propylene stayed at 1/3 for every evaluated candidate; all accepted movement came from Suzuki." ], "prior_attempts": [ { "experiment_id": "chem-sonnet-v2-20261009", "outcome": "stopped before any model call: subscription usage check returned HTTP 401; preserved, not resumed" }, { "experiment_id": "chem-sonnet-v2-20261009-02", "outcome": "stopped after round 4 trials: selection crashed on float rewards; closed by user with round 4 decided offline (closure.json); excluded from new evidence" } ], "rounds": [ { "round": 1, "title": "Campaign checklist and a checking tool", "summary": "From an empty extension: a checklist in instructions.md (read the task files first, keep a validated model on disk, check every batch) plus campaign_check.py, a helper for batch checks, local NDCG and model probing.", "observation": "The first draft copied task file paths verbatim and was stopped by the leakage scan; the critic then rejected a Suzuki-specific test string. Both were repaired before any trial.", "accepted": true, "reason": "first_measured_incumbent", "recorded_offline": false, "score": 0.16666666666666666, "trials_run": 6, "task_rates": { "propylene-active-learning": 0.3333333333333333, "suzuki-condition-screen": 0.0 }, "p_superior": null, "incumbent_score": null, "tokens_per_trial": 901899.0, "complexity": 421, "edit_budget": 2, "components_changed": [ "instructions", "tool" ], "changed_files": [ "instructions.md", "smoke.json", "tools/campaign_check.py" ], "plan": { "problem": "No trial has been measured. archive/ is absent, history.json is [], and parent and candidate instructions.md are both empty. The native agent therefore gets no help with the shared structure of both tasks: seed data, then a few irreversible batched oracle rounds, then a self-contained deterministic model that must pass a discovery gate and a held-out NDCG gate. The most likely losses are predicted, not observed: (a) no local estimate of the held-out NDCG, so the agent can't tell whether the ranking gate passes; (b) rounds spent without a plan for both gates and without a check of the batch; (c) a model that is written late or breaks on unseen levels or NaN; (d) seed data over-trusted as a shortlist for the target regime. All of this is hypothesis.", "edits": [ { "component": "instructions", "hypothesis": "A short, domain-neutral workflow checklist in instructions.md will raise the chance that both gates pass. The checklist covers: (1) read asb campaign, README, MODEL_FORMAT.md and model_template.js in full before round 1; (2) write and validate a baseline final_model.js from the seed data before round 1, and re-validate it after every round; (3) before each asb run, check that the batch has the exact size, is distinct, has not been ordered before, and has the right header; (4) split each round's budget between exploiting the model's top picks and covering under-sampled groups or elements, and track the running hit count against the discovery threshold; (5) use the seed data for how levels behave, not as a shortlist, and normalise within group where the target is per group; (6) pick the final model by cross-validation that matches the held-out distribution, grouped by substrate or alloy element, with the metric computed locally. Falsifiable: pass rate on at least one of the two tasks improves, and no run ends without a validated final_model.js. If the pass rate does not rise and the share of runs without a valid model at the deadline does not fall, the hypothesis is refuted.", "files": [ "instructions.md" ] }, { "component": "tool", "hypothesis": "A stdlib-only helper, tools/campaign_check.py, lowers the cost of the checks the native agent would otherwise write ad hoc, and makes the missing local NDCG available. It has three subcommands. `batch` takes a round CSV, the expected size, the id column and the previous round or results CSVs. It reports the header, size, duplicates and overlap with earlier orders, and it exits non-zero on a problem. `ndcg` takes a CSV of prediction and relevance columns, k, and an optional group column. It computes NDCG@k with the 1/log2(rank+1) discount, the stated tie rule (equal scores keep evaluation order, via a stable sort) and the ideal-ranking normalisation. `probe` runs final_model.js under node on a list of input strings, including edge inputs, and checks that each output is finite and that repeated calls agree. Falsifiable: when the agent uses it, no round is rejected for format or duplicate reasons, and every final model passes probe. If rejected rounds or non-finite outputs still occur with the tool in use, the hypothesis fails.", "files": [ "tools/campaign_check.py", "smoke.json" ] } ], "predictions": { "tasks": { "propylene-active-learning": "improve", "suzuki-condition-screen": "improve" }, "token_effect": "same" }, "why_not_simpler": "Leaving the bundle empty gives no information and no change, so it is not a candidate. Instructions alone would leave the agent hand-writing NDCG with the tie rule and ideal normalisation each run. That is error-prone, and an error there would hide whether the ranking gate passes. A tool alone would never be invoked without a pointer to it. A hook on asb run was rejected as the first lever. It could wrongly block an irreversible-but-valid action, and it can't know the batch size without task-specific logic that would encode task identity. Instructions plus a tool is the smallest additive pair. Neither edits the immutable loop, and both are generic across the two tasks. Both are also reversible by deleting the files.", "regression_risk": "Low to moderate. Instructions could cost tokens or time through extra up-front reading and validation, and a rigid explore/exploit split could slightly reduce discovery hits if the agent follows it mechanically. I will keep the text short and phrase the split as guidance with the discovery count to be tracked, not as fixed percentages. The tool could have bugs. In particular, a wrong NDCG would mislead model selection, so the smoke test should check a hand-computed value, the tie rule and grouping. The tool is advisory and never blocks anything. Because the instructions say nothing about specific designs, they do not encode answers or task identity." }, "attempts": [ { "attempt": 0, "gate_passed": false, "gate_errors": [ "instructions.md: task_text_overlap (['app model format md and app model template', 'asb deliver app final model js validate only', 'model format md and app model template js'])" ], "critic": null }, { "attempt": 1, "gate_passed": true, "gate_errors": [], "critic": { "approved": false, "reasons": [ "tools/campaign_check.py:309 hardcodes BUILTIN_EDGE = [\"\", \"?\", \"Unknown|None|None|None|Unknown\"]. The last string is the five-field pipe-joined input format of one specific task (the Suzuki condition string). The probe feeds it to every model it checks. That is task-specific hardcoding in a tool that is supposed to be domain-neutral. It would be a meaningless input on any other suite, including the other task in this plan.", "instructions.md names task-specific vocabulary as examples: 'alloy element', 'substrate', 'ligands, bases, solvents', 'designs'. It also repeats the task's own relevance wording ('Equal predictions keep evaluation order'). The previous attempt was already rejected by the pre-evaluation gate for task-text overlap (repair-feedback.txt). The paraphrase removes the flagged phrases, but the file still reads as tuned to these two tasks instead of to the generic pattern of seed data, batched oracle rounds, then a model.", "The rest of the change is acceptable. There are no network, model or provider calls. It does not read grader, test or solution files. It does not persist memory or skills across episodes. There are no hooks or unbounded loops: the probe is bounded by --timeout and the vm timeout, and the other subcommands are single passes. It stays within the declared plan (instructions plus one tool and its smoke.json), and the ndcg, batch and probe logic is generic. The two defects above are small and fixable." ], "required_fixes": [ "Remove 'Unknown|None|None|None|Unknown' from BUILTIN_EDGE in tools/campaign_check.py. Keep only domain-neutral edge inputs such as '' and '?'. Have the agent supply any real-format edge cases through --edge-file or --inputs-file, built from the catalog it is working with.", "In instructions.md, replace task-flavoured examples with neutral ones. For example, write 'group folds by the grouping variable that distinguishes held-out items (e.g. a categorical key)' instead of 'substrate or alloy element', and 'levels of each categorical factor' instead of 'elements, ligands, bases, solvents'. Rephrase the tie-rule sentence so it does not echo the task's metric wording.", "Re-run the smoke tests after the edit, including selftest and the batch, ndcg and probe cases, to confirm that dropping the string changes no expected exit codes." ] } }, { "attempt": 2, "gate_passed": true, "gate_errors": [], "critic": { "approved": true, "reasons": [ "Scope matches the declared plan: the diff touches only instructions.md, tools/campaign_check.py and smoke.json (instructions plus one tool, within the edit budget of 2). There are no hooks, MCP servers, memory or skills, so nothing persists task data across episodes.", "The earlier critic's defect is fixed. BUILTIN_EDGE is now just ['', '?'], and the task-specific pipe-joined string is gone. Real-format edge cases come from the agent through --inputs-file or --edge-file.", "The instructions are now domain-neutral. They no longer name 'substrate', 'alloy element', 'ligands, bases, solvents' or 'designs', and they no longer repeat the tasks' tie-rule sentence. They describe the generic pattern of seed data, then batched irreversible rounds, then a deterministic model that must pass a discovery gate and a ranking gate. They contain no composition names, thresholds, batch sizes, measured designs or answers. A different suite with the same structure would still benefit from them.", "I found no access to private truth. The tool reads only files the agent passes to it, and it never opens grader, test or solution files. It has no network, model, provider or billing calls. The only subprocess is a local `node` run for the probe.", "The tool cannot stall the agent. The probe is bounded by --timeout (default 120 s) and a vm timeout of 60 s. The other subcommands are single passes. The tool is advisory and never blocks anything, and the selftest uses a temporary directory that it removes afterwards.", "The change has behavioural effect and is not degenerate or grader gaming. It adds a local NDCG@k (1/log2(rank+1) discount, stable ties, ideal normalisation), a pre-order batch check (header, size, duplicates, overlap with earlier orders, catalog membership) and a model probe (finite, repeatable, call-order independent). The smoke tests cover a hand-computed NDCG value, the tie rule, batch failures and probe accept/reject cases, with expected exit codes." ], "required_fixes": [] } } ], "diff": "--- parent/instructions.md\n+++ candidate/instructions.md\n@@ -0,0 +1,38 @@\n+# Campaign workflow notes\n+\n+These apply to tasks of the form: seed data, a few irreversible batched `asb run` rounds, then a self-contained\n+deterministic `final_model.js` that must pass both a discovery gate (hits among your ordered items) and a\n+held-out ranking (NDCG) gate. Two gates means neither can be left to the end.\n+\n+## Before round 1\n+- Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n+ script in full.\n+- Write a baseline model from the seed data and run the deliver command in its free validation-only mode.\n+ Re-validate after every round, so a valid model always exists well before the deadline.\n+- Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n+\n+## Each round\n+- Accepted rounds are final; rejected requests are free. Before `asb run`, check the file:\n+ `python3 /opt/rsi-harness/tools/campaign_check.py batch round_N.csv --size --id-column --previous round_*_results.csv [--catalog ]`\n+ Fix anything it reports (header, size, duplicates, items already ordered, ids not in the catalog).\n+- Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n+ Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n+ This is guidance, not fixed percentages.\n+- Measurements are noisy; do not read much into a single value that sits near the threshold.\n+\n+## Using the seed data\n+- Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n+ differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n+ score favour easy groups over hard ones.\n+\n+## Choosing and checking the model\n+- Pick the final model by cross-validation that mimics the held-out distribution (group the folds by the grouping variable\n+ that distinguishes held-out items, e.g. a categorical key), with the official metric computed locally:\n+ `python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ] [--threshold ]`\n+ Pass relevance already defined as the task states (clipped/normalised). The tool breaks score ties by input row order;\n+ prefer models that rarely produce ties.\n+- The model must be self-contained, deterministic, finite for every input and independent per input; handle levels\n+ unseen in training with a fallback. Before delivering, run\n+ `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n+ with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n+- `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n--- parent/smoke.json\n+++ candidate/smoke.json\n@@ -0,0 +1,58 @@\n+[\n+ {\n+ \"name\": \"campaign_check selftest (hand-computed NDCG, tie rule, batch and probe checks)\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"selftest\"],\n+ \"stdin\": \"\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 120\n+ },\n+ {\n+ \"name\": \"campaign_check ndcg on a tiny synthetic CSV from stdin\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"p\", \"--rel\", \"r\", \"--k\", \"3\", \"--group\", \"g\"],\n+ \"stdin\": \"p,r,g\\n0.9,1,x\\n0.8,0,x\\n0.7,1,x\\n0.5,0,y\\n0.4,2,y\\n0.3,1,y\\n\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"campaign_check ndcg rejects an unknown column\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"nope\", \"--rel\", \"r\", \"--k\", \"2\"],\n+ \"stdin\": \"p,r\\n1,1\\n2,0\\n\",\n+ \"expect_exit\": 2,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"campaign_check batch accepts a clean round\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n+ \"stdin\": \"well_id\\nw1\\nw2\\nw3\\n\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"campaign_check batch flags duplicates and wrong size\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n+ \"stdin\": \"well_id\\nw1\\nw1\\n\",\n+ \"expect_exit\": 1,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"campaign_check batch flags a wrong header\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"2\", \"--id-column\", \"item_key\"],\n+ \"stdin\": \"id\\nA\\nB\\n\",\n+ \"expect_exit\": 1,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"campaign_check probe reports a missing model as a usage error\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n+ \"stdin\": \"\",\n+ \"expect_exit\": 2,\n+ \"timeout\": 30\n+ }\n+]\n--- parent/tools/campaign_check.py\n+++ candidate/tools/campaign_check.py\n@@ -0,0 +1,366 @@\n+#!/usr/bin/env python3\n+\"\"\"Advisory checks for batched-oracle campaigns (stdlib only, never blocks anything).\n+\n+Subcommands\n+ batch check a round CSV before ordering it (header, size, duplicates, earlier orders, catalog)\n+ ndcg local NDCG@k with 1/log2(rank+1) discount, stable ties, ideal-ranking normalisation\n+ probe run final_model.js under node on sample inputs: finite, deterministic, order-independent\n+ selftest built-in hand-computed checks of the above\n+\n+Exit codes: 0 ok, 1 problem found, 2 usage or I/O error.\n+\"\"\"\n+import argparse\n+import csv\n+import io\n+import json\n+import math\n+import os\n+import shutil\n+import subprocess\n+import sys\n+import tempfile\n+\n+\n+def die(msg, code=2):\n+ sys.stderr.write(\"error: %s\\n\" % msg)\n+ sys.exit(code)\n+\n+\n+def read_text(path):\n+ if path == \"-\":\n+ return sys.stdin.read()\n+ try:\n+ with open(path, encoding=\"utf-8-sig\", newline=\"\") as fh:\n+ return fh.read()\n+ except OSError as exc:\n+ die(\"cannot read %s: %s\" % (path, exc))\n+\n+\n+def read_rows(path):\n+ text = read_text(path)\n+ if text.startswith(\"\"):\n+ text = text[1:]\n+ rows = list(csv.reader(io.StringIO(text)))\n+ rows = [r for r in rows if r and any(c.strip() for c in r)]\n+ if not rows:\n+ die(\"%s is empty\" % path)\n+ return rows[0], rows[1:]\n+\n+\n+def id_values(path, column):\n+ header, body = read_rows(path)\n+ idx = header.index(column) if column in header else 0\n+ return [r[idx].strip() for r in body if len(r) > idx]\n+\n+\n+# ---------------------------------------------------------------- batch\n+def cmd_batch(args):\n+ header, body = read_rows(args.round_csv)\n+ problems, notes = [], []\n+ column = args.id_column or header[0]\n+ if len(header) != 1:\n+ problems.append(\"header has %d columns %r; the round file needs a single column\" % (len(header), header))\n+ if header[0] != column:\n+ problems.append(\"header is %r, expected %r\" % (header[0], column))\n+ ids = [r[0] for r in body]\n+ padded = [i for i in ids if i != i.strip()]\n+ if padded:\n+ problems.append(\"%d ids have leading/trailing whitespace, e.g. %r\" % (len(padded), padded[0]))\n+ ids = [i.strip() for i in ids]\n+ if any(len(r) != 1 for r in body):\n+ problems.append(\"some rows have more than one field\")\n+ if any(not i for i in ids):\n+ problems.append(\"blank ids present\")\n+ notes.append(\"header=%r rows=%d\" % (header, len(ids)))\n+ if args.size is not None and len(ids) != args.size:\n+ problems.append(\"size is %d, expected exactly %d\" % (len(ids), args.size))\n+ seen, dups = set(), []\n+ for i in ids:\n+ if i in seen and i not in dups:\n+ dups.append(i)\n+ seen.add(i)\n+ if dups:\n+ problems.append(\"%d duplicated ids inside the batch, e.g. %s\" % (len(dups), dups[:3]))\n+ prior = set()\n+ for p in args.previous or []:\n+ prior.update(id_values(p, column))\n+ overlap = sorted(seen & prior)\n+ if args.previous:\n+ notes.append(\"compared with %d earlier ids from %d files\" % (len(prior), len(args.previous)))\n+ if overlap:\n+ problems.append(\"%d ids were already ordered, e.g. %s\" % (len(overlap), overlap[:3]))\n+ if args.catalog:\n+ cat = set(id_values(args.catalog, args.catalog_column or column))\n+ notes.append(\"catalog has %d ids\" % len(cat))\n+ unknown = sorted(seen - cat)\n+ if unknown:\n+ problems.append(\"%d ids are not in the catalog, e.g. %s\" % (len(unknown), unknown[:3]))\n+ for n in notes:\n+ print(\"note: \" + n)\n+ for p in problems:\n+ print(\"PROBLEM: \" + p)\n+ print(\"batch %s\" % (\"FAILED\" if problems else \"OK\"))\n+ return 1 if problems else 0\n+\n+\n+# ---------------------------------------------------------------- ndcg\n+def dcg(rels):\n+ return sum(r / math.log2(i + 2) for i, r in enumerate(rels))\n+\n+\n+def ndcg_at_k(preds, rels, k):\n+ \"\"\"Rank by prediction descending; equal predictions keep input order (stable sort).\"\"\"\n+ order = sorted(range(len(preds)), key=lambda i: -preds[i])\n+ got = dcg([rels[i] for i in order[:k]])\n+ ideal = dcg(sorted(rels, reverse=True)[:k])\n+ if ideal <= 0:\n+ return None\n+ return got / ideal\n+\n+\n+def cmd_ndcg(args):\n+ header, body = read_rows(args.csv)\n+ for c in [args.pred, args.rel] + ([args.group] if args.group else []):\n+ if c not in header:\n+ die(\"column %r not in header %r\" % (c, header))\n+ ip, ir = header.index(args.pred), header.index(args.rel)\n+ ig = header.index(args.group) if args.group else None\n+ preds, rels, groups = [], [], []\n+ for n, r in enumerate(body, 2):\n+ try:\n+ p, q = float(r[ip]), float(r[ir])\n+ except (ValueError, IndexError):\n+ die(\"row %d: non-numeric prediction or relevance\" % n)\n+ if math.isnan(p) or math.isnan(q) or math.isinf(p):\n+ die(\"row %d: NaN/inf value\" % n)\n+ preds.append(p)\n+ rels.append(q)\n+ groups.append(r[ig] if ig is not None else None)\n+ if any(q < 0 for q in rels):\n+ print(\"warning: negative relevance present; pass relevance already clipped at 0 if the metric does so\")\n+ k = args.k\n+ if k > len(preds):\n+ print(\"warning: k=%d exceeds %d rows\" % (k, len(preds)))\n+ val = ndcg_at_k(preds, rels, k)\n+ if val is None:\n+ print(\"warning: ideal DCG is 0 (no positive relevance); NDCG undefined, reported as 0\")\n+ val = 0.0\n+ print(\"ndcg@%d = %.6f (n=%d)\" % (k, val, len(preds)))\n+ if args.group:\n+ by = {}\n+ for i, g in enumerate(groups):\n+ by.setdefault(g, []).append(i)\n+ vals = []\n+ for g in sorted(by):\n+ v = ndcg_at_k([preds[i] for i in by[g]], [rels[i] for i in by[g]], k)\n+ vals.append(v)\n+ print(\" group %s: n=%d ndcg@%d=%s\" % (g, len(by[g]), k, \"undefined\" if v is None else \"%.4f\" % v))\n+ ok = [v for v in vals if v is not None]\n+ if ok:\n+ print(\"mean over groups (informational, not the pooled metric) = %.6f\" % (sum(ok) / len(ok)))\n+ if args.threshold is not None:\n+ print(\"threshold %.4f: %s\" % (args.threshold, \"PASS\" if val >= args.threshold else \"BELOW\"))\n+ return 0\n+\n+\n+# ---------------------------------------------------------------- probe\n+PROBE_JS = r\"\"\"\n+const fs = require('fs'), vm = require('vm');\n+const req = JSON.parse(fs.readFileSync(0, 'utf8'));\n+function load() {\n+ const code = fs.readFileSync(req.model, 'utf8');\n+ const mod = { exports: {} };\n+ const sb = { module: mod, exports: mod.exports, console: { log() {}, error() {}, warn() {} } };\n+ vm.createContext(sb);\n+ vm.runInContext(code, sb, { timeout: 60000 });\n+ const f = (typeof sb.predict === 'function') ? sb.predict\n+ : (mod.exports && typeof mod.exports.predict === 'function') ? mod.exports.predict : null;\n+ if (!f) throw new Error('no predict function defined');\n+ return f;\n+}\n+function run(f, x) {\n+ try { return { v: f(x) }; } catch (e) { return { e: String(e && e.message || e) }; }\n+}\n+const out = { load_error: null, results: [] };\n+try {\n+ const f1 = load(), f2 = load();\n+ const xs = req.inputs;\n+ const a = xs.map(x => run(f1, x));\n+ const b = xs.map(x => run(f1, x));\n+ const c = xs.map((x, i) => run(f2, xs[xs.length - 1 - i])).reverse();\n+ xs.forEach((x, i) => {\n+ const same = (p, q) => ('e' in p && 'e' in q) || ('v' in p && 'v' in q && Object.is(p.v, q.v));\n+ out.results.push({\n+ input: x, value: ('v' in a[i]) ? a[i].v : null, error: a[i].e || null,\n+ type: ('v' in a[i]) ? typeof a[i].v : null,\n+ finite: ('v' in a[i]) && typeof a[i].v === 'number' && Number.isFinite(a[i].v),\n+ repeat_same: same(a[i], b[i]), order_same: same(a[i], c[i])\n+ });\n+ });\n+} catch (e) { out.load_error = String(e && e.message || e); }\n+process.stdout.write(JSON.stringify(out));\n+\"\"\"\n+\n+BUILTIN_EDGE = [\"\", \"?\"]\n+\n+\n+def cmd_probe(args):\n+ if not os.path.isfile(args.model):\n+ die(\"model file not found: %s\" % args.model)\n+ node = shutil.which(\"node\")\n+ if not node:\n+ die(\"node not found on PATH\")\n+ inputs = list(args.input or [])\n+ if args.inputs_file:\n+ inputs += [l.rstrip(\"\\r\\n\") for l in read_text(args.inputs_file).splitlines() if l.strip()]\n+ edge_must = []\n+ if args.edge_file:\n+ edge_must = [l.rstrip(\"\\r\\n\") for l in read_text(args.edge_file).splitlines() if l.strip()]\n+ if not inputs and not edge_must:\n+ die(\"give --input and/or --inputs-file (real-format sample inputs)\")\n+ all_inputs = inputs + edge_must + BUILTIN_EDGE\n+ kinds = [\"sample\"] * len(inputs) + [\"edge-required\"] * len(edge_must) + [\"edge-advisory\"] * len(BUILTIN_EDGE)\n+ payload = json.dumps({\"model\": os.path.abspath(args.model), \"inputs\": all_inputs})\n+ try:\n+ proc = subprocess.run([node, \"-e\", PROBE_JS], input=payload, capture_output=True,\n+ text=True, timeout=args.timeout)\n+ except subprocess.TimeoutExpired:\n+ print(\"PROBLEM: node timed out after %ss\" % args.timeout)\n+ return 1\n+ if proc.returncode != 0:\n+ print(\"PROBLEM: node failed: %s\" % proc.stderr.strip()[:500])\n+ return 1\n+ try:\n+ res = json.loads(proc.stdout)\n+ except ValueError:\n+ print(\"PROBLEM: unparseable probe output: %r\" % proc.stdout[:200])\n+ return 1\n+ if res.get(\"load_error\"):\n+ print(\"PROBLEM: model failed to load in an isolated context: %s\" % res[\"load_error\"])\n+ return 1\n+ problems = 0\n+ for r, kind in zip(res[\"results\"], kinds):\n+ bad = []\n+ if r[\"error\"]:\n+ bad.append(\"threw: %s\" % r[\"error\"])\n+ elif not r[\"finite\"]:\n+ bad.append(\"not a finite number (%r)\" % (r[\"value\"],))\n+ if not r[\"repeat_same\"]:\n+ bad.append(\"repeated call differs\")\n+ if not r[\"order_same\"]:\n+ bad.append(\"result depends on call order/fresh load\")\n+ if bad:\n+ tag = \"WARN\" if kind == \"edge-advisory\" else \"PROBLEM\"\n+ if kind != \"edge-advisory\":\n+ problems += 1\n+ print(\"%s [%s] %r: %s\" % (tag, kind, r[\"input\"], \"; \".join(bad)))\n+ vals = [r[\"value\"] for r in res[\"results\"][:len(inputs)] if r[\"finite\"]]\n+ if vals:\n+ print(\"sample outputs: n=%d min=%.6g max=%.6g distinct=%d\" % (len(vals), min(vals), max(vals), len(set(vals))))\n+ if len(set(vals)) == 1 and len(vals) > 1:\n+ print(\"WARN: every sample input got the same score; the ranking would be all ties\")\n+ print(\"probe %s\" % (\"FAILED\" if problems else \"OK\"))\n+ return 1 if problems else 0\n+\n+\n+# ---------------------------------------------------------------- selftest\n+def cmd_selftest(_args):\n+ fails = []\n+\n+ def check(name, cond):\n+ if not cond:\n+ fails.append(name)\n+ print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n+\n+ # hand-computed: dcg = 1 + 0.5 = 1.5 ; ideal = 1 + 1/log2(3)\n+ v = ndcg_at_k([0.9, 0.8, 0.7], [1, 0, 1], 3)\n+ check(\"ndcg hand value\", abs(v - 1.5 / (1 + 1 / math.log2(3))) < 1e-12)\n+ # tie rule: equal scores keep evaluation order -> rels seen [0,1]\n+ v = ndcg_at_k([5, 5, 5], [0, 1, 1], 2)\n+ check(\"ndcg tie keeps input order\", abs(v - (1 / math.log2(3)) / (1 + 1 / math.log2(3))) < 1e-12)\n+ check(\"ndcg perfect ranking is 1\", abs(ndcg_at_k([3, 2, 1], [2, 1, 0], 2) - 1) < 1e-12)\n+ check(\"ndcg undefined without positives\", ndcg_at_k([1, 2], [0, 0], 2) is None)\n+ tmp = tempfile.mkdtemp(prefix=\"campchk_\")\n+ try:\n+ def w(name, text):\n+ p = os.path.join(tmp, name)\n+ with open(p, \"w\") as fh:\n+ fh.write(text)\n+ return p\n+ prev = w(\"prev.csv\", \"id,y\\na,1\\nb,2\\n\")\n+ ns = argparse.Namespace\n+ import contextlib\n+ sink = io.StringIO()\n+ with contextlib.redirect_stdout(sink):\n+ ok = cmd_batch(ns(round_csv=w(\"r1.csv\", \"id\\nc\\nd\\n\"), size=2, id_column=\"id\",\n+ previous=[prev], catalog=None, catalog_column=None))\n+ dup = cmd_batch(ns(round_csv=w(\"r2.csv\", \"id\\nc\\nc\\n\"), size=2, id_column=\"id\",\n+ previous=[prev], catalog=None, catalog_column=None))\n+ old = cmd_batch(ns(round_csv=w(\"r3.csv\", \"id\\na\\nd\\n\"), size=2, id_column=\"id\",\n+ previous=[prev], catalog=None, catalog_column=None))\n+ short = cmd_batch(ns(round_csv=w(\"r4.csv\", \"id\\nc\\n\"), size=2, id_column=\"id\",\n+ previous=[], catalog=None, catalog_column=None))\n+ hdr = cmd_batch(ns(round_csv=w(\"r5.csv\", \"wrong\\nc\\nd\\n\"), size=2, id_column=\"id\",\n+ previous=[], catalog=None, catalog_column=None))\n+ check(\"batch accepts clean round\", ok == 0)\n+ check(\"batch flags in-batch duplicate\", dup == 1)\n+ check(\"batch flags earlier order\", old == 1)\n+ check(\"batch flags wrong size\", short == 1)\n+ check(\"batch flags wrong header\", hdr == 1)\n+ if shutil.which(\"node\"):\n+ model = w(\"m.js\", \"function predict(x){ return x.length * 0.5; }\\n\")\n+ bad = w(\"bad.js\", \"function predict(x){ return x === '' ? NaN : Math.random(); }\\n\")\n+ with contextlib.redirect_stdout(sink):\n+ good = cmd_probe(ns(model=model, input=[\"ab\", \"abcd\"], inputs_file=None,\n+ edge_file=None, timeout=60))\n+ notgood = cmd_probe(ns(model=bad, input=[\"ab\", \"abcd\"], inputs_file=None,\n+ edge_file=None, timeout=60))\n+ check(\"probe accepts a finite deterministic model\", good == 0)\n+ check(\"probe rejects a random model\", notgood == 1)\n+ else:\n+ print(\"skip probe checks (node not found)\")\n+ finally:\n+ shutil.rmtree(tmp, ignore_errors=True)\n+ print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n+ return 1 if fails else 0\n+\n+\n+def main():\n+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n+ sub = ap.add_subparsers(dest=\"cmd\", required=True)\n+\n+ b = sub.add_parser(\"batch\", help=\"check a round CSV before `asb run` (use '-' for stdin)\")\n+ b.add_argument(\"round_csv\")\n+ b.add_argument(\"--size\", type=int, help=\"exact number of rows required\")\n+ b.add_argument(\"--id-column\", help=\"expected header, e.g. the single column name from the task\")\n+ b.add_argument(\"--previous\", nargs=\"*\", help=\"earlier round or results CSVs whose ids may not repeat\")\n+ b.add_argument(\"--catalog\", help=\"CSV listing orderable ids\")\n+ b.add_argument(\"--catalog-column\", help=\"id column in the catalog (default: --id-column)\")\n+ b.set_defaults(fn=cmd_batch)\n+\n+ n = sub.add_parser(\"ndcg\", help=\"local NDCG@k (use '-' for stdin)\")\n+ n.add_argument(\"csv\")\n+ n.add_argument(\"--pred\", required=True, help=\"prediction column\")\n+ n.add_argument(\"--rel\", required=True, help=\"relevance column (already clipped/normalised as the task defines)\")\n+ n.add_argument(\"--k\", type=int, required=True)\n+ n.add_argument(\"--group\", help=\"optional group column; adds per-group NDCG as a diagnostic\")\n+ n.add_argument(\"--threshold\", type=float, help=\"optional pass level to compare against\")\n+ n.set_defaults(fn=cmd_ndcg)\n+\n+ p = sub.add_parser(\"probe\", help=\"run final_model.js under node on sample inputs\")\n+ p.add_argument(\"model\")\n+ p.add_argument(\"--input\", action=\"append\", help=\"sample input string (repeatable)\")\n+ p.add_argument(\"--inputs-file\", help=\"file with one sample input per line\")\n+ p.add_argument(\"--edge-file\", help=\"file with one extra input per line that must also score finitely\")\n+ p.add_argument(\"--timeout\", type=int, default=120)\n+ p.set_defaults(fn=cmd_probe)\n+\n+ s = sub.add_parser(\"selftest\", help=\"hand-computed self checks\")\n+ s.set_defaults(fn=cmd_selftest)\n+\n+ args = ap.parse_args()\n+ sys.exit(args.fn(args))\n+\n+\n+if __name__ == \"__main__\":\n+ main()\n", "files": { "instructions.md": "# Campaign workflow notes\n\nThese apply to tasks of the form: seed data, a few irreversible batched `asb run` rounds, then a self-contained\ndeterministic `final_model.js` that must pass both a discovery gate (hits among your ordered items) and a\nheld-out ranking (NDCG) gate. Two gates means neither can be left to the end.\n\n## Before round 1\n- Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n script in full.\n- Write a baseline model from the seed data and run the deliver command in its free validation-only mode.\n Re-validate after every round, so a valid model always exists well before the deadline.\n- Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n\n## Each round\n- Accepted rounds are final; rejected requests are free. Before `asb run`, check the file:\n `python3 /opt/rsi-harness/tools/campaign_check.py batch round_N.csv --size --id-column --previous round_*_results.csv [--catalog ]`\n Fix anything it reports (header, size, duplicates, items already ordered, ids not in the catalog).\n- Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n This is guidance, not fixed percentages.\n- Measurements are noisy; do not read much into a single value that sits near the threshold.\n\n## Using the seed data\n- Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n score favour easy groups over hard ones.\n\n## Choosing and checking the model\n- Pick the final model by cross-validation that mimics the held-out distribution (group the folds by the grouping variable\n that distinguishes held-out items, e.g. a categorical key), with the official metric computed locally:\n `python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ] [--threshold ]`\n Pass relevance already defined as the task states (clipped/normalised). The tool breaks score ties by input row order;\n prefer models that rarely produce ties.\n- The model must be self-contained, deterministic, finite for every input and independent per input; handle levels\n unseen in training with a fallback. Before delivering, run\n `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n- `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n", "smoke.json": "[\n {\n \"name\": \"campaign_check selftest (hand-computed NDCG, tie rule, batch and probe checks)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"campaign_check ndcg on a tiny synthetic CSV from stdin\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"p\", \"--rel\", \"r\", \"--k\", \"3\", \"--group\", \"g\"],\n \"stdin\": \"p,r,g\\n0.9,1,x\\n0.8,0,x\\n0.7,1,x\\n0.5,0,y\\n0.4,2,y\\n0.3,1,y\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check ndcg rejects an unknown column\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"nope\", \"--rel\", \"r\", \"--k\", \"2\"],\n \"stdin\": \"p,r\\n1,1\\n2,0\\n\",\n \"expect_exit\": 2,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch accepts a clean round\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw2\\nw3\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags duplicates and wrong size\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw1\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags a wrong header\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"2\", \"--id-column\", \"item_key\"],\n \"stdin\": \"id\\nA\\nB\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n \"stdin\": \"\",\n \"expect_exit\": 2,\n \"timeout\": 30\n }\n]\n", "tools/campaign_check.py": "#!/usr/bin/env python3\n\"\"\"Advisory checks for batched-oracle campaigns (stdlib only, never blocks anything).\n\nSubcommands\n batch check a round CSV before ordering it (header, size, duplicates, earlier orders, catalog)\n ndcg local NDCG@k with 1/log2(rank+1) discount, stable ties, ideal-ranking normalisation\n probe run final_model.js under node on sample inputs: finite, deterministic, order-independent\n selftest built-in hand-computed checks of the above\n\nExit codes: 0 ok, 1 problem found, 2 usage or I/O error.\n\"\"\"\nimport argparse\nimport csv\nimport io\nimport json\nimport math\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\n\n\ndef die(msg, code=2):\n sys.stderr.write(\"error: %s\\n\" % msg)\n sys.exit(code)\n\n\ndef read_text(path):\n if path == \"-\":\n return sys.stdin.read()\n try:\n with open(path, encoding=\"utf-8-sig\", newline=\"\") as fh:\n return fh.read()\n except OSError as exc:\n die(\"cannot read %s: %s\" % (path, exc))\n\n\ndef read_rows(path):\n text = read_text(path)\n if text.startswith(\"\"):\n text = text[1:]\n rows = list(csv.reader(io.StringIO(text)))\n rows = [r for r in rows if r and any(c.strip() for c in r)]\n if not rows:\n die(\"%s is empty\" % path)\n return rows[0], rows[1:]\n\n\ndef id_values(path, column):\n header, body = read_rows(path)\n idx = header.index(column) if column in header else 0\n return [r[idx].strip() for r in body if len(r) > idx]\n\n\n# ---------------------------------------------------------------- batch\ndef cmd_batch(args):\n header, body = read_rows(args.round_csv)\n problems, notes = [], []\n column = args.id_column or header[0]\n if len(header) != 1:\n problems.append(\"header has %d columns %r; the round file needs a single column\" % (len(header), header))\n if header[0] != column:\n problems.append(\"header is %r, expected %r\" % (header[0], column))\n ids = [r[0] for r in body]\n padded = [i for i in ids if i != i.strip()]\n if padded:\n problems.append(\"%d ids have leading/trailing whitespace, e.g. %r\" % (len(padded), padded[0]))\n ids = [i.strip() for i in ids]\n if any(len(r) != 1 for r in body):\n problems.append(\"some rows have more than one field\")\n if any(not i for i in ids):\n problems.append(\"blank ids present\")\n notes.append(\"header=%r rows=%d\" % (header, len(ids)))\n if args.size is not None and len(ids) != args.size:\n problems.append(\"size is %d, expected exactly %d\" % (len(ids), args.size))\n seen, dups = set(), []\n for i in ids:\n if i in seen and i not in dups:\n dups.append(i)\n seen.add(i)\n if dups:\n problems.append(\"%d duplicated ids inside the batch, e.g. %s\" % (len(dups), dups[:3]))\n prior = set()\n for p in args.previous or []:\n prior.update(id_values(p, column))\n overlap = sorted(seen & prior)\n if args.previous:\n notes.append(\"compared with %d earlier ids from %d files\" % (len(prior), len(args.previous)))\n if overlap:\n problems.append(\"%d ids were already ordered, e.g. %s\" % (len(overlap), overlap[:3]))\n if args.catalog:\n cat = set(id_values(args.catalog, args.catalog_column or column))\n notes.append(\"catalog has %d ids\" % len(cat))\n unknown = sorted(seen - cat)\n if unknown:\n problems.append(\"%d ids are not in the catalog, e.g. %s\" % (len(unknown), unknown[:3]))\n for n in notes:\n print(\"note: \" + n)\n for p in problems:\n print(\"PROBLEM: \" + p)\n print(\"batch %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- ndcg\ndef dcg(rels):\n return sum(r / math.log2(i + 2) for i, r in enumerate(rels))\n\n\ndef ndcg_at_k(preds, rels, k):\n \"\"\"Rank by prediction descending; equal predictions keep input order (stable sort).\"\"\"\n order = sorted(range(len(preds)), key=lambda i: -preds[i])\n got = dcg([rels[i] for i in order[:k]])\n ideal = dcg(sorted(rels, reverse=True)[:k])\n if ideal <= 0:\n return None\n return got / ideal\n\n\ndef cmd_ndcg(args):\n header, body = read_rows(args.csv)\n for c in [args.pred, args.rel] + ([args.group] if args.group else []):\n if c not in header:\n die(\"column %r not in header %r\" % (c, header))\n ip, ir = header.index(args.pred), header.index(args.rel)\n ig = header.index(args.group) if args.group else None\n preds, rels, groups = [], [], []\n for n, r in enumerate(body, 2):\n try:\n p, q = float(r[ip]), float(r[ir])\n except (ValueError, IndexError):\n die(\"row %d: non-numeric prediction or relevance\" % n)\n if math.isnan(p) or math.isnan(q) or math.isinf(p):\n die(\"row %d: NaN/inf value\" % n)\n preds.append(p)\n rels.append(q)\n groups.append(r[ig] if ig is not None else None)\n if any(q < 0 for q in rels):\n print(\"warning: negative relevance present; pass relevance already clipped at 0 if the metric does so\")\n k = args.k\n if k > len(preds):\n print(\"warning: k=%d exceeds %d rows\" % (k, len(preds)))\n val = ndcg_at_k(preds, rels, k)\n if val is None:\n print(\"warning: ideal DCG is 0 (no positive relevance); NDCG undefined, reported as 0\")\n val = 0.0\n print(\"ndcg@%d = %.6f (n=%d)\" % (k, val, len(preds)))\n if args.group:\n by = {}\n for i, g in enumerate(groups):\n by.setdefault(g, []).append(i)\n vals = []\n for g in sorted(by):\n v = ndcg_at_k([preds[i] for i in by[g]], [rels[i] for i in by[g]], k)\n vals.append(v)\n print(\" group %s: n=%d ndcg@%d=%s\" % (g, len(by[g]), k, \"undefined\" if v is None else \"%.4f\" % v))\n ok = [v for v in vals if v is not None]\n if ok:\n print(\"mean over groups (informational, not the pooled metric) = %.6f\" % (sum(ok) / len(ok)))\n if args.threshold is not None:\n print(\"threshold %.4f: %s\" % (args.threshold, \"PASS\" if val >= args.threshold else \"BELOW\"))\n return 0\n\n\n# ---------------------------------------------------------------- probe\nPROBE_JS = r\"\"\"\nconst fs = require('fs'), vm = require('vm');\nconst req = JSON.parse(fs.readFileSync(0, 'utf8'));\nfunction load() {\n const code = fs.readFileSync(req.model, 'utf8');\n const mod = { exports: {} };\n const sb = { module: mod, exports: mod.exports, console: { log() {}, error() {}, warn() {} } };\n vm.createContext(sb);\n vm.runInContext(code, sb, { timeout: 60000 });\n const f = (typeof sb.predict === 'function') ? sb.predict\n : (mod.exports && typeof mod.exports.predict === 'function') ? mod.exports.predict : null;\n if (!f) throw new Error('no predict function defined');\n return f;\n}\nfunction run(f, x) {\n try { return { v: f(x) }; } catch (e) { return { e: String(e && e.message || e) }; }\n}\nconst out = { load_error: null, results: [] };\ntry {\n const f1 = load(), f2 = load();\n const xs = req.inputs;\n const a = xs.map(x => run(f1, x));\n const b = xs.map(x => run(f1, x));\n const c = xs.map((x, i) => run(f2, xs[xs.length - 1 - i])).reverse();\n xs.forEach((x, i) => {\n const same = (p, q) => ('e' in p && 'e' in q) || ('v' in p && 'v' in q && Object.is(p.v, q.v));\n out.results.push({\n input: x, value: ('v' in a[i]) ? a[i].v : null, error: a[i].e || null,\n type: ('v' in a[i]) ? typeof a[i].v : null,\n finite: ('v' in a[i]) && typeof a[i].v === 'number' && Number.isFinite(a[i].v),\n repeat_same: same(a[i], b[i]), order_same: same(a[i], c[i])\n });\n });\n} catch (e) { out.load_error = String(e && e.message || e); }\nprocess.stdout.write(JSON.stringify(out));\n\"\"\"\n\nBUILTIN_EDGE = [\"\", \"?\"]\n\n\ndef cmd_probe(args):\n if not os.path.isfile(args.model):\n die(\"model file not found: %s\" % args.model)\n node = shutil.which(\"node\")\n if not node:\n die(\"node not found on PATH\")\n inputs = list(args.input or [])\n if args.inputs_file:\n inputs += [l.rstrip(\"\\r\\n\") for l in read_text(args.inputs_file).splitlines() if l.strip()]\n edge_must = []\n if args.edge_file:\n edge_must = [l.rstrip(\"\\r\\n\") for l in read_text(args.edge_file).splitlines() if l.strip()]\n if not inputs and not edge_must:\n die(\"give --input and/or --inputs-file (real-format sample inputs)\")\n all_inputs = inputs + edge_must + BUILTIN_EDGE\n kinds = [\"sample\"] * len(inputs) + [\"edge-required\"] * len(edge_must) + [\"edge-advisory\"] * len(BUILTIN_EDGE)\n payload = json.dumps({\"model\": os.path.abspath(args.model), \"inputs\": all_inputs})\n try:\n proc = subprocess.run([node, \"-e\", PROBE_JS], input=payload, capture_output=True,\n text=True, timeout=args.timeout)\n except subprocess.TimeoutExpired:\n print(\"PROBLEM: node timed out after %ss\" % args.timeout)\n return 1\n if proc.returncode != 0:\n print(\"PROBLEM: node failed: %s\" % proc.stderr.strip()[:500])\n return 1\n try:\n res = json.loads(proc.stdout)\n except ValueError:\n print(\"PROBLEM: unparseable probe output: %r\" % proc.stdout[:200])\n return 1\n if res.get(\"load_error\"):\n print(\"PROBLEM: model failed to load in an isolated context: %s\" % res[\"load_error\"])\n return 1\n problems = 0\n for r, kind in zip(res[\"results\"], kinds):\n bad = []\n if r[\"error\"]:\n bad.append(\"threw: %s\" % r[\"error\"])\n elif not r[\"finite\"]:\n bad.append(\"not a finite number (%r)\" % (r[\"value\"],))\n if not r[\"repeat_same\"]:\n bad.append(\"repeated call differs\")\n if not r[\"order_same\"]:\n bad.append(\"result depends on call order/fresh load\")\n if bad:\n tag = \"WARN\" if kind == \"edge-advisory\" else \"PROBLEM\"\n if kind != \"edge-advisory\":\n problems += 1\n print(\"%s [%s] %r: %s\" % (tag, kind, r[\"input\"], \"; \".join(bad)))\n vals = [r[\"value\"] for r in res[\"results\"][:len(inputs)] if r[\"finite\"]]\n if vals:\n print(\"sample outputs: n=%d min=%.6g max=%.6g distinct=%d\" % (len(vals), min(vals), max(vals), len(set(vals))))\n if len(set(vals)) == 1 and len(vals) > 1:\n print(\"WARN: every sample input got the same score; the ranking would be all ties\")\n print(\"probe %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- selftest\ndef cmd_selftest(_args):\n fails = []\n\n def check(name, cond):\n if not cond:\n fails.append(name)\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n\n # hand-computed: dcg = 1 + 0.5 = 1.5 ; ideal = 1 + 1/log2(3)\n v = ndcg_at_k([0.9, 0.8, 0.7], [1, 0, 1], 3)\n check(\"ndcg hand value\", abs(v - 1.5 / (1 + 1 / math.log2(3))) < 1e-12)\n # tie rule: equal scores keep evaluation order -> rels seen [0,1]\n v = ndcg_at_k([5, 5, 5], [0, 1, 1], 2)\n check(\"ndcg tie keeps input order\", abs(v - (1 / math.log2(3)) / (1 + 1 / math.log2(3))) < 1e-12)\n check(\"ndcg perfect ranking is 1\", abs(ndcg_at_k([3, 2, 1], [2, 1, 0], 2) - 1) < 1e-12)\n check(\"ndcg undefined without positives\", ndcg_at_k([1, 2], [0, 0], 2) is None)\n tmp = tempfile.mkdtemp(prefix=\"campchk_\")\n try:\n def w(name, text):\n p = os.path.join(tmp, name)\n with open(p, \"w\") as fh:\n fh.write(text)\n return p\n prev = w(\"prev.csv\", \"id,y\\na,1\\nb,2\\n\")\n ns = argparse.Namespace\n import contextlib\n sink = io.StringIO()\n with contextlib.redirect_stdout(sink):\n ok = cmd_batch(ns(round_csv=w(\"r1.csv\", \"id\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n dup = cmd_batch(ns(round_csv=w(\"r2.csv\", \"id\\nc\\nc\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n old = cmd_batch(ns(round_csv=w(\"r3.csv\", \"id\\na\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n short = cmd_batch(ns(round_csv=w(\"r4.csv\", \"id\\nc\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n hdr = cmd_batch(ns(round_csv=w(\"r5.csv\", \"wrong\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n check(\"batch accepts clean round\", ok == 0)\n check(\"batch flags in-batch duplicate\", dup == 1)\n check(\"batch flags earlier order\", old == 1)\n check(\"batch flags wrong size\", short == 1)\n check(\"batch flags wrong header\", hdr == 1)\n if shutil.which(\"node\"):\n model = w(\"m.js\", \"function predict(x){ return x.length * 0.5; }\\n\")\n bad = w(\"bad.js\", \"function predict(x){ return x === '' ? NaN : Math.random(); }\\n\")\n with contextlib.redirect_stdout(sink):\n good = cmd_probe(ns(model=model, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n notgood = cmd_probe(ns(model=bad, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n check(\"probe accepts a finite deterministic model\", good == 0)\n check(\"probe rejects a random model\", notgood == 1)\n else:\n print(\"skip probe checks (node not found)\")\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\ndef main():\n ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n sub = ap.add_subparsers(dest=\"cmd\", required=True)\n\n b = sub.add_parser(\"batch\", help=\"check a round CSV before `asb run` (use '-' for stdin)\")\n b.add_argument(\"round_csv\")\n b.add_argument(\"--size\", type=int, help=\"exact number of rows required\")\n b.add_argument(\"--id-column\", help=\"expected header, e.g. the single column name from the task\")\n b.add_argument(\"--previous\", nargs=\"*\", help=\"earlier round or results CSVs whose ids may not repeat\")\n b.add_argument(\"--catalog\", help=\"CSV listing orderable ids\")\n b.add_argument(\"--catalog-column\", help=\"id column in the catalog (default: --id-column)\")\n b.set_defaults(fn=cmd_batch)\n\n n = sub.add_parser(\"ndcg\", help=\"local NDCG@k (use '-' for stdin)\")\n n.add_argument(\"csv\")\n n.add_argument(\"--pred\", required=True, help=\"prediction column\")\n n.add_argument(\"--rel\", required=True, help=\"relevance column (already clipped/normalised as the task defines)\")\n n.add_argument(\"--k\", type=int, required=True)\n n.add_argument(\"--group\", help=\"optional group column; adds per-group NDCG as a diagnostic\")\n n.add_argument(\"--threshold\", type=float, help=\"optional pass level to compare against\")\n n.set_defaults(fn=cmd_ndcg)\n\n p = sub.add_parser(\"probe\", help=\"run final_model.js under node on sample inputs\")\n p.add_argument(\"model\")\n p.add_argument(\"--input\", action=\"append\", help=\"sample input string (repeatable)\")\n p.add_argument(\"--inputs-file\", help=\"file with one sample input per line\")\n p.add_argument(\"--edge-file\", help=\"file with one extra input per line that must also score finitely\")\n p.add_argument(\"--timeout\", type=int, default=120)\n p.set_defaults(fn=cmd_probe)\n\n s = sub.add_parser(\"selftest\", help=\"hand-computed self checks\")\n s.set_defaults(fn=cmd_selftest)\n\n args = ap.parse_args()\n sys.exit(args.fn(args))\n\n\nif __name__ == \"__main__\":\n main()\n" }, "candidate_sha256": "7b8b66f849093bd17d27dd02c87284b3014e0ad2fa545d10a078f8ee9af8b0f9", "parent_sha256": "2262f848a73a4b9229dc986d24e8880141fa350f1ca2e1601a1b6362dfbc4394", "role_cost": { "calls": 7, "list_price_estimate_usd": 0.76 }, "trials": [ { "task": "propylene-active-learning", "trial": "propylene-active-learning__3UZDQrv", "job": "r01-screen", "reward": 1, "model_audit_ok": true, "harness_interventions": 0, "tokens": 948041, "list_price_estimate_usd": 0.5030092 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__NvCQYuj", "job": "r01-screen", "reward": 0, "model_audit_ok": true, "harness_interventions": 0, "tokens": 1290699, "list_price_estimate_usd": 0.6821144 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__79vBo6X", "job": "r01-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 0, "tokens": 640714, "list_price_estimate_usd": 0.32952719999999996 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__GuNbBG9", "job": "r01-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 0, "tokens": 1066282, "list_price_estimate_usd": 0.5444838000000001 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__Gqjc95Q", "job": "r01-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 0, "tokens": 926256, "list_price_estimate_usd": 0.5074054000000001 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__bZWyooS", "job": "r01-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 0, "tokens": 539402, "list_price_estimate_usd": 0.2946284 } ] }, { "round": 2, "title": "Stop hook audit (not evaluated)", "summary": "A one-shot stop hook asking the agent to validate its delivered model on a held-out stand-in before finishing.", "observation": "Lost to a framework gap: the hook script was rejected under hooks/ and again after moving to tools/. No trial was spent. The gap was fixed after the run.", "accepted": false, "reason": "gate_failed", "recorded_offline": false, "score": null, "trials_run": 0, "task_rates": null, "p_superior": null, "incumbent_score": null, "tokens_per_trial": null, "complexity": null, "edit_budget": 2, "components_changed": [ "hook", "instructions", "tool" ], "changed_files": [ "hooks.json", "instructions.md", "smoke.json", "tools/stop_audit.py" ], "plan": { "problem": "The parent bundle passed 1 of 6 trials (propylene 1/3, Suzuki 0/3). The analysis (FM-1) and the cards point to the ranking gate as the likely discriminator for propylene. All five failing trials delivered a model that was never scored on anything resembling the held-out metric. Agents stopped after 2-17 minutes of a 480-minute allowance. The parent's instruction text was only partly followed: the `ndcg` and `selftest` pointers were used in 0 of 6 trials, and baseline-first was followed in 0 of 6. The delivery output says \"the latest submission is scored\", so a model can be re-delivered after a late audit. Nothing in the parent uses that slack. The parent also gives no way to track discovery progress when per-group thresholds are hidden (FM-2, Suzuki). Which gate failed in any trial is an inference, because the verifier shows only a binary reward.", "edits": [ { "component": "hook", "hypothesis": "A one-shot stop hook that fires only after an accepted delivery, and only if time remains, will make the agent run an audit it would otherwise skip. The audit has three steps. (1) Build a held-out stand-in. Treat the first, broad round, or a group-wise split of the ordered items, as the validation set. Fit the model on the rest. Score it with `campaign_check.py ndcg` using the task's relevance definition. (2) Compare the delivered model with at least one structurally different alternative: a lower-dimensional or smoother model, and with versus without the seed data. (3) Re-deliver only if an alternative wins by a stated margin, running `--validate-only` and keeping a backup first. Falsifiable: the propylene pass rate rises above the parent's 1/3, or at least one trial re-delivers a model chosen by a stand-in score; mean wall time rises above the parent's roughly 120-1000 s. If trials ignore the hook message, or re-delivery changes nothing, the hypothesis is refuted.", "files": [ "hooks.json", "hooks/stop_audit.py", "smoke.json" ] }, { "component": "instructions", "hypothesis": "Replace the parent's loose 'seed data, model, check' notes with a short additive section that gives the audit protocol concretely and fixes three gaps. (a) The stand-in validation recipe: score on the broad first round, or on group-held-out folds, using the official relevance transform. A score over the agent's own picks is flagged as biased upward. (b) A hit-progress proxy. Where cutoffs are hidden, count an ordered item as a probable hit if it is in the top few percent of its own group's predicted or observed distribution, using the stated standout rate. Re-plan exploit versus explore from that count after each round. (c) Time handling: run long fits in the background with nohup and poll, because the per-command limit of about 2 minutes, not total time, bound the trials (exit 143/144 in NvCQYuj and Gqjc95Q). The section also shortens the existing baseline-first line to a single imperative. Falsifiable: `ndcg` appears in at least half of trajectories, timeout exits (143/144/124) fall, and Suzuki trials report a hit-count estimate before rounds 3 and 4. If those behaviours do not appear, the hypothesis is refuted.", "files": [ "instructions.md" ] } ], "predictions": { "tasks": { "propylene-active-learning": "improve", "suzuki-condition-screen": "same" }, "token_effect": "higher" }, "why_not_simpler": "Instruction text alone has already been tried. The parent already told agents to compute the metric locally, with a command line. They did not do it (0 of 6), and they also skipped baseline-first (0 of 6). More prose in the same file would face the same compliance gap. Deleting machinery would not help either. `batch` and `probe` were followed in 6 of 6 and caused no harm. The hook is the smallest lever that sits on the agent's critical path, because the agent has to pass the stop event. It needs no behavioural change before round 3, so it cannot break the passing round structure. A pre_tool hook blocking `asb run` until a baseline exists was considered and dropped. The evidence shows no harm from baseline-last, since all six delivered, so it would add friction with no measured benefit.", "regression_risk": "1) The hook could cause churn: replacing a good model with one that wins on a small noisy stand-in. Mitigation: the message requires a margin, a backup, and `--validate-only`, and says keep the current model otherwise. 2) Infinite stop loops. Mitigation: a marker file plus the `stop_hook_active` field, so it blocks at most once. 3) The hook misfires when the task is not a campaign. Mitigation: it checks the transcript for 'deliverable accepted' and for final_model.js, and exits 0 otherwise. 4) Higher token use, perhaps 20-50%, which costs against the token rule. 5) The audit cannot repair the discovery gate, because rounds are irreversible. So Suzuki, where discovery was probably failing, is unlikely to move. The (b) hit-progress guidance may help at the margin. 6) The hook's behaviour depends on the stop event carrying transcript_path. The script must degrade to a plain file check when that is missing." }, "attempts": [ { "attempt": 0, "gate_passed": false, "gate_errors": [ "File outside a known component: hooks/stop_audit.py" ], "critic": null }, { "attempt": 1, "gate_passed": false, "gate_errors": [ "Undeclared change: tools/stop_audit.py" ], "critic": null }, { "attempt": 2, "gate_passed": false, "gate_errors": [ "File outside a known component: hooks/stop_audit.py" ], "critic": null }, { "attempt": 3, "gate_passed": false, "gate_errors": [ "Undeclared change: tools/stop_audit.py" ], "critic": null } ], "diff": "--- parent/hooks.json\n+++ candidate/hooks.json\n@@ -0,0 +1,9 @@\n+{\n+ \"stop\": [\n+ {\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n+ \"timeout\": 30\n+ }\n+ ]\n+}\n--- parent/instructions.md\n+++ candidate/instructions.md\n@@ -7,8 +7,8 @@\n ## Before round 1\n - Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n script in full.\n-- Write a baseline model from the seed data and run the deliver command in its free validation-only mode.\n- Re-validate after every round, so a valid model always exists well before the deadline.\n+- Write a baseline model from the seed data and validate it with the deliver command's validation-only mode, now.\n+ Re-validate after every round.\n - Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n \n ## Each round\n@@ -18,6 +18,10 @@\n - Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n This is guidance, not fixed percentages.\n+- Hit progress when per-group thresholds are hidden: use the stated standout rate (e.g. \"exactly K of N per group\").\n+ After each round count an ordered item as a probable hit if it is in the top K/N fraction of its own group's\n+ observed values (and for candidates, of the group's predicted values). Report that count before planning the next\n+ round and shift toward exploitation when it is behind pace.\n - Measurements are noisy; do not read much into a single value that sits near the threshold.\n \n ## Using the seed data\n@@ -36,3 +40,17 @@\n `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n - `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n+\n+## Audit after delivery\n+- The latest submission is the one scored, so a delivered model can be replaced. Finishing early gains nothing:\n+ the time allowance is far larger than these runs use. The score of a model over your own picks is biased upward;\n+ it does not show held-out ranking quality.\n+- After the first accepted delivery, audit with a held-out stand-in: the broad first round (or the least\n+ model-selected data), or whole groups held out. Fit on the rest, then score with the task's own relevance\n+ transform using `campaign_check.py ndcg`. Compare the delivered model with at least one structurally different\n+ alternative (fewer features or smoother, log or clipped target, with versus without seed data).\n+- Replace the model only if an alternative wins by a stated margin, not by one or two items. Keep a backup copy,\n+ `probe`, run the validation-only deliver, then deliver again. Otherwise keep the delivered model.\n+- A stop hook will ask for this audit once after delivery if time remains. Follow its message. It fires at most once,\n+ so doing the audit yourself earlier is fine; it will still ask once.\n+- Commands are cut off after about two minutes. Run long fits with `nohup ... > log 2>&1 &` and poll the log.\n--- parent/smoke.json\n+++ candidate/smoke.json\n@@ -48,6 +48,30 @@\n \"timeout\": 30\n },\n {\n+ \"name\": \"stop_audit hook selftest (block once after delivery, never loop, no-op otherwise)\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\", \"--selftest\"],\n+ \"stdin\": \"\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 120\n+ },\n+ {\n+ \"name\": \"stop_audit hook lets an already-active stop through\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n+ \"stdin\": \"{\\\"hook_event_name\\\": \\\"Stop\\\", \\\"stop_hook_active\\\": true, \\\"session_id\\\": \\\"smoke\\\", \\\"transcript_path\\\": \\\"/nonexistent/t.jsonl\\\"}\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"stop_audit hook ignores malformed event input\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n+ \"stdin\": \"not json at all\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n--- parent/tools/stop_audit.py\n+++ candidate/tools/stop_audit.py\n@@ -0,0 +1,182 @@\n+#!/usr/bin/env python3\n+\"\"\"One-shot stop hook: after a campaign model was delivered and time remains, ask for a held-out audit.\n+\n+Reads the stop event JSON on stdin. Exit 0 lets the agent stop. Exit 2 blocks the stop once and feeds the\n+audit request (stderr) back to the agent. It never blocks twice per session, never blocks when the event says\n+a stop hook is already active, and never blocks outside a delivered-campaign context.\n+Stdlib only. Run with --selftest for a built-in check using temporary fixtures.\n+\"\"\"\n+import json\n+import os\n+import re\n+import sys\n+import tempfile\n+import time\n+from datetime import datetime, timezone\n+\n+MARKER_ENV = \"RSI_AUDIT_MARKER_DIR\"\n+BUDGET_ENV = \"RSI_AUDIT_BUDGET_S\"\n+DEFAULT_BUDGET_S = 28800\n+MIN_REMAINING_S = 1800\n+DELIVERED = \"deliverable accepted\"\n+MODEL_NAME = \"final_model.js\"\n+\n+MESSAGE = \"\"\"Before stopping: a model has been delivered and time remains. The latest submission is the one scored, so\n+re-delivery is allowed. Spend a short audit on the held-out ranking gate. Everything you measured so far was\n+chosen by your own model, so a score over your own picks is biased upward.\n+\n+1. Build a held-out stand-in. Use the broad first round (or the least model-selected measurements), or hold out whole\n+ groups of the ordered items (group the folds by the key that distinguishes held-out items). Fit on the rest, keeping\n+ the seed data in. Score with the official relevance transform exactly as the task states it:\n+ python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ]\n+2. Compare the delivered model with at least one structurally different alternative on that same stand-in: a\n+ lower-dimensional or smoother model (fewer features, stronger shrinkage, log or clipped target), and the same\n+ model with versus without the seed data. Run long fits in the background (nohup ... &) and poll the log, because\n+ a single command is cut off after about two minutes.\n+3. Re-deliver only if an alternative beats the delivered model by a clear margin (state it, e.g. at least 0.05 NDCG)\n+ and the gap is not one or two items. Then: cp final_model.js final_model.prev.js, write the new file,\n+ python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt,\n+ asb deliver --validate-only, and only then the real deliver.\n+ If nothing wins by the margin, keep the current model: do not re-deliver, and report the stand-in scores.\n+This audit happens once; state the numbers you compared in the final message.\"\"\"\n+\n+\n+def parse_ts(value):\n+ try:\n+ if isinstance(value, (int, float)):\n+ return float(value) / (1000.0 if value > 1e11 else 1.0)\n+ dt = datetime.fromisoformat(str(value).replace(\"Z\", \"+00:00\"))\n+ if dt.tzinfo is None:\n+ dt = dt.replace(tzinfo=timezone.utc)\n+ return dt.timestamp()\n+ except (ValueError, TypeError):\n+ return None\n+\n+\n+def read_transcript(path):\n+ try:\n+ with open(path, encoding=\"utf-8\", errors=\"replace\") as fh:\n+ return fh.read()\n+ except OSError:\n+ return None\n+\n+\n+def first_timestamp(text):\n+ for line in text.splitlines()[:50]:\n+ m = re.search(r'\"timestamp\"\\s*:\\s*(\"[^\"]+\"|[0-9.]+)', line)\n+ if m:\n+ ts = parse_ts(json.loads(m.group(1)))\n+ if ts:\n+ return ts\n+ return None\n+\n+\n+def model_exists(cwd):\n+ for d in (cwd, os.getcwd(), \"/app\"):\n+ if d and os.path.isfile(os.path.join(d, MODEL_NAME)):\n+ return True\n+ return False\n+\n+\n+def marker_path(session):\n+ base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_stop_audit\")\n+ safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session or \"nosession\")[:80]\n+ return base, os.path.join(base, safe + \".done\")\n+\n+\n+def decide(event):\n+ \"\"\"Return True if the stop should be blocked for the audit.\"\"\"\n+ if event.get(\"stop_hook_active\"):\n+ return False\n+ cwd = event.get(\"cwd\") or \"\"\n+ text = None\n+ tp = event.get(\"transcript_path\")\n+ if tp:\n+ text = read_transcript(tp)\n+ if text is not None:\n+ if DELIVERED not in text or MODEL_NAME not in text:\n+ return False\n+ start = first_timestamp(text)\n+ if start is not None:\n+ try:\n+ budget = float(os.environ.get(BUDGET_ENV, DEFAULT_BUDGET_S))\n+ except ValueError:\n+ budget = DEFAULT_BUDGET_S\n+ if time.time() - start > budget - MIN_REMAINING_S:\n+ return False\n+ elif not model_exists(cwd):\n+ return False\n+ return True\n+\n+\n+def main():\n+ if \"--selftest\" in sys.argv:\n+ return selftest()\n+ try:\n+ event = json.loads(sys.stdin.read() or \"{}\")\n+ if not isinstance(event, dict):\n+ return 0\n+ except ValueError:\n+ return 0\n+ try:\n+ if not decide(event):\n+ return 0\n+ base, marker = marker_path(str(event.get(\"session_id\") or \"\"))\n+ if os.path.exists(marker):\n+ return 0\n+ os.makedirs(base, exist_ok=True)\n+ with open(marker, \"w\") as fh:\n+ fh.write(str(time.time()))\n+ except OSError:\n+ return 0\n+ sys.stderr.write(MESSAGE + \"\\n\")\n+ return 2\n+\n+\n+def selftest():\n+ import subprocess\n+ import shutil\n+ tmp = tempfile.mkdtemp(prefix=\"stopaudit_\")\n+ fails = []\n+ try:\n+ env = dict(os.environ, **{MARKER_ENV: os.path.join(tmp, \"m\")})\n+\n+ def run(event):\n+ p = subprocess.run([sys.executable, os.path.abspath(__file__)], input=json.dumps(event),\n+ capture_output=True, text=True, env=env, timeout=30)\n+ return p.returncode\n+\n+ def check(name, cond):\n+ print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n+ if not cond:\n+ fails.append(name)\n+\n+ now = datetime.now(timezone.utc).isoformat()\n+ good = os.path.join(tmp, \"good.jsonl\")\n+ with open(good, \"w\") as fh:\n+ fh.write(json.dumps({\"timestamp\": now, \"text\": \"asb deliver final_model.js\"}) + \"\\n\")\n+ fh.write(json.dumps({\"text\": \"local validation passed\\n\" + DELIVERED + \" (submission 1)\"}) + \"\\n\")\n+ undelivered = os.path.join(tmp, \"undelivered.jsonl\")\n+ with open(undelivered, \"w\") as fh:\n+ fh.write(json.dumps({\"timestamp\": now, \"text\": \"writing final_model.js\"}) + \"\\n\")\n+ late = os.path.join(tmp, \"late.jsonl\")\n+ with open(late, \"w\") as fh:\n+ fh.write(json.dumps({\"timestamp\": \"2020-01-01T00:00:00Z\", \"text\": \"final_model.js \" + DELIVERED}) + \"\\n\")\n+ check(\"active stop hook never blocks\", run({\"stop_hook_active\": True, \"transcript_path\": good, \"session_id\": \"a\"}) == 0)\n+ check(\"no delivery in transcript does not block\", run({\"transcript_path\": undelivered, \"session_id\": \"b\"}) == 0)\n+ check(\"out of time does not block\", run({\"transcript_path\": late, \"session_id\": \"c\"}) == 0)\n+ check(\"garbage input does not block\", subprocess.run([sys.executable, os.path.abspath(__file__)], input=\"not json\",\n+ capture_output=True, text=True, env=env).returncode == 0)\n+ check(\"delivered with time left blocks once\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 2)\n+ check(\"second stop in same session passes\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 0)\n+ os.makedirs(os.path.join(tmp, \"w\"))\n+ open(os.path.join(tmp, \"w\", MODEL_NAME), \"w\").close()\n+ check(\"no transcript but model file present blocks\", run({\"cwd\": os.path.join(tmp, \"w\"), \"session_id\": \"f\"}) == 2)\n+ finally:\n+ shutil.rmtree(tmp, ignore_errors=True)\n+ print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n+ return 1 if fails else 0\n+\n+\n+if __name__ == \"__main__\":\n+ sys.exit(main())\n", "files": { "hooks.json": "{\n \"stop\": [\n {\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"timeout\": 30\n }\n ]\n}\n", "instructions.md": "# Campaign workflow notes\n\nThese apply to tasks of the form: seed data, a few irreversible batched `asb run` rounds, then a self-contained\ndeterministic `final_model.js` that must pass both a discovery gate (hits among your ordered items) and a\nheld-out ranking (NDCG) gate. Two gates means neither can be left to the end.\n\n## Before round 1\n- Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n script in full.\n- Write a baseline model from the seed data and validate it with the deliver command's validation-only mode, now.\n Re-validate after every round.\n- Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n\n## Each round\n- Accepted rounds are final; rejected requests are free. Before `asb run`, check the file:\n `python3 /opt/rsi-harness/tools/campaign_check.py batch round_N.csv --size --id-column --previous round_*_results.csv [--catalog ]`\n Fix anything it reports (header, size, duplicates, items already ordered, ids not in the catalog).\n- Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n This is guidance, not fixed percentages.\n- Hit progress when per-group thresholds are hidden: use the stated standout rate (e.g. \"exactly K of N per group\").\n After each round count an ordered item as a probable hit if it is in the top K/N fraction of its own group's\n observed values (and for candidates, of the group's predicted values). Report that count before planning the next\n round and shift toward exploitation when it is behind pace.\n- Measurements are noisy; do not read much into a single value that sits near the threshold.\n\n## Using the seed data\n- Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n score favour easy groups over hard ones.\n\n## Choosing and checking the model\n- Pick the final model by cross-validation that mimics the held-out distribution (group the folds by the grouping variable\n that distinguishes held-out items, e.g. a categorical key), with the official metric computed locally:\n `python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ] [--threshold ]`\n Pass relevance already defined as the task states (clipped/normalised). The tool breaks score ties by input row order;\n prefer models that rarely produce ties.\n- The model must be self-contained, deterministic, finite for every input and independent per input; handle levels\n unseen in training with a fallback. Before delivering, run\n `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n- `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n\n## Audit after delivery\n- The latest submission is the one scored, so a delivered model can be replaced. Finishing early gains nothing:\n the time allowance is far larger than these runs use. The score of a model over your own picks is biased upward;\n it does not show held-out ranking quality.\n- After the first accepted delivery, audit with a held-out stand-in: the broad first round (or the least\n model-selected data), or whole groups held out. Fit on the rest, then score with the task's own relevance\n transform using `campaign_check.py ndcg`. Compare the delivered model with at least one structurally different\n alternative (fewer features or smoother, log or clipped target, with versus without seed data).\n- Replace the model only if an alternative wins by a stated margin, not by one or two items. Keep a backup copy,\n `probe`, run the validation-only deliver, then deliver again. Otherwise keep the delivered model.\n- A stop hook will ask for this audit once after delivery if time remains. Follow its message. It fires at most once,\n so doing the audit yourself earlier is fine; it will still ask once.\n- Commands are cut off after about two minutes. Run long fits with `nohup ... > log 2>&1 &` and poll the log.\n", "smoke.json": "[\n {\n \"name\": \"campaign_check selftest (hand-computed NDCG, tie rule, batch and probe checks)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"campaign_check ndcg on a tiny synthetic CSV from stdin\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"p\", \"--rel\", \"r\", \"--k\", \"3\", \"--group\", \"g\"],\n \"stdin\": \"p,r,g\\n0.9,1,x\\n0.8,0,x\\n0.7,1,x\\n0.5,0,y\\n0.4,2,y\\n0.3,1,y\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check ndcg rejects an unknown column\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"nope\", \"--rel\", \"r\", \"--k\", \"2\"],\n \"stdin\": \"p,r\\n1,1\\n2,0\\n\",\n \"expect_exit\": 2,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch accepts a clean round\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw2\\nw3\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags duplicates and wrong size\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw1\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags a wrong header\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"2\", \"--id-column\", \"item_key\"],\n \"stdin\": \"id\\nA\\nB\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook selftest (block once after delivery, never loop, no-op otherwise)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\", \"--selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"stop_audit hook lets an already-active stop through\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"{\\\"hook_event_name\\\": \\\"Stop\\\", \\\"stop_hook_active\\\": true, \\\"session_id\\\": \\\"smoke\\\", \\\"transcript_path\\\": \\\"/nonexistent/t.jsonl\\\"}\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook ignores malformed event input\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"not json at all\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n \"stdin\": \"\",\n \"expect_exit\": 2,\n \"timeout\": 30\n }\n]\n", "tools/campaign_check.py": "#!/usr/bin/env python3\n\"\"\"Advisory checks for batched-oracle campaigns (stdlib only, never blocks anything).\n\nSubcommands\n batch check a round CSV before ordering it (header, size, duplicates, earlier orders, catalog)\n ndcg local NDCG@k with 1/log2(rank+1) discount, stable ties, ideal-ranking normalisation\n probe run final_model.js under node on sample inputs: finite, deterministic, order-independent\n selftest built-in hand-computed checks of the above\n\nExit codes: 0 ok, 1 problem found, 2 usage or I/O error.\n\"\"\"\nimport argparse\nimport csv\nimport io\nimport json\nimport math\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\n\n\ndef die(msg, code=2):\n sys.stderr.write(\"error: %s\\n\" % msg)\n sys.exit(code)\n\n\ndef read_text(path):\n if path == \"-\":\n return sys.stdin.read()\n try:\n with open(path, encoding=\"utf-8-sig\", newline=\"\") as fh:\n return fh.read()\n except OSError as exc:\n die(\"cannot read %s: %s\" % (path, exc))\n\n\ndef read_rows(path):\n text = read_text(path)\n if text.startswith(\"\"):\n text = text[1:]\n rows = list(csv.reader(io.StringIO(text)))\n rows = [r for r in rows if r and any(c.strip() for c in r)]\n if not rows:\n die(\"%s is empty\" % path)\n return rows[0], rows[1:]\n\n\ndef id_values(path, column):\n header, body = read_rows(path)\n idx = header.index(column) if column in header else 0\n return [r[idx].strip() for r in body if len(r) > idx]\n\n\n# ---------------------------------------------------------------- batch\ndef cmd_batch(args):\n header, body = read_rows(args.round_csv)\n problems, notes = [], []\n column = args.id_column or header[0]\n if len(header) != 1:\n problems.append(\"header has %d columns %r; the round file needs a single column\" % (len(header), header))\n if header[0] != column:\n problems.append(\"header is %r, expected %r\" % (header[0], column))\n ids = [r[0] for r in body]\n padded = [i for i in ids if i != i.strip()]\n if padded:\n problems.append(\"%d ids have leading/trailing whitespace, e.g. %r\" % (len(padded), padded[0]))\n ids = [i.strip() for i in ids]\n if any(len(r) != 1 for r in body):\n problems.append(\"some rows have more than one field\")\n if any(not i for i in ids):\n problems.append(\"blank ids present\")\n notes.append(\"header=%r rows=%d\" % (header, len(ids)))\n if args.size is not None and len(ids) != args.size:\n problems.append(\"size is %d, expected exactly %d\" % (len(ids), args.size))\n seen, dups = set(), []\n for i in ids:\n if i in seen and i not in dups:\n dups.append(i)\n seen.add(i)\n if dups:\n problems.append(\"%d duplicated ids inside the batch, e.g. %s\" % (len(dups), dups[:3]))\n prior = set()\n for p in args.previous or []:\n prior.update(id_values(p, column))\n overlap = sorted(seen & prior)\n if args.previous:\n notes.append(\"compared with %d earlier ids from %d files\" % (len(prior), len(args.previous)))\n if overlap:\n problems.append(\"%d ids were already ordered, e.g. %s\" % (len(overlap), overlap[:3]))\n if args.catalog:\n cat = set(id_values(args.catalog, args.catalog_column or column))\n notes.append(\"catalog has %d ids\" % len(cat))\n unknown = sorted(seen - cat)\n if unknown:\n problems.append(\"%d ids are not in the catalog, e.g. %s\" % (len(unknown), unknown[:3]))\n for n in notes:\n print(\"note: \" + n)\n for p in problems:\n print(\"PROBLEM: \" + p)\n print(\"batch %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- ndcg\ndef dcg(rels):\n return sum(r / math.log2(i + 2) for i, r in enumerate(rels))\n\n\ndef ndcg_at_k(preds, rels, k):\n \"\"\"Rank by prediction descending; equal predictions keep input order (stable sort).\"\"\"\n order = sorted(range(len(preds)), key=lambda i: -preds[i])\n got = dcg([rels[i] for i in order[:k]])\n ideal = dcg(sorted(rels, reverse=True)[:k])\n if ideal <= 0:\n return None\n return got / ideal\n\n\ndef cmd_ndcg(args):\n header, body = read_rows(args.csv)\n for c in [args.pred, args.rel] + ([args.group] if args.group else []):\n if c not in header:\n die(\"column %r not in header %r\" % (c, header))\n ip, ir = header.index(args.pred), header.index(args.rel)\n ig = header.index(args.group) if args.group else None\n preds, rels, groups = [], [], []\n for n, r in enumerate(body, 2):\n try:\n p, q = float(r[ip]), float(r[ir])\n except (ValueError, IndexError):\n die(\"row %d: non-numeric prediction or relevance\" % n)\n if math.isnan(p) or math.isnan(q) or math.isinf(p):\n die(\"row %d: NaN/inf value\" % n)\n preds.append(p)\n rels.append(q)\n groups.append(r[ig] if ig is not None else None)\n if any(q < 0 for q in rels):\n print(\"warning: negative relevance present; pass relevance already clipped at 0 if the metric does so\")\n k = args.k\n if k > len(preds):\n print(\"warning: k=%d exceeds %d rows\" % (k, len(preds)))\n val = ndcg_at_k(preds, rels, k)\n if val is None:\n print(\"warning: ideal DCG is 0 (no positive relevance); NDCG undefined, reported as 0\")\n val = 0.0\n print(\"ndcg@%d = %.6f (n=%d)\" % (k, val, len(preds)))\n if args.group:\n by = {}\n for i, g in enumerate(groups):\n by.setdefault(g, []).append(i)\n vals = []\n for g in sorted(by):\n v = ndcg_at_k([preds[i] for i in by[g]], [rels[i] for i in by[g]], k)\n vals.append(v)\n print(\" group %s: n=%d ndcg@%d=%s\" % (g, len(by[g]), k, \"undefined\" if v is None else \"%.4f\" % v))\n ok = [v for v in vals if v is not None]\n if ok:\n print(\"mean over groups (informational, not the pooled metric) = %.6f\" % (sum(ok) / len(ok)))\n if args.threshold is not None:\n print(\"threshold %.4f: %s\" % (args.threshold, \"PASS\" if val >= args.threshold else \"BELOW\"))\n return 0\n\n\n# ---------------------------------------------------------------- probe\nPROBE_JS = r\"\"\"\nconst fs = require('fs'), vm = require('vm');\nconst req = JSON.parse(fs.readFileSync(0, 'utf8'));\nfunction load() {\n const code = fs.readFileSync(req.model, 'utf8');\n const mod = { exports: {} };\n const sb = { module: mod, exports: mod.exports, console: { log() {}, error() {}, warn() {} } };\n vm.createContext(sb);\n vm.runInContext(code, sb, { timeout: 60000 });\n const f = (typeof sb.predict === 'function') ? sb.predict\n : (mod.exports && typeof mod.exports.predict === 'function') ? mod.exports.predict : null;\n if (!f) throw new Error('no predict function defined');\n return f;\n}\nfunction run(f, x) {\n try { return { v: f(x) }; } catch (e) { return { e: String(e && e.message || e) }; }\n}\nconst out = { load_error: null, results: [] };\ntry {\n const f1 = load(), f2 = load();\n const xs = req.inputs;\n const a = xs.map(x => run(f1, x));\n const b = xs.map(x => run(f1, x));\n const c = xs.map((x, i) => run(f2, xs[xs.length - 1 - i])).reverse();\n xs.forEach((x, i) => {\n const same = (p, q) => ('e' in p && 'e' in q) || ('v' in p && 'v' in q && Object.is(p.v, q.v));\n out.results.push({\n input: x, value: ('v' in a[i]) ? a[i].v : null, error: a[i].e || null,\n type: ('v' in a[i]) ? typeof a[i].v : null,\n finite: ('v' in a[i]) && typeof a[i].v === 'number' && Number.isFinite(a[i].v),\n repeat_same: same(a[i], b[i]), order_same: same(a[i], c[i])\n });\n });\n} catch (e) { out.load_error = String(e && e.message || e); }\nprocess.stdout.write(JSON.stringify(out));\n\"\"\"\n\nBUILTIN_EDGE = [\"\", \"?\"]\n\n\ndef cmd_probe(args):\n if not os.path.isfile(args.model):\n die(\"model file not found: %s\" % args.model)\n node = shutil.which(\"node\")\n if not node:\n die(\"node not found on PATH\")\n inputs = list(args.input or [])\n if args.inputs_file:\n inputs += [l.rstrip(\"\\r\\n\") for l in read_text(args.inputs_file).splitlines() if l.strip()]\n edge_must = []\n if args.edge_file:\n edge_must = [l.rstrip(\"\\r\\n\") for l in read_text(args.edge_file).splitlines() if l.strip()]\n if not inputs and not edge_must:\n die(\"give --input and/or --inputs-file (real-format sample inputs)\")\n all_inputs = inputs + edge_must + BUILTIN_EDGE\n kinds = [\"sample\"] * len(inputs) + [\"edge-required\"] * len(edge_must) + [\"edge-advisory\"] * len(BUILTIN_EDGE)\n payload = json.dumps({\"model\": os.path.abspath(args.model), \"inputs\": all_inputs})\n try:\n proc = subprocess.run([node, \"-e\", PROBE_JS], input=payload, capture_output=True,\n text=True, timeout=args.timeout)\n except subprocess.TimeoutExpired:\n print(\"PROBLEM: node timed out after %ss\" % args.timeout)\n return 1\n if proc.returncode != 0:\n print(\"PROBLEM: node failed: %s\" % proc.stderr.strip()[:500])\n return 1\n try:\n res = json.loads(proc.stdout)\n except ValueError:\n print(\"PROBLEM: unparseable probe output: %r\" % proc.stdout[:200])\n return 1\n if res.get(\"load_error\"):\n print(\"PROBLEM: model failed to load in an isolated context: %s\" % res[\"load_error\"])\n return 1\n problems = 0\n for r, kind in zip(res[\"results\"], kinds):\n bad = []\n if r[\"error\"]:\n bad.append(\"threw: %s\" % r[\"error\"])\n elif not r[\"finite\"]:\n bad.append(\"not a finite number (%r)\" % (r[\"value\"],))\n if not r[\"repeat_same\"]:\n bad.append(\"repeated call differs\")\n if not r[\"order_same\"]:\n bad.append(\"result depends on call order/fresh load\")\n if bad:\n tag = \"WARN\" if kind == \"edge-advisory\" else \"PROBLEM\"\n if kind != \"edge-advisory\":\n problems += 1\n print(\"%s [%s] %r: %s\" % (tag, kind, r[\"input\"], \"; \".join(bad)))\n vals = [r[\"value\"] for r in res[\"results\"][:len(inputs)] if r[\"finite\"]]\n if vals:\n print(\"sample outputs: n=%d min=%.6g max=%.6g distinct=%d\" % (len(vals), min(vals), max(vals), len(set(vals))))\n if len(set(vals)) == 1 and len(vals) > 1:\n print(\"WARN: every sample input got the same score; the ranking would be all ties\")\n print(\"probe %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- selftest\ndef cmd_selftest(_args):\n fails = []\n\n def check(name, cond):\n if not cond:\n fails.append(name)\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n\n # hand-computed: dcg = 1 + 0.5 = 1.5 ; ideal = 1 + 1/log2(3)\n v = ndcg_at_k([0.9, 0.8, 0.7], [1, 0, 1], 3)\n check(\"ndcg hand value\", abs(v - 1.5 / (1 + 1 / math.log2(3))) < 1e-12)\n # tie rule: equal scores keep evaluation order -> rels seen [0,1]\n v = ndcg_at_k([5, 5, 5], [0, 1, 1], 2)\n check(\"ndcg tie keeps input order\", abs(v - (1 / math.log2(3)) / (1 + 1 / math.log2(3))) < 1e-12)\n check(\"ndcg perfect ranking is 1\", abs(ndcg_at_k([3, 2, 1], [2, 1, 0], 2) - 1) < 1e-12)\n check(\"ndcg undefined without positives\", ndcg_at_k([1, 2], [0, 0], 2) is None)\n tmp = tempfile.mkdtemp(prefix=\"campchk_\")\n try:\n def w(name, text):\n p = os.path.join(tmp, name)\n with open(p, \"w\") as fh:\n fh.write(text)\n return p\n prev = w(\"prev.csv\", \"id,y\\na,1\\nb,2\\n\")\n ns = argparse.Namespace\n import contextlib\n sink = io.StringIO()\n with contextlib.redirect_stdout(sink):\n ok = cmd_batch(ns(round_csv=w(\"r1.csv\", \"id\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n dup = cmd_batch(ns(round_csv=w(\"r2.csv\", \"id\\nc\\nc\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n old = cmd_batch(ns(round_csv=w(\"r3.csv\", \"id\\na\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n short = cmd_batch(ns(round_csv=w(\"r4.csv\", \"id\\nc\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n hdr = cmd_batch(ns(round_csv=w(\"r5.csv\", \"wrong\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n check(\"batch accepts clean round\", ok == 0)\n check(\"batch flags in-batch duplicate\", dup == 1)\n check(\"batch flags earlier order\", old == 1)\n check(\"batch flags wrong size\", short == 1)\n check(\"batch flags wrong header\", hdr == 1)\n if shutil.which(\"node\"):\n model = w(\"m.js\", \"function predict(x){ return x.length * 0.5; }\\n\")\n bad = w(\"bad.js\", \"function predict(x){ return x === '' ? NaN : Math.random(); }\\n\")\n with contextlib.redirect_stdout(sink):\n good = cmd_probe(ns(model=model, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n notgood = cmd_probe(ns(model=bad, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n check(\"probe accepts a finite deterministic model\", good == 0)\n check(\"probe rejects a random model\", notgood == 1)\n else:\n print(\"skip probe checks (node not found)\")\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\ndef main():\n ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n sub = ap.add_subparsers(dest=\"cmd\", required=True)\n\n b = sub.add_parser(\"batch\", help=\"check a round CSV before `asb run` (use '-' for stdin)\")\n b.add_argument(\"round_csv\")\n b.add_argument(\"--size\", type=int, help=\"exact number of rows required\")\n b.add_argument(\"--id-column\", help=\"expected header, e.g. the single column name from the task\")\n b.add_argument(\"--previous\", nargs=\"*\", help=\"earlier round or results CSVs whose ids may not repeat\")\n b.add_argument(\"--catalog\", help=\"CSV listing orderable ids\")\n b.add_argument(\"--catalog-column\", help=\"id column in the catalog (default: --id-column)\")\n b.set_defaults(fn=cmd_batch)\n\n n = sub.add_parser(\"ndcg\", help=\"local NDCG@k (use '-' for stdin)\")\n n.add_argument(\"csv\")\n n.add_argument(\"--pred\", required=True, help=\"prediction column\")\n n.add_argument(\"--rel\", required=True, help=\"relevance column (already clipped/normalised as the task defines)\")\n n.add_argument(\"--k\", type=int, required=True)\n n.add_argument(\"--group\", help=\"optional group column; adds per-group NDCG as a diagnostic\")\n n.add_argument(\"--threshold\", type=float, help=\"optional pass level to compare against\")\n n.set_defaults(fn=cmd_ndcg)\n\n p = sub.add_parser(\"probe\", help=\"run final_model.js under node on sample inputs\")\n p.add_argument(\"model\")\n p.add_argument(\"--input\", action=\"append\", help=\"sample input string (repeatable)\")\n p.add_argument(\"--inputs-file\", help=\"file with one sample input per line\")\n p.add_argument(\"--edge-file\", help=\"file with one extra input per line that must also score finitely\")\n p.add_argument(\"--timeout\", type=int, default=120)\n p.set_defaults(fn=cmd_probe)\n\n s = sub.add_parser(\"selftest\", help=\"hand-computed self checks\")\n s.set_defaults(fn=cmd_selftest)\n\n args = ap.parse_args()\n sys.exit(args.fn(args))\n\n\nif __name__ == \"__main__\":\n main()\n", "tools/stop_audit.py": "#!/usr/bin/env python3\n\"\"\"One-shot stop hook: after a campaign model was delivered and time remains, ask for a held-out audit.\n\nReads the stop event JSON on stdin. Exit 0 lets the agent stop. Exit 2 blocks the stop once and feeds the\naudit request (stderr) back to the agent. It never blocks twice per session, never blocks when the event says\na stop hook is already active, and never blocks outside a delivered-campaign context.\nStdlib only. Run with --selftest for a built-in check using temporary fixtures.\n\"\"\"\nimport json\nimport os\nimport re\nimport sys\nimport tempfile\nimport time\nfrom datetime import datetime, timezone\n\nMARKER_ENV = \"RSI_AUDIT_MARKER_DIR\"\nBUDGET_ENV = \"RSI_AUDIT_BUDGET_S\"\nDEFAULT_BUDGET_S = 28800\nMIN_REMAINING_S = 1800\nDELIVERED = \"deliverable accepted\"\nMODEL_NAME = \"final_model.js\"\n\nMESSAGE = \"\"\"Before stopping: a model has been delivered and time remains. The latest submission is the one scored, so\nre-delivery is allowed. Spend a short audit on the held-out ranking gate. Everything you measured so far was\nchosen by your own model, so a score over your own picks is biased upward.\n\n1. Build a held-out stand-in. Use the broad first round (or the least model-selected measurements), or hold out whole\n groups of the ordered items (group the folds by the key that distinguishes held-out items). Fit on the rest, keeping\n the seed data in. Score with the official relevance transform exactly as the task states it:\n python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ]\n2. Compare the delivered model with at least one structurally different alternative on that same stand-in: a\n lower-dimensional or smoother model (fewer features, stronger shrinkage, log or clipped target), and the same\n model with versus without the seed data. Run long fits in the background (nohup ... &) and poll the log, because\n a single command is cut off after about two minutes.\n3. Re-deliver only if an alternative beats the delivered model by a clear margin (state it, e.g. at least 0.05 NDCG)\n and the gap is not one or two items. Then: cp final_model.js final_model.prev.js, write the new file,\n python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt,\n asb deliver --validate-only, and only then the real deliver.\n If nothing wins by the margin, keep the current model: do not re-deliver, and report the stand-in scores.\nThis audit happens once; state the numbers you compared in the final message.\"\"\"\n\n\ndef parse_ts(value):\n try:\n if isinstance(value, (int, float)):\n return float(value) / (1000.0 if value > 1e11 else 1.0)\n dt = datetime.fromisoformat(str(value).replace(\"Z\", \"+00:00\"))\n if dt.tzinfo is None:\n dt = dt.replace(tzinfo=timezone.utc)\n return dt.timestamp()\n except (ValueError, TypeError):\n return None\n\n\ndef read_transcript(path):\n try:\n with open(path, encoding=\"utf-8\", errors=\"replace\") as fh:\n return fh.read()\n except OSError:\n return None\n\n\ndef first_timestamp(text):\n for line in text.splitlines()[:50]:\n m = re.search(r'\"timestamp\"\\s*:\\s*(\"[^\"]+\"|[0-9.]+)', line)\n if m:\n ts = parse_ts(json.loads(m.group(1)))\n if ts:\n return ts\n return None\n\n\ndef model_exists(cwd):\n for d in (cwd, os.getcwd(), \"/app\"):\n if d and os.path.isfile(os.path.join(d, MODEL_NAME)):\n return True\n return False\n\n\ndef marker_path(session):\n base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_stop_audit\")\n safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session or \"nosession\")[:80]\n return base, os.path.join(base, safe + \".done\")\n\n\ndef decide(event):\n \"\"\"Return True if the stop should be blocked for the audit.\"\"\"\n if event.get(\"stop_hook_active\"):\n return False\n cwd = event.get(\"cwd\") or \"\"\n text = None\n tp = event.get(\"transcript_path\")\n if tp:\n text = read_transcript(tp)\n if text is not None:\n if DELIVERED not in text or MODEL_NAME not in text:\n return False\n start = first_timestamp(text)\n if start is not None:\n try:\n budget = float(os.environ.get(BUDGET_ENV, DEFAULT_BUDGET_S))\n except ValueError:\n budget = DEFAULT_BUDGET_S\n if time.time() - start > budget - MIN_REMAINING_S:\n return False\n elif not model_exists(cwd):\n return False\n return True\n\n\ndef main():\n if \"--selftest\" in sys.argv:\n return selftest()\n try:\n event = json.loads(sys.stdin.read() or \"{}\")\n if not isinstance(event, dict):\n return 0\n except ValueError:\n return 0\n try:\n if not decide(event):\n return 0\n base, marker = marker_path(str(event.get(\"session_id\") or \"\"))\n if os.path.exists(marker):\n return 0\n os.makedirs(base, exist_ok=True)\n with open(marker, \"w\") as fh:\n fh.write(str(time.time()))\n except OSError:\n return 0\n sys.stderr.write(MESSAGE + \"\\n\")\n return 2\n\n\ndef selftest():\n import subprocess\n import shutil\n tmp = tempfile.mkdtemp(prefix=\"stopaudit_\")\n fails = []\n try:\n env = dict(os.environ, **{MARKER_ENV: os.path.join(tmp, \"m\")})\n\n def run(event):\n p = subprocess.run([sys.executable, os.path.abspath(__file__)], input=json.dumps(event),\n capture_output=True, text=True, env=env, timeout=30)\n return p.returncode\n\n def check(name, cond):\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n if not cond:\n fails.append(name)\n\n now = datetime.now(timezone.utc).isoformat()\n good = os.path.join(tmp, \"good.jsonl\")\n with open(good, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"asb deliver final_model.js\"}) + \"\\n\")\n fh.write(json.dumps({\"text\": \"local validation passed\\n\" + DELIVERED + \" (submission 1)\"}) + \"\\n\")\n undelivered = os.path.join(tmp, \"undelivered.jsonl\")\n with open(undelivered, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"writing final_model.js\"}) + \"\\n\")\n late = os.path.join(tmp, \"late.jsonl\")\n with open(late, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": \"2020-01-01T00:00:00Z\", \"text\": \"final_model.js \" + DELIVERED}) + \"\\n\")\n check(\"active stop hook never blocks\", run({\"stop_hook_active\": True, \"transcript_path\": good, \"session_id\": \"a\"}) == 0)\n check(\"no delivery in transcript does not block\", run({\"transcript_path\": undelivered, \"session_id\": \"b\"}) == 0)\n check(\"out of time does not block\", run({\"transcript_path\": late, \"session_id\": \"c\"}) == 0)\n check(\"garbage input does not block\", subprocess.run([sys.executable, os.path.abspath(__file__)], input=\"not json\",\n capture_output=True, text=True, env=env).returncode == 0)\n check(\"delivered with time left blocks once\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 2)\n check(\"second stop in same session passes\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 0)\n os.makedirs(os.path.join(tmp, \"w\"))\n open(os.path.join(tmp, \"w\", MODEL_NAME), \"w\").close()\n check(\"no transcript but model file present blocks\", run({\"cwd\": os.path.join(tmp, \"w\"), \"session_id\": \"f\"}) == 2)\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, "candidate_sha256": "7bdb02aca047294e498a7635fb36849566daadc80ab826da1928192bc0bb1fc7", "parent_sha256": "7b8b66f849093bd17d27dd02c87284b3014e0ad2fa545d10a078f8ee9af8b0f9", "role_cost": { "calls": 6, "list_price_estimate_usd": 2.0 }, "trials": [] }, { "round": 3, "title": "One-shot stop hook audit", "summary": "The same idea, declared correctly: when a model has been delivered and time remains, the stop hook blocks once and asks for a held-out check against one structurally different alternative.", "observation": "Accepted as a credible gain over round 1 (3/6 vs 1/6, posterior 0.87). The hook visibly intervened in every trial. Whether the audit caused any pass was not established.", "accepted": true, "reason": "credible_gain", "recorded_offline": false, "score": 0.5, "trials_run": 6, "task_rates": { "propylene-active-learning": 0.3333333333333333, "suzuki-condition-screen": 0.6666666666666666 }, "p_superior": 0.8671328671328646, "incumbent_score": 0.16666666666666666, "tokens_per_trial": 925800.0, "complexity": 611, "edit_budget": 1, "components_changed": [ "hook", "tool" ], "changed_files": [ "hooks.json", "smoke.json", "tools/stop_audit.py" ], "plan": { "problem": "The parent (round 1) passed 1 of 6 trials: propylene 1/3, Suzuki 0/3. All six agents delivered a valid final_model.js and then stopped after 120-1018 s of a 28800 s allowance. In all five failures the delivered model was never scored against anything like the held-out distribution. The parent's advisory text was largely ignored: `campaign_check.py ndcg` and `selftest` were used in 0 of 6 trials, and baseline-first was followed in 0 of 6. The one lever that sits on the agent's critical path, because the agent must pass the stop event, was never tried in a measured round. Round 2 proposed it (stop hook plus instructions plus tool) but was rejected before any trial by a gate error: `Undeclared change: tools/stop_audit.py`. The plan declared `hooks/stop_audit.py` but the bundle shipped `tools/stop_audit.py`. That error was a process failure, so the hook hypothesis is unresolved, not refuted. Which gate failed in each trial is inferred, because reward is binary.", "edits": [ { "component": "hook", "hypothesis": "A one-shot stop hook (python3 /opt/rsi-harness/tools/stop_audit.py) that exits 2 once, only when the transcript shows 'deliverable accepted' plus final_model.js and more than 30 minutes remain, will make the agent build a held-out stand-in and compare the delivered model with one structurally different alternative before it stops. The stand-in is the broad first round, or group-held-out folds. It is scored with `campaign_check.py ndcg` using the task's relevance transform. The agent re-delivers (after a backup, probe and `--validate-only`) only if the alternative wins by a stated margin, and otherwise keeps the model and reports the numbers. The message is domain-neutral and says a score over the agent's own picks is biased upward. Falsifiable: `ndcg` appears in at least 4 of 6 post-change trajectories, and the propylene pass rate rises above 1/3. If trials ignore the message, if re-delivery never happens or never changes the model, or if a passing-style model is replaced by a worse one, the hypothesis is refuted. The files must match the plan's declared paths exactly: the script ships under tools/ and the plan declares tools/stop_audit.py.", "files": [ "hooks.json", "tools/stop_audit.py", "smoke.json" ] } ], "predictions": { "tasks": { "propylene-active-learning": "improve", "suzuki-condition-screen": "same" }, "token_effect": "higher" }, "why_not_simpler": "Instruction text is the smaller edit, but it has been tried and the compliance evidence is bad. The parent already gave the exact `ndcg` command line and baseline-first, and both were ignored in 6 of 6 trials (analysis FM-1, FM-4). More prose in the same file would face the same gap. Removing machinery would not help either: `batch` and `probe` were used 6/6 with no harm. A pre_tool block on `asb run` was considered and rejected, because late baselines caused no observed harm and it would add friction before the irreversible rounds. A skill would be advisory and need the agent to invoke it, which the 0/6 `ndcg` usage suggests will not happen. The hook is the only option here that changes behaviour without requiring the agent to remember anything. It touches only the end of the run, after delivery, so it cannot damage the round structure. It is also within the budget of 1: the instructions edit from round 2 is dropped, and the hook message carries the protocol.", "regression_risk": "1) Churn: the agent replaces a good model with one that wins on a small, noisy stand-in. Mitigation: the message requires a stated margin, a backup copy, a probe and `--validate-only`, and says to keep the current model when nothing clears the margin. 2) Stop loops: mitigated by the `stop_hook_active` check and a per-session marker file, so it blocks at most once. 3) A misfire outside a campaign: it requires 'deliverable accepted' and final_model.js in the transcript, otherwise it exits 0. If transcript_path is missing it falls back to a file check, as in round 2. 4) Higher token use, perhaps 20-50%, because a second fit-and-compare pass is added. 5) The audit cannot repair the discovery gate, because rounds are irreversible, so Suzuki (3/3 failures, likely discovery) is unlikely to move. 6) The hook is untested in a real session. If the harness runs hooks with a different event shape, it silently no-ops, which leaves behaviour equal to the parent. 7) Gate risk: the plan must list exactly hooks.json, tools/stop_audit.py and smoke.json." }, "attempts": [ { "attempt": 0, "gate_passed": true, "gate_errors": [], "critic": { "approved": false, "reasons": [ "The candidate changes instructions.md, which the plan does not declare. The plan's only edit is the hook, with files hooks.json, tools/stop_audit.py and smoke.json. Its why_not_simpler says \"the instructions edit from round 2 is dropped, and the hook message carries the protocol\". The diff adds a new \"## After delivery\" section to instructions.md anyway. That is a change beyond the declared plan. It also gives two components (hook and instructions) against a stated budget of one.", "The same class of error rejected round 2, where the bundle and plan disagreed on which files changed. The plan's regression_risk item 7 says the plan must list exactly hooks.json, tools/stop_audit.py and smoke.json. The shipped bundle does not match that.", "The instructions addition also duplicates the hook's message. It promises that a stop hook will run, and if the hook silently no-ops the instructions are misleading. The plan explicitly chose to avoid more advisory prose because that was ignored in 6 of 6 trials.", "The hook itself looks acceptable on the other criteria. It is bounded: it blocks at most once per session, checks stop_hook_active, has a 30 s timeout, and fails open on bad input. Its message is domain-neutral, and it makes no network, model or provider calls. The remaining concern is minor: the marker file is under /tmp, keyed by session id. The rejection is for the undeclared instructions.md edit." ], "required_fixes": [ "Revert instructions.md to the parent's content, so the bundle changes only hooks.json, tools/stop_audit.py and smoke.json, exactly as the plan declares. Alternatively, have the plan declare instructions.md and justify the second component. The plan currently says that edit was dropped.", "Keep the declared paths exactly matching the shipped paths: tools/stop_audit.py, hooks.json and smoke.json.", "Keep the hook's message and behaviour domain-neutral and bounded as they are now: one block per session, a stop_hook_active check, and fail-open." ] } }, { "attempt": 1, "gate_passed": true, "gate_errors": [], "critic": { "approved": true, "reasons": [ "The changed files are hooks.json, tools/stop_audit.py and smoke.json. These are exactly the paths the plan declares, so the earlier path-mismatch error is fixed. instructions.md and tools/campaign_check.py have the same line counts as the parent, and the diff shows no edits to them.", "The hook is bounded. It blocks a stop at most once per session, using a marker file and the stop_hook_active check. It exits 0 on malformed input, a missing transcript or OS errors. It has a 30 s timeout and does no network or model work.", "It has no task-specific hardcoding. The audit message names no task, dataset, design or answer, and it contains no measured values. It reuses the existing `campaign_check.py ndcg` and `probe` tools and the `asb deliver --validate-only` flow that the parent's instructions already describe.", "It does not touch private truth, tests or grader files, and it makes no network, provider or billing changes.", "It persists nothing across episodes except a session-keyed marker file in the temp directory. That file holds a timestamp, not task data. No memory or skills are added.", "Smoke tests cover the main paths: the built-in selftest (block once, never loop, ignore malformed input, skip when time is short or nothing was delivered), an active stop hook, and garbage stdin. The selftest uses temporary fixtures only.", "The litmus test is acceptable. The hook only fires for a campaign-style delivery, matched by 'deliverable accepted' plus final_model.js. In that case it asks for a held-out stand-in comparison, which applies to any task of the parent's seed-data, irreversible-rounds and ranking-gate family. Elsewhere it no-ops. The message stays domain-neutral. The re-delivery margin rule and the 'keep the current model' fallback limit the risk of churn.", "The change is not degenerate. It adds an enforcement point on the stop path, which the plan argues is the only lever that does not rely on the agent following advisory text." ], "required_fixes": [] } } ], "diff": "--- parent/hooks.json\n+++ candidate/hooks.json\n@@ -0,0 +1,9 @@\n+{\n+ \"stop\": [\n+ {\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n+ \"timeout\": 30\n+ }\n+ ]\n+}\n--- parent/smoke.json\n+++ candidate/smoke.json\n@@ -48,6 +48,30 @@\n \"timeout\": 30\n },\n {\n+ \"name\": \"stop_audit hook selftest (block once after delivery, never loop, no-op otherwise)\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\", \"--selftest\"],\n+ \"stdin\": \"\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 120\n+ },\n+ {\n+ \"name\": \"stop_audit hook lets an already-active stop through\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n+ \"stdin\": \"{\\\"hook_event_name\\\": \\\"Stop\\\", \\\"stop_hook_active\\\": true, \\\"session_id\\\": \\\"smoke\\\", \\\"transcript_path\\\": \\\"/nonexistent/t.jsonl\\\"}\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"stop_audit hook ignores malformed event input\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n+ \"stdin\": \"not json at all\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n--- parent/tools/stop_audit.py\n+++ candidate/tools/stop_audit.py\n@@ -0,0 +1,182 @@\n+#!/usr/bin/env python3\n+\"\"\"One-shot stop hook: after a campaign model was delivered and time remains, ask for a held-out audit.\n+\n+Reads the stop event JSON on stdin. Exit 0 lets the agent stop. Exit 2 blocks the stop once and feeds the\n+audit request (stderr) back to the agent. It never blocks twice per session, never blocks when the event says\n+a stop hook is already active, and never blocks outside a delivered-campaign context.\n+Stdlib only. Run with --selftest for a built-in check using temporary fixtures.\n+\"\"\"\n+import json\n+import os\n+import re\n+import sys\n+import tempfile\n+import time\n+from datetime import datetime, timezone\n+\n+MARKER_ENV = \"RSI_AUDIT_MARKER_DIR\"\n+BUDGET_ENV = \"RSI_AUDIT_BUDGET_S\"\n+DEFAULT_BUDGET_S = 28800\n+MIN_REMAINING_S = 1800\n+DELIVERED = \"deliverable accepted\"\n+MODEL_NAME = \"final_model.js\"\n+\n+MESSAGE = \"\"\"Before stopping: a model has been delivered and time remains. The latest submission is the one scored, so\n+re-delivery is allowed. Spend a short audit on the held-out ranking gate. Everything you measured so far was\n+chosen by your own model, so a score over your own picks is biased upward.\n+\n+1. Build a held-out stand-in. Use the broad first round (or the least model-selected measurements), or hold out whole\n+ groups of the ordered items (group the folds by the key that distinguishes held-out items). Fit on the rest, keeping\n+ the seed data in. Score with the official relevance transform exactly as the task states it:\n+ python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ]\n+2. Compare the delivered model with at least one structurally different alternative on that same stand-in: a\n+ lower-dimensional or smoother model (fewer features, stronger shrinkage, log or clipped target), and the same\n+ model with versus without the seed data. Run long fits in the background (nohup ... &) and poll the log, because\n+ a single command is cut off after about two minutes.\n+3. Re-deliver only if an alternative beats the delivered model by a clear margin (state it, e.g. at least 0.05 NDCG)\n+ and the gap is not one or two items. Then: cp final_model.js final_model.prev.js, write the new file,\n+ python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt,\n+ asb deliver --validate-only, and only then the real deliver.\n+ If nothing wins by the margin, keep the current model: do not re-deliver, and report the stand-in scores.\n+This audit happens once; state the numbers you compared in the final message.\"\"\"\n+\n+\n+def parse_ts(value):\n+ try:\n+ if isinstance(value, (int, float)):\n+ return float(value) / (1000.0 if value > 1e11 else 1.0)\n+ dt = datetime.fromisoformat(str(value).replace(\"Z\", \"+00:00\"))\n+ if dt.tzinfo is None:\n+ dt = dt.replace(tzinfo=timezone.utc)\n+ return dt.timestamp()\n+ except (ValueError, TypeError):\n+ return None\n+\n+\n+def read_transcript(path):\n+ try:\n+ with open(path, encoding=\"utf-8\", errors=\"replace\") as fh:\n+ return fh.read()\n+ except OSError:\n+ return None\n+\n+\n+def first_timestamp(text):\n+ for line in text.splitlines()[:50]:\n+ m = re.search(r'\"timestamp\"\\s*:\\s*(\"[^\"]+\"|[0-9.]+)', line)\n+ if m:\n+ ts = parse_ts(json.loads(m.group(1)))\n+ if ts:\n+ return ts\n+ return None\n+\n+\n+def model_exists(cwd):\n+ for d in (cwd, os.getcwd(), \"/app\"):\n+ if d and os.path.isfile(os.path.join(d, MODEL_NAME)):\n+ return True\n+ return False\n+\n+\n+def marker_path(session):\n+ base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_stop_audit\")\n+ safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session or \"nosession\")[:80]\n+ return base, os.path.join(base, safe + \".done\")\n+\n+\n+def decide(event):\n+ \"\"\"Return True if the stop should be blocked for the audit.\"\"\"\n+ if event.get(\"stop_hook_active\"):\n+ return False\n+ cwd = event.get(\"cwd\") or \"\"\n+ text = None\n+ tp = event.get(\"transcript_path\")\n+ if tp:\n+ text = read_transcript(tp)\n+ if text is not None:\n+ if DELIVERED not in text or MODEL_NAME not in text:\n+ return False\n+ start = first_timestamp(text)\n+ if start is not None:\n+ try:\n+ budget = float(os.environ.get(BUDGET_ENV, DEFAULT_BUDGET_S))\n+ except ValueError:\n+ budget = DEFAULT_BUDGET_S\n+ if time.time() - start > budget - MIN_REMAINING_S:\n+ return False\n+ elif not model_exists(cwd):\n+ return False\n+ return True\n+\n+\n+def main():\n+ if \"--selftest\" in sys.argv:\n+ return selftest()\n+ try:\n+ event = json.loads(sys.stdin.read() or \"{}\")\n+ if not isinstance(event, dict):\n+ return 0\n+ except ValueError:\n+ return 0\n+ try:\n+ if not decide(event):\n+ return 0\n+ base, marker = marker_path(str(event.get(\"session_id\") or \"\"))\n+ if os.path.exists(marker):\n+ return 0\n+ os.makedirs(base, exist_ok=True)\n+ with open(marker, \"w\") as fh:\n+ fh.write(str(time.time()))\n+ except OSError:\n+ return 0\n+ sys.stderr.write(MESSAGE + \"\\n\")\n+ return 2\n+\n+\n+def selftest():\n+ import subprocess\n+ import shutil\n+ tmp = tempfile.mkdtemp(prefix=\"stopaudit_\")\n+ fails = []\n+ try:\n+ env = dict(os.environ, **{MARKER_ENV: os.path.join(tmp, \"m\")})\n+\n+ def run(event):\n+ p = subprocess.run([sys.executable, os.path.abspath(__file__)], input=json.dumps(event),\n+ capture_output=True, text=True, env=env, timeout=30)\n+ return p.returncode\n+\n+ def check(name, cond):\n+ print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n+ if not cond:\n+ fails.append(name)\n+\n+ now = datetime.now(timezone.utc).isoformat()\n+ good = os.path.join(tmp, \"good.jsonl\")\n+ with open(good, \"w\") as fh:\n+ fh.write(json.dumps({\"timestamp\": now, \"text\": \"asb deliver final_model.js\"}) + \"\\n\")\n+ fh.write(json.dumps({\"text\": \"local validation passed\\n\" + DELIVERED + \" (submission 1)\"}) + \"\\n\")\n+ undelivered = os.path.join(tmp, \"undelivered.jsonl\")\n+ with open(undelivered, \"w\") as fh:\n+ fh.write(json.dumps({\"timestamp\": now, \"text\": \"writing final_model.js\"}) + \"\\n\")\n+ late = os.path.join(tmp, \"late.jsonl\")\n+ with open(late, \"w\") as fh:\n+ fh.write(json.dumps({\"timestamp\": \"2020-01-01T00:00:00Z\", \"text\": \"final_model.js \" + DELIVERED}) + \"\\n\")\n+ check(\"active stop hook never blocks\", run({\"stop_hook_active\": True, \"transcript_path\": good, \"session_id\": \"a\"}) == 0)\n+ check(\"no delivery in transcript does not block\", run({\"transcript_path\": undelivered, \"session_id\": \"b\"}) == 0)\n+ check(\"out of time does not block\", run({\"transcript_path\": late, \"session_id\": \"c\"}) == 0)\n+ check(\"garbage input does not block\", subprocess.run([sys.executable, os.path.abspath(__file__)], input=\"not json\",\n+ capture_output=True, text=True, env=env).returncode == 0)\n+ check(\"delivered with time left blocks once\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 2)\n+ check(\"second stop in same session passes\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 0)\n+ os.makedirs(os.path.join(tmp, \"w\"))\n+ open(os.path.join(tmp, \"w\", MODEL_NAME), \"w\").close()\n+ check(\"no transcript but model file present blocks\", run({\"cwd\": os.path.join(tmp, \"w\"), \"session_id\": \"f\"}) == 2)\n+ finally:\n+ shutil.rmtree(tmp, ignore_errors=True)\n+ print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n+ return 1 if fails else 0\n+\n+\n+if __name__ == \"__main__\":\n+ sys.exit(main())\n", "files": { "hooks.json": "{\n \"stop\": [\n {\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"timeout\": 30\n }\n ]\n}\n", "instructions.md": "# Campaign workflow notes\n\nThese apply to tasks of the form: seed data, a few irreversible batched `asb run` rounds, then a self-contained\ndeterministic `final_model.js` that must pass both a discovery gate (hits among your ordered items) and a\nheld-out ranking (NDCG) gate. Two gates means neither can be left to the end.\n\n## Before round 1\n- Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n script in full.\n- Write a baseline model from the seed data and run the deliver command in its free validation-only mode.\n Re-validate after every round, so a valid model always exists well before the deadline.\n- Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n\n## Each round\n- Accepted rounds are final; rejected requests are free. Before `asb run`, check the file:\n `python3 /opt/rsi-harness/tools/campaign_check.py batch round_N.csv --size --id-column --previous round_*_results.csv [--catalog ]`\n Fix anything it reports (header, size, duplicates, items already ordered, ids not in the catalog).\n- Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n This is guidance, not fixed percentages.\n- Measurements are noisy; do not read much into a single value that sits near the threshold.\n\n## Using the seed data\n- Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n score favour easy groups over hard ones.\n\n## Choosing and checking the model\n- Pick the final model by cross-validation that mimics the held-out distribution (group the folds by the grouping variable\n that distinguishes held-out items, e.g. a categorical key), with the official metric computed locally:\n `python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ] [--threshold ]`\n Pass relevance already defined as the task states (clipped/normalised). The tool breaks score ties by input row order;\n prefer models that rarely produce ties.\n- The model must be self-contained, deterministic, finite for every input and independent per input; handle levels\n unseen in training with a fallback. Before delivering, run\n `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n- `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n", "smoke.json": "[\n {\n \"name\": \"campaign_check selftest (hand-computed NDCG, tie rule, batch and probe checks)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"campaign_check ndcg on a tiny synthetic CSV from stdin\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"p\", \"--rel\", \"r\", \"--k\", \"3\", \"--group\", \"g\"],\n \"stdin\": \"p,r,g\\n0.9,1,x\\n0.8,0,x\\n0.7,1,x\\n0.5,0,y\\n0.4,2,y\\n0.3,1,y\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check ndcg rejects an unknown column\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"nope\", \"--rel\", \"r\", \"--k\", \"2\"],\n \"stdin\": \"p,r\\n1,1\\n2,0\\n\",\n \"expect_exit\": 2,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch accepts a clean round\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw2\\nw3\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags duplicates and wrong size\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw1\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags a wrong header\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"2\", \"--id-column\", \"item_key\"],\n \"stdin\": \"id\\nA\\nB\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook selftest (block once after delivery, never loop, no-op otherwise)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\", \"--selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"stop_audit hook lets an already-active stop through\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"{\\\"hook_event_name\\\": \\\"Stop\\\", \\\"stop_hook_active\\\": true, \\\"session_id\\\": \\\"smoke\\\", \\\"transcript_path\\\": \\\"/nonexistent/t.jsonl\\\"}\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook ignores malformed event input\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"not json at all\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n \"stdin\": \"\",\n \"expect_exit\": 2,\n \"timeout\": 30\n }\n]\n", "tools/campaign_check.py": "#!/usr/bin/env python3\n\"\"\"Advisory checks for batched-oracle campaigns (stdlib only, never blocks anything).\n\nSubcommands\n batch check a round CSV before ordering it (header, size, duplicates, earlier orders, catalog)\n ndcg local NDCG@k with 1/log2(rank+1) discount, stable ties, ideal-ranking normalisation\n probe run final_model.js under node on sample inputs: finite, deterministic, order-independent\n selftest built-in hand-computed checks of the above\n\nExit codes: 0 ok, 1 problem found, 2 usage or I/O error.\n\"\"\"\nimport argparse\nimport csv\nimport io\nimport json\nimport math\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\n\n\ndef die(msg, code=2):\n sys.stderr.write(\"error: %s\\n\" % msg)\n sys.exit(code)\n\n\ndef read_text(path):\n if path == \"-\":\n return sys.stdin.read()\n try:\n with open(path, encoding=\"utf-8-sig\", newline=\"\") as fh:\n return fh.read()\n except OSError as exc:\n die(\"cannot read %s: %s\" % (path, exc))\n\n\ndef read_rows(path):\n text = read_text(path)\n if text.startswith(\"\"):\n text = text[1:]\n rows = list(csv.reader(io.StringIO(text)))\n rows = [r for r in rows if r and any(c.strip() for c in r)]\n if not rows:\n die(\"%s is empty\" % path)\n return rows[0], rows[1:]\n\n\ndef id_values(path, column):\n header, body = read_rows(path)\n idx = header.index(column) if column in header else 0\n return [r[idx].strip() for r in body if len(r) > idx]\n\n\n# ---------------------------------------------------------------- batch\ndef cmd_batch(args):\n header, body = read_rows(args.round_csv)\n problems, notes = [], []\n column = args.id_column or header[0]\n if len(header) != 1:\n problems.append(\"header has %d columns %r; the round file needs a single column\" % (len(header), header))\n if header[0] != column:\n problems.append(\"header is %r, expected %r\" % (header[0], column))\n ids = [r[0] for r in body]\n padded = [i for i in ids if i != i.strip()]\n if padded:\n problems.append(\"%d ids have leading/trailing whitespace, e.g. %r\" % (len(padded), padded[0]))\n ids = [i.strip() for i in ids]\n if any(len(r) != 1 for r in body):\n problems.append(\"some rows have more than one field\")\n if any(not i for i in ids):\n problems.append(\"blank ids present\")\n notes.append(\"header=%r rows=%d\" % (header, len(ids)))\n if args.size is not None and len(ids) != args.size:\n problems.append(\"size is %d, expected exactly %d\" % (len(ids), args.size))\n seen, dups = set(), []\n for i in ids:\n if i in seen and i not in dups:\n dups.append(i)\n seen.add(i)\n if dups:\n problems.append(\"%d duplicated ids inside the batch, e.g. %s\" % (len(dups), dups[:3]))\n prior = set()\n for p in args.previous or []:\n prior.update(id_values(p, column))\n overlap = sorted(seen & prior)\n if args.previous:\n notes.append(\"compared with %d earlier ids from %d files\" % (len(prior), len(args.previous)))\n if overlap:\n problems.append(\"%d ids were already ordered, e.g. %s\" % (len(overlap), overlap[:3]))\n if args.catalog:\n cat = set(id_values(args.catalog, args.catalog_column or column))\n notes.append(\"catalog has %d ids\" % len(cat))\n unknown = sorted(seen - cat)\n if unknown:\n problems.append(\"%d ids are not in the catalog, e.g. %s\" % (len(unknown), unknown[:3]))\n for n in notes:\n print(\"note: \" + n)\n for p in problems:\n print(\"PROBLEM: \" + p)\n print(\"batch %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- ndcg\ndef dcg(rels):\n return sum(r / math.log2(i + 2) for i, r in enumerate(rels))\n\n\ndef ndcg_at_k(preds, rels, k):\n \"\"\"Rank by prediction descending; equal predictions keep input order (stable sort).\"\"\"\n order = sorted(range(len(preds)), key=lambda i: -preds[i])\n got = dcg([rels[i] for i in order[:k]])\n ideal = dcg(sorted(rels, reverse=True)[:k])\n if ideal <= 0:\n return None\n return got / ideal\n\n\ndef cmd_ndcg(args):\n header, body = read_rows(args.csv)\n for c in [args.pred, args.rel] + ([args.group] if args.group else []):\n if c not in header:\n die(\"column %r not in header %r\" % (c, header))\n ip, ir = header.index(args.pred), header.index(args.rel)\n ig = header.index(args.group) if args.group else None\n preds, rels, groups = [], [], []\n for n, r in enumerate(body, 2):\n try:\n p, q = float(r[ip]), float(r[ir])\n except (ValueError, IndexError):\n die(\"row %d: non-numeric prediction or relevance\" % n)\n if math.isnan(p) or math.isnan(q) or math.isinf(p):\n die(\"row %d: NaN/inf value\" % n)\n preds.append(p)\n rels.append(q)\n groups.append(r[ig] if ig is not None else None)\n if any(q < 0 for q in rels):\n print(\"warning: negative relevance present; pass relevance already clipped at 0 if the metric does so\")\n k = args.k\n if k > len(preds):\n print(\"warning: k=%d exceeds %d rows\" % (k, len(preds)))\n val = ndcg_at_k(preds, rels, k)\n if val is None:\n print(\"warning: ideal DCG is 0 (no positive relevance); NDCG undefined, reported as 0\")\n val = 0.0\n print(\"ndcg@%d = %.6f (n=%d)\" % (k, val, len(preds)))\n if args.group:\n by = {}\n for i, g in enumerate(groups):\n by.setdefault(g, []).append(i)\n vals = []\n for g in sorted(by):\n v = ndcg_at_k([preds[i] for i in by[g]], [rels[i] for i in by[g]], k)\n vals.append(v)\n print(\" group %s: n=%d ndcg@%d=%s\" % (g, len(by[g]), k, \"undefined\" if v is None else \"%.4f\" % v))\n ok = [v for v in vals if v is not None]\n if ok:\n print(\"mean over groups (informational, not the pooled metric) = %.6f\" % (sum(ok) / len(ok)))\n if args.threshold is not None:\n print(\"threshold %.4f: %s\" % (args.threshold, \"PASS\" if val >= args.threshold else \"BELOW\"))\n return 0\n\n\n# ---------------------------------------------------------------- probe\nPROBE_JS = r\"\"\"\nconst fs = require('fs'), vm = require('vm');\nconst req = JSON.parse(fs.readFileSync(0, 'utf8'));\nfunction load() {\n const code = fs.readFileSync(req.model, 'utf8');\n const mod = { exports: {} };\n const sb = { module: mod, exports: mod.exports, console: { log() {}, error() {}, warn() {} } };\n vm.createContext(sb);\n vm.runInContext(code, sb, { timeout: 60000 });\n const f = (typeof sb.predict === 'function') ? sb.predict\n : (mod.exports && typeof mod.exports.predict === 'function') ? mod.exports.predict : null;\n if (!f) throw new Error('no predict function defined');\n return f;\n}\nfunction run(f, x) {\n try { return { v: f(x) }; } catch (e) { return { e: String(e && e.message || e) }; }\n}\nconst out = { load_error: null, results: [] };\ntry {\n const f1 = load(), f2 = load();\n const xs = req.inputs;\n const a = xs.map(x => run(f1, x));\n const b = xs.map(x => run(f1, x));\n const c = xs.map((x, i) => run(f2, xs[xs.length - 1 - i])).reverse();\n xs.forEach((x, i) => {\n const same = (p, q) => ('e' in p && 'e' in q) || ('v' in p && 'v' in q && Object.is(p.v, q.v));\n out.results.push({\n input: x, value: ('v' in a[i]) ? a[i].v : null, error: a[i].e || null,\n type: ('v' in a[i]) ? typeof a[i].v : null,\n finite: ('v' in a[i]) && typeof a[i].v === 'number' && Number.isFinite(a[i].v),\n repeat_same: same(a[i], b[i]), order_same: same(a[i], c[i])\n });\n });\n} catch (e) { out.load_error = String(e && e.message || e); }\nprocess.stdout.write(JSON.stringify(out));\n\"\"\"\n\nBUILTIN_EDGE = [\"\", \"?\"]\n\n\ndef cmd_probe(args):\n if not os.path.isfile(args.model):\n die(\"model file not found: %s\" % args.model)\n node = shutil.which(\"node\")\n if not node:\n die(\"node not found on PATH\")\n inputs = list(args.input or [])\n if args.inputs_file:\n inputs += [l.rstrip(\"\\r\\n\") for l in read_text(args.inputs_file).splitlines() if l.strip()]\n edge_must = []\n if args.edge_file:\n edge_must = [l.rstrip(\"\\r\\n\") for l in read_text(args.edge_file).splitlines() if l.strip()]\n if not inputs and not edge_must:\n die(\"give --input and/or --inputs-file (real-format sample inputs)\")\n all_inputs = inputs + edge_must + BUILTIN_EDGE\n kinds = [\"sample\"] * len(inputs) + [\"edge-required\"] * len(edge_must) + [\"edge-advisory\"] * len(BUILTIN_EDGE)\n payload = json.dumps({\"model\": os.path.abspath(args.model), \"inputs\": all_inputs})\n try:\n proc = subprocess.run([node, \"-e\", PROBE_JS], input=payload, capture_output=True,\n text=True, timeout=args.timeout)\n except subprocess.TimeoutExpired:\n print(\"PROBLEM: node timed out after %ss\" % args.timeout)\n return 1\n if proc.returncode != 0:\n print(\"PROBLEM: node failed: %s\" % proc.stderr.strip()[:500])\n return 1\n try:\n res = json.loads(proc.stdout)\n except ValueError:\n print(\"PROBLEM: unparseable probe output: %r\" % proc.stdout[:200])\n return 1\n if res.get(\"load_error\"):\n print(\"PROBLEM: model failed to load in an isolated context: %s\" % res[\"load_error\"])\n return 1\n problems = 0\n for r, kind in zip(res[\"results\"], kinds):\n bad = []\n if r[\"error\"]:\n bad.append(\"threw: %s\" % r[\"error\"])\n elif not r[\"finite\"]:\n bad.append(\"not a finite number (%r)\" % (r[\"value\"],))\n if not r[\"repeat_same\"]:\n bad.append(\"repeated call differs\")\n if not r[\"order_same\"]:\n bad.append(\"result depends on call order/fresh load\")\n if bad:\n tag = \"WARN\" if kind == \"edge-advisory\" else \"PROBLEM\"\n if kind != \"edge-advisory\":\n problems += 1\n print(\"%s [%s] %r: %s\" % (tag, kind, r[\"input\"], \"; \".join(bad)))\n vals = [r[\"value\"] for r in res[\"results\"][:len(inputs)] if r[\"finite\"]]\n if vals:\n print(\"sample outputs: n=%d min=%.6g max=%.6g distinct=%d\" % (len(vals), min(vals), max(vals), len(set(vals))))\n if len(set(vals)) == 1 and len(vals) > 1:\n print(\"WARN: every sample input got the same score; the ranking would be all ties\")\n print(\"probe %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- selftest\ndef cmd_selftest(_args):\n fails = []\n\n def check(name, cond):\n if not cond:\n fails.append(name)\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n\n # hand-computed: dcg = 1 + 0.5 = 1.5 ; ideal = 1 + 1/log2(3)\n v = ndcg_at_k([0.9, 0.8, 0.7], [1, 0, 1], 3)\n check(\"ndcg hand value\", abs(v - 1.5 / (1 + 1 / math.log2(3))) < 1e-12)\n # tie rule: equal scores keep evaluation order -> rels seen [0,1]\n v = ndcg_at_k([5, 5, 5], [0, 1, 1], 2)\n check(\"ndcg tie keeps input order\", abs(v - (1 / math.log2(3)) / (1 + 1 / math.log2(3))) < 1e-12)\n check(\"ndcg perfect ranking is 1\", abs(ndcg_at_k([3, 2, 1], [2, 1, 0], 2) - 1) < 1e-12)\n check(\"ndcg undefined without positives\", ndcg_at_k([1, 2], [0, 0], 2) is None)\n tmp = tempfile.mkdtemp(prefix=\"campchk_\")\n try:\n def w(name, text):\n p = os.path.join(tmp, name)\n with open(p, \"w\") as fh:\n fh.write(text)\n return p\n prev = w(\"prev.csv\", \"id,y\\na,1\\nb,2\\n\")\n ns = argparse.Namespace\n import contextlib\n sink = io.StringIO()\n with contextlib.redirect_stdout(sink):\n ok = cmd_batch(ns(round_csv=w(\"r1.csv\", \"id\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n dup = cmd_batch(ns(round_csv=w(\"r2.csv\", \"id\\nc\\nc\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n old = cmd_batch(ns(round_csv=w(\"r3.csv\", \"id\\na\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n short = cmd_batch(ns(round_csv=w(\"r4.csv\", \"id\\nc\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n hdr = cmd_batch(ns(round_csv=w(\"r5.csv\", \"wrong\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n check(\"batch accepts clean round\", ok == 0)\n check(\"batch flags in-batch duplicate\", dup == 1)\n check(\"batch flags earlier order\", old == 1)\n check(\"batch flags wrong size\", short == 1)\n check(\"batch flags wrong header\", hdr == 1)\n if shutil.which(\"node\"):\n model = w(\"m.js\", \"function predict(x){ return x.length * 0.5; }\\n\")\n bad = w(\"bad.js\", \"function predict(x){ return x === '' ? NaN : Math.random(); }\\n\")\n with contextlib.redirect_stdout(sink):\n good = cmd_probe(ns(model=model, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n notgood = cmd_probe(ns(model=bad, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n check(\"probe accepts a finite deterministic model\", good == 0)\n check(\"probe rejects a random model\", notgood == 1)\n else:\n print(\"skip probe checks (node not found)\")\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\ndef main():\n ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n sub = ap.add_subparsers(dest=\"cmd\", required=True)\n\n b = sub.add_parser(\"batch\", help=\"check a round CSV before `asb run` (use '-' for stdin)\")\n b.add_argument(\"round_csv\")\n b.add_argument(\"--size\", type=int, help=\"exact number of rows required\")\n b.add_argument(\"--id-column\", help=\"expected header, e.g. the single column name from the task\")\n b.add_argument(\"--previous\", nargs=\"*\", help=\"earlier round or results CSVs whose ids may not repeat\")\n b.add_argument(\"--catalog\", help=\"CSV listing orderable ids\")\n b.add_argument(\"--catalog-column\", help=\"id column in the catalog (default: --id-column)\")\n b.set_defaults(fn=cmd_batch)\n\n n = sub.add_parser(\"ndcg\", help=\"local NDCG@k (use '-' for stdin)\")\n n.add_argument(\"csv\")\n n.add_argument(\"--pred\", required=True, help=\"prediction column\")\n n.add_argument(\"--rel\", required=True, help=\"relevance column (already clipped/normalised as the task defines)\")\n n.add_argument(\"--k\", type=int, required=True)\n n.add_argument(\"--group\", help=\"optional group column; adds per-group NDCG as a diagnostic\")\n n.add_argument(\"--threshold\", type=float, help=\"optional pass level to compare against\")\n n.set_defaults(fn=cmd_ndcg)\n\n p = sub.add_parser(\"probe\", help=\"run final_model.js under node on sample inputs\")\n p.add_argument(\"model\")\n p.add_argument(\"--input\", action=\"append\", help=\"sample input string (repeatable)\")\n p.add_argument(\"--inputs-file\", help=\"file with one sample input per line\")\n p.add_argument(\"--edge-file\", help=\"file with one extra input per line that must also score finitely\")\n p.add_argument(\"--timeout\", type=int, default=120)\n p.set_defaults(fn=cmd_probe)\n\n s = sub.add_parser(\"selftest\", help=\"hand-computed self checks\")\n s.set_defaults(fn=cmd_selftest)\n\n args = ap.parse_args()\n sys.exit(args.fn(args))\n\n\nif __name__ == \"__main__\":\n main()\n", "tools/stop_audit.py": "#!/usr/bin/env python3\n\"\"\"One-shot stop hook: after a campaign model was delivered and time remains, ask for a held-out audit.\n\nReads the stop event JSON on stdin. Exit 0 lets the agent stop. Exit 2 blocks the stop once and feeds the\naudit request (stderr) back to the agent. It never blocks twice per session, never blocks when the event says\na stop hook is already active, and never blocks outside a delivered-campaign context.\nStdlib only. Run with --selftest for a built-in check using temporary fixtures.\n\"\"\"\nimport json\nimport os\nimport re\nimport sys\nimport tempfile\nimport time\nfrom datetime import datetime, timezone\n\nMARKER_ENV = \"RSI_AUDIT_MARKER_DIR\"\nBUDGET_ENV = \"RSI_AUDIT_BUDGET_S\"\nDEFAULT_BUDGET_S = 28800\nMIN_REMAINING_S = 1800\nDELIVERED = \"deliverable accepted\"\nMODEL_NAME = \"final_model.js\"\n\nMESSAGE = \"\"\"Before stopping: a model has been delivered and time remains. The latest submission is the one scored, so\nre-delivery is allowed. Spend a short audit on the held-out ranking gate. Everything you measured so far was\nchosen by your own model, so a score over your own picks is biased upward.\n\n1. Build a held-out stand-in. Use the broad first round (or the least model-selected measurements), or hold out whole\n groups of the ordered items (group the folds by the key that distinguishes held-out items). Fit on the rest, keeping\n the seed data in. Score with the official relevance transform exactly as the task states it:\n python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ]\n2. Compare the delivered model with at least one structurally different alternative on that same stand-in: a\n lower-dimensional or smoother model (fewer features, stronger shrinkage, log or clipped target), and the same\n model with versus without the seed data. Run long fits in the background (nohup ... &) and poll the log, because\n a single command is cut off after about two minutes.\n3. Re-deliver only if an alternative beats the delivered model by a clear margin (state it, e.g. at least 0.05 NDCG)\n and the gap is not one or two items. Then: cp final_model.js final_model.prev.js, write the new file,\n python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt,\n asb deliver --validate-only, and only then the real deliver.\n If nothing wins by the margin, keep the current model: do not re-deliver, and report the stand-in scores.\nThis audit happens once; state the numbers you compared in the final message.\"\"\"\n\n\ndef parse_ts(value):\n try:\n if isinstance(value, (int, float)):\n return float(value) / (1000.0 if value > 1e11 else 1.0)\n dt = datetime.fromisoformat(str(value).replace(\"Z\", \"+00:00\"))\n if dt.tzinfo is None:\n dt = dt.replace(tzinfo=timezone.utc)\n return dt.timestamp()\n except (ValueError, TypeError):\n return None\n\n\ndef read_transcript(path):\n try:\n with open(path, encoding=\"utf-8\", errors=\"replace\") as fh:\n return fh.read()\n except OSError:\n return None\n\n\ndef first_timestamp(text):\n for line in text.splitlines()[:50]:\n m = re.search(r'\"timestamp\"\\s*:\\s*(\"[^\"]+\"|[0-9.]+)', line)\n if m:\n ts = parse_ts(json.loads(m.group(1)))\n if ts:\n return ts\n return None\n\n\ndef model_exists(cwd):\n for d in (cwd, os.getcwd(), \"/app\"):\n if d and os.path.isfile(os.path.join(d, MODEL_NAME)):\n return True\n return False\n\n\ndef marker_path(session):\n base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_stop_audit\")\n safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session or \"nosession\")[:80]\n return base, os.path.join(base, safe + \".done\")\n\n\ndef decide(event):\n \"\"\"Return True if the stop should be blocked for the audit.\"\"\"\n if event.get(\"stop_hook_active\"):\n return False\n cwd = event.get(\"cwd\") or \"\"\n text = None\n tp = event.get(\"transcript_path\")\n if tp:\n text = read_transcript(tp)\n if text is not None:\n if DELIVERED not in text or MODEL_NAME not in text:\n return False\n start = first_timestamp(text)\n if start is not None:\n try:\n budget = float(os.environ.get(BUDGET_ENV, DEFAULT_BUDGET_S))\n except ValueError:\n budget = DEFAULT_BUDGET_S\n if time.time() - start > budget - MIN_REMAINING_S:\n return False\n elif not model_exists(cwd):\n return False\n return True\n\n\ndef main():\n if \"--selftest\" in sys.argv:\n return selftest()\n try:\n event = json.loads(sys.stdin.read() or \"{}\")\n if not isinstance(event, dict):\n return 0\n except ValueError:\n return 0\n try:\n if not decide(event):\n return 0\n base, marker = marker_path(str(event.get(\"session_id\") or \"\"))\n if os.path.exists(marker):\n return 0\n os.makedirs(base, exist_ok=True)\n with open(marker, \"w\") as fh:\n fh.write(str(time.time()))\n except OSError:\n return 0\n sys.stderr.write(MESSAGE + \"\\n\")\n return 2\n\n\ndef selftest():\n import subprocess\n import shutil\n tmp = tempfile.mkdtemp(prefix=\"stopaudit_\")\n fails = []\n try:\n env = dict(os.environ, **{MARKER_ENV: os.path.join(tmp, \"m\")})\n\n def run(event):\n p = subprocess.run([sys.executable, os.path.abspath(__file__)], input=json.dumps(event),\n capture_output=True, text=True, env=env, timeout=30)\n return p.returncode\n\n def check(name, cond):\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n if not cond:\n fails.append(name)\n\n now = datetime.now(timezone.utc).isoformat()\n good = os.path.join(tmp, \"good.jsonl\")\n with open(good, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"asb deliver final_model.js\"}) + \"\\n\")\n fh.write(json.dumps({\"text\": \"local validation passed\\n\" + DELIVERED + \" (submission 1)\"}) + \"\\n\")\n undelivered = os.path.join(tmp, \"undelivered.jsonl\")\n with open(undelivered, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"writing final_model.js\"}) + \"\\n\")\n late = os.path.join(tmp, \"late.jsonl\")\n with open(late, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": \"2020-01-01T00:00:00Z\", \"text\": \"final_model.js \" + DELIVERED}) + \"\\n\")\n check(\"active stop hook never blocks\", run({\"stop_hook_active\": True, \"transcript_path\": good, \"session_id\": \"a\"}) == 0)\n check(\"no delivery in transcript does not block\", run({\"transcript_path\": undelivered, \"session_id\": \"b\"}) == 0)\n check(\"out of time does not block\", run({\"transcript_path\": late, \"session_id\": \"c\"}) == 0)\n check(\"garbage input does not block\", subprocess.run([sys.executable, os.path.abspath(__file__)], input=\"not json\",\n capture_output=True, text=True, env=env).returncode == 0)\n check(\"delivered with time left blocks once\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 2)\n check(\"second stop in same session passes\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 0)\n os.makedirs(os.path.join(tmp, \"w\"))\n open(os.path.join(tmp, \"w\", MODEL_NAME), \"w\").close()\n check(\"no transcript but model file present blocks\", run({\"cwd\": os.path.join(tmp, \"w\"), \"session_id\": \"f\"}) == 2)\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, "candidate_sha256": "b232ad942c6f44e92710c1bf6a8c148a76d813fdc9a918bc1c68f88066815528", "parent_sha256": "7b8b66f849093bd17d27dd02c87284b3014e0ad2fa545d10a078f8ee9af8b0f9", "role_cost": { "calls": 6, "list_price_estimate_usd": 1.41 }, "trials": [ { "task": "propylene-active-learning", "trial": "propylene-active-learning__k3Tgise", "job": "r03-screen", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 888513, "list_price_estimate_usd": 0.424365 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__kfqHbhm", "job": "r03-screen", "reward": 0, "model_audit_ok": true, "harness_interventions": 1, "tokens": 868376, "list_price_estimate_usd": 0.47436880000000003 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__DNstK4u", "job": "r03-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 1, "tokens": 969926, "list_price_estimate_usd": 0.5111182000000001 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__LGkcbM5", "job": "r03-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 1, "tokens": 1302786, "list_price_estimate_usd": 0.5334584000000001 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__9uRcEiy", "job": "r03-topup", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 860331, "list_price_estimate_usd": 0.46048479999999997 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__djh5T3X", "job": "r03-topup", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 664868, "list_price_estimate_usd": 0.37399699999999997 } ] }, { "round": 4, "title": "Discovery ledger in the instructions", "summary": "Before each experiment round, write down hits so far, the target, and hits still needed per remaining round, then shift toward neighbours of the best results when behind.", "observation": "4/6 but not accepted: one extra pass is inside the noise band and the candidate was neither simpler nor cheaper. Predicted a Propylene gain; the extra pass came on Suzuki.", "accepted": false, "reason": "unresolved_in_noise_band", "recorded_offline": false, "score": 0.6666666666666666, "trials_run": 6, "task_rates": { "propylene-active-learning": 0.3333333333333333, "suzuki-condition-screen": 1.0 }, "p_superior": 0.7039627039627053, "incumbent_score": 0.5, "tokens_per_trial": 1111677.3333333333, "complexity": 634, "edit_budget": 1, "components_changed": [ "instructions" ], "changed_files": [ "instructions.md" ], "plan": { "problem": "The parent passes 3/6 (propylene 1/3, Suzuki 2/3). Propylene has stayed at 1/3 across rounds 1 and 3. The stop-hook audit ran in 6 of 6 round-3 trials but only diagnoses the final model. It cannot repair the discovery gate, which is fixed once rounds 2 and 3 are ordered. The existing checklist says to \"track the running hit count\", but no trial turned that into a per-round hit target. Suzuki cutoffs are hidden, and three round-1 trials said they could not count standouts. LGkcbM5 failed with 5 hits against a threshold of 7 after a random round 1 and a Pd-rich ridge top-40. Passing propylene runs reached 9 hits and used models that clip doses to the measured range. This is inferred from final messages, not observed gate output.", "edits": [ { "component": "instructions", "hypothesis": "Adding one short, domain-neutral 'discovery ledger' section to instructions.md will raise the discovery-gate pass rate, mainly on propylene. Before each of rounds 2 to N it asks the agent to write down three numbers: probable hits so far, the threshold, and hits needed per remaining round. A probable hit is an observed value above the stated cutoff or, where cutoffs are hidden, in the top few percent of its own group's observed distribution at the stated standout rate. The section then sets rules. If the needed rate exceeds what the previous round delivered, the next round is mostly the model's top-ranked unordered items plus unordered neighbours (one factor changed) of every probable hit. Round 1 is a model-free stratified design that covers every level of each factor, with extra weight on single-factor coverage. Elements or levels that gave near-zero results at their lowest dose are excluded from later rounds. The section adds a model-form note taken from the passing trials: a regularised additive base with a smooth residual term, group intercepts for per-group targets, and inputs clipped to the measured range. It also adds two friction notes: pandas is not installed, so use csv and numpy; and the tool name is capital-B Bash. Falsifiable: propylene trials report hits-so-far before rounds 2 and 3 and reach at least 7 hits in at least 2 of 3 trials. The parent's pass rate of 1/3 rises, with Suzuki at least as good as 2/3. Refuted if propylene is still at most 1/3 with a shortfall below 7 hits in any failing trial, or if no ledger appears.", "files": [ "instructions.md" ] } ], "predictions": { "tasks": { "propylene-active-learning": "improve", "suzuki-condition-screen": "same" }, "token_effect": "same" }, "why_not_simpler": "Removing the stop hook is not supported: it fired once per trial, never looped, cost about 24k extra tokens per trial, and the round-3 parent beat round 1 with 87% credibility. Tweaking the hook message or margin only changes a post-delivery step, and the audit cannot change discovery after round 3. A change to campaign_check.py is the wrong place for this, because the checks that exist (batch, probe) are already used, `ndcg` is not, and a new subcommand would still depend on the agent deciding how to spend rounds. A hook before `asb run` is possible, but it would have to parse task-specific CSV state, which risks hard-coding. The existing checklist line 'track the running count of hits' was too vague to change behaviour, so a concrete ledger with a per-round target is the smallest edit that targets the failing step. It stays within edit_budget 1 because it touches one file.", "regression_risk": "Low to moderate. (1) Over-exploitation: excluding 'dead' levels or neighbour-chasing could narrow coverage and hurt held-out NDCG, as with NvCQYuj's narrow lookup table. The text must keep one explicit coverage clause and say 'guidance, not fixed percentages', as the parent already does. (2) The hit proxy for hidden cutoffs is noisy and may mislead Suzuki, where 2 of 3 trials already pass. It is advisory only, and Suzuki is predicted 'same'. (3) Longer instructions add about 300-400 tokens of input per step. This is offset by the saved pandas and tool-name retries. (4) The model-form note could anchor the agent and reduce variety. It is phrased as a pattern seen working, not a required recipe, and it contains no measured values or task-specific designs. (5) With 3 trials per task, noise of roughly ±35-40 points can hide or fake an effect, so the result is weak evidence either way." }, "attempts": [ { "attempt": 0, "gate_passed": true, "gate_errors": [], "critic": { "approved": true, "reasons": [ "The change is confined to the declared plan: one instructions.md edit. Line counts match the parent for hooks.json, smoke.json, tools/stop_audit.py and tools/campaign_check.py. The candidate's instructions.md matches candidate.diff exactly.", "No task-specific hardcoding or dispatch. The new 'Discovery ledger' section uses generic wording (items, groups, levels, factors, thresholds). It has no task names, no measured designs, no answers and no numeric thresholds. Task-instruction text was not copied verbatim.", "No access to private truth, solutions, tests or grader files. No network, model, provider or billing changes. No new hooks, tools, memory or skills. Nothing persists task data across episodes, and no loops or hooks were added that could stall the agent.", "The edit has real behavioural effect. It adds a per-round hit-target ledger, a rule to shift toward exploiting hits when behind, and a model-free stratified round 1. It also adds guidance on model form and two environment notes (pandas is missing; the tool name is capital-B Bash). This is not a degenerate edit.", "It is not grader gaming. The hit proxy is explicitly labelled advisory and noisy. The model-form note is phrased as a pattern to compare by grouped cross-validation, not a required recipe.", "It would still help an unfamiliar task from a different suite. The ledger, balanced round 1 and range-clipped models apply to any multi-round batched discovery-plus-ranking campaign. The pandas and Bash notes are environment facts that cost steps in earlier trials.", "Minor wording leans toward the seen domains, such as 'lowest dose' and 'elements'. It is advisory and does not amount to dispatch or hardcoding." ], "required_fixes": [] } } ], "diff": "--- parent/instructions.md\n+++ candidate/instructions.md\n@@ -20,6 +20,31 @@\n This is guidance, not fixed percentages.\n - Measurements are noisy; do not read much into a single value that sits near the threshold.\n \n+## Discovery ledger (write it down before every round after the first)\n+The discovery gate is decided by how rounds 2..N are spent, and the final-model audit cannot repair it. Before\n+building each later round, write three numbers in your notes:\n+1. Probable hits so far among ordered items. A probable hit is an observed value above the stated cutoff; where the\n+ cutoff is hidden, count an item whose value is in the top few percent of its own group's observed values, at the\n+ standout rate the task states. The proxy is noisy and advisory.\n+2. The discovery threshold.\n+3. Hits still needed per remaining round (threshold minus hits, divided by rounds left).\n+Rules:\n+- If the needed rate is higher than the previous round delivered, make the next round mostly the model's top-ranked\n+ unordered items plus unordered neighbours (one factor changed) of every probable hit. Keep a small explicit\n+ coverage share so levels never tried are not lost; this is guidance, not fixed percentages.\n+- Round 1 is a model-free stratified design: every level of every factor appears, with extra weight on single-factor\n+ coverage and, for per-group targets, a balanced count per group. Do not rely on a random draw.\n+- After round 1, drop elements or levels that gave near-zero results even at their lowest dose or best pairing, and\n+ do not order them again.\n+- Model form seen working in runs that passed: a regularised additive base with a smooth residual term (e.g. ridge\n+ plus a kernel/GP on residuals), a variance-stabilising target transform such as log(y + small constant), group\n+ intercepts when targets are per group, and inputs clipped to the measured range so extrapolation stays bounded.\n+ This is a pattern, not a required recipe; compare it with alternatives by grouped cross-validation.\n+\n+## Environment friction\n+- `pandas` is not installed: read data with `csv` and `numpy` (or plain Python).\n+- The tool name is `Bash` with a capital B; lowercase `bash` is rejected.\n+\n ## Using the seed data\n - Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n", "files": { "hooks.json": "{\n \"stop\": [\n {\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"timeout\": 30\n }\n ]\n}\n", "instructions.md": "# Campaign workflow notes\n\nThese apply to tasks of the form: seed data, a few irreversible batched `asb run` rounds, then a self-contained\ndeterministic `final_model.js` that must pass both a discovery gate (hits among your ordered items) and a\nheld-out ranking (NDCG) gate. Two gates means neither can be left to the end.\n\n## Before round 1\n- Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n script in full.\n- Write a baseline model from the seed data and run the deliver command in its free validation-only mode.\n Re-validate after every round, so a valid model always exists well before the deadline.\n- Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n\n## Each round\n- Accepted rounds are final; rejected requests are free. Before `asb run`, check the file:\n `python3 /opt/rsi-harness/tools/campaign_check.py batch round_N.csv --size --id-column --previous round_*_results.csv [--catalog ]`\n Fix anything it reports (header, size, duplicates, items already ordered, ids not in the catalog).\n- Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n This is guidance, not fixed percentages.\n- Measurements are noisy; do not read much into a single value that sits near the threshold.\n\n## Discovery ledger (write it down before every round after the first)\nThe discovery gate is decided by how rounds 2..N are spent, and the final-model audit cannot repair it. Before\nbuilding each later round, write three numbers in your notes:\n1. Probable hits so far among ordered items. A probable hit is an observed value above the stated cutoff; where the\n cutoff is hidden, count an item whose value is in the top few percent of its own group's observed values, at the\n standout rate the task states. The proxy is noisy and advisory.\n2. The discovery threshold.\n3. Hits still needed per remaining round (threshold minus hits, divided by rounds left).\nRules:\n- If the needed rate is higher than the previous round delivered, make the next round mostly the model's top-ranked\n unordered items plus unordered neighbours (one factor changed) of every probable hit. Keep a small explicit\n coverage share so levels never tried are not lost; this is guidance, not fixed percentages.\n- Round 1 is a model-free stratified design: every level of every factor appears, with extra weight on single-factor\n coverage and, for per-group targets, a balanced count per group. Do not rely on a random draw.\n- After round 1, drop elements or levels that gave near-zero results even at their lowest dose or best pairing, and\n do not order them again.\n- Model form seen working in runs that passed: a regularised additive base with a smooth residual term (e.g. ridge\n plus a kernel/GP on residuals), a variance-stabilising target transform such as log(y + small constant), group\n intercepts when targets are per group, and inputs clipped to the measured range so extrapolation stays bounded.\n This is a pattern, not a required recipe; compare it with alternatives by grouped cross-validation.\n\n## Environment friction\n- `pandas` is not installed: read data with `csv` and `numpy` (or plain Python).\n- The tool name is `Bash` with a capital B; lowercase `bash` is rejected.\n\n## Using the seed data\n- Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n score favour easy groups over hard ones.\n\n## Choosing and checking the model\n- Pick the final model by cross-validation that mimics the held-out distribution (group the folds by the grouping variable\n that distinguishes held-out items, e.g. a categorical key), with the official metric computed locally:\n `python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ] [--threshold ]`\n Pass relevance already defined as the task states (clipped/normalised). The tool breaks score ties by input row order;\n prefer models that rarely produce ties.\n- The model must be self-contained, deterministic, finite for every input and independent per input; handle levels\n unseen in training with a fallback. Before delivering, run\n `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n- `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n", "smoke.json": "[\n {\n \"name\": \"campaign_check selftest (hand-computed NDCG, tie rule, batch and probe checks)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"campaign_check ndcg on a tiny synthetic CSV from stdin\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"p\", \"--rel\", \"r\", \"--k\", \"3\", \"--group\", \"g\"],\n \"stdin\": \"p,r,g\\n0.9,1,x\\n0.8,0,x\\n0.7,1,x\\n0.5,0,y\\n0.4,2,y\\n0.3,1,y\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check ndcg rejects an unknown column\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"nope\", \"--rel\", \"r\", \"--k\", \"2\"],\n \"stdin\": \"p,r\\n1,1\\n2,0\\n\",\n \"expect_exit\": 2,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch accepts a clean round\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw2\\nw3\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags duplicates and wrong size\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw1\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags a wrong header\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"2\", \"--id-column\", \"item_key\"],\n \"stdin\": \"id\\nA\\nB\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook selftest (block once after delivery, never loop, no-op otherwise)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\", \"--selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"stop_audit hook lets an already-active stop through\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"{\\\"hook_event_name\\\": \\\"Stop\\\", \\\"stop_hook_active\\\": true, \\\"session_id\\\": \\\"smoke\\\", \\\"transcript_path\\\": \\\"/nonexistent/t.jsonl\\\"}\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook ignores malformed event input\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"not json at all\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n \"stdin\": \"\",\n \"expect_exit\": 2,\n \"timeout\": 30\n }\n]\n", "tools/campaign_check.py": "#!/usr/bin/env python3\n\"\"\"Advisory checks for batched-oracle campaigns (stdlib only, never blocks anything).\n\nSubcommands\n batch check a round CSV before ordering it (header, size, duplicates, earlier orders, catalog)\n ndcg local NDCG@k with 1/log2(rank+1) discount, stable ties, ideal-ranking normalisation\n probe run final_model.js under node on sample inputs: finite, deterministic, order-independent\n selftest built-in hand-computed checks of the above\n\nExit codes: 0 ok, 1 problem found, 2 usage or I/O error.\n\"\"\"\nimport argparse\nimport csv\nimport io\nimport json\nimport math\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\n\n\ndef die(msg, code=2):\n sys.stderr.write(\"error: %s\\n\" % msg)\n sys.exit(code)\n\n\ndef read_text(path):\n if path == \"-\":\n return sys.stdin.read()\n try:\n with open(path, encoding=\"utf-8-sig\", newline=\"\") as fh:\n return fh.read()\n except OSError as exc:\n die(\"cannot read %s: %s\" % (path, exc))\n\n\ndef read_rows(path):\n text = read_text(path)\n if text.startswith(\"\"):\n text = text[1:]\n rows = list(csv.reader(io.StringIO(text)))\n rows = [r for r in rows if r and any(c.strip() for c in r)]\n if not rows:\n die(\"%s is empty\" % path)\n return rows[0], rows[1:]\n\n\ndef id_values(path, column):\n header, body = read_rows(path)\n idx = header.index(column) if column in header else 0\n return [r[idx].strip() for r in body if len(r) > idx]\n\n\n# ---------------------------------------------------------------- batch\ndef cmd_batch(args):\n header, body = read_rows(args.round_csv)\n problems, notes = [], []\n column = args.id_column or header[0]\n if len(header) != 1:\n problems.append(\"header has %d columns %r; the round file needs a single column\" % (len(header), header))\n if header[0] != column:\n problems.append(\"header is %r, expected %r\" % (header[0], column))\n ids = [r[0] for r in body]\n padded = [i for i in ids if i != i.strip()]\n if padded:\n problems.append(\"%d ids have leading/trailing whitespace, e.g. %r\" % (len(padded), padded[0]))\n ids = [i.strip() for i in ids]\n if any(len(r) != 1 for r in body):\n problems.append(\"some rows have more than one field\")\n if any(not i for i in ids):\n problems.append(\"blank ids present\")\n notes.append(\"header=%r rows=%d\" % (header, len(ids)))\n if args.size is not None and len(ids) != args.size:\n problems.append(\"size is %d, expected exactly %d\" % (len(ids), args.size))\n seen, dups = set(), []\n for i in ids:\n if i in seen and i not in dups:\n dups.append(i)\n seen.add(i)\n if dups:\n problems.append(\"%d duplicated ids inside the batch, e.g. %s\" % (len(dups), dups[:3]))\n prior = set()\n for p in args.previous or []:\n prior.update(id_values(p, column))\n overlap = sorted(seen & prior)\n if args.previous:\n notes.append(\"compared with %d earlier ids from %d files\" % (len(prior), len(args.previous)))\n if overlap:\n problems.append(\"%d ids were already ordered, e.g. %s\" % (len(overlap), overlap[:3]))\n if args.catalog:\n cat = set(id_values(args.catalog, args.catalog_column or column))\n notes.append(\"catalog has %d ids\" % len(cat))\n unknown = sorted(seen - cat)\n if unknown:\n problems.append(\"%d ids are not in the catalog, e.g. %s\" % (len(unknown), unknown[:3]))\n for n in notes:\n print(\"note: \" + n)\n for p in problems:\n print(\"PROBLEM: \" + p)\n print(\"batch %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- ndcg\ndef dcg(rels):\n return sum(r / math.log2(i + 2) for i, r in enumerate(rels))\n\n\ndef ndcg_at_k(preds, rels, k):\n \"\"\"Rank by prediction descending; equal predictions keep input order (stable sort).\"\"\"\n order = sorted(range(len(preds)), key=lambda i: -preds[i])\n got = dcg([rels[i] for i in order[:k]])\n ideal = dcg(sorted(rels, reverse=True)[:k])\n if ideal <= 0:\n return None\n return got / ideal\n\n\ndef cmd_ndcg(args):\n header, body = read_rows(args.csv)\n for c in [args.pred, args.rel] + ([args.group] if args.group else []):\n if c not in header:\n die(\"column %r not in header %r\" % (c, header))\n ip, ir = header.index(args.pred), header.index(args.rel)\n ig = header.index(args.group) if args.group else None\n preds, rels, groups = [], [], []\n for n, r in enumerate(body, 2):\n try:\n p, q = float(r[ip]), float(r[ir])\n except (ValueError, IndexError):\n die(\"row %d: non-numeric prediction or relevance\" % n)\n if math.isnan(p) or math.isnan(q) or math.isinf(p):\n die(\"row %d: NaN/inf value\" % n)\n preds.append(p)\n rels.append(q)\n groups.append(r[ig] if ig is not None else None)\n if any(q < 0 for q in rels):\n print(\"warning: negative relevance present; pass relevance already clipped at 0 if the metric does so\")\n k = args.k\n if k > len(preds):\n print(\"warning: k=%d exceeds %d rows\" % (k, len(preds)))\n val = ndcg_at_k(preds, rels, k)\n if val is None:\n print(\"warning: ideal DCG is 0 (no positive relevance); NDCG undefined, reported as 0\")\n val = 0.0\n print(\"ndcg@%d = %.6f (n=%d)\" % (k, val, len(preds)))\n if args.group:\n by = {}\n for i, g in enumerate(groups):\n by.setdefault(g, []).append(i)\n vals = []\n for g in sorted(by):\n v = ndcg_at_k([preds[i] for i in by[g]], [rels[i] for i in by[g]], k)\n vals.append(v)\n print(\" group %s: n=%d ndcg@%d=%s\" % (g, len(by[g]), k, \"undefined\" if v is None else \"%.4f\" % v))\n ok = [v for v in vals if v is not None]\n if ok:\n print(\"mean over groups (informational, not the pooled metric) = %.6f\" % (sum(ok) / len(ok)))\n if args.threshold is not None:\n print(\"threshold %.4f: %s\" % (args.threshold, \"PASS\" if val >= args.threshold else \"BELOW\"))\n return 0\n\n\n# ---------------------------------------------------------------- probe\nPROBE_JS = r\"\"\"\nconst fs = require('fs'), vm = require('vm');\nconst req = JSON.parse(fs.readFileSync(0, 'utf8'));\nfunction load() {\n const code = fs.readFileSync(req.model, 'utf8');\n const mod = { exports: {} };\n const sb = { module: mod, exports: mod.exports, console: { log() {}, error() {}, warn() {} } };\n vm.createContext(sb);\n vm.runInContext(code, sb, { timeout: 60000 });\n const f = (typeof sb.predict === 'function') ? sb.predict\n : (mod.exports && typeof mod.exports.predict === 'function') ? mod.exports.predict : null;\n if (!f) throw new Error('no predict function defined');\n return f;\n}\nfunction run(f, x) {\n try { return { v: f(x) }; } catch (e) { return { e: String(e && e.message || e) }; }\n}\nconst out = { load_error: null, results: [] };\ntry {\n const f1 = load(), f2 = load();\n const xs = req.inputs;\n const a = xs.map(x => run(f1, x));\n const b = xs.map(x => run(f1, x));\n const c = xs.map((x, i) => run(f2, xs[xs.length - 1 - i])).reverse();\n xs.forEach((x, i) => {\n const same = (p, q) => ('e' in p && 'e' in q) || ('v' in p && 'v' in q && Object.is(p.v, q.v));\n out.results.push({\n input: x, value: ('v' in a[i]) ? a[i].v : null, error: a[i].e || null,\n type: ('v' in a[i]) ? typeof a[i].v : null,\n finite: ('v' in a[i]) && typeof a[i].v === 'number' && Number.isFinite(a[i].v),\n repeat_same: same(a[i], b[i]), order_same: same(a[i], c[i])\n });\n });\n} catch (e) { out.load_error = String(e && e.message || e); }\nprocess.stdout.write(JSON.stringify(out));\n\"\"\"\n\nBUILTIN_EDGE = [\"\", \"?\"]\n\n\ndef cmd_probe(args):\n if not os.path.isfile(args.model):\n die(\"model file not found: %s\" % args.model)\n node = shutil.which(\"node\")\n if not node:\n die(\"node not found on PATH\")\n inputs = list(args.input or [])\n if args.inputs_file:\n inputs += [l.rstrip(\"\\r\\n\") for l in read_text(args.inputs_file).splitlines() if l.strip()]\n edge_must = []\n if args.edge_file:\n edge_must = [l.rstrip(\"\\r\\n\") for l in read_text(args.edge_file).splitlines() if l.strip()]\n if not inputs and not edge_must:\n die(\"give --input and/or --inputs-file (real-format sample inputs)\")\n all_inputs = inputs + edge_must + BUILTIN_EDGE\n kinds = [\"sample\"] * len(inputs) + [\"edge-required\"] * len(edge_must) + [\"edge-advisory\"] * len(BUILTIN_EDGE)\n payload = json.dumps({\"model\": os.path.abspath(args.model), \"inputs\": all_inputs})\n try:\n proc = subprocess.run([node, \"-e\", PROBE_JS], input=payload, capture_output=True,\n text=True, timeout=args.timeout)\n except subprocess.TimeoutExpired:\n print(\"PROBLEM: node timed out after %ss\" % args.timeout)\n return 1\n if proc.returncode != 0:\n print(\"PROBLEM: node failed: %s\" % proc.stderr.strip()[:500])\n return 1\n try:\n res = json.loads(proc.stdout)\n except ValueError:\n print(\"PROBLEM: unparseable probe output: %r\" % proc.stdout[:200])\n return 1\n if res.get(\"load_error\"):\n print(\"PROBLEM: model failed to load in an isolated context: %s\" % res[\"load_error\"])\n return 1\n problems = 0\n for r, kind in zip(res[\"results\"], kinds):\n bad = []\n if r[\"error\"]:\n bad.append(\"threw: %s\" % r[\"error\"])\n elif not r[\"finite\"]:\n bad.append(\"not a finite number (%r)\" % (r[\"value\"],))\n if not r[\"repeat_same\"]:\n bad.append(\"repeated call differs\")\n if not r[\"order_same\"]:\n bad.append(\"result depends on call order/fresh load\")\n if bad:\n tag = \"WARN\" if kind == \"edge-advisory\" else \"PROBLEM\"\n if kind != \"edge-advisory\":\n problems += 1\n print(\"%s [%s] %r: %s\" % (tag, kind, r[\"input\"], \"; \".join(bad)))\n vals = [r[\"value\"] for r in res[\"results\"][:len(inputs)] if r[\"finite\"]]\n if vals:\n print(\"sample outputs: n=%d min=%.6g max=%.6g distinct=%d\" % (len(vals), min(vals), max(vals), len(set(vals))))\n if len(set(vals)) == 1 and len(vals) > 1:\n print(\"WARN: every sample input got the same score; the ranking would be all ties\")\n print(\"probe %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- selftest\ndef cmd_selftest(_args):\n fails = []\n\n def check(name, cond):\n if not cond:\n fails.append(name)\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n\n # hand-computed: dcg = 1 + 0.5 = 1.5 ; ideal = 1 + 1/log2(3)\n v = ndcg_at_k([0.9, 0.8, 0.7], [1, 0, 1], 3)\n check(\"ndcg hand value\", abs(v - 1.5 / (1 + 1 / math.log2(3))) < 1e-12)\n # tie rule: equal scores keep evaluation order -> rels seen [0,1]\n v = ndcg_at_k([5, 5, 5], [0, 1, 1], 2)\n check(\"ndcg tie keeps input order\", abs(v - (1 / math.log2(3)) / (1 + 1 / math.log2(3))) < 1e-12)\n check(\"ndcg perfect ranking is 1\", abs(ndcg_at_k([3, 2, 1], [2, 1, 0], 2) - 1) < 1e-12)\n check(\"ndcg undefined without positives\", ndcg_at_k([1, 2], [0, 0], 2) is None)\n tmp = tempfile.mkdtemp(prefix=\"campchk_\")\n try:\n def w(name, text):\n p = os.path.join(tmp, name)\n with open(p, \"w\") as fh:\n fh.write(text)\n return p\n prev = w(\"prev.csv\", \"id,y\\na,1\\nb,2\\n\")\n ns = argparse.Namespace\n import contextlib\n sink = io.StringIO()\n with contextlib.redirect_stdout(sink):\n ok = cmd_batch(ns(round_csv=w(\"r1.csv\", \"id\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n dup = cmd_batch(ns(round_csv=w(\"r2.csv\", \"id\\nc\\nc\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n old = cmd_batch(ns(round_csv=w(\"r3.csv\", \"id\\na\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n short = cmd_batch(ns(round_csv=w(\"r4.csv\", \"id\\nc\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n hdr = cmd_batch(ns(round_csv=w(\"r5.csv\", \"wrong\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n check(\"batch accepts clean round\", ok == 0)\n check(\"batch flags in-batch duplicate\", dup == 1)\n check(\"batch flags earlier order\", old == 1)\n check(\"batch flags wrong size\", short == 1)\n check(\"batch flags wrong header\", hdr == 1)\n if shutil.which(\"node\"):\n model = w(\"m.js\", \"function predict(x){ return x.length * 0.5; }\\n\")\n bad = w(\"bad.js\", \"function predict(x){ return x === '' ? NaN : Math.random(); }\\n\")\n with contextlib.redirect_stdout(sink):\n good = cmd_probe(ns(model=model, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n notgood = cmd_probe(ns(model=bad, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n check(\"probe accepts a finite deterministic model\", good == 0)\n check(\"probe rejects a random model\", notgood == 1)\n else:\n print(\"skip probe checks (node not found)\")\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\ndef main():\n ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n sub = ap.add_subparsers(dest=\"cmd\", required=True)\n\n b = sub.add_parser(\"batch\", help=\"check a round CSV before `asb run` (use '-' for stdin)\")\n b.add_argument(\"round_csv\")\n b.add_argument(\"--size\", type=int, help=\"exact number of rows required\")\n b.add_argument(\"--id-column\", help=\"expected header, e.g. the single column name from the task\")\n b.add_argument(\"--previous\", nargs=\"*\", help=\"earlier round or results CSVs whose ids may not repeat\")\n b.add_argument(\"--catalog\", help=\"CSV listing orderable ids\")\n b.add_argument(\"--catalog-column\", help=\"id column in the catalog (default: --id-column)\")\n b.set_defaults(fn=cmd_batch)\n\n n = sub.add_parser(\"ndcg\", help=\"local NDCG@k (use '-' for stdin)\")\n n.add_argument(\"csv\")\n n.add_argument(\"--pred\", required=True, help=\"prediction column\")\n n.add_argument(\"--rel\", required=True, help=\"relevance column (already clipped/normalised as the task defines)\")\n n.add_argument(\"--k\", type=int, required=True)\n n.add_argument(\"--group\", help=\"optional group column; adds per-group NDCG as a diagnostic\")\n n.add_argument(\"--threshold\", type=float, help=\"optional pass level to compare against\")\n n.set_defaults(fn=cmd_ndcg)\n\n p = sub.add_parser(\"probe\", help=\"run final_model.js under node on sample inputs\")\n p.add_argument(\"model\")\n p.add_argument(\"--input\", action=\"append\", help=\"sample input string (repeatable)\")\n p.add_argument(\"--inputs-file\", help=\"file with one sample input per line\")\n p.add_argument(\"--edge-file\", help=\"file with one extra input per line that must also score finitely\")\n p.add_argument(\"--timeout\", type=int, default=120)\n p.set_defaults(fn=cmd_probe)\n\n s = sub.add_parser(\"selftest\", help=\"hand-computed self checks\")\n s.set_defaults(fn=cmd_selftest)\n\n args = ap.parse_args()\n sys.exit(args.fn(args))\n\n\nif __name__ == \"__main__\":\n main()\n", "tools/stop_audit.py": "#!/usr/bin/env python3\n\"\"\"One-shot stop hook: after a campaign model was delivered and time remains, ask for a held-out audit.\n\nReads the stop event JSON on stdin. Exit 0 lets the agent stop. Exit 2 blocks the stop once and feeds the\naudit request (stderr) back to the agent. It never blocks twice per session, never blocks when the event says\na stop hook is already active, and never blocks outside a delivered-campaign context.\nStdlib only. Run with --selftest for a built-in check using temporary fixtures.\n\"\"\"\nimport json\nimport os\nimport re\nimport sys\nimport tempfile\nimport time\nfrom datetime import datetime, timezone\n\nMARKER_ENV = \"RSI_AUDIT_MARKER_DIR\"\nBUDGET_ENV = \"RSI_AUDIT_BUDGET_S\"\nDEFAULT_BUDGET_S = 28800\nMIN_REMAINING_S = 1800\nDELIVERED = \"deliverable accepted\"\nMODEL_NAME = \"final_model.js\"\n\nMESSAGE = \"\"\"Before stopping: a model has been delivered and time remains. The latest submission is the one scored, so\nre-delivery is allowed. Spend a short audit on the held-out ranking gate. Everything you measured so far was\nchosen by your own model, so a score over your own picks is biased upward.\n\n1. Build a held-out stand-in. Use the broad first round (or the least model-selected measurements), or hold out whole\n groups of the ordered items (group the folds by the key that distinguishes held-out items). Fit on the rest, keeping\n the seed data in. Score with the official relevance transform exactly as the task states it:\n python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ]\n2. Compare the delivered model with at least one structurally different alternative on that same stand-in: a\n lower-dimensional or smoother model (fewer features, stronger shrinkage, log or clipped target), and the same\n model with versus without the seed data. Run long fits in the background (nohup ... &) and poll the log, because\n a single command is cut off after about two minutes.\n3. Re-deliver only if an alternative beats the delivered model by a clear margin (state it, e.g. at least 0.05 NDCG)\n and the gap is not one or two items. Then: cp final_model.js final_model.prev.js, write the new file,\n python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt,\n asb deliver --validate-only, and only then the real deliver.\n If nothing wins by the margin, keep the current model: do not re-deliver, and report the stand-in scores.\nThis audit happens once; state the numbers you compared in the final message.\"\"\"\n\n\ndef parse_ts(value):\n try:\n if isinstance(value, (int, float)):\n return float(value) / (1000.0 if value > 1e11 else 1.0)\n dt = datetime.fromisoformat(str(value).replace(\"Z\", \"+00:00\"))\n if dt.tzinfo is None:\n dt = dt.replace(tzinfo=timezone.utc)\n return dt.timestamp()\n except (ValueError, TypeError):\n return None\n\n\ndef read_transcript(path):\n try:\n with open(path, encoding=\"utf-8\", errors=\"replace\") as fh:\n return fh.read()\n except OSError:\n return None\n\n\ndef first_timestamp(text):\n for line in text.splitlines()[:50]:\n m = re.search(r'\"timestamp\"\\s*:\\s*(\"[^\"]+\"|[0-9.]+)', line)\n if m:\n ts = parse_ts(json.loads(m.group(1)))\n if ts:\n return ts\n return None\n\n\ndef model_exists(cwd):\n for d in (cwd, os.getcwd(), \"/app\"):\n if d and os.path.isfile(os.path.join(d, MODEL_NAME)):\n return True\n return False\n\n\ndef marker_path(session):\n base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_stop_audit\")\n safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session or \"nosession\")[:80]\n return base, os.path.join(base, safe + \".done\")\n\n\ndef decide(event):\n \"\"\"Return True if the stop should be blocked for the audit.\"\"\"\n if event.get(\"stop_hook_active\"):\n return False\n cwd = event.get(\"cwd\") or \"\"\n text = None\n tp = event.get(\"transcript_path\")\n if tp:\n text = read_transcript(tp)\n if text is not None:\n if DELIVERED not in text or MODEL_NAME not in text:\n return False\n start = first_timestamp(text)\n if start is not None:\n try:\n budget = float(os.environ.get(BUDGET_ENV, DEFAULT_BUDGET_S))\n except ValueError:\n budget = DEFAULT_BUDGET_S\n if time.time() - start > budget - MIN_REMAINING_S:\n return False\n elif not model_exists(cwd):\n return False\n return True\n\n\ndef main():\n if \"--selftest\" in sys.argv:\n return selftest()\n try:\n event = json.loads(sys.stdin.read() or \"{}\")\n if not isinstance(event, dict):\n return 0\n except ValueError:\n return 0\n try:\n if not decide(event):\n return 0\n base, marker = marker_path(str(event.get(\"session_id\") or \"\"))\n if os.path.exists(marker):\n return 0\n os.makedirs(base, exist_ok=True)\n with open(marker, \"w\") as fh:\n fh.write(str(time.time()))\n except OSError:\n return 0\n sys.stderr.write(MESSAGE + \"\\n\")\n return 2\n\n\ndef selftest():\n import subprocess\n import shutil\n tmp = tempfile.mkdtemp(prefix=\"stopaudit_\")\n fails = []\n try:\n env = dict(os.environ, **{MARKER_ENV: os.path.join(tmp, \"m\")})\n\n def run(event):\n p = subprocess.run([sys.executable, os.path.abspath(__file__)], input=json.dumps(event),\n capture_output=True, text=True, env=env, timeout=30)\n return p.returncode\n\n def check(name, cond):\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n if not cond:\n fails.append(name)\n\n now = datetime.now(timezone.utc).isoformat()\n good = os.path.join(tmp, \"good.jsonl\")\n with open(good, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"asb deliver final_model.js\"}) + \"\\n\")\n fh.write(json.dumps({\"text\": \"local validation passed\\n\" + DELIVERED + \" (submission 1)\"}) + \"\\n\")\n undelivered = os.path.join(tmp, \"undelivered.jsonl\")\n with open(undelivered, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"writing final_model.js\"}) + \"\\n\")\n late = os.path.join(tmp, \"late.jsonl\")\n with open(late, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": \"2020-01-01T00:00:00Z\", \"text\": \"final_model.js \" + DELIVERED}) + \"\\n\")\n check(\"active stop hook never blocks\", run({\"stop_hook_active\": True, \"transcript_path\": good, \"session_id\": \"a\"}) == 0)\n check(\"no delivery in transcript does not block\", run({\"transcript_path\": undelivered, \"session_id\": \"b\"}) == 0)\n check(\"out of time does not block\", run({\"transcript_path\": late, \"session_id\": \"c\"}) == 0)\n check(\"garbage input does not block\", subprocess.run([sys.executable, os.path.abspath(__file__)], input=\"not json\",\n capture_output=True, text=True, env=env).returncode == 0)\n check(\"delivered with time left blocks once\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 2)\n check(\"second stop in same session passes\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 0)\n os.makedirs(os.path.join(tmp, \"w\"))\n open(os.path.join(tmp, \"w\", MODEL_NAME), \"w\").close()\n check(\"no transcript but model file present blocks\", run({\"cwd\": os.path.join(tmp, \"w\"), \"session_id\": \"f\"}) == 2)\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, "candidate_sha256": "c92c6c3bb603e3d2914581512abc2b90b9fcfa709740007a7fb630078fb6271b", "parent_sha256": "b232ad942c6f44e92710c1bf6a8c148a76d813fdc9a918bc1c68f88066815528", "role_cost": { "calls": 4, "list_price_estimate_usd": 1.13 }, "trials": [ { "task": "propylene-active-learning", "trial": "propylene-active-learning__BtrjPXY", "job": "r04-screen", "reward": 0, "model_audit_ok": true, "harness_interventions": 1, "tokens": 1038082, "list_price_estimate_usd": 0.5381166000000001 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__Z46VbQp", "job": "r04-screen", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 1421519, "list_price_estimate_usd": 0.6538411999999999 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__aCihm6k", "job": "r04-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 1, "tokens": 746226, "list_price_estimate_usd": 0.3819516 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__qEzUmRt", "job": "r04-topup", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 628110, "list_price_estimate_usd": 0.34073840000000005 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__abD62aX", "job": "r04-topup", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 1544820, "list_price_estimate_usd": 0.7890583999999999 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__xpuprVU", "job": "r04-topup", "reward": 1, "model_audit_ok": true, "harness_interventions": 1, "tokens": 1291307, "list_price_estimate_usd": 0.6289614 } ] }, { "round": 5, "title": "Hook before each experiment batch", "summary": "A pre_tool hook on asb run that blocks once per round with a data-derived digest and asks for a hit ledger before spending the round.", "observation": "Passed the screen 2/2, then failed all 4 top-up trials (2/6). Not accepted. The critic made the proposer remove a supporting instructions note as over budget, arguably too strict.", "accepted": false, "reason": "unresolved_in_noise_band", "recorded_offline": false, "score": 0.3333333333333333, "trials_run": 6, "task_rates": { "propylene-active-learning": 0.3333333333333333, "suzuki-condition-screen": 0.3333333333333333 }, "p_superior": 0.2960372960372962, "incumbent_score": 0.5, "tokens_per_trial": 1483015.6666666667, "complexity": 1000, "edit_budget": 1, "components_changed": [ "hook", "tool" ], "changed_files": [ "hooks.json", "smoke.json", "tools/round_gate.py" ], "plan": { "problem": "The parent passes about 50% overall (P 1/3, S 2/3 in round 3). Propylene (P) is stuck at 1/3 across rounds 1, 3 and 4. Two of the propylene failures were discovery shortfalls: LGkcbM5 with 5 hits and aCihm6k with 6 hits against the 7 needed. Both ran a one-slot-per-element sweep in round 1 and only exploited in round 2 or 3. In Suzuki (S) every agent said it could not count standouts because the cutoffs are hidden. The only attempt to fix this, the round-4 \"discovery ledger\" instructions section, was followed literally by 1 of 6 trials, so it never had a chance to work. Hooks are the only channel with 12/12 compliance, but the parent's only hook fires after delivery, which is too late to affect discovery. I can't tell which gate failed in the other four P failures or the S failures, so this plan targets only the discovery-timing failures and the missing hit estimate.", "edits": [ { "component": "hook", "hypothesis": "A once-per-round pre_tool hook on the Bash command `asb run` will make the agent state a hit ledger before it spends rounds 2 and later, and rounds 2+ will then shift toward neighbours of the best results. The hook is tools/round_gate.py, registered in hooks.json as a pre_tool entry with matcher Bash. It reads the event JSON and acts only if the command contains `asb run`. It then globs round_*_results.csv in the cwd, and passes only if there is at least one prior results file. The first time per session and per prior-round count, it exits 2. It cannot know the threshold, so its stderr message gives only numbers computed from the data. For each results file it reports the row count, the top 5 values, and how many values lie in the top 10% of all values observed so far. If the id column has structured fields, it adds one line per field level, giving the best value seen and its count. The message asks the agent to write three numbers: hits so far against the task's stated discovery target, hits still needed, and hits needed per remaining round. If the needed rate exceeds what the last round delivered, at least about 60% of the next batch should be unordered near neighbours of the best items, changing one factor at a time, plus the model's top picks. Levels whose best result is near the floor should be dropped. For group-relative targets with hidden cutoffs, use the top few percent of each group's observed values as the hit proxy. The second `asb run` attempt in the same state passes, so a rejected-then-retried request is never blocked twice and no round is lost. Falsifiable: in post-change trajectories the ledger numbers appear before rounds 2 and 3 in at least 5 of 6 trials, up from 1 of 6 under the round-4 text. P trials reach at least 7 hits in at least 2 of 3. S pass rate stays at or above the parent's 2/3. It is refuted if P stays at or below 1/3 with a shortfall below 7 hits in any failing trial despite the ledger being written, or if any S trial regresses because of over-exploitation.", "files": [ "tools/round_gate.py", "hooks.json", "smoke.json" ] } ], "predictions": { "tasks": { "propylene-active-learning": "improve", "suzuki-condition-screen": "same" }, "token_effect": "higher" }, "why_not_simpler": "The simpler lever, more instruction text, was tried in round 4 and followed in 1 of 6 trials, so another paragraph is unlikely to change behaviour. Deleting the stop hook would save tokens but would not address the discovery failures. It would also drop the audit that preserved known-good models in passing trials such as k3Tgise, qEzUmRt and the passing S trials. A larger tool such as an automatic batch designer would be a rewrite, and it would encode design choices the evidence cannot yet support. This plan adds one small non-destructive nudge at the only moment that matters, just before an irreversible round is spent. The hook computes only generic descriptive numbers, so it contains no task identity, thresholds or answers.", "regression_risk": "The risks are listed below. (1) A blocked `asb run` costs one extra step per round, and the second attempt passes. The hook must never block twice for the same state, must exit 0 on malformed input, and must exit 0 when the command is not `asb run`. It must also exit 0 when the command is a repeat of the latest accepted batch, a harmless re-run. Smoke tests must cover these cases offline. (2) Over-exploitation could hurt the held-out ranking gate by shrinking coverage. This is the largest risk. Round-4 S passes were exploit-heavy (36 of 48 in djh5T3X), so S should be fine, but the message says explicitly to keep some under-sampled coverage. (3) The agent may over-trust the top-10% proxy. The message labels it advisory and noisy, and does not set the number of hits. (4) The hook does nothing about FM-A (ranking-gate failures: 79vBo6X, GuNbBG9, DNstK4u, BtrjPXY), so P may still fail there. Because the attribution of the earlier gain is weak, small differences in either direction are within the roughly ±35 point noise of 3 trials per task." }, "attempts": [ { "attempt": 0, "gate_passed": false, "gate_errors": [ "tools/round_gate.py: literal_composition" ], "critic": null }, { "attempt": 1, "gate_passed": false, "gate_errors": [ "smoke:round_gate hook selftest (block once per round state, retry passes, repeats and other commands pass) failed (exit 1): " ], "critic": null }, { "attempt": 2, "gate_passed": true, "gate_errors": [], "critic": { "approved": false, "reasons": [ "The change goes beyond the declared plan. plan.json declares one hook edit touching only tools/round_gate.py, hooks.json and smoke.json, and search-directive.json sets edit_budget 1. The candidate also edits instructions.md with a new bullet that describes the hook and asks for a ledger. That is a second mechanism on the instructions channel. The plan's own reasoning says that channel was followed in only 1 of 6 trials, so the edit also muddies attribution for the hook hypothesis.", "The pre-evaluation gate rejected this candidate. repair-feedback.txt says the smoke test 'round_gate hook selftest' failed with exit 1 and no output. The candidate round_gate.py is identical to the diff under review, with 353 lines and the same selftest. I traced the selftest by hand and found no definite bug, but I could not run it. Nothing shows the failure was repaired, so the candidate cannot be accepted as passing the gate.", "The hook message prescribes 'at least about 60% of the next batch' as near neighbours. The existing instructions say the split is 'guidance, not fixed percentages'. This is a fixed allocation rule baked into a blocking hook, and it risks over-exploitation. The plan itself names over-exploitation as the largest risk, because it could starve coverage for the held-out ranking gate. The proxy rule of top few percent per group is also specific to the two known campaigns' hit definitions rather than a general principle.", "Risk of stalling is bounded: each state blocks at most once and the retry passes. Even so, the hook's marker files persist in the shared temp directory, keyed by session id and results path. If session_id is missing they fall back to the shared key 'nosession', so a marker could suppress or misfire the block in a later episode on the same machine. The hook should not depend on state that outlives the episode." ], "required_fixes": [ "Remove the instructions.md edit so the candidate matches the single declared hook edit (round_gate.py, hooks.json, smoke.json). Alternatively, amend the plan to declare it, but that would exceed edit_budget 1.", "Fix the round_gate.py --selftest so it exits 0 in the offline gate, and re-run the smoke suite. The gate showed no stdout or stderr on failure, so make failures print which check failed. Confirm the repaired selftest passes before resubmitting.", "Drop the fixed '60%' allocation from the hook message. Use a qualitative nudge that fits the existing 'guidance, not fixed percentages' stance, or state it as an explicit advisory with the coverage requirement given equal weight.", "Make the marker state episode-local. Key it to a per-process or per-session value only (for example a parent-pid or session-scoped directory), with no fallback to a shared 'nosession' key. Alternatively, derive the once-per-round state from the results files themselves, for example by hashing the contents of the latest results file." ] } }, { "attempt": 3, "gate_passed": true, "gate_errors": [], "critic": { "approved": true, "reasons": [ "The change matches the declared plan. It adds one pre_tool hook (tools/round_gate.py), its hooks.json entry and smoke tests. instructions.md and the other tools are unchanged. The diff is 393 lines for round_gate.py, which matches the candidate file.", "There is no task-specific hardcoding or dispatch. The hook reacts to `asb run`, which the parent's instructions already name, and to the agent's own round_*_results.csv files. It computes only descriptive statistics: top values, a top-10% line, and the best value per factor level. It contains no thresholds, designs, answers or task identity. The plan's example numbers are not in the code.", "It uses no private truth, grader files, network, model or provider. The only files it reads are the agent's own round results, the batch file named in the command, and seed CSVs under ./data that the agent already has.", "The stall risk is bounded. Each results state blocks at most once, using a hash-keyed marker in the temp directory. The retry passes, as do repeats of the latest batch, non-`asb run` commands, a missing session id and malformed input. Any exception exits 0. There are no loops, the hook timeout is 30 s, and the selftest timeout is 120 s.", "The marker files are keyed to the session and the data hash, and live in the temp directory. They are not memory or skills, and they carry no task data into later episodes.", "The edit has a real behavioural effect. It inserts a ledger prompt before irreversible rounds, which is the point the parent's post-delivery stop hook cannot reach. The guidance is generic and should still help an unfamiliar batched-discovery task. It is also explicitly advisory and keeps coverage in mind. The smoke tests cover the pass, block and malformed-input paths offline." ], "required_fixes": [] } } ], "diff": "--- parent/hooks.json\n+++ candidate/hooks.json\n@@ -1,4 +1,12 @@\n {\n+ \"pre_tool\": [\n+ {\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n+ \"matcher\": \"Bash\",\n+ \"timeout\": 30\n+ }\n+ ],\n \"stop\": [\n {\n \"command\": \"python3\",\n--- parent/smoke.json\n+++ candidate/smoke.json\n@@ -72,6 +72,38 @@\n \"timeout\": 30\n },\n {\n+ \"name\": \"round_gate hook selftest (block once per round state, retry passes, repeats and other commands pass)\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/round_gate.py\", \"--selftest\"],\n+ \"stdin\": \"\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 120\n+ },\n+ {\n+ \"name\": \"round_gate hook ignores a command that is not asb run\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n+ \"stdin\": \"{\\\"hook_event_name\\\": \\\"PreToolUse\\\", \\\"tool_name\\\": \\\"Bash\\\", \\\"tool_input\\\": {\\\"command\\\": \\\"ls\\\"}, \\\"cwd\\\": \\\"/nonexistent\\\", \\\"session_id\\\": \\\"smoke\\\"}\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"round_gate hook lets asb run through when no earlier round exists\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n+ \"stdin\": \"{\\\"hook_event_name\\\": \\\"PreToolUse\\\", \\\"tool_name\\\": \\\"Bash\\\", \\\"tool_input\\\": {\\\"command\\\": \\\"asb run /nonexistent/round_1.csv --output /nonexistent/round_1_results.csv\\\"}, \\\"cwd\\\": \\\"/nonexistent\\\", \\\"session_id\\\": \\\"smoke\\\"}\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n+ \"name\": \"round_gate hook ignores malformed event input\",\n+ \"command\": \"python3\",\n+ \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n+ \"stdin\": \"not json at all\",\n+ \"expect_exit\": 0,\n+ \"timeout\": 30\n+ },\n+ {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n--- parent/tools/round_gate.py\n+++ candidate/tools/round_gate.py\n@@ -0,0 +1,393 @@\n+#!/usr/bin/env python3\n+\"\"\"Once-per-round pre_tool hook on Bash: before an irreversible `asb run`, ask for a hit ledger.\n+\n+Reads the pre_tool event JSON on stdin. It acts only when the Bash command contains `asb run` and at least one\n+earlier round_*_results.csv exists. It then summarises the results so far (descriptive numbers only, no thresholds\n+of its own) and exits 2 once for that state, feeding the summary and a request back to the agent on stderr.\n+The retried `asb run` in the same state passes, so no round is ever lost. Exit 0 on anything unexpected, on\n+commands that are not `asb run`, and when the batch is a repeat of the latest accepted one.\n+Stdlib only. Run with --selftest for a built-in check using temporary fixtures.\n+\"\"\"\n+import csv\n+import glob\n+import hashlib\n+import json\n+import math\n+import os\n+import re\n+import shlex\n+import sys\n+import tempfile\n+\n+MARKER_ENV = \"RSI_GATE_MARKER_DIR\"\n+RUN_RE = re.compile(r\"\\basb\\s+run\\b\")\n+ROUND_RE = re.compile(r\"round_(\\d+)_results\\.csv$\")\n+SEP_RE = re.compile(r\"[-_:;/ ]+\")\n+NUM_RE = re.compile(r\"^[+-]?\\d*\\.?\\d+$\")\n+MAX_LEVEL_FIELDS = 6\n+MAX_LEVELS = 12\n+MAX_CARDINALITY = 30\n+\n+\n+def to_float(text):\n+ try:\n+ v = float(text)\n+ except (TypeError, ValueError):\n+ return None\n+ return v if math.isfinite(v) else None\n+\n+\n+def run_args(command):\n+ \"\"\"Tokens following `asb run` in the command, or [] if they cannot be parsed.\"\"\"\n+ m = RUN_RE.search(command)\n+ if not m:\n+ return []\n+ tail = command[m.end():]\n+ tail = re.split(r\"&&|\\|\\||;|\\n\", tail)[0]\n+ try:\n+ return shlex.split(tail)\n+ except ValueError:\n+ return tail.split()\n+\n+\n+def candidate_dirs(event, command):\n+ dirs = []\n+ try:\n+ here = os.getcwd()\n+ except OSError:\n+ here = None\n+ for d in (event.get(\"cwd\"), here):\n+ if d and isinstance(d, str) and os.path.isdir(d):\n+ dirs.append(os.path.realpath(d))\n+ for m in re.finditer(r\"\\bcd\\s+([^\\s;&|]+)\", command):\n+ p = os.path.expanduser(m.group(1).strip(\"'\\\"\"))\n+ if not os.path.isabs(p) and dirs:\n+ p = os.path.join(dirs[0], p)\n+ if os.path.isdir(p):\n+ dirs.insert(0, os.path.realpath(p))\n+ for tok in run_args(command):\n+ if tok.startswith(\"-\"):\n+ continue\n+ base = os.path.dirname(tok)\n+ if base:\n+ p = base if os.path.isabs(base) else os.path.join(dirs[0] if dirs else \"\", base)\n+ if os.path.isdir(p):\n+ dirs.append(os.path.realpath(p))\n+ seen, out = set(), []\n+ for d in dirs:\n+ if d not in seen:\n+ seen.add(d)\n+ out.append(d)\n+ return out\n+\n+\n+def find_results(dirs):\n+ \"\"\"round number -> path, keeping the first directory that has each round.\"\"\"\n+ found = {}\n+ for d in dirs:\n+ for path in glob.glob(os.path.join(d, \"round_*_results.csv\")):\n+ m = ROUND_RE.search(os.path.basename(path))\n+ if m and int(m.group(1)) not in found:\n+ found[int(m.group(1))] = path\n+ return [found[k] for k in sorted(found)]\n+\n+\n+def read_results(path):\n+ \"\"\"Return (id_column_name, [(id, value)]) using the first column as id and the last numeric column as value.\"\"\"\n+ try:\n+ with open(path, newline=\"\", encoding=\"utf-8\", errors=\"replace\") as fh:\n+ rows = list(csv.reader(fh))\n+ except OSError:\n+ return None, []\n+ if len(rows) < 2:\n+ return None, []\n+ header, body = rows[0], [r for r in rows[1:] if r]\n+ best_col = None\n+ for col in range(len(header) - 1, 0, -1):\n+ good = sum(1 for r in body if col < len(r) and to_float(r[col]) is not None)\n+ if body and good >= 0.8 * len(body):\n+ best_col = col\n+ break\n+ if best_col is None:\n+ return header[0], []\n+ out = []\n+ for r in body:\n+ v = to_float(r[best_col]) if best_col < len(r) else None\n+ if v is not None:\n+ out.append((r[0], v))\n+ return header[0], out\n+\n+\n+def read_ids(path):\n+ try:\n+ with open(path, newline=\"\", encoding=\"utf-8\", errors=\"replace\") as fh:\n+ rows = list(csv.reader(fh))\n+ except OSError:\n+ return None\n+ return [r[0] for r in rows[1:] if r]\n+\n+\n+def is_repeat(args, dirs, latest_ids):\n+ \"\"\"True when the batch file named in the command has exactly the ids of the latest accepted round.\"\"\"\n+ for tok in args:\n+ if tok.startswith(\"-\") or not tok.lower().endswith(\".csv\"):\n+ continue\n+ paths = [tok] if os.path.isabs(tok) else [os.path.join(d, tok) for d in dirs]\n+ for p in paths:\n+ if os.path.isfile(p):\n+ ids = read_ids(p)\n+ return ids is not None and bool(ids) and set(ids) == latest_ids\n+ return False\n+ return False\n+\n+\n+def load_catalog(dirs, id_name):\n+ \"\"\"Optional join: the largest csv under /data with the id column and low-cardinality factor columns.\"\"\"\n+ best = None\n+ for d in dirs:\n+ for path in glob.glob(os.path.join(d, \"data\", \"*.csv\")):\n+ try:\n+ with open(path, newline=\"\", encoding=\"utf-8\", errors=\"replace\") as fh:\n+ rows = list(csv.DictReader(fh))\n+ except OSError:\n+ continue\n+ if rows and id_name in rows[0] and len(rows[0]) > 2 and (best is None or len(rows) > len(best)):\n+ best = rows\n+ if not best:\n+ return {}\n+ fields = []\n+ for name in best[0]:\n+ if name == id_name:\n+ continue\n+ levels = {r.get(name) for r in best}\n+ if 2 <= len(levels) <= MAX_CARDINALITY:\n+ fields.append(name)\n+ return {r[id_name]: {f: r.get(f) for f in fields} for r in best if r.get(id_name) is not None}\n+\n+\n+def factors_for(item_id, catalog):\n+ \"\"\"Map field name -> list of levels for this item, derived from the id itself or the catalog join.\"\"\"\n+ if item_id in catalog:\n+ return {k: [v] for k, v in catalog[item_id].items() if v not in (None, \"\")}\n+ if \"|\" in item_id:\n+ return {\"field%d\" % (i + 1): [p] for i, p in enumerate(item_id.split(\"|\"))}\n+ parts = [p for p in SEP_RE.split(item_id) if p]\n+ if len(parts) > 1:\n+ words = [p for p in parts if not NUM_RE.match(p)]\n+ if words:\n+ return {\"part\": words}\n+ return {}\n+\n+\n+def summarise(paths, dirs):\n+ per_file, allrows = [], []\n+ id_name = None\n+ for p in paths:\n+ name, rows = read_results(p)\n+ id_name = id_name or name\n+ per_file.append((os.path.basename(p), rows))\n+ allrows.extend(rows)\n+ if not allrows:\n+ return None\n+ vals = sorted(v for _, v in allrows)\n+ cut = vals[min(len(vals) - 1, int(math.floor(0.9 * len(vals))))]\n+ lines = [\"Results so far: %d items over %d round(s); top-10%% line of all observed values = %.4g.\"\n+ % (len(allrows), len(per_file), cut)]\n+ for fname, rows in per_file:\n+ vs = sorted((v for _, v in rows), reverse=True)\n+ top = \", \".join(\"%.4g\" % v for v in vs[:5])\n+ n_top = sum(1 for v in vs if v >= cut)\n+ lines.append(\" %s: %d rows, top 5 = [%s], %d at or above the top-10%% line\" % (fname, len(rows), top, n_top))\n+ catalog = load_catalog(dirs, id_name) if id_name else {}\n+ stats = {}\n+ for item_id, v in allrows:\n+ for field, levels in factors_for(item_id, catalog).items():\n+ for lv in set(levels):\n+ n, best = stats.setdefault(field, {}).get(lv, (0, None))\n+ stats[field][lv] = (n + 1, v if best is None else max(best, v))\n+ shown = 0\n+ for field in sorted(stats):\n+ levels = {lv: nb for lv, nb in stats[field].items() if nb[0] < len(allrows)}\n+ if len(levels) < 2 or len(levels) > MAX_CARDINALITY or shown >= MAX_LEVEL_FIELDS:\n+ continue\n+ shown += 1\n+ ranked = sorted(levels.items(), key=lambda kv: (-kv[1][1], kv[0]))[:MAX_LEVELS]\n+ lines.append(\" by %s (level: best value, n tried): %s\" % (\n+ field, \"; \".join(\"%s: %.4g, %d\" % (lv, b, n) for lv, (n, b) in ranked)))\n+ return \"\\n\".join(lines)\n+\n+\n+MESSAGE = \"\"\"Before the next irreversible round: write a short hit ledger, then re-issue the same `asb run`.\n+(This check appears once per round; the retry goes through unchanged and nothing is lost.)\n+\n+{summary}\n+\n+Ledger, in your reply, with real numbers:\n+ 1. Hits so far against the task's stated discovery target (use the task's own threshold or rule).\n+ 2. Hits still needed, and hits needed per remaining round.\n+ 3. Hits the last round actually delivered.\n+If the needed rate is above what the last round delivered, lean the next batch toward unordered near neighbours of\n+the best items so far (changing one factor at a time) plus the model's top picks, and drop levels whose best result\n+is near the floor. Coverage of under-sampled levels matters just as much, because the held-out ranking gate needs it:\n+this is guidance, not a fixed split. If the needed rate is at or below the delivered rate, keep your current split.\n+If the discovery target is relative to a group whose cutoffs are hidden, count as a hit proxy the top few percent\n+of each group's own observed values. The proxy is advisory and noisy; do not treat a value near a line as a hit.\"\"\"\n+\n+\n+def state_tag(paths):\n+ \"\"\"Hash of the contents of every results file, so the once-per-round state follows the data itself.\"\"\"\n+ h = hashlib.sha1()\n+ for p in paths:\n+ h.update(os.path.basename(p).encode(\"utf-8\", \"replace\"))\n+ try:\n+ with open(p, \"rb\") as fh:\n+ h.update(fh.read())\n+ except OSError:\n+ pass\n+ return h.hexdigest()[:16]\n+\n+\n+def marker_path(session, tag):\n+ base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_round_gate\")\n+ safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session)[:60]\n+ return base, os.path.join(base, \"%s_%s.done\" % (safe, tag))\n+\n+\n+def evaluate(event):\n+ \"\"\"Return the message to block with, or None to let the call through.\"\"\"\n+ command = \"\"\n+ ti = event.get(\"tool_input\")\n+ if isinstance(ti, dict):\n+ command = str(ti.get(\"command\") or \"\")\n+ if not command or not RUN_RE.search(command):\n+ return None\n+ dirs = candidate_dirs(event, command)\n+ paths = find_results(dirs)\n+ if not paths:\n+ return None\n+ summary = summarise(paths, dirs)\n+ if summary is None:\n+ return None\n+ _, last_rows = read_results(paths[-1])\n+ if is_repeat(run_args(command), dirs, {i for i, _ in last_rows}):\n+ return None\n+ session = str(event.get(\"session_id\") or \"\")\n+ if not session:\n+ return None # no per-session key means no safe place for once-per-round state: never block\n+ base, marker = marker_path(session, state_tag(paths))\n+ if os.path.exists(marker):\n+ return None\n+ os.makedirs(base, exist_ok=True)\n+ with open(marker, \"w\") as fh:\n+ fh.write(\"1\")\n+ return MESSAGE.format(summary=summary)\n+\n+\n+def handle(raw):\n+ \"\"\"(exit code, stderr text) for one raw event; anything unexpected lets the call through.\"\"\"\n+ try:\n+ event = json.loads(raw or \"{}\")\n+ if not isinstance(event, dict):\n+ return 0, \"\"\n+ msg = evaluate(event)\n+ except Exception:\n+ return 0, \"\"\n+ return (2, msg + \"\\n\") if msg else (0, \"\")\n+\n+\n+def main():\n+ if \"--selftest\" in sys.argv:\n+ return selftest()\n+ code, text = handle(sys.stdin.read())\n+ if text:\n+ sys.stderr.write(text)\n+ return code\n+\n+\n+def selftest():\n+ \"\"\"In-process checks on temporary fixtures; every failure is printed to stdout with the offending output.\"\"\"\n+ import shutil\n+ import traceback\n+ fails = []\n+ tmp = None\n+ old_cwd = None\n+ old_marker = os.environ.get(MARKER_ENV)\n+ try:\n+ tmp = os.path.realpath(tempfile.mkdtemp(prefix=\"roundgate_\"))\n+ try:\n+ old_cwd = os.getcwd()\n+ except OSError:\n+ old_cwd = None\n+ os.chdir(tmp)\n+ os.environ[MARKER_ENV] = os.path.join(tmp, \"m\")\n+ last = {\"out\": None}\n+\n+ def run(event, raw=None):\n+ code, err = handle(json.dumps(event) if raw is None else raw)\n+ last[\"out\"] = (code, err)\n+ return code, err\n+\n+ def check(name, cond):\n+ print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name), flush=True)\n+ if not cond:\n+ fails.append(name)\n+ print(\" last result (exit code, stderr): %r\" % (last[\"out\"],), flush=True)\n+\n+ def ev(cmd, cwd, sid=\"s1\"):\n+ return {\"hook_event_name\": \"PreToolUse\", \"tool_name\": \"Bash\", \"tool_input\": {\"command\": cmd},\n+ \"cwd\": cwd, \"session_id\": sid}\n+\n+ w = os.path.join(tmp, \"w\")\n+ os.makedirs(os.path.join(w, \"data\"))\n+ with open(os.path.join(w, \"round_1_results.csv\"), \"w\") as fh:\n+ fh.write(\"item,score\\nred_big,1.9\\nred_small,1.2\\nblue_big,0.5\\nblue_small,1.8\\n\")\n+ with open(os.path.join(w, \"round_2.csv\"), \"w\") as fh:\n+ fh.write(\"item\\nred_medium\\nblue_medium\\n\")\n+ with open(os.path.join(w, \"round_1.csv\"), \"w\") as fh:\n+ fh.write(\"item\\nred_big\\nred_small\\nblue_big\\nblue_small\\n\")\n+ cmd = \"asb run round_2.csv --output round_2_results.csv\"\n+ check(\"non-run command passes\", run(ev(\"ls -la\", w))[0] == 0)\n+ check(\"malformed input passes\", run({}, raw=\"not json\")[0] == 0)\n+ check(\"no earlier results passes\", run(ev(cmd, os.path.join(tmp, \"empty\")))[0] == 0)\n+ check(\"repeat of latest batch passes\", run(ev(\"asb run round_1.csv --output round_1_results.csv\", w, \"r\"))[0] == 0)\n+ code, err = run(ev(cmd, w))\n+ check(\"first run after a round blocks\", code == 2)\n+ check(\"message has numbers and ledger request\", \"top-10%\" in err and \"Ledger\" in err and \"big: 1.9\" in err)\n+ check(\"retry in the same state passes\", run(ev(cmd, w))[0] == 0)\n+ check(\"other session blocks again\", run(ev(cmd, w, \"s2\"))[0] == 2)\n+ check(\"missing session id never blocks\", run(ev(cmd, w, \"\"))[0] == 0)\n+ with open(os.path.join(w, \"round_2_results.csv\"), \"w\") as fh:\n+ fh.write(\"item,score\\nred_medium,2.2\\nblue_medium,1.0\\n\")\n+ cmd3 = \"cd %s && asb run round_3.csv --output round_3_results.csv\" % w\n+ check(\"new round count blocks again\", run(ev(cmd3, tmp))[0] == 2)\n+ check(\"and then passes on retry\", run(ev(cmd3, tmp))[0] == 0)\n+ # opaque ids joined with a catalog under data/\n+ w2 = os.path.join(tmp, \"w2\")\n+ os.makedirs(os.path.join(w2, \"data\"))\n+ with open(os.path.join(w2, \"data\", \"cat.csv\"), \"w\") as fh:\n+ fh.write(\"well_id,fa,fb\\na1,L1,B1\\na2,L2,B1\\na3,L1,B2\\na4,L2,B2\\n\")\n+ with open(os.path.join(w2, \"round_1_results.csv\"), \"w\") as fh:\n+ fh.write(\"well_id,score\\na1,80\\na2,10\\na3,60\\na4,5\\n\")\n+ code, err = run(ev(\"asb run round_2.csv\", w2, \"s3\"))\n+ check(\"opaque ids with catalog blocks and shows levels\", code == 2 and \"by fa\" in err and \"L1: 80, 2\" in err)\n+ except Exception:\n+ fails.append(\"selftest raised\")\n+ print(\"FAIL selftest raised:\\n\" + traceback.format_exc(), flush=True)\n+ finally:\n+ if old_cwd:\n+ try:\n+ os.chdir(old_cwd)\n+ except OSError:\n+ pass\n+ if old_marker is None:\n+ os.environ.pop(MARKER_ENV, None)\n+ else:\n+ os.environ[MARKER_ENV] = old_marker\n+ if tmp:\n+ shutil.rmtree(tmp, ignore_errors=True)\n+ print(\"selftest %s\" % (\"FAILED: \" + \", \".join(fails) if fails else \"OK\"), flush=True)\n+ return 1 if fails else 0\n+\n+\n+if __name__ == \"__main__\":\n+ sys.exit(main())\n", "files": { "hooks.json": "{\n \"pre_tool\": [\n {\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n \"matcher\": \"Bash\",\n \"timeout\": 30\n }\n ],\n \"stop\": [\n {\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"timeout\": 30\n }\n ]\n}\n", "instructions.md": "# Campaign workflow notes\n\nThese apply to tasks of the form: seed data, a few irreversible batched `asb run` rounds, then a self-contained\ndeterministic `final_model.js` that must pass both a discovery gate (hits among your ordered items) and a\nheld-out ranking (NDCG) gate. Two gates means neither can be left to the end.\n\n## Before round 1\n- Read the campaign description (`asb campaign`), the data README, and the model format spec plus the template\n script in full.\n- Write a baseline model from the seed data and run the deliver command in its free validation-only mode.\n Re-validate after every round, so a valid model always exists well before the deadline.\n- Note both pass criteria and how relevance, ties and the discount are defined; reproduce the metric locally.\n\n## Each round\n- Accepted rounds are final; rejected requests are free. Before `asb run`, check the file:\n `python3 /opt/rsi-harness/tools/campaign_check.py batch round_N.csv --size --id-column --previous round_*_results.csv [--catalog ]`\n Fix anything it reports (header, size, duplicates, items already ordered, ids not in the catalog).\n- Split each round between the model's top picks and covering groups or levels that are under-sampled so far.\n Track the running count of hits against the discovery threshold and shift toward exploitation when behind.\n This is guidance, not fixed percentages.\n- Measurements are noisy; do not read much into a single value that sits near the threshold.\n\n## Using the seed data\n- Use the seed data for how the levels of each categorical factor behave, not as a shortlist: the target regime can\n differ from the seed regime. Where targets are defined per group, normalise within group and do not let a global\n score favour easy groups over hard ones.\n\n## Choosing and checking the model\n- Pick the final model by cross-validation that mimics the held-out distribution (group the folds by the grouping variable\n that distinguishes held-out items, e.g. a categorical key), with the official metric computed locally:\n `python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ] [--threshold ]`\n Pass relevance already defined as the task states (clipped/normalised). The tool breaks score ties by input row order;\n prefer models that rarely produce ties.\n- The model must be self-contained, deterministic, finite for every input and independent per input; handle levels\n unseen in training with a fallback. Before delivering, run\n `python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt`\n with a few real-format inputs (taken from the catalog), then the validation-only deliver, then the real deliver.\n- `python3 /opt/rsi-harness/tools/campaign_check.py selftest` checks the tool itself. The tool is advisory only.\n", "smoke.json": "[\n {\n \"name\": \"campaign_check selftest (hand-computed NDCG, tie rule, batch and probe checks)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"campaign_check ndcg on a tiny synthetic CSV from stdin\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"p\", \"--rel\", \"r\", \"--k\", \"3\", \"--group\", \"g\"],\n \"stdin\": \"p,r,g\\n0.9,1,x\\n0.8,0,x\\n0.7,1,x\\n0.5,0,y\\n0.4,2,y\\n0.3,1,y\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check ndcg rejects an unknown column\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"ndcg\", \"-\", \"--pred\", \"nope\", \"--rel\", \"r\", \"--k\", \"2\"],\n \"stdin\": \"p,r\\n1,1\\n2,0\\n\",\n \"expect_exit\": 2,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch accepts a clean round\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw2\\nw3\\n\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags duplicates and wrong size\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"3\", \"--id-column\", \"well_id\"],\n \"stdin\": \"well_id\\nw1\\nw1\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check batch flags a wrong header\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"batch\", \"-\", \"--size\", \"2\", \"--id-column\", \"item_key\"],\n \"stdin\": \"id\\nA\\nB\\n\",\n \"expect_exit\": 1,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook selftest (block once after delivery, never loop, no-op otherwise)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\", \"--selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"stop_audit hook lets an already-active stop through\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"{\\\"hook_event_name\\\": \\\"Stop\\\", \\\"stop_hook_active\\\": true, \\\"session_id\\\": \\\"smoke\\\", \\\"transcript_path\\\": \\\"/nonexistent/t.jsonl\\\"}\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"stop_audit hook ignores malformed event input\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/stop_audit.py\"],\n \"stdin\": \"not json at all\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"round_gate hook selftest (block once per round state, retry passes, repeats and other commands pass)\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/round_gate.py\", \"--selftest\"],\n \"stdin\": \"\",\n \"expect_exit\": 0,\n \"timeout\": 120\n },\n {\n \"name\": \"round_gate hook ignores a command that is not asb run\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n \"stdin\": \"{\\\"hook_event_name\\\": \\\"PreToolUse\\\", \\\"tool_name\\\": \\\"Bash\\\", \\\"tool_input\\\": {\\\"command\\\": \\\"ls\\\"}, \\\"cwd\\\": \\\"/nonexistent\\\", \\\"session_id\\\": \\\"smoke\\\"}\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"round_gate hook lets asb run through when no earlier round exists\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n \"stdin\": \"{\\\"hook_event_name\\\": \\\"PreToolUse\\\", \\\"tool_name\\\": \\\"Bash\\\", \\\"tool_input\\\": {\\\"command\\\": \\\"asb run /nonexistent/round_1.csv --output /nonexistent/round_1_results.csv\\\"}, \\\"cwd\\\": \\\"/nonexistent\\\", \\\"session_id\\\": \\\"smoke\\\"}\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"round_gate hook ignores malformed event input\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/round_gate.py\"],\n \"stdin\": \"not json at all\",\n \"expect_exit\": 0,\n \"timeout\": 30\n },\n {\n \"name\": \"campaign_check probe reports a missing model as a usage error\",\n \"command\": \"python3\",\n \"args\": [\"/opt/rsi-harness/tools/campaign_check.py\", \"probe\", \"/nonexistent/final_model.js\", \"--input\", \"x\"],\n \"stdin\": \"\",\n \"expect_exit\": 2,\n \"timeout\": 30\n }\n]\n", "tools/campaign_check.py": "#!/usr/bin/env python3\n\"\"\"Advisory checks for batched-oracle campaigns (stdlib only, never blocks anything).\n\nSubcommands\n batch check a round CSV before ordering it (header, size, duplicates, earlier orders, catalog)\n ndcg local NDCG@k with 1/log2(rank+1) discount, stable ties, ideal-ranking normalisation\n probe run final_model.js under node on sample inputs: finite, deterministic, order-independent\n selftest built-in hand-computed checks of the above\n\nExit codes: 0 ok, 1 problem found, 2 usage or I/O error.\n\"\"\"\nimport argparse\nimport csv\nimport io\nimport json\nimport math\nimport os\nimport shutil\nimport subprocess\nimport sys\nimport tempfile\n\n\ndef die(msg, code=2):\n sys.stderr.write(\"error: %s\\n\" % msg)\n sys.exit(code)\n\n\ndef read_text(path):\n if path == \"-\":\n return sys.stdin.read()\n try:\n with open(path, encoding=\"utf-8-sig\", newline=\"\") as fh:\n return fh.read()\n except OSError as exc:\n die(\"cannot read %s: %s\" % (path, exc))\n\n\ndef read_rows(path):\n text = read_text(path)\n if text.startswith(\"\"):\n text = text[1:]\n rows = list(csv.reader(io.StringIO(text)))\n rows = [r for r in rows if r and any(c.strip() for c in r)]\n if not rows:\n die(\"%s is empty\" % path)\n return rows[0], rows[1:]\n\n\ndef id_values(path, column):\n header, body = read_rows(path)\n idx = header.index(column) if column in header else 0\n return [r[idx].strip() for r in body if len(r) > idx]\n\n\n# ---------------------------------------------------------------- batch\ndef cmd_batch(args):\n header, body = read_rows(args.round_csv)\n problems, notes = [], []\n column = args.id_column or header[0]\n if len(header) != 1:\n problems.append(\"header has %d columns %r; the round file needs a single column\" % (len(header), header))\n if header[0] != column:\n problems.append(\"header is %r, expected %r\" % (header[0], column))\n ids = [r[0] for r in body]\n padded = [i for i in ids if i != i.strip()]\n if padded:\n problems.append(\"%d ids have leading/trailing whitespace, e.g. %r\" % (len(padded), padded[0]))\n ids = [i.strip() for i in ids]\n if any(len(r) != 1 for r in body):\n problems.append(\"some rows have more than one field\")\n if any(not i for i in ids):\n problems.append(\"blank ids present\")\n notes.append(\"header=%r rows=%d\" % (header, len(ids)))\n if args.size is not None and len(ids) != args.size:\n problems.append(\"size is %d, expected exactly %d\" % (len(ids), args.size))\n seen, dups = set(), []\n for i in ids:\n if i in seen and i not in dups:\n dups.append(i)\n seen.add(i)\n if dups:\n problems.append(\"%d duplicated ids inside the batch, e.g. %s\" % (len(dups), dups[:3]))\n prior = set()\n for p in args.previous or []:\n prior.update(id_values(p, column))\n overlap = sorted(seen & prior)\n if args.previous:\n notes.append(\"compared with %d earlier ids from %d files\" % (len(prior), len(args.previous)))\n if overlap:\n problems.append(\"%d ids were already ordered, e.g. %s\" % (len(overlap), overlap[:3]))\n if args.catalog:\n cat = set(id_values(args.catalog, args.catalog_column or column))\n notes.append(\"catalog has %d ids\" % len(cat))\n unknown = sorted(seen - cat)\n if unknown:\n problems.append(\"%d ids are not in the catalog, e.g. %s\" % (len(unknown), unknown[:3]))\n for n in notes:\n print(\"note: \" + n)\n for p in problems:\n print(\"PROBLEM: \" + p)\n print(\"batch %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- ndcg\ndef dcg(rels):\n return sum(r / math.log2(i + 2) for i, r in enumerate(rels))\n\n\ndef ndcg_at_k(preds, rels, k):\n \"\"\"Rank by prediction descending; equal predictions keep input order (stable sort).\"\"\"\n order = sorted(range(len(preds)), key=lambda i: -preds[i])\n got = dcg([rels[i] for i in order[:k]])\n ideal = dcg(sorted(rels, reverse=True)[:k])\n if ideal <= 0:\n return None\n return got / ideal\n\n\ndef cmd_ndcg(args):\n header, body = read_rows(args.csv)\n for c in [args.pred, args.rel] + ([args.group] if args.group else []):\n if c not in header:\n die(\"column %r not in header %r\" % (c, header))\n ip, ir = header.index(args.pred), header.index(args.rel)\n ig = header.index(args.group) if args.group else None\n preds, rels, groups = [], [], []\n for n, r in enumerate(body, 2):\n try:\n p, q = float(r[ip]), float(r[ir])\n except (ValueError, IndexError):\n die(\"row %d: non-numeric prediction or relevance\" % n)\n if math.isnan(p) or math.isnan(q) or math.isinf(p):\n die(\"row %d: NaN/inf value\" % n)\n preds.append(p)\n rels.append(q)\n groups.append(r[ig] if ig is not None else None)\n if any(q < 0 for q in rels):\n print(\"warning: negative relevance present; pass relevance already clipped at 0 if the metric does so\")\n k = args.k\n if k > len(preds):\n print(\"warning: k=%d exceeds %d rows\" % (k, len(preds)))\n val = ndcg_at_k(preds, rels, k)\n if val is None:\n print(\"warning: ideal DCG is 0 (no positive relevance); NDCG undefined, reported as 0\")\n val = 0.0\n print(\"ndcg@%d = %.6f (n=%d)\" % (k, val, len(preds)))\n if args.group:\n by = {}\n for i, g in enumerate(groups):\n by.setdefault(g, []).append(i)\n vals = []\n for g in sorted(by):\n v = ndcg_at_k([preds[i] for i in by[g]], [rels[i] for i in by[g]], k)\n vals.append(v)\n print(\" group %s: n=%d ndcg@%d=%s\" % (g, len(by[g]), k, \"undefined\" if v is None else \"%.4f\" % v))\n ok = [v for v in vals if v is not None]\n if ok:\n print(\"mean over groups (informational, not the pooled metric) = %.6f\" % (sum(ok) / len(ok)))\n if args.threshold is not None:\n print(\"threshold %.4f: %s\" % (args.threshold, \"PASS\" if val >= args.threshold else \"BELOW\"))\n return 0\n\n\n# ---------------------------------------------------------------- probe\nPROBE_JS = r\"\"\"\nconst fs = require('fs'), vm = require('vm');\nconst req = JSON.parse(fs.readFileSync(0, 'utf8'));\nfunction load() {\n const code = fs.readFileSync(req.model, 'utf8');\n const mod = { exports: {} };\n const sb = { module: mod, exports: mod.exports, console: { log() {}, error() {}, warn() {} } };\n vm.createContext(sb);\n vm.runInContext(code, sb, { timeout: 60000 });\n const f = (typeof sb.predict === 'function') ? sb.predict\n : (mod.exports && typeof mod.exports.predict === 'function') ? mod.exports.predict : null;\n if (!f) throw new Error('no predict function defined');\n return f;\n}\nfunction run(f, x) {\n try { return { v: f(x) }; } catch (e) { return { e: String(e && e.message || e) }; }\n}\nconst out = { load_error: null, results: [] };\ntry {\n const f1 = load(), f2 = load();\n const xs = req.inputs;\n const a = xs.map(x => run(f1, x));\n const b = xs.map(x => run(f1, x));\n const c = xs.map((x, i) => run(f2, xs[xs.length - 1 - i])).reverse();\n xs.forEach((x, i) => {\n const same = (p, q) => ('e' in p && 'e' in q) || ('v' in p && 'v' in q && Object.is(p.v, q.v));\n out.results.push({\n input: x, value: ('v' in a[i]) ? a[i].v : null, error: a[i].e || null,\n type: ('v' in a[i]) ? typeof a[i].v : null,\n finite: ('v' in a[i]) && typeof a[i].v === 'number' && Number.isFinite(a[i].v),\n repeat_same: same(a[i], b[i]), order_same: same(a[i], c[i])\n });\n });\n} catch (e) { out.load_error = String(e && e.message || e); }\nprocess.stdout.write(JSON.stringify(out));\n\"\"\"\n\nBUILTIN_EDGE = [\"\", \"?\"]\n\n\ndef cmd_probe(args):\n if not os.path.isfile(args.model):\n die(\"model file not found: %s\" % args.model)\n node = shutil.which(\"node\")\n if not node:\n die(\"node not found on PATH\")\n inputs = list(args.input or [])\n if args.inputs_file:\n inputs += [l.rstrip(\"\\r\\n\") for l in read_text(args.inputs_file).splitlines() if l.strip()]\n edge_must = []\n if args.edge_file:\n edge_must = [l.rstrip(\"\\r\\n\") for l in read_text(args.edge_file).splitlines() if l.strip()]\n if not inputs and not edge_must:\n die(\"give --input and/or --inputs-file (real-format sample inputs)\")\n all_inputs = inputs + edge_must + BUILTIN_EDGE\n kinds = [\"sample\"] * len(inputs) + [\"edge-required\"] * len(edge_must) + [\"edge-advisory\"] * len(BUILTIN_EDGE)\n payload = json.dumps({\"model\": os.path.abspath(args.model), \"inputs\": all_inputs})\n try:\n proc = subprocess.run([node, \"-e\", PROBE_JS], input=payload, capture_output=True,\n text=True, timeout=args.timeout)\n except subprocess.TimeoutExpired:\n print(\"PROBLEM: node timed out after %ss\" % args.timeout)\n return 1\n if proc.returncode != 0:\n print(\"PROBLEM: node failed: %s\" % proc.stderr.strip()[:500])\n return 1\n try:\n res = json.loads(proc.stdout)\n except ValueError:\n print(\"PROBLEM: unparseable probe output: %r\" % proc.stdout[:200])\n return 1\n if res.get(\"load_error\"):\n print(\"PROBLEM: model failed to load in an isolated context: %s\" % res[\"load_error\"])\n return 1\n problems = 0\n for r, kind in zip(res[\"results\"], kinds):\n bad = []\n if r[\"error\"]:\n bad.append(\"threw: %s\" % r[\"error\"])\n elif not r[\"finite\"]:\n bad.append(\"not a finite number (%r)\" % (r[\"value\"],))\n if not r[\"repeat_same\"]:\n bad.append(\"repeated call differs\")\n if not r[\"order_same\"]:\n bad.append(\"result depends on call order/fresh load\")\n if bad:\n tag = \"WARN\" if kind == \"edge-advisory\" else \"PROBLEM\"\n if kind != \"edge-advisory\":\n problems += 1\n print(\"%s [%s] %r: %s\" % (tag, kind, r[\"input\"], \"; \".join(bad)))\n vals = [r[\"value\"] for r in res[\"results\"][:len(inputs)] if r[\"finite\"]]\n if vals:\n print(\"sample outputs: n=%d min=%.6g max=%.6g distinct=%d\" % (len(vals), min(vals), max(vals), len(set(vals))))\n if len(set(vals)) == 1 and len(vals) > 1:\n print(\"WARN: every sample input got the same score; the ranking would be all ties\")\n print(\"probe %s\" % (\"FAILED\" if problems else \"OK\"))\n return 1 if problems else 0\n\n\n# ---------------------------------------------------------------- selftest\ndef cmd_selftest(_args):\n fails = []\n\n def check(name, cond):\n if not cond:\n fails.append(name)\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n\n # hand-computed: dcg = 1 + 0.5 = 1.5 ; ideal = 1 + 1/log2(3)\n v = ndcg_at_k([0.9, 0.8, 0.7], [1, 0, 1], 3)\n check(\"ndcg hand value\", abs(v - 1.5 / (1 + 1 / math.log2(3))) < 1e-12)\n # tie rule: equal scores keep evaluation order -> rels seen [0,1]\n v = ndcg_at_k([5, 5, 5], [0, 1, 1], 2)\n check(\"ndcg tie keeps input order\", abs(v - (1 / math.log2(3)) / (1 + 1 / math.log2(3))) < 1e-12)\n check(\"ndcg perfect ranking is 1\", abs(ndcg_at_k([3, 2, 1], [2, 1, 0], 2) - 1) < 1e-12)\n check(\"ndcg undefined without positives\", ndcg_at_k([1, 2], [0, 0], 2) is None)\n tmp = tempfile.mkdtemp(prefix=\"campchk_\")\n try:\n def w(name, text):\n p = os.path.join(tmp, name)\n with open(p, \"w\") as fh:\n fh.write(text)\n return p\n prev = w(\"prev.csv\", \"id,y\\na,1\\nb,2\\n\")\n ns = argparse.Namespace\n import contextlib\n sink = io.StringIO()\n with contextlib.redirect_stdout(sink):\n ok = cmd_batch(ns(round_csv=w(\"r1.csv\", \"id\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n dup = cmd_batch(ns(round_csv=w(\"r2.csv\", \"id\\nc\\nc\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n old = cmd_batch(ns(round_csv=w(\"r3.csv\", \"id\\na\\nd\\n\"), size=2, id_column=\"id\",\n previous=[prev], catalog=None, catalog_column=None))\n short = cmd_batch(ns(round_csv=w(\"r4.csv\", \"id\\nc\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n hdr = cmd_batch(ns(round_csv=w(\"r5.csv\", \"wrong\\nc\\nd\\n\"), size=2, id_column=\"id\",\n previous=[], catalog=None, catalog_column=None))\n check(\"batch accepts clean round\", ok == 0)\n check(\"batch flags in-batch duplicate\", dup == 1)\n check(\"batch flags earlier order\", old == 1)\n check(\"batch flags wrong size\", short == 1)\n check(\"batch flags wrong header\", hdr == 1)\n if shutil.which(\"node\"):\n model = w(\"m.js\", \"function predict(x){ return x.length * 0.5; }\\n\")\n bad = w(\"bad.js\", \"function predict(x){ return x === '' ? NaN : Math.random(); }\\n\")\n with contextlib.redirect_stdout(sink):\n good = cmd_probe(ns(model=model, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n notgood = cmd_probe(ns(model=bad, input=[\"ab\", \"abcd\"], inputs_file=None,\n edge_file=None, timeout=60))\n check(\"probe accepts a finite deterministic model\", good == 0)\n check(\"probe rejects a random model\", notgood == 1)\n else:\n print(\"skip probe checks (node not found)\")\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\ndef main():\n ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)\n sub = ap.add_subparsers(dest=\"cmd\", required=True)\n\n b = sub.add_parser(\"batch\", help=\"check a round CSV before `asb run` (use '-' for stdin)\")\n b.add_argument(\"round_csv\")\n b.add_argument(\"--size\", type=int, help=\"exact number of rows required\")\n b.add_argument(\"--id-column\", help=\"expected header, e.g. the single column name from the task\")\n b.add_argument(\"--previous\", nargs=\"*\", help=\"earlier round or results CSVs whose ids may not repeat\")\n b.add_argument(\"--catalog\", help=\"CSV listing orderable ids\")\n b.add_argument(\"--catalog-column\", help=\"id column in the catalog (default: --id-column)\")\n b.set_defaults(fn=cmd_batch)\n\n n = sub.add_parser(\"ndcg\", help=\"local NDCG@k (use '-' for stdin)\")\n n.add_argument(\"csv\")\n n.add_argument(\"--pred\", required=True, help=\"prediction column\")\n n.add_argument(\"--rel\", required=True, help=\"relevance column (already clipped/normalised as the task defines)\")\n n.add_argument(\"--k\", type=int, required=True)\n n.add_argument(\"--group\", help=\"optional group column; adds per-group NDCG as a diagnostic\")\n n.add_argument(\"--threshold\", type=float, help=\"optional pass level to compare against\")\n n.set_defaults(fn=cmd_ndcg)\n\n p = sub.add_parser(\"probe\", help=\"run final_model.js under node on sample inputs\")\n p.add_argument(\"model\")\n p.add_argument(\"--input\", action=\"append\", help=\"sample input string (repeatable)\")\n p.add_argument(\"--inputs-file\", help=\"file with one sample input per line\")\n p.add_argument(\"--edge-file\", help=\"file with one extra input per line that must also score finitely\")\n p.add_argument(\"--timeout\", type=int, default=120)\n p.set_defaults(fn=cmd_probe)\n\n s = sub.add_parser(\"selftest\", help=\"hand-computed self checks\")\n s.set_defaults(fn=cmd_selftest)\n\n args = ap.parse_args()\n sys.exit(args.fn(args))\n\n\nif __name__ == \"__main__\":\n main()\n", "tools/round_gate.py": "#!/usr/bin/env python3\n\"\"\"Once-per-round pre_tool hook on Bash: before an irreversible `asb run`, ask for a hit ledger.\n\nReads the pre_tool event JSON on stdin. It acts only when the Bash command contains `asb run` and at least one\nearlier round_*_results.csv exists. It then summarises the results so far (descriptive numbers only, no thresholds\nof its own) and exits 2 once for that state, feeding the summary and a request back to the agent on stderr.\nThe retried `asb run` in the same state passes, so no round is ever lost. Exit 0 on anything unexpected, on\ncommands that are not `asb run`, and when the batch is a repeat of the latest accepted one.\nStdlib only. Run with --selftest for a built-in check using temporary fixtures.\n\"\"\"\nimport csv\nimport glob\nimport hashlib\nimport json\nimport math\nimport os\nimport re\nimport shlex\nimport sys\nimport tempfile\n\nMARKER_ENV = \"RSI_GATE_MARKER_DIR\"\nRUN_RE = re.compile(r\"\\basb\\s+run\\b\")\nROUND_RE = re.compile(r\"round_(\\d+)_results\\.csv$\")\nSEP_RE = re.compile(r\"[-_:;/ ]+\")\nNUM_RE = re.compile(r\"^[+-]?\\d*\\.?\\d+$\")\nMAX_LEVEL_FIELDS = 6\nMAX_LEVELS = 12\nMAX_CARDINALITY = 30\n\n\ndef to_float(text):\n try:\n v = float(text)\n except (TypeError, ValueError):\n return None\n return v if math.isfinite(v) else None\n\n\ndef run_args(command):\n \"\"\"Tokens following `asb run` in the command, or [] if they cannot be parsed.\"\"\"\n m = RUN_RE.search(command)\n if not m:\n return []\n tail = command[m.end():]\n tail = re.split(r\"&&|\\|\\||;|\\n\", tail)[0]\n try:\n return shlex.split(tail)\n except ValueError:\n return tail.split()\n\n\ndef candidate_dirs(event, command):\n dirs = []\n try:\n here = os.getcwd()\n except OSError:\n here = None\n for d in (event.get(\"cwd\"), here):\n if d and isinstance(d, str) and os.path.isdir(d):\n dirs.append(os.path.realpath(d))\n for m in re.finditer(r\"\\bcd\\s+([^\\s;&|]+)\", command):\n p = os.path.expanduser(m.group(1).strip(\"'\\\"\"))\n if not os.path.isabs(p) and dirs:\n p = os.path.join(dirs[0], p)\n if os.path.isdir(p):\n dirs.insert(0, os.path.realpath(p))\n for tok in run_args(command):\n if tok.startswith(\"-\"):\n continue\n base = os.path.dirname(tok)\n if base:\n p = base if os.path.isabs(base) else os.path.join(dirs[0] if dirs else \"\", base)\n if os.path.isdir(p):\n dirs.append(os.path.realpath(p))\n seen, out = set(), []\n for d in dirs:\n if d not in seen:\n seen.add(d)\n out.append(d)\n return out\n\n\ndef find_results(dirs):\n \"\"\"round number -> path, keeping the first directory that has each round.\"\"\"\n found = {}\n for d in dirs:\n for path in glob.glob(os.path.join(d, \"round_*_results.csv\")):\n m = ROUND_RE.search(os.path.basename(path))\n if m and int(m.group(1)) not in found:\n found[int(m.group(1))] = path\n return [found[k] for k in sorted(found)]\n\n\ndef read_results(path):\n \"\"\"Return (id_column_name, [(id, value)]) using the first column as id and the last numeric column as value.\"\"\"\n try:\n with open(path, newline=\"\", encoding=\"utf-8\", errors=\"replace\") as fh:\n rows = list(csv.reader(fh))\n except OSError:\n return None, []\n if len(rows) < 2:\n return None, []\n header, body = rows[0], [r for r in rows[1:] if r]\n best_col = None\n for col in range(len(header) - 1, 0, -1):\n good = sum(1 for r in body if col < len(r) and to_float(r[col]) is not None)\n if body and good >= 0.8 * len(body):\n best_col = col\n break\n if best_col is None:\n return header[0], []\n out = []\n for r in body:\n v = to_float(r[best_col]) if best_col < len(r) else None\n if v is not None:\n out.append((r[0], v))\n return header[0], out\n\n\ndef read_ids(path):\n try:\n with open(path, newline=\"\", encoding=\"utf-8\", errors=\"replace\") as fh:\n rows = list(csv.reader(fh))\n except OSError:\n return None\n return [r[0] for r in rows[1:] if r]\n\n\ndef is_repeat(args, dirs, latest_ids):\n \"\"\"True when the batch file named in the command has exactly the ids of the latest accepted round.\"\"\"\n for tok in args:\n if tok.startswith(\"-\") or not tok.lower().endswith(\".csv\"):\n continue\n paths = [tok] if os.path.isabs(tok) else [os.path.join(d, tok) for d in dirs]\n for p in paths:\n if os.path.isfile(p):\n ids = read_ids(p)\n return ids is not None and bool(ids) and set(ids) == latest_ids\n return False\n return False\n\n\ndef load_catalog(dirs, id_name):\n \"\"\"Optional join: the largest csv under /data with the id column and low-cardinality factor columns.\"\"\"\n best = None\n for d in dirs:\n for path in glob.glob(os.path.join(d, \"data\", \"*.csv\")):\n try:\n with open(path, newline=\"\", encoding=\"utf-8\", errors=\"replace\") as fh:\n rows = list(csv.DictReader(fh))\n except OSError:\n continue\n if rows and id_name in rows[0] and len(rows[0]) > 2 and (best is None or len(rows) > len(best)):\n best = rows\n if not best:\n return {}\n fields = []\n for name in best[0]:\n if name == id_name:\n continue\n levels = {r.get(name) for r in best}\n if 2 <= len(levels) <= MAX_CARDINALITY:\n fields.append(name)\n return {r[id_name]: {f: r.get(f) for f in fields} for r in best if r.get(id_name) is not None}\n\n\ndef factors_for(item_id, catalog):\n \"\"\"Map field name -> list of levels for this item, derived from the id itself or the catalog join.\"\"\"\n if item_id in catalog:\n return {k: [v] for k, v in catalog[item_id].items() if v not in (None, \"\")}\n if \"|\" in item_id:\n return {\"field%d\" % (i + 1): [p] for i, p in enumerate(item_id.split(\"|\"))}\n parts = [p for p in SEP_RE.split(item_id) if p]\n if len(parts) > 1:\n words = [p for p in parts if not NUM_RE.match(p)]\n if words:\n return {\"part\": words}\n return {}\n\n\ndef summarise(paths, dirs):\n per_file, allrows = [], []\n id_name = None\n for p in paths:\n name, rows = read_results(p)\n id_name = id_name or name\n per_file.append((os.path.basename(p), rows))\n allrows.extend(rows)\n if not allrows:\n return None\n vals = sorted(v for _, v in allrows)\n cut = vals[min(len(vals) - 1, int(math.floor(0.9 * len(vals))))]\n lines = [\"Results so far: %d items over %d round(s); top-10%% line of all observed values = %.4g.\"\n % (len(allrows), len(per_file), cut)]\n for fname, rows in per_file:\n vs = sorted((v for _, v in rows), reverse=True)\n top = \", \".join(\"%.4g\" % v for v in vs[:5])\n n_top = sum(1 for v in vs if v >= cut)\n lines.append(\" %s: %d rows, top 5 = [%s], %d at or above the top-10%% line\" % (fname, len(rows), top, n_top))\n catalog = load_catalog(dirs, id_name) if id_name else {}\n stats = {}\n for item_id, v in allrows:\n for field, levels in factors_for(item_id, catalog).items():\n for lv in set(levels):\n n, best = stats.setdefault(field, {}).get(lv, (0, None))\n stats[field][lv] = (n + 1, v if best is None else max(best, v))\n shown = 0\n for field in sorted(stats):\n levels = {lv: nb for lv, nb in stats[field].items() if nb[0] < len(allrows)}\n if len(levels) < 2 or len(levels) > MAX_CARDINALITY or shown >= MAX_LEVEL_FIELDS:\n continue\n shown += 1\n ranked = sorted(levels.items(), key=lambda kv: (-kv[1][1], kv[0]))[:MAX_LEVELS]\n lines.append(\" by %s (level: best value, n tried): %s\" % (\n field, \"; \".join(\"%s: %.4g, %d\" % (lv, b, n) for lv, (n, b) in ranked)))\n return \"\\n\".join(lines)\n\n\nMESSAGE = \"\"\"Before the next irreversible round: write a short hit ledger, then re-issue the same `asb run`.\n(This check appears once per round; the retry goes through unchanged and nothing is lost.)\n\n{summary}\n\nLedger, in your reply, with real numbers:\n 1. Hits so far against the task's stated discovery target (use the task's own threshold or rule).\n 2. Hits still needed, and hits needed per remaining round.\n 3. Hits the last round actually delivered.\nIf the needed rate is above what the last round delivered, lean the next batch toward unordered near neighbours of\nthe best items so far (changing one factor at a time) plus the model's top picks, and drop levels whose best result\nis near the floor. Coverage of under-sampled levels matters just as much, because the held-out ranking gate needs it:\nthis is guidance, not a fixed split. If the needed rate is at or below the delivered rate, keep your current split.\nIf the discovery target is relative to a group whose cutoffs are hidden, count as a hit proxy the top few percent\nof each group's own observed values. The proxy is advisory and noisy; do not treat a value near a line as a hit.\"\"\"\n\n\ndef state_tag(paths):\n \"\"\"Hash of the contents of every results file, so the once-per-round state follows the data itself.\"\"\"\n h = hashlib.sha1()\n for p in paths:\n h.update(os.path.basename(p).encode(\"utf-8\", \"replace\"))\n try:\n with open(p, \"rb\") as fh:\n h.update(fh.read())\n except OSError:\n pass\n return h.hexdigest()[:16]\n\n\ndef marker_path(session, tag):\n base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_round_gate\")\n safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session)[:60]\n return base, os.path.join(base, \"%s_%s.done\" % (safe, tag))\n\n\ndef evaluate(event):\n \"\"\"Return the message to block with, or None to let the call through.\"\"\"\n command = \"\"\n ti = event.get(\"tool_input\")\n if isinstance(ti, dict):\n command = str(ti.get(\"command\") or \"\")\n if not command or not RUN_RE.search(command):\n return None\n dirs = candidate_dirs(event, command)\n paths = find_results(dirs)\n if not paths:\n return None\n summary = summarise(paths, dirs)\n if summary is None:\n return None\n _, last_rows = read_results(paths[-1])\n if is_repeat(run_args(command), dirs, {i for i, _ in last_rows}):\n return None\n session = str(event.get(\"session_id\") or \"\")\n if not session:\n return None # no per-session key means no safe place for once-per-round state: never block\n base, marker = marker_path(session, state_tag(paths))\n if os.path.exists(marker):\n return None\n os.makedirs(base, exist_ok=True)\n with open(marker, \"w\") as fh:\n fh.write(\"1\")\n return MESSAGE.format(summary=summary)\n\n\ndef handle(raw):\n \"\"\"(exit code, stderr text) for one raw event; anything unexpected lets the call through.\"\"\"\n try:\n event = json.loads(raw or \"{}\")\n if not isinstance(event, dict):\n return 0, \"\"\n msg = evaluate(event)\n except Exception:\n return 0, \"\"\n return (2, msg + \"\\n\") if msg else (0, \"\")\n\n\ndef main():\n if \"--selftest\" in sys.argv:\n return selftest()\n code, text = handle(sys.stdin.read())\n if text:\n sys.stderr.write(text)\n return code\n\n\ndef selftest():\n \"\"\"In-process checks on temporary fixtures; every failure is printed to stdout with the offending output.\"\"\"\n import shutil\n import traceback\n fails = []\n tmp = None\n old_cwd = None\n old_marker = os.environ.get(MARKER_ENV)\n try:\n tmp = os.path.realpath(tempfile.mkdtemp(prefix=\"roundgate_\"))\n try:\n old_cwd = os.getcwd()\n except OSError:\n old_cwd = None\n os.chdir(tmp)\n os.environ[MARKER_ENV] = os.path.join(tmp, \"m\")\n last = {\"out\": None}\n\n def run(event, raw=None):\n code, err = handle(json.dumps(event) if raw is None else raw)\n last[\"out\"] = (code, err)\n return code, err\n\n def check(name, cond):\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name), flush=True)\n if not cond:\n fails.append(name)\n print(\" last result (exit code, stderr): %r\" % (last[\"out\"],), flush=True)\n\n def ev(cmd, cwd, sid=\"s1\"):\n return {\"hook_event_name\": \"PreToolUse\", \"tool_name\": \"Bash\", \"tool_input\": {\"command\": cmd},\n \"cwd\": cwd, \"session_id\": sid}\n\n w = os.path.join(tmp, \"w\")\n os.makedirs(os.path.join(w, \"data\"))\n with open(os.path.join(w, \"round_1_results.csv\"), \"w\") as fh:\n fh.write(\"item,score\\nred_big,1.9\\nred_small,1.2\\nblue_big,0.5\\nblue_small,1.8\\n\")\n with open(os.path.join(w, \"round_2.csv\"), \"w\") as fh:\n fh.write(\"item\\nred_medium\\nblue_medium\\n\")\n with open(os.path.join(w, \"round_1.csv\"), \"w\") as fh:\n fh.write(\"item\\nred_big\\nred_small\\nblue_big\\nblue_small\\n\")\n cmd = \"asb run round_2.csv --output round_2_results.csv\"\n check(\"non-run command passes\", run(ev(\"ls -la\", w))[0] == 0)\n check(\"malformed input passes\", run({}, raw=\"not json\")[0] == 0)\n check(\"no earlier results passes\", run(ev(cmd, os.path.join(tmp, \"empty\")))[0] == 0)\n check(\"repeat of latest batch passes\", run(ev(\"asb run round_1.csv --output round_1_results.csv\", w, \"r\"))[0] == 0)\n code, err = run(ev(cmd, w))\n check(\"first run after a round blocks\", code == 2)\n check(\"message has numbers and ledger request\", \"top-10%\" in err and \"Ledger\" in err and \"big: 1.9\" in err)\n check(\"retry in the same state passes\", run(ev(cmd, w))[0] == 0)\n check(\"other session blocks again\", run(ev(cmd, w, \"s2\"))[0] == 2)\n check(\"missing session id never blocks\", run(ev(cmd, w, \"\"))[0] == 0)\n with open(os.path.join(w, \"round_2_results.csv\"), \"w\") as fh:\n fh.write(\"item,score\\nred_medium,2.2\\nblue_medium,1.0\\n\")\n cmd3 = \"cd %s && asb run round_3.csv --output round_3_results.csv\" % w\n check(\"new round count blocks again\", run(ev(cmd3, tmp))[0] == 2)\n check(\"and then passes on retry\", run(ev(cmd3, tmp))[0] == 0)\n # opaque ids joined with a catalog under data/\n w2 = os.path.join(tmp, \"w2\")\n os.makedirs(os.path.join(w2, \"data\"))\n with open(os.path.join(w2, \"data\", \"cat.csv\"), \"w\") as fh:\n fh.write(\"well_id,fa,fb\\na1,L1,B1\\na2,L2,B1\\na3,L1,B2\\na4,L2,B2\\n\")\n with open(os.path.join(w2, \"round_1_results.csv\"), \"w\") as fh:\n fh.write(\"well_id,score\\na1,80\\na2,10\\na3,60\\na4,5\\n\")\n code, err = run(ev(\"asb run round_2.csv\", w2, \"s3\"))\n check(\"opaque ids with catalog blocks and shows levels\", code == 2 and \"by fa\" in err and \"L1: 80, 2\" in err)\n except Exception:\n fails.append(\"selftest raised\")\n print(\"FAIL selftest raised:\\n\" + traceback.format_exc(), flush=True)\n finally:\n if old_cwd:\n try:\n os.chdir(old_cwd)\n except OSError:\n pass\n if old_marker is None:\n os.environ.pop(MARKER_ENV, None)\n else:\n os.environ[MARKER_ENV] = old_marker\n if tmp:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED: \" + \", \".join(fails) if fails else \"OK\"), flush=True)\n return 1 if fails else 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n", "tools/stop_audit.py": "#!/usr/bin/env python3\n\"\"\"One-shot stop hook: after a campaign model was delivered and time remains, ask for a held-out audit.\n\nReads the stop event JSON on stdin. Exit 0 lets the agent stop. Exit 2 blocks the stop once and feeds the\naudit request (stderr) back to the agent. It never blocks twice per session, never blocks when the event says\na stop hook is already active, and never blocks outside a delivered-campaign context.\nStdlib only. Run with --selftest for a built-in check using temporary fixtures.\n\"\"\"\nimport json\nimport os\nimport re\nimport sys\nimport tempfile\nimport time\nfrom datetime import datetime, timezone\n\nMARKER_ENV = \"RSI_AUDIT_MARKER_DIR\"\nBUDGET_ENV = \"RSI_AUDIT_BUDGET_S\"\nDEFAULT_BUDGET_S = 28800\nMIN_REMAINING_S = 1800\nDELIVERED = \"deliverable accepted\"\nMODEL_NAME = \"final_model.js\"\n\nMESSAGE = \"\"\"Before stopping: a model has been delivered and time remains. The latest submission is the one scored, so\nre-delivery is allowed. Spend a short audit on the held-out ranking gate. Everything you measured so far was\nchosen by your own model, so a score over your own picks is biased upward.\n\n1. Build a held-out stand-in. Use the broad first round (or the least model-selected measurements), or hold out whole\n groups of the ordered items (group the folds by the key that distinguishes held-out items). Fit on the rest, keeping\n the seed data in. Score with the official relevance transform exactly as the task states it:\n python3 /opt/rsi-harness/tools/campaign_check.py ndcg preds.csv --pred --rel --k [--group ]\n2. Compare the delivered model with at least one structurally different alternative on that same stand-in: a\n lower-dimensional or smoother model (fewer features, stronger shrinkage, log or clipped target), and the same\n model with versus without the seed data. Run long fits in the background (nohup ... &) and poll the log, because\n a single command is cut off after about two minutes.\n3. Re-deliver only if an alternative beats the delivered model by a clear margin (state it, e.g. at least 0.05 NDCG)\n and the gap is not one or two items. Then: cp final_model.js final_model.prev.js, write the new file,\n python3 /opt/rsi-harness/tools/campaign_check.py probe --inputs-file samples.txt,\n asb deliver --validate-only, and only then the real deliver.\n If nothing wins by the margin, keep the current model: do not re-deliver, and report the stand-in scores.\nThis audit happens once; state the numbers you compared in the final message.\"\"\"\n\n\ndef parse_ts(value):\n try:\n if isinstance(value, (int, float)):\n return float(value) / (1000.0 if value > 1e11 else 1.0)\n dt = datetime.fromisoformat(str(value).replace(\"Z\", \"+00:00\"))\n if dt.tzinfo is None:\n dt = dt.replace(tzinfo=timezone.utc)\n return dt.timestamp()\n except (ValueError, TypeError):\n return None\n\n\ndef read_transcript(path):\n try:\n with open(path, encoding=\"utf-8\", errors=\"replace\") as fh:\n return fh.read()\n except OSError:\n return None\n\n\ndef first_timestamp(text):\n for line in text.splitlines()[:50]:\n m = re.search(r'\"timestamp\"\\s*:\\s*(\"[^\"]+\"|[0-9.]+)', line)\n if m:\n ts = parse_ts(json.loads(m.group(1)))\n if ts:\n return ts\n return None\n\n\ndef model_exists(cwd):\n for d in (cwd, os.getcwd(), \"/app\"):\n if d and os.path.isfile(os.path.join(d, MODEL_NAME)):\n return True\n return False\n\n\ndef marker_path(session):\n base = os.environ.get(MARKER_ENV) or os.path.join(tempfile.gettempdir(), \"rsi_stop_audit\")\n safe = re.sub(r\"[^A-Za-z0-9_.-]\", \"_\", session or \"nosession\")[:80]\n return base, os.path.join(base, safe + \".done\")\n\n\ndef decide(event):\n \"\"\"Return True if the stop should be blocked for the audit.\"\"\"\n if event.get(\"stop_hook_active\"):\n return False\n cwd = event.get(\"cwd\") or \"\"\n text = None\n tp = event.get(\"transcript_path\")\n if tp:\n text = read_transcript(tp)\n if text is not None:\n if DELIVERED not in text or MODEL_NAME not in text:\n return False\n start = first_timestamp(text)\n if start is not None:\n try:\n budget = float(os.environ.get(BUDGET_ENV, DEFAULT_BUDGET_S))\n except ValueError:\n budget = DEFAULT_BUDGET_S\n if time.time() - start > budget - MIN_REMAINING_S:\n return False\n elif not model_exists(cwd):\n return False\n return True\n\n\ndef main():\n if \"--selftest\" in sys.argv:\n return selftest()\n try:\n event = json.loads(sys.stdin.read() or \"{}\")\n if not isinstance(event, dict):\n return 0\n except ValueError:\n return 0\n try:\n if not decide(event):\n return 0\n base, marker = marker_path(str(event.get(\"session_id\") or \"\"))\n if os.path.exists(marker):\n return 0\n os.makedirs(base, exist_ok=True)\n with open(marker, \"w\") as fh:\n fh.write(str(time.time()))\n except OSError:\n return 0\n sys.stderr.write(MESSAGE + \"\\n\")\n return 2\n\n\ndef selftest():\n import subprocess\n import shutil\n tmp = tempfile.mkdtemp(prefix=\"stopaudit_\")\n fails = []\n try:\n env = dict(os.environ, **{MARKER_ENV: os.path.join(tmp, \"m\")})\n\n def run(event):\n p = subprocess.run([sys.executable, os.path.abspath(__file__)], input=json.dumps(event),\n capture_output=True, text=True, env=env, timeout=30)\n return p.returncode\n\n def check(name, cond):\n print(\"%s %s\" % (\"ok \" if cond else \"FAIL\", name))\n if not cond:\n fails.append(name)\n\n now = datetime.now(timezone.utc).isoformat()\n good = os.path.join(tmp, \"good.jsonl\")\n with open(good, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"asb deliver final_model.js\"}) + \"\\n\")\n fh.write(json.dumps({\"text\": \"local validation passed\\n\" + DELIVERED + \" (submission 1)\"}) + \"\\n\")\n undelivered = os.path.join(tmp, \"undelivered.jsonl\")\n with open(undelivered, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": now, \"text\": \"writing final_model.js\"}) + \"\\n\")\n late = os.path.join(tmp, \"late.jsonl\")\n with open(late, \"w\") as fh:\n fh.write(json.dumps({\"timestamp\": \"2020-01-01T00:00:00Z\", \"text\": \"final_model.js \" + DELIVERED}) + \"\\n\")\n check(\"active stop hook never blocks\", run({\"stop_hook_active\": True, \"transcript_path\": good, \"session_id\": \"a\"}) == 0)\n check(\"no delivery in transcript does not block\", run({\"transcript_path\": undelivered, \"session_id\": \"b\"}) == 0)\n check(\"out of time does not block\", run({\"transcript_path\": late, \"session_id\": \"c\"}) == 0)\n check(\"garbage input does not block\", subprocess.run([sys.executable, os.path.abspath(__file__)], input=\"not json\",\n capture_output=True, text=True, env=env).returncode == 0)\n check(\"delivered with time left blocks once\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 2)\n check(\"second stop in same session passes\", run({\"transcript_path\": good, \"session_id\": \"d\"}) == 0)\n os.makedirs(os.path.join(tmp, \"w\"))\n open(os.path.join(tmp, \"w\", MODEL_NAME), \"w\").close()\n check(\"no transcript but model file present blocks\", run({\"cwd\": os.path.join(tmp, \"w\"), \"session_id\": \"f\"}) == 2)\n finally:\n shutil.rmtree(tmp, ignore_errors=True)\n print(\"selftest %s\" % (\"FAILED\" if fails else \"OK\"))\n return 1 if fails else 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, "candidate_sha256": "43170da1c00926f99d7c379021c6cad27068c3909140e8aa10e6e938651cc24a", "parent_sha256": "b232ad942c6f44e92710c1bf6a8c148a76d813fdc9a918bc1c68f88066815528", "role_cost": { "calls": 8, "list_price_estimate_usd": 2.34 }, "trials": [ { "task": "propylene-active-learning", "trial": "propylene-active-learning__bZHmd7u", "job": "r05-screen", "reward": 1, "model_audit_ok": true, "harness_interventions": 3, "tokens": 2021464, "list_price_estimate_usd": 0.8853184 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__mE6faYA", "job": "r05-screen", "reward": 1, "model_audit_ok": true, "harness_interventions": 4, "tokens": 1722967, "list_price_estimate_usd": 0.7755443999999999 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__BEvBqMd", "job": "r05-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 3, "tokens": 1177229, "list_price_estimate_usd": 0.5831182 }, { "task": "propylene-active-learning", "trial": "propylene-active-learning__FVUQSma", "job": "r05-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 3, "tokens": 1213586, "list_price_estimate_usd": 0.6683166 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__7XSJYR5", "job": "r05-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 4, "tokens": 1296847, "list_price_estimate_usd": 0.6305639999999999 }, { "task": "suzuki-condition-screen", "trial": "suzuki-condition-screen__qqDRKCm", "job": "r05-topup", "reward": 0, "model_audit_ok": true, "harness_interventions": 4, "tokens": 1466001, "list_price_estimate_usd": 0.6870642000000001 } ] } ] }