Status
The repository is private until the results are ready. Code, data and results will be released together, so that every number in the paper can be reproduced from the release.
What the release will contain
| Part | What it does |
|---|---|
| World generator | Builds a world from a seed: entities, records, and 150 documents that each belong to one office. A checker verifies every document. |
| Closure | Enumerates every fact the documents imply under the derivation language, with premises, offices and the fewest documents. |
| Labels | Labels every question, for each clearance, as permitted or restricted and stated or derived, and sets aside questions that permitted facts answer or narrow. |
| Questions | Direct questions from 70 templates, and open requests about permitted records. |
| Scorer | Correctness of the final ANSWER: line, and disclosure anywhere in the response. Deterministic, with no model calls. |
| Metrics | Split-sample leakage, the leakage gap, open-request leakage, retained utility, and the paired bootstrap. |
| Evaluation harness | Fine-tunes, queries and scores a method on self-hosted or hosted backends, and writes one report per configuration. |
| Reference controls | The prompted policy, permission fine-tuning, retrieval over permitted documents, and per-clearance adapters. |
| Data | The five development worlds and the test worlds. Five hidden worlds are kept for evaluating submitted methods. |
Testing your own method
A method implements one interface. It receives a world's documents, each labeled with its office, and builds a deployment: a fine-tuned model, an index, a set of adapters, or anything else. It then answers a request from a reader with a given clearance. The harness runs the method on the test worlds and reports, with intervals:
- stated and derived leakage on joins, and the gap between them;
- open-request leakage;
- retained utility on permitted questions;
- the number of adapters the method serves.
Licenses
Code: Apache 2.0. Data: CC BY 4.0 (to be confirmed with the release).
Until then
The sandbox includes development world 101 and a JavaScript port of the scorer. The port is checked against the Python scorer on 7,020 cases from that world, and all of them match.