Abstract
Organizations fine-tune language models on internal documents and restrict what each user may see. Access restrictions are defined over documents. A fine-tuned model, however, can combine facts from different documents, so a user may obtain a restricted fact that no document they may see states. In database security, such a path is called an inference channel. We introduce NeedToKnow, a benchmark that measures whether access controls on fine-tuned models withhold facts derived from restricted documents as reliably as facts those documents state. NeedToKnow generates synthetic organizations whose 150 documents are divided among five offices. For each organization, it enumerates every fact the documents imply and records which facts, and which offices, each one depends on. A model is fine-tuned on all documents, and readers with six clearances ask for restricted facts that are either stated in one document or derived from several. Responses are scored by exact matching, so the benchmark runs offline without human or model judges. We evaluate four access controls on four open-weight models from 9B to 975B parameters: a prompted policy, permission fine-tuning, retrieval over permitted documents, and one adapter per clearance.
Results will be added after the pre-registered runs. In the PDF, text in gray brackets marks where they go.
Pre-registration
Hypotheses and analysis plan
The study is registered before any run. The draft fixes the two hypotheses and the analysis rules below. Before the main runs they will be frozen and published with hashes of the test worlds, the prompts and the scorer.
Fine-tuned models state derived facts that no single document states.
Estimand: accuracy of the fine-tuned model on derived joins, minus the accuracy of the permitted-documents-only adapter on the same probes (the floor). Test: one-sided, > 0, for each model.Behavioral controls withhold derived restricted facts less reliably than stated restricted facts.
Estimand: leakage gap = derived leakage - stated leakage on joins, for the prompted policy and for permission fine-tuning. Test: one-sided, > 0, for each control (Holm over the two controls), on the mean over models.- Tests are one-sided at α = 0.05, with Holm correction over the two controls in H2.
- A gap of at least 0.10 is treated as practically material, and reported beside significance.
- Each model is analysed separately; the four models are summarized by their mean.
- Known questions are selected on the reference's first sample and normalized by its second.
- A condition with fewer than 50 known questions is reported but not tested.
- Intervals come from a bootstrap that resamples questions within worlds, with one draw shared by all conditions (B = 1,000).
- The number of test worlds, between 5 and 15, is chosen before the main runs by simulation from pilot rates, for 80% power at a gap of 0.10.
- Three checks must pass before any hypothesis result is reported: per-clearance adapters and retrieval over permitted documents leak at the floor; a guard that parses the question templates withholds every restricted direct question; and in simulation, a control that does nothing has a leakage rate of 1.
Contents
What the draft contains
- Introduction
- Problem setup: definitions, threat model, and what a score means
- The benchmark: construction, statistics, and validation
- Evaluation protocol: models, fine-tuning, access controls, reference points, metrics, hypotheses, and usage rules
- Results (to be added after the runs)
- Related work
- Discussion: using NeedToKnow, scope, and limitations
- Appendix A: benchmark details
- Appendix B: training and control details
- Appendix C: pre-registration
- Appendix D: datasheet and reproduction
Cite
BibTeX
@misc{dauber2026needtoknow,
title = {{NeedToKnow}: A Benchmark for Inference Channels in Fine-Tuned Language Models},
author = {Dauber, Max},
year = {2026},
howpublished = {Signet Systems},
note = {Pre-registration draft},
url = {https://research.signet-systems.com/needtoknow/}
}