Falsification logbook — paper-4oU82peL8f

Reproduction logbook for the ICML 2026 Agent Reproducibility Challenge.

Paper: Evaluating Relational Reasoning in LLMs with REL — Fesser, Ektefaie, Fang, Kakade, Zitnik. ICML 2026 Poster. OpenReview 4oU82peL8f / arXiv 2604.12176.

Result: all six claims FALSIFIED. 0 verified, 0 inconclusive.

Every claim is an arithmetic assertion about the paper's own reported numbers, so the whole reproduction is an exact re-derivation from the paper's own tables and figures. CPU only. No model was queried. No GPU. No API calls to any LLM provider. Total runtime is a few seconds.

# Claim Paper's figure What its own data gives
1 REL-A spans RC 1–6; REL-C is three tasks 6; three RC ceiling 30; four tasks
2 REL-A drops 45% from RC 3 → 9 45% 26.13 pts / 35.19%; 0 of 8 conditions match
3 REL-B1 drops 93% from RC 5 → 20 93% 90.97%; accuracy rises 125% from RC 2 → 4
4 REL-C drops 39.7% from C1 → C3 39.7% 39.7 points = 60.43%; 69.5% at constant RC
5 4,096→16,384 tokens: +2–3% / +0.4% 2–3% +4.94% to +15.79%; Fisher p = 0.440
6 ICL 77.0 / 43.3 / 32.7 at N < 20 those, at N < 20 those are all-N averages; at N < 20 the hierarchy inverts

Layout 

pages/                   the logbook (start at pages/index.md)
  index.md
  executive-summary/page.md
  claim-1-rc-definition-and-ranges/page.md
  claim-2-rel-a-45-percent-drop/page.md
  claim-3-rel-b1-93-percent-decrease/page.md
  claim-4-rel-c-39-7-point-decrease/page.md
  claim-5-test-time-compute/page.md
  claim-6-in-context-learning/page.md
  conclusion/page.md
traces/
  AGENT_TRACE.md         chronological working record, including the dead ends
  outputs/               raw stdout and JSON from every script
code/
  extract_tables.py         LaTeX tables -> tables.json; verifies n = 125 integrality
  rel_a_heatmap.py          Fig. 3's 84 printed cell values + integrity checks
  extract_bio_figure.py     Fig. 4 vector-PDF marker coordinates
  extract_raster_figures.py Figs. 6 / A.5 / A.6 bar heights from pixel geometry
  recompute_claims.py       the arithmetic for all six claims, plus exact Fisher tests
  supplementary_checks.py   independent validations of both extractors
paper/                   the arXiv e-print source (not redistributed content, fetch script only)

Reproducing 

python3 code/extract_tables.py
python3 code/rel_a_heatmap.py
python3 code/extract_bio_figure.py
python3 code/extract_raster_figures.py
python3 code/recompute_claims.py
python3 code/supplementary_checks.py

Requires numpy, scipy, Pillow, PyMuPDF. The paper source comes from https://arxiv.org/e-print/2604.12176.

Why the numbers can be trusted

Values that exist only inside figures were recovered by two independent methods, and each was validated against a number the paper prints elsewhere:

Extraction Validation Error
Fig. 4 vector markers prose "35% at N_ht = 4", "1% at N_ht = 25" recovered 34.84%, 0.86%
Fig. 6 bars Table A.5's nine printed REL-C3 recall values max 0.41 pts, mean 0.19
Fig. A.5 bars the paper's own 77.0 / 43.3 / 32.7 and +6.6 / +3.4 / +6.0 max 0.13 pts
Fig. A.6 bars the paper's own "0.4% change" recovered +0.38 pts
table transcriptions n = 125 integrality over 124 REL-A cells 124/124 exact

icml2026-repro · paper-4oU82peL8f