Reviewed public release package. This report contains only the selected public benchmark materials and disclosed evaluation outputs.
A static, searchable report of a completed Beam evaluation, not a live inference endpoint. Open the viewer to inspect every public question and final answer, or run the offline verifier to reproduce the reference-agreement score.
Can a model return the right number while following a strict answer contract? This small experiment evaluated Beam on 64 deterministically selected, non-commerce questions from the public GSM8K test split.
The sole mismatch deserves inspection rather than concealment. In
gsm8k-test-0403, a 900-watt air conditioner is used five fewer hours daily for
30 days. Beam returned 135, consistent with 135 kilowatt-hours saved.
The reference says 81, but its worked solution treats usage as reduced
to five hours instead of by five hours. The question also labels energy
with power units. This apparent reference error was identified after the run;
the original 63/64 metric remains unchanged. No adjusted 100% score is claimed.
Model: reflection/Beam-1.0-501B-A23B-FP8, revision
acc3abd45767664dd1891b794eeec8c439367fb7.
Temperature 0, at most 2,048 output tokens per request, concurrency 1, six requests
per minute. Only the public question and the declared system prompt were sent.
Gold answers and reference solutions were excluded from requests.
The system prompt is preserved in benchmark.json.
The required final answer is exactly {"answer":"NUMBER"}, where NUMBER is a
plain decimal string. The included scorer normalizes decimal formatting without
rounding. Invalid, failed, and skipped cases stay in the planned-case denominator.
This is not a full or standard GSM8K score. The subset is small and filtered; published benchmark questions may have occurred in model training. Matching reference answers is not independent mathematical adjudication or evidence of unseen-problem generalization.
With Python 3.13 or newer, run:
python -B verify.py
No packages, credentials, network requests, or model calls are needed. The manifest checks every included file; the verifier independently recomputes the published reference agreement from the selected questions and final answers. Hashes establish file integrity, not authorship or proof of model execution.
The published code is the scorer and offline verifier, not the original inference harness or deployment system. Infrastructure and private operator materials are intentionally excluded. The exported final-answer strings are unchanged from the completed run; response identifiers and operational metadata are omitted. The original, fuller execution evidence is retained separately.
GSM8K: OpenAI, Training Verifiers to Solve Math Word Problems (Cobbe et al., 2021).
3101c7d5072418e28b9008a6636bde82a006892cgrade_school_math/data/test.jsonl, 1,319 questions.benchmark.json, leaving 789 eligible
questions; select the first 64 by ascending SHA-256 of
beam-reasoning-check-v1|case_id.LICENSE-GSM8K.txt.Public availability is not a privacy certification. This package contains public academic word problems and numeric final answers; it does not contain customer records, private catalogs, prompts from another application, credentials, or deployment configuration.
The dataset retains its upstream MIT license. No additional open-source license is assigned to the implementation; other rights remain with their respective owners. File integrity and technical disclosure screening do not establish ownership or grant additional rights.