Validating specs¶
The automath check command group validates question specifications without
rendering anything. It powers two workflows:
- the live validation in the
automathoneditor (Progressions and Scripts panels), and - the
/check/qspecand/check/qblock/batchendpoints ofautomathd.
Two subcommands are available: qspec (one spec line) and qblock (a whole
questions: block, including multi-line pipe/mix/pool notation).
One level up, a whole script document — structure included, not only its
questions — is validated by the
/check/document endpoint of
automathd. It has no CLI counterpart: on the command line, rendering the
script is the more direct way to find out whether it holds together.
automath check qspec¶
Validate a single question-spec line — the full body that would follow the
leading (n) count in a YAML questions block.
Exit code 0 on a valid spec, 1 otherwise (the error message is printed to
stderr).
A trailing # comment is dropped before the spec is read, so a line copied out
of a questions: block checks the same way on its own. A spec that is nothing
but a comment is an error — there is no spec to check.
The spec is read exactly as the render reads a line of a block, with its
leading (n) as the count if it has one, (1) otherwise — the count the
Progressions panel gives it in an assessment draft. So a spec that validates
here is never refused by the render for its notation: {multi_direct} 2-9×2-9
(a one-entry group reads as a mix, which needs a source group) and
multi_direct {2-9×2-9} (after a lone identifier, only a | group) are
errors. Mix notation proper is not accepted in a single spec.
Warnings¶
A spec can be perfectly valid and still be unlikely to give what you meant —
typically a constraint that leaves only a handful of drawable values. Those
remarks are printed to stderr as Warning: lines and do not change the
exit code.
Warning: order_of_2ops variant 24 (a:b-c): only 2 of the 13 values of the
singleton source are usable with result=positive. Consider a dedicated line
for this variant, or exclude it with variant=^24.
What a line will give — --report¶
A warning says this line is unlikely to give what you want. --report says
what it will give: how many questions the line can draw, the range its
results fall in, and the hardest calculation of each kind it can pose, with
an example of each. On stdout, and only when asked for.
It is not a sample. Where a question type can enumerate every question a line is capable of, the extremes printed are the real ones — which is the point: a handful of drawn questions says nothing about what sits at the edge of a range, and the edge is exactly what nobody can foresee from the range itself.
expr_eval: shapes selected: 1. Questions they can draw: 4000.
expr_eval: the result falls between 125 and 1 185.
expr_eval: it multiplies within the tables, the most it asks being 12 × 11,
in 12x + 88, x = 11.
expr_eval: it multiplies beyond the tables, the most it asks being 11 × 99,
in 11x + 21, x = 99.
Difficulty is read as a demand, not as a size: it is easier to multiply by 1 000 than by 79, and easier still to multiply 9,6 by 10 — the point moves and the digits stay. The bands are listed lowest first, so the list shows the most trivial question a line can ask as plainly as the hardest.
A line that cannot work is refused rather than described, exactly as it is
without --report. A question type with nothing to say about its own output
prints nothing — the flag is a hook, and only the types that implement it
answer. --report and --batch are exclusive.
Batch mode¶
With --batch, one qspec is read per line from stdin (blank lines and
comment-only lines ignored) and a JSON array is written to stdout — one object
per remaining line, in input order. The command always exits 0 in batch mode,
regardless of individual results. This is far cheaper than spawning automath
once per qspec.
A valid entry carries an extra warnings array when there is something to
report: {"valid": true, "warnings": ["<message>", "..."]}. The key is
omitted when the list would be empty.
automath check qblock¶
Validate a whole questions: block. Unlike qspec, qblock understands the
multi-line pipe/mix/pool notation: it walks the logical structure and validates
every identifier and every source mentioned, returning the full list of errors
per block.
A block is checked as deeply as qspec checks a single line. Every
(identifier, sources, options) pairing a line can resolve to — the cartesian
product, since mix notation shuffles its sources before pairing them — goes
through the same per-question-type checks: the number of sources the type
accepts, the steps option, and the type's own diagnostics. So an arity
mistake or an unusable option is reported here and not left to the render.
The notation itself is read by the same code as the render's, so its rules are
enforced identically, with the same messages: a mix line's (n) (a pool
line's weight) must equal its number of identifiers and of sources, | and
; cannot be combined, a mix needs a source group, and nothing but -- …
options may follow a source group.
Only --batch is supported. Pass a JSON array of {name, text} objects on
stdin (one per block — for example, one per YAML series in a file) and read a
JSON array of {name, errors} results in the same order from stdout. The
command always exits 0.
echo '[{"name": "series_a", "text": "questions:\n - (5) multi_direct 2-9×2-9\n"}]' \
| automath check qblock --batch
Each error carries a line field (1-based, relative to the start of the block)
so an editor can map the message back to the right line of the YAML. Comment
lines count: they are emptied, not removed, so the numbers keep matching the
text the author is editing.
A block that is valid yet questionable also carries a warnings array of the
same shape, present only when there is something to report:
Warnings do not make a block invalid: what they describe renders, just probably not the way its author meant.