Difficulty

Task families come with graded difficulty levels (ten for the single-answer families). Whether a level is hard for a given model is an empirical question, and it has not been answered yet.

The table below is a placeholder. Numbers are published only after they are measured. Early runs without tools mix wrong answers with format failures, so the two are reported in separate columns.

What the levels are designed to change

• Lower levels use random instances of growing size, which a short script can solve.
• Middle levels use larger inputs only.
• The top levels change the rules or the instance distribution so that the lower-level strategy fails by construction; each such failure is covered by a test.

That is a design intent, not a measured result.

What has been run

Early runs exist, but each cell had only a handful of attempts, which is too few to quote a rate, so none is shown. Three things can be said without a number:

• Without tools, many failures were format failures rather than wrong answers, which is why the table separates the two.
• With a code tool, a model solved most tasks in several families at their hardest levels in those small runs. Those families are treated as a baseline and demo set, not as hard.
• One interactive family did not look saturated at its upper levels, but the sample is too small to say how hard it is. A second interactive family has not been run on any model.

The table fills in only after enough attempts per cell have been run to support a rate. Nothing has been checked by a blind human solve or an expert audit yet.