Structure is not the same as factual correctness
Two dimensions that a reasoning-adaptation benchmark needs to keep separate.
Separate the axes
An explanation may connect several elements coherently while relying on a false premise. Another may state one correct fact without connecting it to anything else. Calling the first “more structured” does not make it more factually correct.
This distinction matters when constructing SOLO-informed contrasts. If structure and correctness always rise together, the tutor’s response cannot tell us which property it is responding to.
A construction rule to investigate
For each task, write down the factual claims and relationships in a candidate explanation separately. Audit which claims are correct, which elements are relevant, and which relationships the learner explicitly demonstrates.
| Dimension | Question for the item audit |
|---|---|
| Task facts | What information is available to the tutor? |
| Correctness | Which assertions are true of the program? |
| Structure | Which relevant elements are identified or related? |
| Expression | Does wording, length, or confidence introduce a cue? |
This table is a design aid, not a validated rating rubric.
A limit worth preserving
The benchmark should report the controlled contrasts it actually constructs. It should not claim a general separation of structure and correctness merely because the item labels are different.
References & reading
Connected thinking.
Related reading 3
When relational-looking output no longer implies relational understanding
Adaptive AI tutoringSOLO transition structures and the resolution of adaptation
Adaptive AI tutoringSame Error, Different Learner
Adaptive AI tutoringLinked from 3
Backlinks from declared content relationships.
NoteWhen relational-looking output no longer implies relational understanding ↗NoteSOLO transition structures and the resolution of adaptation ↗ProjectSame Error, Different Learner ↗