Same Error, Different Learner

Research question

When the programming task, erroneous code, observed error, and tutoring prompt are held constant, does changing only the structure of demonstrated learner reasoning cause an LLM tutor to change its behavior?

The contrast is deliberately narrow. A tutor might give different answers to different programming problems simply because the problems differ. This project asks whether a change in what the learner demonstrates is itself enough to change the pedagogical response.

Why it matters

A response can sound supportive while remaining insensitive to the learner’s explanation. Likewise, a tutor can avoid revealing a solution without offering useful scaffolding. The benchmark therefore treats solution leakage as one measurement among several.

The broader goal is to distinguish responding to a problem from responding to a learner.

Current hypothesis

Working hypothesis: some tutoring behaviors may respond to broad changes in demonstrated reasoning while failing to distinguish finer transitions. This is a proposal to test, not an empirical finding.

A useful outcome would be an account of where adaptation occurs, where it remains indistinguishable from ordinary response variation, and which pedagogical choices change.

Methods

For each controlled problem instance:

$$Y_i = M(P,T,C,E,R_i)$$

• $P$: tutoring prompt, held constant.
• $T$: programming task, held constant.
• $C$: erroneous code, held constant.
• $E$: observed error, held constant.
• $R_i$: demonstrated learner reasoning, varied.
• $Y_i$: tutoring response, observed.

“Constant” needs to include formatting and the placement of each prompt component. Each contrast should preserve all content outside the learner-reasoning field. Model versions, sampling parameters, run order, and repeated trials would also need to be recorded.

Why the unit of analysis needs care

A pair of responses is not enough to establish sensitivity to reasoning. Sampling variability, annotation disagreement, and differences between task instances can all affect the comparison. The analysis plan should specify the sampling unit, repetitions, coding rules, and uncertainty estimates before drawing conclusions.

Experiment

The proposed conditions are SOLO-informed: Prestructural, transitional 1A, Unistructural, transitional 2A, Multistructural, transitional 3A, and Relational. The intermediate labels describe candidate operational contrasts within this project. They should not be presented as validated diagnostic categories.

The accompanying transition note asks how to vary structure without changing factual correctness, verbosity, or the amount of task information available to the tutor.

An illustrative erroneous program:

def total(values):
    for value in values:
        running = 0
        running += value
    return running

The intended task is to sum a non-empty sequence of numbers. A trace of this example can support several different demonstrations of reasoning without changing the underlying error. It is not a released benchmark item.

Results

No results are reported here. The interactive graphic illustrates a proposed comparison and does not call a model, score a tutor, or estimate adaptation.

Candidate coding dimensions include diagnostic questioning, scaffolding, engagement with the demonstrated explanation, solution leakage, and preservation of the learner’s next substantive step. Definitions and annotation procedures remain to be finalized.

Open questions

1. What response difference counts as pedagogical adaptation rather than paraphrase?
2. How can the reasoning contrast remain independent of factual correctness?
3. What is the smallest meaningful distinction in demonstrated reasoning that changes tutor behavior?
4. How much of the result depends on the task family, model, or tutoring prompt?
5. How should annotation uncertainty affect claims about the resolution of adaptation?

Artifacts & timeline

The current artifact is a working research design. No code repository, dataset download, registration, or manuscript link has been supplied yet.

The intended sequence is question → operational definitions → contrast construction → annotation protocol → pilot → revision → benchmark release → publication. These are phases of work, not a claim that each has been completed.

References & reading

1. Biggs & Collis (1982). Evaluating the Quality of Learning: The SOLO Taxonomy. Academic Press. ↗