Vincent Dimitri OngueneA public research studio
⌘ K
Menu
← Research project
A research walkthrough 01 / 07

Same Error, Different Learner

Same error.
Different learner.
Same tutor?

An AI tutor sees the bug. Does it also see what the learner already understands?

Vincent Dimitri Onguene · Preliminary proof of concept

Meet the learners ↓
01 / The same Python program● ● ●
age = input("Enter your age: ")
next_age = age + 1
print("Next year you will be", next_age)

TypeError: can only concatenate str (not "int") to str

P
Prestructural reasoning

“Maybe print() doesn't work properly when the result is a number.”

R
Relational reasoning

“input() returns age as a string, while 1 is an integer. The + operator is therefore being applied to two incompatible types for the intended arithmetic operation, which is why Python raises the TypeError.”

Should these explanations receive the same next step?
4debugging tasks
4reasoning profiles
3 × 3models × repeats
144recorded responses
Read the reasoning, not the label 02 / 07

Four ways to explain
the very same failure.

SOLO means Structure of the Observed Learning Outcome. These letters describe the structure demonstrated in an explanation. They are not fixed student identities, age groups, or diagnoses.

Response structure / adapted diagramPrestructural
THE PROBLEMCueEXPLANATIONResponseIRRELEVANT IDEAK1K2K3K4K5HYPOTHETICAL / NOT GIVENTASK-RELEVANT KNOWLEDGE
× irrelevant idea● relevant, given○ hypothetical, not givengreen links = used · arcs = connected
P / Prestructural

Missing the relevant idea

“Maybe print() doesn't work properly when the result is a number.”

The explanation focuses on an irrelevant feature rather than the cause of this problem.

A cue is linked to an irrelevant idea. The relevant pieces have not been used.

Open a knowledge component

K1: input() returns a string.

How to read this diagram — and how it relates to the original

This is an original teaching adaptation of the SOLO response-structure diagrams in Biggs & Collis (1982). A cue is the problem presented; the response is the learner’s explanation. Here K1–K5 refer to the age example above. The selected links show the researcher-constructed contrast, rather than a diagnostic score.

In the original diagram, × marks irrelevant or inappropriate information, ● marks relevant information given in the display, and ○ marks relevant hypothetical information not given. This adaptation uses muted filled circles for available relevant facts and green links to show which are expressed. The hollow circle illustrates hypothetical information beyond the supplied task; none of these four conditions uses it. Extended Abstract, which goes beyond the given system, is outside these four experimental conditions.

Biggs & Collis, Evaluating the Quality of Learning ↗

The counterfactual design 03 / 07

Change one thing.
Watch what follows.

Within a task and model, the tutor gets the same programming problem. Only the learner’s natural-language reasoning changes. The SOLO label itself is never included in the tutor prompt.

Held fixed within each task
  • PTutoring prompt
  • TProgramming task
  • CErroneous code
  • EError / observed problem
VariedRs

Demonstrated reasoning

PUMR

One explanation per query

ObservedYs

The tutor’s response

Three repeated queries per task × reasoning × model condition.

Ys = M(P, T, C, E, Rs)

A letter can mean two things. “Student P” is shorthand for the Prestructural condition. In the equation, P means the fixed tutoring prompt. R is a Relational profile; Rs is a learner explanation; “Response” in the structure diagram means the explanation being classified.

What this isolates. Differences in demonstrated reasoning, within these constructed examples. It does not establish real students’ abilities, learning gains, or a validated general-purpose tutoring policy.

Read the exact shared tutor instruction
You are an experienced programming tutor helping an introductory Python student debug their code. Your goal is to help the student learn and make progress on the problem. Respond directly to the student based only on the programming task, their code, the error, and the reasoning they have provided. Keep your response concise and natural, as if you were tutoring the student in a one-on-one session.

Each query then supplied Programming Task, Programming Code, Error/Problem, and Student Reasoning. The four task PDFs preserve the queries and screenshots of the resulting replies.

The evidence workbench 04 / 07

One bug. Four explanations.
What does the tutor do?

Choose a task and reasoning profile, then compare all three tutors. Switch repetitions to see the replies collected under the same condition. These are recorded outputs, not live model calls.

Types · TypeErrorFixed code

The age that won’t add up

Write a Python program that asks the user for their age and prints how old they will be next year. Assume the user enters 20.

age = input("Enter your age: ")next_age = age + 1print("Next year you will be", next_age)

TypeError: can only concatenate str (not "int") to str

Illustrative trace

A step-by-step explanation of this fixed program. No code is executed in your browser.

Change the demonstrated reasoning

Prestructural

“Maybe print() doesn't work properly when the result is a number.”

The explanation focuses on an irrelevant feature rather than the cause of this problem.

Task-relevant knowledge components
  1. input() returns a string.
  2. 1 is an integer.
  3. + is applied to age and 1.
  4. The types are incompatible for the intended arithmetic.
  5. That incompatibility causes the TypeError.
Compare the recorded replies

Task 1 · P / Prestructural

Each panel preserves the screenshot fragments following its numbered Output marker. Click a screenshot for full size. The original headings vary (for example “GPT 5.6 High” and “GPT 5.6 sol”); the manuscript groups these as GPT-5.6 (Sol). Some Task 2 GPT headings are omitted in the record and grouped by the repeated sequence. Inspect all 144 records and attribution notes →

A measurement, not the whole story 05 / 07

Helpful answer.
Whose next step?

Solution leakage means supplying the learner’s next substantive programming step, construct, expression, or valid repair. A prompt can narrow the search while still leaving that step to the learner.

Try the coding distinction

For the age bug, which reply supplies the decisive repair?

These two short examples are teaching illustrations, not additional experimental responses.

Choose a reply to see the coding rationale.

What the study distinguishes

Variation ≠ calibration.

A tutor can acknowledge a learner’s correct causal reasoning, change its tone, and still give away the same repair.

Likewise, withholding a solution does not by itself show that a learner received useful guidance or learned more.

See the preliminary findings ↓
A concrete contrast / Task 1, GPT-5.6, first response

To P: corrects the idea that print() caused the error.

To R: confirms the learner’s explanation of incompatible types.

Both replies supply the decisive conversion / repair.

This is the paired qualitative example discussed in the manuscript, not a claim that the two responses are identical.

Inspect P’s reply ↗ Inspect R’s reply ↗
Preliminary findings / author-coded 06 / 07

Different leakage rates.
No simple adaptation story.

The updated manuscript reports 144 responses: 4 tasks × 4 reasoning profiles × 3 models × 3 repeated queries. These findings describe this small constructed proof of concept.

Decisive repair supplied48 responses per model
0%50%100%
GPT-5.6 (Sol)48/48 100%

95% Wilson interval: 92.6–100%

Gemini 3.1 Pro12/48 25%

95% Wilson interval: 14.9–38.8%

Qwen 3.8 Max47/48 97.9%

95% Wilson interval: 89.1–99.6%

Solid bars = reported leakage proportion. Fine rules = reported 95% Wilson intervals. Descriptive uncertainty does not turn these constructed tasks into a representative sample of learners or tutoring situations.

Read the finding carefully

Less leakage is one result.
Better learning is another question.

GPT-5.6 and Qwen supplied the decisive repair in nearly every recorded reply. Gemini did so less often, and its leakage classification varied more across repeated queries.

The manuscript does not establish that lower leakage causes better learning, or that one model is the best tutor.

Follow the evidence ↗
Two other dimensions, as reported in the updated manuscript
ModelP → R changeStable conditions
GPT-5.6 (Sol)0 percentage points16/16
Gemini 3.1 Pro0 percentage points6/16
Qwen 3.8 Max+8.3 percentage points15/16

P → R change: mean within-task change in leakage, from Prestructural to Relational. The manuscript reports exact task-blocked permutation p = 1.000 for all three models. This does not prove absence of adaptation.

Stable conditions: task × reasoning conditions whose three replies shared the same binary leakage label. It measures agreement in leakage classification, not identical wording or an identical tutoring policy.

Source versions, coding limits, and what has been checked

The updated two-page manuscript, Same Error, Different Learner, pp. 1–2, supplies the four-task totals and results here: GPT-5.6 48/48; Gemini 12/48; Qwen 47/48. Its four-task scope supersedes the older 20-slide presentation’s three-task / 108-response summary. The older deck also has an inconsistent GPT-4o label; the updated manuscript identifies GPT-5.6 (Sol).

All 144 numbered response records have been retained from the four task PDFs. The manuscript reports manual coding, but a response-level coding table and analysis script were not supplied. These aggregate statistics and tests are reproduced as author-reported findings; the website does not independently recode replies or reproduce the inferential analysis. Model sampling settings and run timestamps are not fully specified in the supplied material.

Gemini’s average 0-point P → R change combines opposing task-level patterns: the manuscript reports −33.3 points on one task and +33.3 on another. An average can hide changes in opposite directions.

From proof of concept to benchmark 07 / 07

A sharper question.
A stronger next study.

The preliminary result is a reason to look more closely at tutoring behavior. The next phase needs independently validated reasoning contrasts and measures beyond whether a repair was revealed.

01 / Validate the contrast

Does the response demonstrate the intended structure?

Proposed: expert review of task realism, blind SOLO classification, agreement measurement, and adjudication before freezing benchmark items.

02 / Broaden the evidence

More tasks. Real learner reasoning.

Extend beyond four researcher-authored tasks, three models, and three repeats per condition. Record version, sampling settings, and collection context consistently.

03 / Measure the pedagogy

What next step does the learner get to take?

Examine diagnostic questions, scaffolding, acknowledgment of demonstrated understanding, and learner outcomes alongside leakage.

The question is not only
“Did the tutor fix the code?”
“Did the tutor respond to the reasoning?”
Explore all 144 replies ↗Discuss the research or partner →Start again ↑
Sources and reading notes
  • Same Error, Different Learner.pdf, two-page manuscript: current method, 144-response results, limitations, and qualitative comparison.
  • Same-Error-Different-Learner-Same-Tutor.pdf, 20-slide deck: earlier narrative and three-task results, used for context rather than the current totals.
  • Task1 (5).pdf (25 pages), Task 2 (5).pdf (31 pages), Task 3 (2).pdf (24 pages), and Task 4 (3).pdf (38 pages): task prompts, reasoning conditions, and recorded output screenshots.
  • Biggs, J. B., & Collis, K. F. (1982). Evaluating the Quality of Learning: The SOLO Taxonomy. Structural inspiration for the interactive teaching diagram.

The diagram, code traces, navigation, and coding exercise are explanatory additions. They are not extra experimental observations. Task descriptions and knowledge-component labels are lightly condensed for display; the learner reasoning preserves the supplied wording.

01 / 07 · The question

Follow a question.

Search notes, projects, essays, and research streams.

Open full search →