Why solution leakage is not enough to measure adaptation

A necessary distinction

Consider two tutors that never reveal the final corrected program. One asks every learner to “think carefully about the loop.” The other asks a question that engages directly with a relationship the learner has already described.

Both might receive the same solution-leakage label. That shared label cannot establish whether either tutor adapted to the learner’s demonstrated reasoning.

Working definition

Solution leakage: supplying the learner’s next substantive programming step or repair, removing the need for the learner to generate it.

This is a proposed operational definition for the counterfactual benchmark. It will need examples, boundary cases, and an annotation protocol before it can support reliable measurement.

A question can leak a solution. “What happens if you move the reset outside the loop?” already names a repair. A direct explanation can preserve useful work if it addresses a prerequisite rather than the next substantive step. Surface form alone does not settle the classification.

Adaptation needs a separate account

The benchmark should record both what is supplied and how the response engages with the demonstrated explanation. Possible dimensions include:

• Whether the tutor acknowledges the learner’s identified elements.
• Whether it asks the learner to connect those elements.
• Whether it requests a diagnostic trace before intervening.
• Whether the learner still has a substantive inference to make.

These dimensions are candidates for an evaluation design, not established measures.

What would change my mind?

If a carefully validated leakage measure captured all meaningful differences between tutor responses in the controlled design, the separate adaptation dimensions might add little. The analysis should allow for that possibility rather than assume the constructs must diverge.

The next step is to connect this distinction with reasoning transitions and the proposed observe-first tutor.