From programming tutors to physical-world learning companions

A question for later systems

What would change if a tutor could observe a learner working through a handwritten derivation, assembling an object, or drawing a diagram?

The ambition is broader than adding a camera or voice interface. A system would need to decide which parts of the visible activity are meaningful evidence and when an intervention would preserve the learner’s opportunity to reason.

What the current work can contribute

The counterfactual benchmark may help sharpen a distinction between reacting to an error and engaging with demonstrated reasoning. It cannot establish that the same approach works across perception, handwriting, or physical tasks.

Open design problems

• What observation has enough context to be interpretable?
• How does the learner correct a mistaken interpretation?
• When should the system wait?
• Which records should stay local, and which should not be retained at all?

These questions belong to the proposed observe-first tutor, not to a claim that a multimodal system has already been built.