Why does exercise material need renewing?

Gains follow the material worked on

Castro, Nadeau and Kendall took up a long-standing difficulty in the treatment of anomia: trained words improve, untrained words much less so. Their work examines the phonological proximity between the items worked on and those that are not, using data from a trial of phonomotor treatment.

The stakes are as follows. If generalisation depends on the kinship between items, then the choice of material is not a matter of organisation: it determines the extent of the benefit. A patient who works for months on a stable set of cards improves on that set of cards.

The observation is not specific to these authors. Limited generalisation is a recurring finding in language rehabilitation, to the point that it constitutes one of the main arguments against the idea that targeted training alone would restore a function. The contribution of this work is to examine a precise explanation rather than to observe the phenomenon yet again.

Castro et al., Aphasiology, 2022↗

Retaking the same test measures something else

The second finding comes from quite another field. Calamia, Markon and Tranel aggregated nearly 1,600 effect sizes from the neuropsychological literature to quantify what is known as the retest effect: the improvement in score produced by the mere repetition of a test, with no real improvement in the function being assessed.

The average effect comes to around a quarter of a standard deviation. That is not negligible at the scale of an assessment, and the effect is all the more marked when the interval between the two administrations is short. In other words, an improvement observed between two closely spaced assessments on the same material contains a share of familiarity that nothing allows one to isolate after the fact.

Calamia et al., The Clinical Neuropsychologist, 2012↗

Two distinct literatures, one shared consequence

It is worth being clear on this point, because the temptation to merge these two results is strong. Castro deals with the generalisation of a treatment in aphasiology; Calamia deals with the validity of a repeated measure in neuropsychology. They are two different objects, studied in two fields that rarely cite each other.

They do not demonstrate the same thing and need not be presented as a single result. What they share is a practical consequence: in both cases, keeping the material stable degrades the very thing one is after. It limits the extent of progress in Castro; it distorts its measurement in Calamia.

Caution is all the more warranted because the two studies have a different status. Calamia’s is a meta-analysis covering a very large number of effects, and its order of magnitude is solid. Castro’s is a secondary analysis of trial data, with the ambition of explaining limited generalisation rather than measuring it once more. The latter sheds light on a mechanism; it does not establish a rule.

Finally, neither says at what pace to renew. The question of the right interval remains open, and this page cannot settle it.

How to spot it in session

The drift described here has an awkward peculiarity: it looks like success. The patient answers better, faster, with less hesitation. Nothing in the session signals that the improvement bears on the material rather than on the function.

A few clues exist nonetheless. A widening gap between performance on the usual material and performance on unfamiliar material is the main one. A response latency that collapses while the stated difficulty has not changed is another. The mismatch reported by those close to the patient also counts: when family and friends perceive no change while the scores keep rising, the hypothesis deserves to be raised.

None of these signs is a test, and the literature cited here proposes no screening procedure. They are markers drawn from experience, presented as such.

What this means in everyday practice

The first consequence concerns the work itself. A fixed stock of exercises ends up training the memory of that stock rather than the target function, and this drift is all the harder to spot because the patient is, in fact, giving correct answers.

The second concerns assessment. Reusing in an assessment material that has served in rehabilitation amounts to measuring, in part, the training received on that very material. The principle of separating training material from assessment material is long established; these studies give it an order of magnitude. The corollary also holds for comparisons over time: two closely spaced assessments on the same tests do not compare like two independent measures, and the observed difference overestimates real progress by an amount that nothing allows one to subtract after the fact.

The third is the most concrete, and the one least talked about: producing new material costs preparation time, outside the sessions and outside what is paid for. The constraint is not theoretical, it is logistical, and that is probably why it is so little discussed in the literature itself.

Renewing does not mean changing at random. What these studies suggest is varying the material while keeping the instructions, the level of difficulty and the target function constant. A brand-new series whose parameters have not moved goes on training the same thing; a series that changes content and difficulty at the same time makes progress unreadable. The useful variation bears on the items, not on the task.

One must also accept that the patient will do less well. On new material, scores drop compared with familiar material, and that drop is not a regression: it is the disappearance of the familiarity share. This is a delicate moment to get through, for the patient and for those around them, and it is best announced in advance.

How doMind puts this into practice

This is precisely the problem doMind seeks to reduce. The platform generates variants of the same activity from a pool of media, in French, English and Spanish, which makes it possible to renew the material without rebuilding the exercise. The clinician keeps the choice of level and targets; what changes is the time it takes to obtain a brand-new series. Discover the tool.

Book a demonstration →