An experiment we did on the recalibration of visuomotor simultaneity perception to vision-lead and movement-lead temporal discrepancies was published as part of the research topic “time and causality” in Frontiers in Psychology. This is the first in a series of experiments where we use signals from early movement onset to predict the timing of a button press (see Figure below) and thus be able to present visual stimuli even before the button press.
In this first paper, we demonstrate that humans can adapt their perception of simultaneity to both movement-lead and vision-lead temporal discrepancies. However, a more complete picture of temporal recalibration arises only if follow-up work (under review) is taken into account. Therefore, I do not want to comment on the results themselves at this moment in time.
Instead I want to make some remarks about the idea of “tricking” the underlying temporal structure of cause and effect by predicting action and presenting visual events even before the voluntary movement event happens. In the study of sensorimotor timing and time perception, the relative timing of voluntary actions and sensory events cannot be manipulated as easily as, for instance, spatial offsets or the relative timing of external sensory events. This is because the human decides when to act, and this decision furthermore depends on previous sensory inputs, if they are early enough to react to them. How is it possible to present a signal just before a voluntary action?
I got inspired by previous research by Stetson et al. (2006), who estimated the time of a button press from previous rhythmic actions or a “go-signal” to probabilistically time visual stimuli to occur before a voluntary button press. This is a clever idea. Yet, there is likely too much noise in the prediction to use this kind of estimate for recalibration. Also, I couldn’t help feeling that the existence of a clearly perceptible lead event could be a confound in this kind of research. Therefore, I got very excited when I read Dennett & Kinsbourne’s (1992) article on temporal consciousness, where, among other things, they discuss Grey Walter’s reports on using pre-motor brain activity in neurosurgical patients to trigger events in the real world, thereby finding a “shortcut” for inherent sensorimotor latencies.
It is well-established that neural correlates of voluntary action occur even before humans become aware of committing to the execution of an action (Readiness Potential or Contingent Negative Variation, e.g., Libet et al., 1983). Even if it seems odd that Grey Walter’s work on using these to generate actions, which sounds rather revolutionary, has never been presented in written form, the reports were encouraging enough to start a project on using brain-computer-interface (BCI) in real-time for the generation of psychophysical stimuli. This work was done together with Nicholas Del Grosso and Michael Barnett-Cowan at the Max Planck Institute for Biological Cybernetics in Tübingen in 2010/2011. We used an EEG system there and received both support and encouragement for this project from researchers in EEG imaging and BCI, meaning that the idea was, at least superficially, not completely crazy.
Still, at the end of the day, this project turned out to be too ambitious. I was humbled when realising how noisy EEG data are on a trial-by-trial basis and have huge respect for BCI researchers using EEG. These algorithms involve cutting edge machine learning techniques that detect non-linear interactions in time, space and in the frequency domain together, using built-in knowledge about the underlying neural processes and training both the participant to the algorithm and the algorithm to the participant. The filters can thus detect the neural correlate of an action that does not even take place in real time, on a trial-by-trial basis (ca. 80% classification success). Yet, for our purposes, i.e., to reliably predict a motor event before it takes place with a more or less constant temporal offset, these algorithms did not really work. The strongest time domain signals (e.g., P300) occur after the action, not before, and any correlate prior to action will likely be detected with considerable variability concerning the exact time of detection, if at all. Of course, it is possible that our limited experience with either EEG measurement or non-linear filtering was the root of the problem. Yet, from the literature and my experience, I think that there are currently no tools sensitive enough to solve our problem to predict a button press in real time with high temporal accuracy on a trial-by-trial using EEG.
Thus, when I moved back into the realm of behavioural prediction, it was at first out of need. In retrospect, however, there were also conceptual, not just technical reasons to discontinue research using BCI for this kind of stimulus generation. In our current research (as described in the paper), we use Phantom force-feedback devices (Sensable Technologies Inc.) to haptically display buttons. These devices track participants’ movements as well as display forces, so when participants initiate the press movement, to press the button, this early movement onset can be registered and used to predict the timing of a button press in real time, in order to display a visual flash between 200 ms and 10 ms beforehand (with some error, of course). Using this technique, we encountered a problem. Participants started to construct causal shortcuts to explain why events reliably happened before they performed an action. For instance, they speculated about changes in the sensitivity of the button. When we tried to address this problem by making participants perform a reach through the air before they pressed the button and used this movement to predict the button press, they started speculating about a light barrier installed in the set-up. Only when, verbally/cognitively, we provided participants with alternative causal origins for these early flashes, such as another person performing the same experiment in a different room or the computer randomly producing flashes at certain times, participants stopped deriving causal shortcuts and just did the task. (This corresponds to a relaxation of the criterion of exclusivity; Wegner & Wheatley, 1999)
In this kind of research, the perception of causality and the perception of time are intricately linked, and it is hard to eliminate possible confounds between the two, as a change in the one is likely preceded, followed or accompanied by a change in the other. The best way of controlling these appears to be both on the level of stimulus and on the level of subjects’ beliefs about the causal structure underlying the task. Therefore, I am rather convinced that, had we succeeded with the BCI approach, this effort would have been in vain. Subjects would have also constructed external causal shortcuts, such as a light barrier, to explain the fact that stimuli occur prior to their movment.
- Dennett, D. C., & Kinsbourne, M. (1992). Time and the observer: The where and when of consciousness in the brain. Behavioral and Brain Sciences, 15, 193–247.-
- Libet, B., C. A. Gleason, E.W. Wright & D.K. Pearl (1983): Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). The unconscious initiation of a freely voluntary act. Brain 106 (3): 623-42.
- Rohde, M., & Ernst, M.O. (2013): To lead and to lag – forward and backward recalibration of perceived visuo-motor simultaneity. Front Psychology 3, 599. Research topic: Time and Causality – Frontiers in Perception Science.
- Stetson, C., Cui, X., Montague, P. R., and Eagleman, D. M. (2006). Motor-sensory recalibration leads to an illusory reversal of action and sensation. Neuron 51, 651–659.
- Wegner, D. M., & Wheatley, T. P. (1999). Apparent mental causation: Sources of the experience of will. American Psychologist, 54, 480-492.