bioRxiv2020
It is argued in [1] that [2] was able to classify EEG responses to visual stimuli solely because of the temporal correlation that exists in all EEG data and the use of a block design. While one of the analyses in [1] is correct, i.e., that low-frequency slow EEG activity can inflate classifier performance in block-designed studies [2], as we already discussed in [3], we here show that the main claim in [1] is drastically overstated and their other analyses are seriously flawed by wrong methodological choices. Our counter-analyses clearly demonstrate that the data in [2] show small temporal correlation and that such a correlation minimally contributes to classification accuracy. Thus, [1]s analysis and criticism of block-design studies does not generalize to our case or, possibly, to other cases. To validate our counter-claims, we evaluate the performance of several state-of-the-art classification methods on the dataset in [2] (after properly filtering the data) reaching about 50% classification accuracy over 40 classes, lower than in [2], but still significant. We then investigate the influence of EEG temporal correlation on classification accuracy by testing the same models in two additional experimental settings: one that replicates [1]s rapid-design experiment, and another one that examines the data between blocks while subjects are shown a blank screen. In both cases, classification accuracy is at or near chance, in contrast to what [1] reports, indicating a negligible contribution of temporal correlation to classification accuracy. We, instead, are able to replicate the results in [1] only when intentionally contaminating our data by inducing a temporal correlation. This suggests that what Li et al. [1] demonstrate is simply that their data are strongly contaminated by temporal correlation and low signal-to-noise ratio. We argue that the reason why Li et al. in [1] observe such high correlation in EEG data is their unconventional experimental design and settings that violate the basic cognitive neuroscience study design recommendations, first and foremost the one of limiting the experiments duration, as instead done in [2]. The reduced stimulus-driven neural activity, the removal of breaks and the prolonged duration of experiments in [1], removed the very neural responses that one would hope to classify, leaving only the amplified slow EEG activity consistent with a temporal correlation. Furthermore, the influence of temporal correlation on classification performance in [1] is exacerbated by their choice to perform per-subject classification rather than the more commonly-used and appropriate pooled subject classification as in [2]. Our analyses and reasoning in this paper refute the claims of the "perils and pitfalls of block-design" in [1]. Finally, we conclude the paper by examining a number of other oversimplistic statements, inconsistencies, misinterpretation of machine learning concepts, speculations and misleading claims in [1]. NoteThis paper was prepared as a response to [1] before its publication and we were not given access to the code (although its authors had agreed, through the PAMI EiC, to share it with us). For this reason, in the experiments presented in this work we employed our own implementation of their model.