Performance assessment in brain-computer interface-based augmentative and alternative communication

BioMedical Engineering OnLine

David E Thompson²  Stefanie Blain-Moraes¹  Jane E Huggins¹ 

[1] Department of Physical Medicine and Rehabilitation, University of Michigan, Ann Arbor, MI, USA

[2] Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA

关键词: Information transfer rate;    Outcome measures;    Augmentative and alternative communication;    Brain-computer interface;   

Others  :  797881
DOI  :  10.1186/1475-925X-12-43

 received in 2012-11-27, accepted in 2013-04-17,  发布年份 2013

【 摘 要 】

A large number of incommensurable metrics are currently used to report the performance of brain-computer interfaces (BCI) used for augmentative and alterative communication (AAC). The lack of standard metrics precludes the comparison of different BCI-based AAC systems, hindering rapid growth and development of this technology. This paper presents a review of the metrics that have been used to report performance of BCIs used for AAC from January 2005 to January 2012. We distinguish between Level 1 metrics used to report performance at the output of the BCI Control Module, which translates brain signals into logical control output, and Level 2 metrics at the Selection Enhancement Module, which translates logical control to semantic control. We recommend that: (1) the commensurate metrics Mutual Information or Information Transfer Rate (ITR) be used to report Level 1 BCI performance, as these metrics represent information throughput, which is of interest in BCIs for AAC; 2) the BCI-Utility metric be used to report Level 2 BCI performance, as it is capable of handling all current methods of improving BCI performance; (3) these metrics should be supplemented by information specific to each unique BCI configuration; and (4) studies involving Selection Enhancement Modules should report performance at both Level 1 and Level 2 in the BCI system. Following these recommendations will enable efficient comparison between both BCI Control and Selection Enhancement Modules, accelerating research and development of BCI-based AAC systems.

【 授权许可】

   
2013 Thompson et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

Figure 2.

Figure 3.

【 参考文献 】

• [1]Venkatagiri H: Clinical implications of an augmentative and alternative communication taxonomy. Augment Altern Commun 2002, 18:45-57.
• [2]Glennen SL, DeCoste DC: Augmentative and Alternative Communication Systems. In The Handbook of Augmentative and Alternative Communication. San Diego, CA: Cengage Learning; 1997.
• [3]Tai K, Blain S, Chau T: A review of emerging access technologies for individuals with severe motor impairments. Assist Technol 2008, 20:204-219.
• [4]Mason SG, Birch GE: A general framework for brain-computer interface design. IEEE Trans Neural Syst Rehabil Eng 2003, 11:70-85.
• [5]Farwell LA, Donchin E: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 1988, 70:510-523.
• [6]Townsend G, LaPallo B, Boulay C, Krusienski D, Frye G, Hauser C, Schwartz N, Vaughan T, Wolpaw J, Sellers E: A novel P300-based brain-computer interface stimulus presentation paradigm: moving beyond rows and columns. Clin Neurophysiol 2010, 121:1109-1120.
• [7]Jin J, Allison BZ, Sellers EW, Brunner C, Horki P, Wang XY, Neuper C: An adaptive P300-based control system. J Neural Eng 2011., 8036006
• [8]Takano K, Komatsu T, Hata N, Nakajima Y, Kansaku K: Visual stimuli for the P300 brain-computer interface: A comparison of white/gray and green/blue flicker matrices. Clin Neurophysiol 2009, 120:1562-1566.
• [9]McFarland DJ, Sarnacki WA, Townsend G, Vaughan T, Wolpaw JR: The P300-based brain-computer interface (BCI): Effects of stimulus rate. Clin Neurophysiol 2011, 122:731-737.
• [10]Vidaurre C, Sannelli C, Muller KR, Blankertz B: Co-adaptive calibration to improve BCI efficiency. J Neural Eng 2011., 8025009
• [11]Vidaurre C, Schlogl A, Cabeza R, Scherer R, Pfurthscheller G: A fully on-line adaptive BCI. IEEE Trans Biomed Eng 2006, 53:1214-1219.
• [12]Mason SG, Bashashati A, Fatourechi M, Navarro KF, Birch GE: A Comprehensive Survey of Brain Interface Technology Designs. Ann Biomed Eng 2006, 35:137-169.
• [13]Blankertz B, Dornhege G, Krauledat M, Schoroder M, Williamson J, Murray-Smith R, Muller KR: The Berlin Brain-Computer Interface Presents the Novel Mental Typewriter Hex-O-Spell. 2006. [3rd International Brain-Computer Interface Workshop and Training Course]
• [14]Ryan DB, Frye GE, Townsend G, Berry DR, Mesa-G S, Gates NA, Sellers EW: Predictive spelling with a P300-Based brain-computer interface: increasing the rate of communication. Int J Human-Computer Inter 2011, 27:69-84.
• [15]Furdea A, Halder S, Krusienski DJ, Bross D, Nijboer F, Birbaumer N, Kubler A: An auditory oddball (P300) spelling system for brain-computer interfaces. Psychophysiology 2009, 46:617-625.
• [16]Dal Seno B, Matteucci M, Mainardi LT: The utility metric: a novel method to assess the overall performance of discrete brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 2010, 18:20-29.
• [17]Bianchi L, Quitadamo LR, Garreffa G, Cardarilli GC, Marciani MG: Performances evaluation and optimization of brain computer interface systems in a copy spelling task. IEEE Trans Neural Syst Rehabil Eng 2007, 15:207-216.
• [18]Beukelman DR, Mirenda P: Augmentative and Alternative Communication: Management of Severe Communication Disorders in Children and Adults. Baltimore, MD: Paul H Brookes; 1992.
• [19]Goodenough-Trepagnier C, Rosen MJ: Predicitive assessmet for communication aid prescription: Motor-determined maximum communication rate. In The Vocally Impaired: Clinical Practice and Research. Edited by Bernstein L. Philadelphia: Grune & Stratton; 1988.
• [20]Koester HH, Levine SP: Learning and performance of able-bodied individuals using scanning systems with and without word prediction. Assist Technol 1994, 6:42-53.
• [21]Koester HH, Levine SP: Modeling the speed of text entry with a word prediction interface. IEEE Trans Rehabil Eng 1994, 2:177-187. Sep
• [22]Lesher G, Moulton B: Techniques for augmenting scanning communication. Augment Altern Commun 1998, 14:81-81.
• [23]Hill K, Romich B: A rate index for augmentative and alternative communication. Int J Speech Technol 2002, 5:57-64.
• [24]Waller A, Dennis F, Brodie J, Cairns AY: Evaluating the use of TalksBac, a predictive communication device for nonfluent adults with aphasia. Int J Lang Commun Disord 1998, 33:45-70.
• [25]Scherer MJ, Gray DB, Quatrano LA, Lieberman ML: The impact of assistive technology on the lives of people with disabilities. In Designing and Using Assistive Technology: The Human Perspective. Balitmore, MD: Paul H. Brookes Publishing Co; 1996:99-115.
• [26]Wolpaw JR: Brain–computer interfaces as new brain output pathways. J Physiol 2007, 579:613-619.
• [27]Friedrich EVC, Scherer R, Sonnleitner K, Neuper C: Impact of auditory distraction on user performance in a brain-computer interface driven by different mental tasks. Clin Neurophysiol 2011, 122:2003-2009.
• [28]Hsu WY, Lin CY, Kuo WF, Liou M, Sun YN, Tsai ACH, Hsu HJ, Chen PH, Chen IR: Unsupervised fuzzy c-means clustering for motor imagery EEG recognition. Int J Innovative Comput Inform Control 2011, 7:4965-4976.
• [29]Hsu WY: EEG-based motor imagery classification using enhanced active segment selection and adaptive classifier. Comput Biol Med 2011, 41:633-639.
• [30]Bobrov P, Frolov A, Cantor C, Fedulova I, Bakhnyan M, Zhavoronkov A: Brain-Computer Interface Based on Generation of Visual Images. PLoS One 2011, 6:e20674.
• [31]Arvaneh M, Guan CT, Ang KK, Quek C: Optimizing the channel selection and classification accuracy in EEG-Based BCI. IEEE Trans Biomed Eng 2011, 58:1865-1873.
• [32]Ikegami S, Takano K, Saeki N, Kansaku K: Operation of a P300-based brain-computer interface by individuals with cervical spinal cord injury. Clin Neurophysiol 2011, 122:991-996.
• [33]Krusienski DJ, Shih JJ: Control of a visual keyboard using an electrocorticographic brain-computer interface. Neurorehabil Neural Repair 2011, 25:323-331.
• [34]Cecotti H, Rivet B, Congedo M, Jutten C, Bertrand O, Maby E, Mattout J: A robust sensor-selection method for P300 brain-computer interfaces. J Neural Eng 2011., 8016001
• [35]Li YQ, Nam CS, Shadden BB, Johnson SL: A P300-Based Brain-Computer Interface: Effects of Interface Type and Screen Size. Int J Hum-Comput Interact 2011, 27:52-68.
• [36]Brunner P, Joshi S, Briskin S, Wolpaw JR, Bischof H, Schalk G: Does the “P300” speller depend on eye gaze? J Neural Eng 2010, 7:056013.
• [37]Sellers EW, Vaughan TM, Wolpaw JR: A brain-computer interface for long-term independent home use. Amyotroph Lateral Scler 2010, 11:449-455.
• [38]Hashimoto Y, Ushiba J, Kimura A, Liu MG, Tomita Y: Change in brain activity through virtual reality-based brain-machine communication in a chronic tetraplegic subject with muscular dystrophy. BMC Neurosci 2010, 11:9. BioMed Central Full Text
• [39]Zhang D, Maye A, Gao XR, Hong B, Engel AK, Gao SK: An independent brain-computer interface using covert non-spatial visual selective attention. J Neural Eng 2010., 7016010
• [40]Cabrera AF, Farina D, Dremstrup K: Comparison of feature selection and classification methods for a brain-computer interface driven by non-motor imagery. Med Biol Eng Comput 2010, 48:123-132.
• [41]Guger C, Daban S, Sellers EW, Holzner C, Krausz G, Carabalona R, Gramatica F, Edlinger G: How many people are able to control a P300-based brain-computer interface (BCI)? Neurosci Lett 2009, 462:94-98.
• [42]Fazel-Rezai R, Abhari K: A region-based P300 speller for brain-computer interface. Can J Elect Comput Eng 2009, 34:81-85.
• [43]Kayagil TA, Bai O, Henriquez CS, Lin P, Furlani SJ, Vorbach S, Hallett M: A binary method for simple and accurate two-dimensional cursor control from EEG with minimal subject training. J Neuroeng Rehabil 2009., 6 BioMed Central Full Text
• [44]Friedrich EVC, McFarland DJ, Neuper C, Vaughan TM, Brunner P, Wolpaw JR: A scanning protocol for a sensorimotor rhythm-based brain-computer interface. Biol Psychol 2009, 80:169-175.
• [45]Sano A, Bakardjian H: Movement-related cortical evoked potentials using four-limb imagery. Int J Neurosci 2009, 119:639-663.
• [46]Guo F, Hong B, Gao X, Gao S: A brain-computer interface using motion-onset visual evoked potential. J Neural Eng 2008, 5:477-485.
• [47]Blankertz B, Losch F, Krauledat M, Dornhege G, Curio G, Muller K-R: The Berlin brain–computer interface: accurate performance from first-session in BCI-NaÏve subjects. IEEE Trans Biomed Eng 2008, 55:2452-2462.
• [48]Bai O, Lin P, Vorbach S, Floeter MK, Hattori N, Hallett M: A high performance sensorimotor beta rhythm-based brain-computer interface associated with human natural motor behavior. J Neural Eng 2008, 5:24-35.
• [49]Chatterjee A, Aggarwal V, Ramos A, Acharya S, Thakor NV: A brain-computer interface with vibrotactile biofeedback for haptic information. J Neuroeng Rehabil 2007., 4 BioMed Central Full Text
• [50]Liao X, Yao D, Li C: Transductive SVM for reducing the training effort in BCI. J Neural Eng 2007, 4:246-254.
• [51]Wei Q, Fei M, Wang Y, Gao X, Gao S: Feature combination for classifying single-trial ECoG during motor imagery of different sessions. Progress Natural Sci 2007, 17:851-858.
• [52]Dornhege G, Blankertz B, Krauledat M, Losch F, Curio G, Mueller K-R: Combined optimization of spatial and temporal filters for improving brain-computer interfacing. IEEE Trans Biomed Eng 2006, 53:2274-2281.
• [53]Mahmoudi B, Erfanian A: Electro-encephalogram based brain-computer interface: improved performance by mental practice and concentration skills. Med Biol Eng Comput 2006, 44:959-969.
• [54]Phothisonothai M, Nakagawa M: EEG-based classification of new imagery tasks using three-layer feedforward neural network classifier for brain-computer interface. J Physical Soc Japan 2006, 75:104801. 10401-6
• [55]Ince NF, Arica S, Tewfik A: Classification of single trial motor imagery EEG recordings with subject adapted non-dyadic arbitrary time-frequency tilings. J Neural Eng 2006, 3:235-244.
• [56]Neuper C, Scherer R, Reiner M, Pfurtscheller G: Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Cogn Brain Res 2005, 25:668-677.
• [57]Burke DR, Kelly SR, de Chazal P, Reilly RB, Finucane C: A parametric feature extraction and classification strategy for brain-computer interfacing. IEEE Trans Neural Syst Rehabil Eng 2005, 13:12-17.
• [58]Kamousi B, Liu ZM, He B: Classification of motor imagery tasks for brain-computer interface applications by means of two equivalent dipoles analysis. IEEE Trans Neural Syst Rehabil Eng 2005, 13:166-171.
• [59]Kelly SP, Lalor EC, Reilly RB, Foxe JJ: Visual spatial attention tracking using high-density SSVEP data for independent brain-computer communication. IEEE Trans Neural Syst Rehabil Eng 2005, 13:172-178.
• [60]Treder MS, Schmidt NM, Blankertz B: Gaze-independent brain-computer interfaces based on covert attention and feature attention. J Neural Eng 2011., 8
• [61]Chen M, Guan J, Liu H: Enabling fast brain-computer interaction by single-trial extraction of visual evoked potentials. J Med Syst 2011, 35:1323-1331.
• [62]Kaufmann T, Schulz SM, Gruenzinger C, Kuebler A: Flashing characters with famous faces improves ERP-based brain-computer interface performance. J Neural Eng 2011., 8056016
• [63]Garcia Cossio E, Fernandez C, Eugenia Gaviria M, Palacio C, Alvaran L, Torres Villa RA: P300 based Brain computer interface for alternative communication: a case study with two teenagers with motor disabilities. Rev Fac Ing-Univ Antioquia 2011, 60:9-19.
• [64]Kim DW, Hwang HJ, Lim JH, Lee YH, Jung KY, Im CH: Classification of selective attention to auditory stimuli: toward vision-free brain-computer interfacing. J Neurosci Methods 2011, 197:180-185.
• [65]Pires G, Nunes U, Castelo-Branco M: Statistical spatial filtering for a P300-based BCI: Tests in able-bodied, and patients with cerebral palsy and amyotrophic lateral sclerosis. J Neurosci Methods 2011, 195:270-281.
• [66]Mugler EM, Ruf CA, Halder S, Bensch M, Kubler A: Design and Implementation of a P300-Based Brain-Computer Interface for Controlling an Internet Browser. IEEE Trans Neural Syst Rehabil Eng 2011, 18:599-609.
• [67]Lee PL, Sie JJ, Liu YJ, Wu CH, Lee MH, Shu CH, Li PH, Sun CW, Shyu KK: An SSVEP-actuated brain computer interface using phase-tagged flickering sequences: a cursor system. Ann Biomed Eng 2010, 38:2383-2397.
• [68]Nam CS, Li YQ, Johnson S: Evaluation of P300-Based Brain-Computer Interface in Real-World Contexts. Int J Human-Comput Interact 2010, 26:621-637.
• [69]Chen CW, Ju MS, Sun YN, Lin CCK: Model analyses of visual biofeedback training for EEG-based brain-computer interface. J Comput Neurosci 2009, 27:357-368.
• [70]Bin G, Gao X, Yan Z, Hong B, Gao S: An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method. J Neural Eng 2009., 6046002
• [71]Klobassa DS, Vaughan TM, Brunner P, Schwartz NE, Wolpaw JR, Neuper C, Sellers EW: Toward a high-throughput auditory P300-based brain-computer interface. Clin Neurophysiol 2009, 120:1252-1261.
• [72]Lee P-L, Hsieh J-C, Wu C-H, Shyu K-K, Wu Y-T: Brain computer interface using flash onset and offset visual evoked potentials. Clin Neurophysiol 2008, 119:605-616.
• [73]Hoffmann U, Vesin J-M, Ebrahimi T, Diserens K: An efficient P300-based brain-computer interface for disabled subjects. J Neurosci Methods 2008, 167:115-125.
• [74]Sellers EW, Krusienski DJ, McFarland DJ, Vaughan TM, Wolpaw JR: A P300 event-related potential brain-computer interface (BCI): The effects of matrix size and inter stimulus interval on performance. Biol Psychol 2006, 73:242-252.
• [75]Lee P-L, Hsieh J-C, Wu C-H, Shyu K-K, Chen S-S, Yeh T-C, Wu Y-T: The brain computer interface using flash visual evoked potential and independent component analysis. Ann Biomed Eng 2006, 34:1641-1654.
• [76]Lee PL, Wu CH, Hsieh JC, Wu YT: Visual evoked potential actuated brain computer interface: a brain-actuated cursor system. Electron Lett 2005, 41:832-834.
• [77]Lee P-L, Yeh C-L, Cheng JY-S, Yang C-Y, Lan G-Y: An SSVEP-Based BCI Using High Duty-Cycle Visual Flicker. IEEE Trans Biomed Eng 2011, 58:3350-3359.
• [78]Volosyak I, Valbuena D, Luth T, Malechka T, Graser A: BCI Demographics II: How Many (and What Kinds of) People Can Use a High-Frequency SSVEP BCI? IEEE Trans Neural Syst Rehabil Eng 2011, 19:232-239.
• [79]Takahashi H, Yoshikawa T, Furuhashi T: Error Control for Performance Improvement of Brain-Computer Interface: Reliability-Based Automatic Repeat Request. Ieice Transactions on Information and Systems 2011, E94D:1243-1252.
• [80]Volosyak I: SSVEP-based Bremen-BCI interface-boosting information transfer rates. J Neural Eng 2011., 8036020
• [81]Wang YT, Wang YJ, Jung TP: A cell-phone-based brain-computer interface for communication in daily life. J Neural Eng 2011., 8025018
• [82]Blankertz B, Dornhege G, Krauledat M, Mueller K-R, Curio G: The non-invasive Berlin brain-computer interface: fast acquisition of effective performance in untrained subjects. Neuroimage 2007, 37:539-550.
• [83]Royer AS, Rose ML, He B: Goal selection versus process control while learning to use a brain-computer interface. J Neural Eng 2011., 8036012
• [84]Vlek RJ, Schaefer RS, Gielen C, Farquhar JDR, Desain P: Sequenced subjective accents for brain-computer interfaces. J Neural Eng 2011., 8036002
• [85]Solis-Escalante T, Müller-Putz G, Brunner C, Kaiser V, Pfurtscheller G: Analysis of sensorimotor rhythms for the implementation of a brain switch for healthy subjects. Biomed Signal Process Control 2010, 5:15-20.
• [86]Liu Y, Zhou ZT, Hu DW: Gaze independent brain-computer speller with covert visual search tasks. Clin Neurophysiol 2011, 122:1127-1136.
• [87]Treder MS, Blankertz B: (C)overt attention and visual speller design in an ERP-based brain-computer interface. Behav Brain Funct 2010., 6 BioMed Central Full Text
• [88]Nijboer F, Furdea A, Gunst I, Mellinger J, McFarland DJ, Birbaumer N, Kuebler A: An auditory brain-computer interface (BCI). J Neurosci Methods 2008, 167:43-50.
• [89]Lee S, Lim H: Brain-Operated Typewriter using the Language Prediction Model. KSII Trans Internet Inf Syst 2011, 5:1770-1782.
• [90]Zickler C, Riccio A, Leotta F, Hillian-Tress S, Halder S, Holz E, Staiger-Saelzer P, Hoogerwerf E-J, Desideri L, Mattia D, Kuebler A: A Brain-Computer Interface as Input Channel for a Standard Assistive Technology Software. Clin EEG Neurosci 2011, 42:236-244.
• [91]Jutai JW, Fuhrer MJ, Demers L, Scherer MJ, DeRuyter F: Toward a Taxonomy of Assistive Technology Device Outcomes. Am J Phys Med Rehabil 2005, 84:294-302.
• [92]Scherer MJ: Living in the State of Stuck: How Technology Impacts the Lives of People with Disabilities. Cambridge, MA: Brookline Books; 1993.
• [93]Scherer MJ: Outcomes of assistive technology use on quality of life. Disabil Rehabil 1996, 18:439-448.
• [94]Sellers EW, Donchin E: A P300-based brain-computer interface: Initial tests by ALS patients. Clin Neurophysiol 2006, 117:538-548.
• [95]Wolpaw JR: An EEG-based brain-computer interface for cursor control. Electroencephalogr Clin Neurophysiol 1991, 78:252-259.
• [96]Gao Y, Black MH: A quantitative comparison of linear and non-linear models of motor cortical activity for the encoding and decoding of arm motions. Int IEEE EMBS Conf Neural Eng 2003, 189-192.
• [97]Nykopp T: Statistical modelling issues for the adaptive brain interface. Helsinki, Finland: Helsinki University of Technology; 2001.
• [98]Blankertz B, Muller K, Curio G, Vaughan TM, Schalk G, Wolpaw J, Schlogl A, Neuper C, Pfurtscheller G, Hinterberger T, Schroder M, Birbaumer N: The BCI Competition 2003: progress and perspectives in detection and discrimination of EEG single trials. IEEE Trans Biomed Eng 2003, 2004(51):1044-1052.
• [99]Cohen J: A coefficient of agreement for nominal scales. Educ Psychol Meas 1960, 20:37-46.
• [100]Schlogl A, Lee FY: Characterization of Four-Class Motor Imagery EEG Data for the BCI-Competition 2005. J Neural Eng 2005, 2:14-22.
• [101]Gwet K: Inter-rater reliability: Dependency on trait prevalence and marginal homogeneity. Statistical Methods for Inter-Rater Reliability Assessment Series 2002, 2:1-9.
• [102]Obermaier B, Neuper C, Guger C, Pfurtscheller G: Information transfer rate in a five-classes brain-computer interface. IEEE Trans Neural Syst Rehabil Eng 2001, 9:283-288.
• [103]Schlogl A, Neuper C, Pfurtscheller G: Estimating the mutual information of an EEG-based brain-computer interface. Biomed Tech 2002, 47:3-8.
• [104]Wolpaw J, Ramoser H: EEG-based communication: improved accuracy by response verification. IEEE Trans Rehabil Eng 1998, 6:326-333.
• [105]Kronegg J, Voloshynovskiy S, Pun T: Analysis of bit-rate definitions for Brain-Computer Interfaces. Las Vegas: Conference on Human-Computer Interactions; 2005.
• [106]Buttfield A, Ferrez PW, Millan JR: Towards a robust BCI: Error potentials and online learning. IEEE Trans Neural Syst Rehabil Eng 2006, 14:164-168.
• [107]Baker B: Minspeak: A semantic compaction system that makes self-expression easier for communicatively disabled individuals. Byte 1982, 7:186-202.
• [108]Zhang D, Wang Y, Gao X, Hong B, Gao S: An algorithm for idle-state detection in motor-imagery-based brain-computer interface. Comput Intell Neurosci 2007. 39714
• [109]Thompson DE, Gruis KL, Huggins JE: A plug-and-play brain-computer interface to operate commercial assistive technology. Disabil Rehabil Assist Technol 2013. in press PMID: 23590556
• [110]Mason SG, Kronegg J, Huggins J, Fatourechi M, Schlogl A: Evaluating the Performance of Self-Paced Brain-Computer Interface Technology. 2006. http://ipl.ece.ubc.ca/bci_files/self_paced_tech_report-2006-05-19.pdf webcite
• [111]Panicker RC, Puthusserypady S: Ying Sun: An Asynchronous P300 BCI With SSVEP-Based Control State Detection. IEEE Trans Biomed Eng 2011, 58:1781-1788.
• [112]Davis J, Goadrich M: The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on Machine learning. New York, NY, USA: ACM; 2006:233-240. [ICML’06]
• [113]Schogl A, Anderer P: Artefact detection in sleep EEG by the use of Kalman filtering. Vienna, Austria: Proceedings EMBEC; 1999. [EMBEC]
• [114]Simpson RC, Koesten HH: Adaptive one-switch row-column scanning. IEEE Trans Rehabil Eng 1999, 7:464-473.