A technique for interrater reliability evaluation of a mobile game aimed for executive functions stimulation

Main Article Content

Bernardo Cerqueira
Debora Barbosa
João Mossmann
Jorge Barbosa


This work presents an interrater reliability evaluation of a mobile game aimed for the executive function’s stimulation, specifically the inhibitory control. The educational Exergame <name suppressed for blind review> was presented in a previous piece of work, which showed the development of the game and its application in a researching intervention with children, in the 6 to 10 age-group, in a school environment. Subsequently, the game had its code reworked for being able to be used across different platforms, hence culminating in the present work. The methodology in this paper consists in a mixed qualitative-quantitative evaluation through questionnaires with four domain professionals experienced in the executive functions field. The statistical measurement used was based on the Kappa coefficient and average percentage among the judges. As the results indicate, there was a substantial agreement (k=0,659; P-value=0,000) between the raters, as well as a high percentage of agreement in general on the mobile game’s capability of Executive Functions stimulation for children.


Download data is not yet available.


Metrics Loading ...

Article Details

How to Cite
Cerqueira, B., Nice Ferrari Barbosa, D., Mossmann, J., & Barbosa, J. (2019). A technique for interrater reliability evaluation of a mobile game aimed for executive functions stimulation. International Journal of Innovation Education and Research, 7(5), 338-354. https://doi.org/10.31686/ijier.Vol7.Iss5.1518


Anonymous 2017. Details omitted for double-blind reviewing.

Anonymous 2018. Details omitted for double-blind reviewing.

Baba, Y. & Tschang, F., Product development in japanese tv game software: The case of an innovative game. International Journal of Innovation Management, 05(04), 2001, 487–515.

Bernier, A. et al., From external regulation to self-regulation: Early parenting precursors of young children’s executive functioning. Child development, 81(1), 2010, 326-339.

Buelow, M.; et al., A. The influence of video games on executive functions in college students. Computers in Human Behavior, 45, 2015, 228-234.

Bull, R. et al., A comparison of performance on the Towers of London and Hanoi in young children. Journal of Child Psychology and Psychiatry, 45(4), 2004, 743-754.

Bull, R. et al., Short-term memory, working memory, and executive functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years. Developmental neuropsychology, 33(3), 2008, 205-228.

Carlson, S. et al., Individual differences in executive functioning and theory of mind: An investigation of inhibitory control and planning ability. Journal of experimental child psychology, 87(4), 2004, 299-319.

Chen, P., Krauss, A., Interrater agreement. The sage encyclopedia of social science research methods, 2, 2004, 511-513.

Christopher, M. et al., Predicting word reading and comprehension with executive function and speed measures across development: A latent variable analysis. Journal of Experimental Psychology: General, 141(3), 2012, 470.

Conklin, H. et al., Working memory performance in typically developing children and adolescents: Behavioral evidence of protracted frontal lobe development. Developmental neuropsychology, 31(1), 2007, 103-128.

de Jong., Effects of training working memory in adolescents with a below average IQ. Workshop on Enhancing Executive Functions in Education in Nijmegen, May 20, 2014 Nijmegen, NL.

Diamond, A., & Lee, K., Interventions shown to aid Executive Function development in children 4 to 12 years old. Science, 333(6045), 2011, 959-964.

Diamond, A., Executive functions. Annual review of psychology, 64. 2013, 135-168.

Diamond, A; Ling, D., Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Developmental cognitive neuroscience, 18. 2016, 34-48.

Fleiss, J. L.; Cohen, J.M., The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 1973, 613-619.

Holmes, J.; et al., Adaptive training leads to sustained enhancement of poor working memory in children. Developmental science, 12(4), 2009, 1-7.

Hughes, C. & Ensor, R., Executive function and theory of mind: Predictive relations from ages 2 to 4. Developmental psychology, 43(6), 2007, 1447.

Institute of Digital Media and Child Development Working Group on Games for Health et al., Games for health for children—Current status and needed research. Games for Health Journal, 5(1), 2016, 1-12.

Jurado, M., Rosselli, M., The elusive nature of executive functions: a review of our current understanding. Neuropsychology review, 17(3), 2007, 213-233.

Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics, 2000, 159–174.

Likert, R., A technique for the measurement of attitudes. Archives of Psychology. New York: Columbia University Press, 1932.

Mansur-Alves, M; Saldanha-Silva, R., Treinar memória de trabalho promove mudanças em inteligência fluida? Temas em Psicologia, 25 (2), 2017,787-807.

Monette, S et al. The role of the executive functions in school achievement at the end of Grade 1. Journal of experimental child psychology, 109(2), 2011, 158-173.

Müller, U. et al., The effect of labeling on preschool children's performance in the Dimensional Change Card Sort Task. Cognitive Development, 23(3), 2008, 395-408.

Posner, K. et al., Measuring interrater reliability among multiple raters: an example of methods for nominal data. Statistics in medicine, 9(9), 1990, 1103-1115.

R Core Team., R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2018, URL .

Toll, S. et al., Executive functions as predictors of math learning disabilities. Journal of Learning Disabilities, 44(6), 2011, 521-532.

Tschang, T., Videogames as interactive experiential products and their manner of development. International Journal of Innovation Management, 9(1), 2005, 103-131.

Welsh et al., The development of cognitive skills and gains in academic school readiness for children from low-income families. Journal of Educational Psychology, 102(1), 2010, 43-53.

DB Error: Unknown column 'Array' in 'where clause'