Authors:
(1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil;
(2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil;
(3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil;
(4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil.
Table of Links
Abstract and 1 Introduction
2. Background and Related Work
3. Systematic Mapping
4. Systematic Mapping Results
5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements
6. Limitations
7. Concluding Remarks, Acknowledgements, and References
Abstract
Context: While most research shows positive effects of gamification, the focus on its adverse effects is considerably smaller and further understanding of these effects is needed. Objective: To provide a comprehensive overview on research reporting negative effects of game design elements and to provide insights into the awareness of developers on these effects and into how they could be considered in practice. Method: We conducted a systematic mapping study of the negative effects of game design elements on education/learning systems. We also held a focus group discussion with developers of a gamified software, discussing the mapping study results with regard to their awareness and perceptions on the reported negative effects in practice. Results: The mapping study revealed 87 papers reporting undesired effects of game design elements. We found that badges, leaderboards, competitions, and points are the game design elements most often reported as causing negative effects. The most cited negative effects were lack of effect, worsened performance, motivational issues, lack of understanding, and irrelevance. The ethical issues of gaming the system and cheating were also often reported. As part of our results, we map the relations between game design elements and the negative effects that they may cause. The focus group revealed that developers were not aware of many of the possible negative effects and that they consider this type of information useful. The discussion revealed their agreement on some of those potential negative effects and also some positive counterparts. Conclusions: Gamification, when properly applied, can have positive effects on education/learning software. However, gamified software is also prone to generate harmful effects. Revealing and discussing potentially negative effects can help to make more informed decisions considering their trade-off with respect to the expected benefits. Keywords: gamification, negative effects, education, learning, systematic mapping, snowballing, focus group
1. Introduction
There are plenty of digital platforms for education with a massive number of users, as Duolingo, a language teaching service used by 300 million people worldwide. It boasts that it gives opportunities for people to learn new languages, no matter their financial standing [1].
Duolingo and other services worldwide use gamification – applying gameplaying elements to non-game contexts [2] that are typically tedious, discouraging, or inefficient – as a strategy to make their objectives more achievable. It’s a strategy with a strong presence in education, and other domains [3], representing a market predicted to grow over 30% through 2019-2025, with an expected value of more than 32 billion in 2025 [4]. This context means that a significant demand for gamified software exists, which calls for software engineers to develop them.
Software development is not a trivial task, and it is more complicated in the case of gamified software solutions. These cases require specialized expertise, going beyond what is expected by an average software engineer [5], for instance:
• Effective gamification requires knowledge of human psychology, similar to how serious games require knowledge regarding the subject they deal with. This necessity arises because gamified software aims to change human behavior (see Volkswagen’s Fun Theory videos ([6],[7],[8]);
• Software engineers need a good understanding of the game design mechanics used as tools and how they contribute to functional and non-functional requirements;
• Software engineers face the fact that gamified software has a more limited design space and different objectives to focus on than a game [9].
Hence, selecting the correct gamification elements when designing gamified software is strongly related to requirements engineering and can affect the overall project success [10]. Moreover, defects in requirement are the most expensive to fix when found in production [11][12]. Indeed, given that gamification deals with changing human behavior, when gamified software is ill-specified, it may not hit the intended target or even be counterproductive, which can have serious consequences when applied to education.
Education is the main target of the present work for important reasons. For instance, approximately 617 million children and adolescents of primary and lower secondary school age (roughly 55% of the global total) have not reached minimum reading and mathematics proficiency in 2015 [13]. The reasons for this global learning crisis are manifold, such as inequality and poverty, but the poor quality of education is one of the critical causes. In this context, applying gamification to education and learning represents a promising means to allow educators to make learning fun, contextualize learning quickly, speak the language of young people, and directly deal with soft skills, improving education quality. However, gamification is also prone to generate harmful effects, usually unknown to designers and engineers. These unexpected effects happen because current gamification research lacks a critical lens capable of exploring unintended design consequences [3].
In this paper we extend our previous systematic mapping study efforts on adverse, unintended gamification effects in education and learning software (such as modified learning management software, course wares and digital learning environments) [14]. Our search strategy is driven by broad research questions and uses a hybrid search strategy [15], combining database search with backward and forward snowballing. The extension comprises providing more details on the mapping study, complementing the search strategy to gather evidence reported until the end of 2020 (previously 2020 was not completely covered), leading our set of identified studies to increase from 77 to 87. As far as we know, this paper is the most precise, comprehensive, and up-to-date systematic mapping study that organizes evidence regarding the adverse effects of game design elements in gamified education software. Furthermore, we conducted a focus group discussion with developers of a gamified software, discussing the mapping study results with regard to their awareness and perceptions on the reported negative effects.
The mapping study reported several different negative effects, allowing to gather valuable information for software engineers and designers of gamified education/learning software, such as the game design elements that have most often been reported being involved with adverse effects; the most common negative effects on students; the adverse consequences affecting teachers; the relation between game design elements and negative effects; the fields in which research on negative effects has been conducted; and the types of empirical studies conducted to assess the adverse effects.
The study revealed that badges, leaderboards, competitions, and points are the game design elements most often reported as causing negative effects. The most reported negative effects were lack of effect, worsened performance, motivational issues, lack of understanding, and irrelevance. The ethical issues of gaming the system and cheating were also often reported. We also mapped the relations between game design elements and the negative effects that they may cause, so that these effects can be pragmatically considered when designing gamified software.
The focus group revealed that developers were not aware of many of the possible negative effects and that they consider this type of information useful. We report on the discussions held with the practitioners on the use of the identified game design elements and their negative effects. Nevertheless, they recall that this information should be discussed within the context of the gamified software, as in this focus group, to analyze trade-offs between expected positive effects and potential negative ones. Sharing this discussion provides further insights into the usefulness of our mapping study results from the point of view of the practitioners.
The organization of the paper is as follows. Section 2 presents the fundamentals of gamification and gamification effects and related work. We explain our systematic mapping protocol and its execution in Section 3. The results answering the research questions are presented Section 4. In Section 5 we describe the focus group its results. Finally, Sections 6 and 7 discuss the limitations and present the conclusions.
