Table of Links
Abstract and 1 Introduction
2.1 Software Testing
2.2 Gamification of Software Testing
3 Gamifying Continuous Integration and 3.1 Challenges in Teaching Software Testing
3.2 Gamification Elements of Gamekins
3.3 Gamified Elements and the Testing Curriculum
4 Experiment Setup and 4.1 Software Testing Course
4.2 Integration of Gamekins and 4.3 Participants
4.4 Data Analysis
4.5 Threats to Validity
5.1 RQ1: How did the students use Gamekins during the course?
5.2 RQ2: What testing behavior did the students exhibit?
5.3 RQ3: How did the students perceive the integration of Gamekins into their projects?
6 Related Work
7 Conclusions, Acknowledgments, and References
3.2 Gamification Elements of Gamekins
We aim to improve testing courses by incentivizing students to test more and better using the CI system underlying their coding tasks. For this, we developed a tool, Gamekins [47] [2], which incorporates gamification elements into CI based on static code, coverage, and mutation analysis. The combination of underlying quality metrics is intended to shift the students’ focus away from solely relying on one metric. To enforce better commit and testing behavior, Gamekins provides gamification elements only after new commits.
The general workflow of Gamekins is as follows[3]: Developers commit code changes to a version control system, which triggers a job in the Jenkins CI environment. Jenkins executes a build job, which includes running the project’s tests, and then invokes Gamekins. Gamekins utilizes information from the run and the version control system to update and generate challenges, quests, achievements, and the leaderboard.
3.2.1 Challenges. Gamekins offers a variety of seven distinct challenge types, each designed to provide developers with specific test and quality-related tasks to complete:[3]
• Build Challenge: This challenge requires developers to resolve a build failure. To prevent misuse of Gamekins by repeatedly breaking and fixing the build for points, this challenge is generated only once per week.
• Test Challenge: This generic challenge requires developers to write at least one additional test without specifying the specific code area to target.
• Class Coverage Challenge: This challenge type focuses on increasing code coverage for a selected Java class.
• Method Coverage Challenge: This challenge focuses on improving the coverage of a specific target method.
• Line Coverage Challenge: This challenge focuses on improving the coverage of a specific line of code that is not fully covered. The task is to either cover or increase the branch coverage of the line. • Mutation Challenge: This challenge focuses on detecting a specific mutant by adding or improving a test, and therefore goes beyond a simple check of the robustness of the test suite to closing test gaps [38]. The mutants are generated with PIT[4], a popular mutation testing tool for Java.
• Smell Challenge: This challenge involves analyzing a target class using SonarLint[5], a code quality tool. The challenge is to choose one of the detected smells in either the source or test files and remove it.
In order to generate new challenges, Gamekins ensures that there is always a (configurable) fixed number of current challenges. When generating a new challenge, Gamekins only considers recently changed classes based on the commit history. This ensures that the challenge is relevant to the functionality currently considered by the developer. To generate a challenge, Gamekins first ranks all classes in the project under test based on the current code coverage, then probabilistically selects a class for the challenge, giving higher chances to classes with lower coverage; finally, a challenge type is randomly selected, and a random instance for this type is generated.
If a developer already has open challenges, Gamekins checks if these challenges have been solved. If they have been solved, Gamekins will generate new challenges to replace them. Additionally, Gamekins also checks if all currently open challenges are still applicable, for example by verifying if the specific code fragment for a challenge has not been deleted. To keep developers informed, all current challenges are displayed on a dedicated page in Jenkins, which provides an overview of open challenges and their status (Fig. 1). Clicking on a challenge provides additional information, including the code snippet mentioned in the challenge description, as well as an explanation of what needs to be done. This information is particularly important for Mutation Challenges, where both the modified source code and the original code are shown.
To remove a current challenge from the list, participants can click on the Reject button. When rejecting a challenge, participants are required to provide a reason, which aims to limit rejections and helps to improve Gamekins. Some possible reasons for rejecting a challenge include cases where the challenge is unnecessary due to defensive programming or when reported code smells are intentionally present in the code. In some cases, challenges may be
automatically rejected if, for example, the target line or class has been removed. If a Class Coverage Challenge is rejected, this will block the future generation of any challenges in the class that the rejected challenge was targeting. However, this can be undone in the list of rejected challenges if necessary.
3.2.2 Quests. Quests are a way to group multiple challenges together. They consist of a series of individual challenges that need to be solved one after the other. In a quest, only the current challenge that needs to be solved next is enabled and can be expanded to show more details (Fig. 2). The successive challenges in the quest are disabled and cannot be viewed until it is their turn to be solved. To earn points for a quest, all steps must be solved in succession. Gamekins supports a total of nine different types of quests3].
Quests are created by checking the prerequisites of potential quests, such as ensuring that a sufficient number of required challenges can be generated. One quest is then randomly chosen from the available options. The individual steps or challenges within a quest are generated as if they were standalone challenges but with predetermined classes based on the quest type. Participants have the option to reject quests, or quests can be automatically rejected by Gamekins if at least one of the challenges within the quest becomes unsolvable. In this case, participants will receive points for the challenges they have already solved within the quest, but not for the steps themselves.
3.2.3 Leaderboard. By completing challenges and quests, users earn points, and their point rankings are displayed on a leaderboard in Jenkins. The leaderboard not only shows the points but also the total number of completed challenges and achievements, fostering competition. Participants can personalize their experience by selecting one of the 50 avatars to be displayed on the leaderboard.
3.2.4 Achievements. Developers are rewarded for their test achievements, which are based on specific behaviors or actions they perform, regardless of specific challenges. These achievements have
varying difficulty, ranging from easy tasks, like having a test in a project, to more challenging ones, like achieving 100 % coverage. Currently, Gamekins implements 63 achievements, and developers can view obtained and open achievements (Fig. 4). Some achievements are kept secret and are only revealed once they are completed.
3.3 Gamified Elements and the Testing Curriculum
Besides offering general incentives, the integration of Gamekins into coursework allows to address several learning objectives outlined in the Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering [2], such as:
• Static analysis (VAV.rev.3) and refactoring (PRO.evo.3), e.g., with Smell Challenges
• Unit (VAV.tst.1) and integration (VAV.tst.5) testing with Challenges and Quests
• Testing edge cases and boundary conditions with Mutation Challenges (VAV.tst.2) • Coverage analysis with Coverage Challenges (VAV.tst.3)
• Regression testing (VAV.tst.10), testing tools and automation (VAV.tst.11) with Gamekins as CI plugin
• Build processes (PRO.cm.4) and analyzing failure reports (VAV.par.1) with Gamekins as CI plugin
[2] Available at https://gamekins.org
[3] Detailed information can be found in [47]
[4] https://pitest.org/
[5] https://www.sonarlint.org/
Authors:
(1) Philipp Straubinger, University of Passau, Germany;
(2) Gordon Fraser, University of Passau, Germany.
