From Chaos to Control: Building a Standardized Dev Environment for 1,000+ Students | HackerNoon

Table of Links

1 Introduction

2 Course Structure and Conditions

2.1 Conditions

2.2 Syllabus

3 Lectures

4 Practical Part

4.1 Engagement Mechanisms

4.2 Technical Setup and Automated Assessment

4.3 Selected Exercises and Tools

5 Check Your Proof by Example

6 Exams

7 Related Work

8 Conclusion, Acknowledgements, and References

4.2 Technical Setup and Automated Assessment

Automated Assessment In WS19, we used an improved version of the testing infrastructure introduced in [4]. However, this system could neither manage online exams nor mark non-programming tasks. We thus switched to a newly written open-source tool developed at TUM called ArTEMiS [22]. ArTEMis is a highly scalable, automated assessment management system and is programming language independent – it only expects test runners to produce tests results adhering to the Apache Ant JUnit XML schema. It already offered support for a few imperative programming languages, and we added support for Haskell.[10]

As ArTEMis takes care of most things, including automated test execution and score management, and offers an exam mode and good support for grading non-programming tasks, the only thing that was left to do was writing the testing code. For the most part, we verified the results computed by a student’s submission by comparing them to those computed by a sample solution written by us. In some cases, we also tested for efficiency using timeouts. Our tests were powered by the following libraries:

1. QuickCheck [7]: Although QuickCheck can automatically generate test data (using the typeclass Arbitrary), most tests and benchmarks used custom input generators. This was necessary to increase coverage and eliminate non-applicable inputs for tests with preconditions. We also used custom shrinkers to provide better feedback to students in case of a failure. In both cases, the flexible combinators provided by QuickCheck made this a straightforward task.
2. SmallCheck [34]: The exhaustive testing facilities provided by SmallCheck mainly served as a complementary tool that provided small counterexamples for, in many cases, obvious deficiencies.
3. Tasty[11]: We used Tasty to put QuickCheck, SmallCheck, unit tests, and the checking of “Check Your Proof” proofs (see Section 5) into one common framework that is capable of generating results interpretable by ArTEMiS. We used the unit testing facilities of Tasty to complement our tests with corner cases. Integration of proof checking was pleasantly straightforward: one only has to provide a suitable instance for Tasty’s IsTest typeclass.[12] Moreover, Tasty supports timeouts for individual test cases. solving the issue of truncated test reports mentioned in [4].

Development Environment and Online Tutorials In previous iterations, there were no recommendations for students regarding the development environment they should use for the practical part of the course. However, due to the COVID-19 pandemic, this was no longer an option: we needed a way for students to share their code in read and write mode to other peers (pair-programming) and teaching assistants (for feedback purposes) during online tutorials. Moreover, as explained in Section 4.1, we wanted students to use a linter. Finally, we had negative experiences with students installing no compiler at all and instead (mis)using our submission server as a compiler backend. We thus introduced a strict policy for the technical setup to be used during the tutorials. Detailed installation instructions can be found online[13]. Here we briefly list the key components and our experiences:

1. IDE: We used VSCodium[14] due to its cross-platform support, rich library of extensions, widespread adoption, and free open-source software philosophy. We did not receive any negative feedback by students and, besides few exceptions, no major installation problems were reported.
2. Build and dependency manager: We used Stack[15] for these purposes. Since Stack provides curated sets of packages and compiler versions that are checked for compatibility, deterministic builds are guaranteed. Students hence showed little struggle to compile and execute their programs.

3. Linter: We used HLint[16], which, among other things, provides suggestions for alternative functions and simplified code and spots redundancies. Students reacted curiously and positively to these suggestions and discussed them vividly during pair-programming sessions.

4. API search engine: Due to Haskell’s strong type system, searching its API by type signature often returns better results than searching for function names. For this purpose, we introduced Hoogle[17] to our students and let them install an extension that integrates Hoogle searches into their IDE. Unfortunately, we have no data to report on whether this enriched students’ programming experiences.

5. Real-time collaboration: We used VSLiveShare[18] to run our pair-programming sessions (groups of 3–4 students). We can report positively on its connection stability and usability. Unfortunately, the plugin requires a Microsoft or GitHub account, but since almost all students already signed up to the latter before, this requirement passed uncontroversially. Nevertheless, for future iterations, we plan to investigate open-source alternatives.

All in all, we can report positively on the setup. Students were more knowledgeable about their programming environment (compilation, dependency management, etc.) than in previous iterations. Misappropriation of the submission server as a compiler backend also stopped.

[10] It now also supports OCaml.

[11] https://hackage.haskell.org/package/tasty

[12] The code can be found in the repository in resources/cyp integration/test/hs/Test.hs

[13] https://www21.in.tum.de/teaching/fpv/WS20/installation.html

[14] VSCodium provides free open-source software binaries of VSCode https://vscodium.com/

[15] https://www.haskellstack.org/

[16] https://github.com/ndmitchell/hlint

[17] https://hoogle.haskell.org/

[18] https://visualstudio.microsoft.com/services/live-share/