Table of Links
Abstract and 1. Introduction
-
Prior Work and 2.1 Educational Objectives of Learning Activities
2.2 Multiscale Design
2.3 Assessing Creative Visual Design
2.4 Learning Analytics and Dashboards
-
Research Artifact/Probe
3.1 Multiscale Design Environment
3.2 Integrating a Design Analytics Dashboard with the Multiscale Design Environment
-
Methodology and Context
4.1 Course Contexts
4.2 Instructor interviews
-
Findings
5.1 Gaining Insights and Informing Pedagogical Action
5.2 Support for Exploration, Understanding, and Validation of Analytics
5.3 Using Analytics for Assessment and Feedback
5.4 Analytics as a Potential Source of Self-Reflection for Students
-
Discussion + Implications: Contextualizing: Analytics to Support Design Education
6.1 Indexicality: Demonstrating Design Analytics by Linking to Instances
6.2 Supporting Assessment and Feedback in Design Courses through Multiscale Design Analytics
6.3 Limitations of Multiscale Design Analytics
-
Conclusion and References
A. Interview Questions
6.2 Supporting Assessment and Feedback in Design Courses through Multiscale Design Analytics
For learning analytics to be effective in open-ended, project-based contexts, there is a need to assess complex characteristics that can give insights into students’ creative strategies and abilities [13]. Toward addressing this need, our study investigates how multiscale design analytics support instructors’ assessment efforts in creative project-based learning contexts of design courses.
Instructors in our study reported that multiscale design analytics can support them directly or indirectly in assessment and feedback processes. Instructors found that multiscale analytics have the potential to inform pedagogical intervention, based on whether or not students are able to effectively utilize the design environment. In I9’s words, “So if this number is extremely low for everybody…then maybe you need to [give] a tutorial [on the design environment].” Instructors shared that providing these analytics to students can help them reflect and improve their multiscale design skills. For example, in I1’s words, “I would love students to explore more zoom levels…they don’t really utilize being able to…zooming in to certain parts and elaborating.”
Implications. Our study demonstrates the potential of multiscale design analytics—which measure complex characteristics of design work—to assist instructors in assessing student work. The organization principles that design instructors expect their students to demonstrate map to the create category in Bloom’s revised taxonomy, i.e., “put elements together” into a “new, coherent whole” [2, 46]. As I4 expressed, “I think that I would definitely like to assign scales as a part of the rubric to say, I would like to see the big picture from out here, and then when you zoom in, see more.” Likewise, I1 expressed that cluster analytics could help students reflect on their design representation and become “more aware about how they separate”. We thus find that multiscale design analytics empower instructors in assessing students’ holistic thinking and creative capabilities.
We advocate for future research that investigates further development of multiscale design analytics, visual annotations, and dashboard interactions, as well as more diverse and in-depth studies, in order to develop new knowledge about how to support instructors, in developing effective pedagogical interventions, and students, in learning how to do design that involves thinking about and presenting complex information. This line of research has the potential to create new educational avenues for teaching how to present complex information that supports audiences in micro and macro readings—i.e., details and overviews [21]—and the formation of mental models [32] and maps. Since conveying and understanding such information is vital in so many areas of society, this mission has the potential for broad impact that benefits society, through the work these students will perform throughout their careers.
Further, in design course contexts—where providing frequent feedback is vital—AI-based analytics demonstrate their utility in scaling the assessment. For example, I9 finds using analytics “better than having to go to every [design] and look for every single issue or having a much larger rubric [to run] by.” Instructors in diverse project-based learning contexts—e.g., arts and humanities [23]—engage students in creative, open-ended work. Thus, these contexts are similarly expected to benefit from analytics based on assessments of complex characteristics.
The current research provides evidence for multiscale design measures to serve as descriptive analytics, i.e., analytics that provide insights into student work [76]. Going ahead, with data from the past iterations of a course, these analytics have the potential to function as prescriptive analytics, i.e., provide instructors and students alerts and suggestions based on computational modeling of the relationship between analytics and students’ course performance [6, 76]. On demand feedback through analytics has the potential to stimulate students’ learning-by-doing. Further, incorporating multiscale design analytics in widely distributed tools, such as Photoshop and Illustrator, has the potential to bring widespread benefits, as students use these tools in diverse design course contexts.
6.3 Limitations of Multiscale Design Analytics
Multiscale design analytics are not a panacea. On the one hand, our findings show that the present multiscale design analytics provide value to instructors, in situated course contexts. I9 would like to see that students are able to effectively use the multiscale design environment. I2 values students’ presentation of structure at different levels. I1 would “love students to explore more zoom levels”, as she does not see them “zooming in to certain parts and elaborating”.
On the other hand, multiscale design analytics were not found to serve as a catch-all measure for design. I4 talks about not wanting all designs to look the same “like you don’t want to go somewhere and see every painting looks the same”. I1 says, “I would rather not control…how they see spatial clusters”.
We build theory using creative cognition’s family resemblance principle, according to which, no particular characteristics are required for a work to be deemed creative [70]. Rather, a family of traits tends to serve as indicative. We find that multiscale design, as measured here, functions as one such design creativity trait. As another fruitful avenue for future research, we identify deriving analytics for families of design creativity traits—for example, feasibility, originality, and aesthetics [3, 20]; and gestalts principles, e.g., proximity, closure, continuity, symmetry, parallelism, and similarity of color, size, and orientation [79]—and applying these traits in education and even crowd-sourced design contexts. According to the family resemblance principle, as no particular trait is sufficient, design creativity analytics will never be perfect. But inasmuch as they work well enough, they can provide instructors, students, and other designers with insights so as to (1) provide first-order assessment; and (2) stimulate ongoing work.
:::info
Authors:
(1) Ajit Jain, Texas A&M University, USA; Current affiliation: Audigent;
(2) Andruid Kerne, Texas A&M University, USA; Current affiliation: University of Illinois Chicago;
(3) Nic Lupfer, Texas A&M University, USA; Current affiliation: Mapware;
(4) Gabriel Britain, Texas A&M University, USA; Current affiliation: Microsoft;
(5) Aaron Perrine, Texas A&M University, USA;
(6) Yoonsuck Choe, Texas A&M University, USA;
(7) John Keyser, Texas A&M University, USA;
(8) Ruihong Huang, Texas A&M University, USA;
(9) Jinsil Seo, Texas A&M University, USA;
(10) Annie Sungkajun, Illinois State University, USA;
(11) Robert Lightfoot, Texas A&M University, USA;
(12) Timothy McGuire, Texas A&M University, USA.
:::
:::info
This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license.
:::
