Fitbit Leaderboards: Surprising Gains for the Sedentary, Setbacks for the Highly Active | HackerNoon

Table of Links

Abstract and 1 Introduction

2. Background and 2.1. Leaderboards

3. Effect of Leaderboards on Healthful Physical Activity and 3.1. Competition

3.2. Social Influence

3.3. Moderating Effects of Prior Activity Levels and Leaderboard Size

4. Data and Model

4.1. Data

4.2. Model

5. Estimation and Robustness of the Main Effects of Leaderboards

5.2. Robustness Check for Leaderboard Initiation

5.3. Fitbit Compliance

5.4. Fitbit Attrition, Leaderboard De-Adoption, and Additional Robustness Checks

6. Heterogeneous Effect of Leaderboards

6.1. Heterogeneity by Prior Activity Levels

6.2. Interaction of Leaderboard Size, Rank, and Prior Activity Levels

6.3. Summary of Findings from Heterogeneous Effect Analysis

7. Conclusions and Discussion, Endnotes, and References

Abstract

Health wearables in combination with gamification enable interventions that have the potential to increase physical activity—a key determinant of health. However, the extant literature does not provide conclusive evidence on the benefits of gamification, and there are persistent concerns that competition-based gamification approaches will only benefit those who are highly active at the expense of those who are sedentary. We investigate the effect of Fitbit leaderboards on the number of steps taken by the user. Using a unique data set of Fitbit wearable users, some of whom participate in a leaderboard, we find that leaderboards lead to a 370 (3.5%) step increase in the users’ daily physical activity. However, we find that the benefits of leaderboards are highly heterogeneous. Surprisingly, we find that those who were highly active prior to adoption are hurt by leaderboards and walk 630 fewer steps daily after adoption (a 5% relative decrease). In contrast, those who were sedentary prior to adoption benefited substantially from leaderboards and walked an additional 1,300 steps daily after adoption (a 15% relative increase). We find that these effects emerge because sedentary individuals benefit even when leaderboards are small and when they do not rank first on them. In contrast, highly active individuals are harmed by smaller leaderboards and only see benefit when they rank highly on large leaderboards. We posit that this unexpected divergence in effects could be due to the underappreciated potential of noncompetition dynamics (e.g., changes in expectations for exercise) to benefit sedentary users, but harm more active ones.

1. Introduction

The evidence on the health[1] benefits of physical activity is irrefutable (Warburton et al. 2006). Yet, a significant portion of the world population is not sufficiently active.[2] This lack of physical activity contributes significantly to chronic disease and to most of the leading causes of death in the United States.3 Prior research suggests that behavioral barriers are one of the most important contributing factors to this trend. Mitchell et al. (2013) suggest that “for many adults, the ‘costs’ of exercise (e.g., time, uncomfortable feelings) loom so large that they never start” and that the lack of physical activity is “a problem of both initiation and maintenance” (p. 658). Recognizing that changing health behaviors is often challenging and new strategies are needed, research situated mostly in the health and economics literature has evaluated a plethora of economic and non-economic interventions for overcoming motivational barriers to increasing physical activity (Charness and Gneezy 2009, Mitchell et al. 2013). The conclusion from this literature is that, although many interventions can drive short-run gains in physical activity, these benefits are fleeting and motivating meaningful, and sustained increases in physical activity is elusive. More so, many of these interventions (e.g., daily payments) are difficult to implement on a population scale.

One contemporary phenomenon with the potential to address persistent limitations of prior approaches and unlock new interventions that can improve the motivation of individuals to exercise is the rapid consumer adoption of health wearables (Swan 2013, Lupton 2016). A health wearable, sometimes referred to as an activity tracker, is “a wearable device or a computer application that records a person’s daily physical activity, together with other data relating to their fitness or health, such as the number of calories burned, heart rate, etc.”4 Despite the rapid adoption of health wearables and their potential for motivating individuals to engage in healthy activities, scholars suggest it is unlikely that the measurement capabilities that the health wearables provide would significantly impact health on their own (Patel et al. 2015, Sullivan and Lachman 2017). Rather, they suggest that for health wearables to impact health behavior, the information they collect “must be presented back to the user in a manner that can be understood, that motivates action, and that sustains that motivation toward improved health” (Patel et al. 2015, p. 460).

Particularly promising in this regard is combining granular physical activity data from health wearables with gamification approaches. Gamification is defined as the “use of game design elements in nongame contexts” (Deterding et al. 2011). Some examples of game design elements are badges, rules-based competition, leaderboards, points, ranking, reputation, rewards, teams, and time pressure (Deterding et al. 2011). Coupling gamification with health wearables has the potential to improve motivation by converting the usually mundane action of physical activity into the more enjoyable activity of collecting rewards or competing with other individuals (Hamari et al. 2014a, Johnson et al. 2016). More so, gamification approaches can provide immediate positive reinforcement that helps individuals get over the initial hurdles of engaging in exercise and could also help them sustain higher levels of activity in the longer term (Mitchell et al. 2013, Shameli et al. 2017). In addition, the broad adoption of health wearables unlocks more robust gamification interventions by providing an objective and common source of measurement and a form factor that enables real-time feedback while engaging in physical activity (Johnson et al. 2016).

Although coupling gamification with health wearables has the potential to generate sustained increases in physical activity, the evidence on the benefits of gamification is mixed. Hamari et al. (2014b) reviewed 24 empirical gamification studies, primarily within education contexts, but reported mixed effects on outcomes. Moreover, these studies used interviews or surveys to measure outcomes. Hamari and Koivisto (2013), the only empirical study in a health context in the aforementioned literature review, used surveys to measure outcomes and these outcomes were not health related (e.g., continued use intention for the gamification service and the intention to recommend service to others). Johnson et al. (2016) conducted a systematic literature review on the impact of gamification on health and well-being. Of the 19 empirical papers they reviewed, 59% reported positive results and 41% reported mixed results. Both Hamari et al. (2014b) and Johnson et al. (2016) also noted that the quality of evidence was moderate to low.

The significant potential benefits of coupling gamification with health wearables and the narrow focus and lack of evidentiary quality of prior works motivate this research study. We evaluate the benefits of leaderboards that allow users to view the performance of others who also agree to share their activity levels and, in most cases, to compete with them. We focus on the potential benefits of leaderboards because they are one of the most common gamification features available with modern health wearables. This increases the policy and practical relevance of our results. Another reason we focus on leaderboards is that they exemplify the theoretical tensions surrounding gamification interventions; scholars suggest that gamification features, and leaderboards in particular, are likely to have heterogeneous effects on individuals (Deci et al. 1981, Santhanam et al. 2016, Sullivan and Lachman 2017). Specifically, a central concern with competition-based gamification interventions like leaderboards is that they will lead to motivational benefits only for those who are already highly active (and need the increased motivation the least), whereas actually harming the least physically active in the population (Patel et al. 2015, Wu et al. 2015, Shameli et al. 2017).5 With these dynamics in mind, our first objective is to evaluate the average impact on physical activity of leaderboard adoption by individuals wearing health wearables. Our second research objective is to evaluate the potential for leaderboard effect heterogeneity by (i) the activity level of the focal user prior to adoption, (ii) the number of active participants on the leaderboard, and (iii) the rank of the focal user on the leaderboard in the prior period.

We engage in an intensive data collection effort to estimate the average benefit of leaderboards and the heterogeneity in these benefits. For approximately 500 individuals observed over a two-year time period, we capture leaderboard adoption data and granular measures of physical activity continuously captured by Fitbit Charge HR health wearables. For those individuals with leaderboards, the data set also includes activity data and rank of all participants in the leaderboard. We supplement these data from health wearables with periodic surveys (every six months on average) capturing a rich array of individual characteristics (psychological attributes, frequency of technology use, etc.). Leveraging variation in physical activity and adoption of leaderboards over time and between individuals, we use a difference-in-differences (DID) estimation approach to evaluate the effect of leaderboards on daily physical activity as measured by the user’s step count, as well as heterogeneity in these effects.

We find that leaderboard adoption results in an average daily increase of 370 steps. This main effect is resilient to various tests for the assumption of common trends between those who adopt and do not adopt, estimation of several falsification tests, and other robustness checks. These initial results, however, mask important heterogeneity in the benefits of leaderboard adoption. When we take into account an individual’s prior activity levels, we find a stark divergence in leaderboard effects. Individuals who were highly active prior to adoption, instead of benefiting from leaderboards, experienced a significant decrease in their average daily physical activity after leaderboard adoption (a decrease of 631 steps daily). Moreover, these negative effects persisted (and actually increased in magnitude) in the 10 weeks following leaderboard adoption. In contrast, users who were less active prior to adoption had large and significant positive impacts on their daily step counts—On average, their activity increased by 1,365 steps daily (an approximately 15% increase), and these increases also persisted well after the adoption decision (10 weeks after adoption).

Examining this trend further, we find significant nuance in how leaderboard size impacts sedentary versus highly active users. Specifically, the key distinction between these groups is that previously sedentary individuals can reap significant benefit from small leaderboards (only one or two other members) and even if they do not rank first. In contrast, those who were highly active (prior to adoption) see the most significant harm when leaderboards are small. Our interpretation of these results is that individuals who are already on the high end of the physical activity distribution can become complacent on small leaderboards where, more often than not, they are paired with those less active than themselves.[6] In contrast, sedentary individuals who are at the lower end of the distribution of physical activity often encounter (even on small leaderboards) peers who are more active than themselves, who can positively impact their reference point for exercise, and who can hold them accountable if their activity levels slump. However, these benefits for sedentary users diminish if leaderboards become too large; the marginal benefit of an additional leaderboard member diminishes at three times the rate for sedentary users relative to highly active ones. One explanation for this effect is that the benefits of social influence that accrue to sedentary users (e.g., positive impact on their exercise reference points) diminish as leaderboards become larger and less intimate.

Our research contributes to streams of work at the intersection of information systems, economics, and healthcare. Specifically, we contribute to the literature on the economics of health information technology (IT) and specifically to the nascent streams of work evaluating economic and health implications of widespread adoption of health wearables (Handel and Kolstad 2017) and the broader potential of digital platforms to unlock interventions that leverage social norms and reciprocity to improve health (Liu et al. 2019a; b; Sun et al. 2019). Currently, the evidence on benefits from health wearables does not align with their promise. Piwek et al. (2016, p. 2) suggest that “current empirical evidence is not supportive” of health benefits from health wearables. Recent studies using large samples and robust causal approaches find little or no benefit on health outcomes of using health wearables (Lewis et al. 2015, Finkelstein et al. 2016, Jakicic et al. 2016). However, scholars have argued that a limitation of prior works is that they do not adequately consider the role of innovative technology decision aids and behavioral interventions enabled by broad adoption of health wearables (Patel et al. 2015). Our study addresses these limitations of prior work and finds that, on average, leaderboards promote healthful activity. However, our results also caution that these benefits may be highly nuanced with considerable variation in gains. In some cases, individuals may opt into variants of these interventions with either no benefit to them or, in some cases, negative effects on their physical activity.

We also contribute to the behavioral economics and information systems (IS) literature on gamification, especially, within the healthcare context. Despite mixed evidence of benefits and numerous open empirical and theoretical questions (Liu et al. 2017, Treiblmaier et al. 2018, James et al. 2019), gamification is spreading into a number of decision contexts. For instance, two recent IS papers have examined the impact of gamification within the retail context (Pamuru et al. 2021, Ho et al. 2022). Our study is differentiated with extant literature in several ways. First, our study is an individual-level intervention within healthcare in which the combination of unique data and rigorous estimation approaches results in more conservative estimates of average treatment effects of leaderboards; prior work showing positive effects of similar gamification interventions has found treatment effects five times our estimates (Shameli et al. 2017). Second, our results suggest that the mixed evidence of prior work may be explained, in part, by significant heterogeneity in gamification impacts. Not only are we able to provide more nuance in our study for gamification’s impact (Ho et al. 2022), we also provide evidence on a substantively important issue in the medical literature, that is, the impact on the previously less active users (Patel et al. 2015). In our setting, the relatively conservative estimates of the average effect of leaderboards mask robust heterogeneous effects that are large in magnitude, statistically significant, and persistent over time. These heterogeneous effects support our theoretical conjecture that competition and social influence are key mechanisms underlying leaderboard effects but also highlight that these mechanisms can result in unexpected motivational and de-motivational effects. Specifically, we identify a divergence of benefit for sedentary versus highly active users that is opposite to the expectation for competition-based gamification in the literature. These findings point to the underappreciated role of social influence benefiting sedentary users but harming more active ones. These results not only have significant managerial implications for firms in the health wearable and gamification spaces, but also for policy makers, healthcare entities, and employers interested in improving health.