We all know that loyalty programs and reward systems like cashback in e-commerce are great tools to encourage repeat purchases. We aimed to increase the frequency of purchases by paying our premium users cashback earlier than they normally do within 15 days. However, this well-intentioned move led to an unexpected result:
In this article, I will talk about our A/B test that we ran and the unexpected result we observed in user behaviour.
Why We Ran This A/B Test? 👀
Making data-driven decisions is crucial in e-commerce. We had been receiving user feedback stating that cashback processing times were too long and that customers often forgot to use their cashback due to the delayed payout period. These concerns led us to explore ways to optimise cashback frequency to make the program more effective.
We aimed to evaluate whether adjusting cashback timing could positively influence shopping frequency without negatively impacting business outcomes.
Our Hypothesis 📚
Faster cashback payments would encourage users to shop more frequently, ultimately increasing engagement and revenue.
- Objective of the Test:
-
Understand the relationship between the timing of users’ cashback purchases and their purchase frequency
-
See if users who receive cashback more frequently use the platform
-
Increase sales volume
-
Steps and Decision-Making Process:
- Initial Data Analysis: We examined past cashback transaction data to identify patterns in user behaviour.
- Considering Refund Policy: Since our company’s refund policy allowed returns within 15 days, cashback timing was aligned with this restriction to minimise refund exploitation.
- Test Groups:
- Control Group: Cashback payments every 15 days (standard practice).
- Test Group 1: Cashback credited on the same day.
- Test Group 2: Cashback credited every 7 days.
- Test Duration: We set a time frame to observe changes in behaviour and measure key outcomes.
- Key Metrics: We tracked user transaction frequency, average order value, refund rates, and fraud attempts to assess the impact of different cashback timing strategies
However, the findings we obtained at the end of the test surprised us.
Why Did Users Start Frauding? 🐍
When examining the results of the A/B test, we noticed the following:
- Some users in Test Group 1 and 2 began manipulating our system to earn cashback rewards illegitimately.
- An increase in small, frequent purchases, order completion followed by cancellations, and refund exploitation.
- Certain users would receive cashback and then cancel their transactions to extract additional money from the system.
- A significant rise in refund rates indicated that users were attempting to game the system by returning purchases after receiving cashback.
- An increase in daily transactions per user, likely driven by users attempting to maximise cashback payouts.
The biggest fraud risk emerged in the group where cashback was deposited on the same day. Users saw the incentive offered by the system as an “open door” and began to abuse it.
What Can We Do to Prevent Fraud?
This unexpected situation showed that we need to consider fraud risks when planning the timing of cashback payments. We evaluated the following solutions:
- Cashback Limits: Determining the maximum cashback amount in a certain period.
- Fraud Detection Systems: Using AI-based systems to identify anomalies in the user’s shopping habits.
- Strict Order and Return Controls: Examining users who make a large number of orders and cancellations in a short period of time.
- Different Cashback Structure: Making cashback payments more balanced and spread over time.
Learning from Unexpected Outcomes 🙃
This experience taught us that every user incentive should be designed with potential exploitation risks in mind. Before launching any program, we need to ask ourselves:
“How could users manipulate this system for unintended gains?”
Otherwise, a well-intentioned improvement could end up backfiring! 🚀
Thank you for your time; sharing is caring! 🌍
