Optimizely Web Experimentation : Understanding Bucketing with a use case

 

Introduction

In Web experimentation bucketing is used to assign users to different variations in an experiment. This is done in order to ensure consistent experiences provide statistically valid outcomes.

Bucketing is deterministic , it is sticky and it is scalable which works very well with dynamic websites and Single Page Apps.

In this blog, I am taking some examples from the utility sector such as billing, outages, service requests etc to simplify the understanding the concepts of bucketing. I hope this will be helpful for beginners to understand and apply these concept to real world use cases.

While the use cases that I am quoting from Utility sector is just a reference, the basic principles are the same and they can be applied across other verticals like retail, finance, health care etc.  

Bucketing 

Concept	Explanation	Use Case	Importance
Deterministic	Same user always sees the same variation	A returning customer always sees the same version of the 'Pay Bill' page	It Ensures consistent experience across sessions/devices
Sticky	Variation assignment will be changed only if traffic allocation changes	A customer in a 'Smart Meter Info' experiment will see other variations only if the traffic is reallocated	This prevents results being skewed because of rebucketing
Hashing (MurmurHash)	The user's Unique ID for eg. UserID is hashed  & used for variation assignment	User ID 12345 is mapped to variation B in the 'Outage Push Notification ' experiment	This has two benefits, one it uses randomization & the second it helps in consistency
Impact of Traffic Allocation	Changing traffic allocation percentage can re-bucket users	Increasing traffic from 40% to 100% in an experiment for eg 'High Energy Usage Insights' may reassign users	Traffic changes should be done early in the experiment to avoid invalid outcomes
Targeting Audience	Users must satisfy audience criteria to enter experiment	For e.g. Users with high-consumption will only see variation A of 'Energy Saving Tips'	Personalized experiments can be carried out
Scalable	Works across large number of users and concurrent experiments	Utility website with 200K monthly users running 4 experiments	Supports enterprise-grade experimentation

End to End Flow

Below is the process used by Optimizley to decide on which variation is displayed to which user :

Best Practices

• Set traffic allocation early in the experiment and avoid any changes in the middle of the experiment 
• Use consistent splits unless restarting the experiment
• Ensure required data (e.g., user tier, region) is available at activation

Summary

Bucketing is a core of reliable experimentation in Optimizely. It provides various mechanisms to ensure the users are consistently assigned to variations , it ensures precise measurement of the behavior and outcomes of the experiment . We have used utility sector use cases to explain these concepts but the same principles apply to an dynamic website say for eg a retail website's checkout, product details page or even personalized dashboard page.

I hope this post for useful in some way in understanding the bucketing concept.