(Link at bottom)
During preparations for our new training course onAnalytics and User Experience, I discovered recent research that invalidates many simplistic approaches to website metrics. I’m ashamed to admit that it took me two years to appreciate these results, but the papers were written by economists that I normally don’t follow.
Even though this research was done by economists and not usability people, it’s the most important new insight into web user behaviorsince information foraging and information scent.
The basic finding is that users’ behavior is much lumpier than previously recognized: for any given user, periods of unusually high Internet activityalternate with periods of low activity.
So, once again, this means that you can’t conclude that exposure to a specific stimulus causes a certain behavior, even if you observe increased occurrences of this behavior after the exposure. As I’ve always said, correlation doesn’t prove causation because hidden covariants might exist. We now know that user activity bias is one such covariant — and that it’s very strong.
Do Promo Videos Make People Visit Websites?
At Yahoo, Randall A. Lewis and his colleagues ran an experiment in which they asked visitors to Amazon’s Mechanical Turk service to watch a promotional video for Yahoo. The following chart shows whether these users visited any of the sites within the Yahoo network on the day they watched the video, as well as during the two weeks before and after their video exposure.
The chart’s heavy blue line shows the behavior over time of users who watched a promotional video for Yahoo while visiting Amazon Mechanical Turk. Day 0 is the day the users saw the video; clearly, they were dramatically more likely to visit Yahoo on that day.
Given the users’ average behavior during the previous two weeks, the data shows a fabulous lift of 144% in their Yahoo visits on the day they saw a promotional video for this site. Even better, the impact of the video stays with users for some time: there’s a more modest 43% increase in user visits the following day (Day +1).
A naïve reader of such an analytics report would now conclude that this promotional video was a superb marketing tool. Clearly, the company should invest heavily in showing this video more widely.
Not so fast. Close inspection of the chart also shows a lift in visits on the last few days beforeusers watched the promotional video. On Day -1, there was a 36% lift in Yahoo visits. How can that be? Unless you believe in time travel, how can a promotional video make users visit a site before they’ve seen that video?
Even worse, look at the chart’s thin orange line. This line shows the behavior of users who visited Amazon’s Mechanical Turk service but wereshown an unrelated video that didn’t mention Yahoo. As the chart shows, these users had essentially the same behavior pattern as users who saw the promo video.
So, on the day the control group watched the unrelated video, they, too, experienced a huge lift in their desire to visit Yahoo. And these extra Yahoo visits also persisted for some time.
It strains belief to think that a video that never mentioned Yahoo would have enhanced viewers’ impressions of the Yahoo brand. Clearly, watching the unrelated video didn’t cause the Yahoo site visits; given that the two videos had essentially the same effect, we can conclude that the promotional video didn’t motivate the Yahoo visits either.
And yet, Yahoo visits rose dramatically. If this wasn’t because of the promotional video, what caused the extra visits? Users in both the stimulus and control groups were simply more active on the Internet the day they watched the videos.
This phenomenon is called activity bias: some days, people do a lot online; other days, they do very little.
On very active days, people are more likely to do both Activity A and Activity B, no matter what A or B might be. (In this experiment, “A” was visiting Mechanical Turk to watch a video and “B” was visiting Yahoo.)
Crucially, even if there is no relationship between A and B, the very fact that you observe users doing A means that they are likely to be having one of their more active days and therefore are also more likely to do B.
Do Search Ads Make People Buy?
Thomas Blake and his colleagues at eBay further illustrated the activity bias effect in their experiments with search engine advertising on Google and Bing.
Before the experiments, eBay had run a broad variety of search ads and recorded both good clickthrough rates and significant sales to users who clicked the ads. Does this make the ads worth running? Not necessarily.
Now that we know about activity bias, we recognize that the very fact that users clicked on eBay advertisements means that those users were probably having one of their high-activity days. (On low-activity days, people search less and click fewer ads.) On a high-activity day, users would also be more likely to buy stuff on eBay.
Thus, just because ad clicks and product purchases happened on the same day doesn’t mean that the ad caused the sale. It’s also possible that both events were caused by users having a particularly active day and doing a lot on the web.
As a first experiment, Blake et al. simply switched off all advertising for branded keywords on Google and Bing. (Branded searches are those where a user’s query includes the name of the company or one of its brands — such as a search for “eBay shoes” instead of simply “shoes” or “buy shoes.”)
Although visits driven by ads obviously stopped, people still kept coming to eBay. The authors’ analysis shows that only 0.5% of the expected clicks from branded search ads on Bing were lost, whereas 3% of the Google clicks were lost. In either case, the vast majority of those users who would have clicked an ad still arrived at eBay through other means, usually by clicking an organic search result.
Remember, these were branded searches; users had already decided to consider eBay, as evidenced by the fact that they included the company name in their queries. So, not surprisingly, it’s a waste of money to advertise further to people who have already decided to visit the website.
(I believe that some companies run branded search ads to reduce the number of ads that people see from competitors, who might divert prospective customers at the last minute. However, this is a very expensive way of suppressing the competition that might not be worth it for companies with a decent brand reputation. If people have decided they want to check your offers, they’re unlikely to be diverted by other companies’ ads.)
In a second more elaborate study, Blake et al. tested the effectiveness of non-branded keyword advertising. In this case, they stopped the search ads in 65 randomly chosen U.S. metropolitan areas. They matched these geographical regions with others as closely as possible to create a set of 68 control metropolitan areas in which search advertising continued as usual. In total, the authors estimate that the paid search advertising added 0.4% to sales and that the return on investment was so close to zero as to be statistically insignificant.
This is not to say that only 0.4% of sales were to people who had clicked search ads; sales to ad clickers were much higher (though eBay keeps the actual number secret). The key point, however, is that many of these sales would have occurred even without the ad. In those regions where the search ads were run, many users clicked the ads because it was easy and they were right in front of them. Users are lazy, as we know from countless usability studies. This drove up the “attributed sales” for the ads. But in those regions where no ads ran, users reached the site in other ways and made an almost equal number of purchases.
In an additional analysis, the authors considered whether people buying on eBay had made a purchase there during the previous year. For people who hadn’t made a purchase during that period, the search ads increased sales by much more than the general estimate. For people who had made one or two purchases during the past year, there was also a small sales lift from the search ads. But, for customers who had made three or more purchases in the past year, the sales lift from search ads was statistically too close to zero to be significant.
This finding matches the branded keyword study’s result: the benefit of search ads comes from exposure to customers who don’t know you or don’t remember you. People who know the brand are less likely to be swayed by such advertising.
The key lesson from this study? Activity bias comes back to haunt marketing managers who run simplistic analyses of “attributed sales” to advertising, assuming that sales are caused by whatever happened to be the user’s last click. Many users who both click ads and make purchases would have done the latter even if they hadn’t seen an ad. A controlled experiment is the only way to discover an ad’s true impact.
Why Does Activity Bias Exist?
Several different experiments have found strong evidence of activity bias that was so prominent that it greatly distorted the conclusion one would draw from a simplistic view of website analytics.
Why is activity bias so pronounced in user behavior? We don’t know; further research is needed. However, I can certainly speculate. Here are some possible reasons for activity bias:
• Some days, people have plenty of time to kill on the computer. Other days, they might be on vacation or a business trip, have a looming deadline, or have other reasons to minimize their time online.
• Being on the computer can be captivatingin its own right. You sit down to check one thing, and by the time you look up, an hour has passed and you’ve visited twenty other websites. One thing leads to another. Other times, you need to check one thing and then immediately go discuss it with your boss. You never get time to be pulled into the mesmerizing realm of the web.
• Sometimes, external factors — such as writing a research paper or planning a vacation — drive people to heavy web use.
• People’s moods impact web use; in a good mood, users might be happy with a user experience that might otherwise disgust them and lead them to turn off the online world.
• Real world events drive or hinder Internet use. Anything from network outages (zero use) to hurricanes (heavy use while people are cooped up, assuming they can connect).
As I said, the reasons here are speculation. What we do know is that activity bias is real, and users’ online behavior is lumpy. We should recognize this, take advantage of it in our designs when possible, and definitely control for it when we use analytics data.
References
Thomas Blake, Chris Nosko, and Steven Tadelis: “Consumer heterogeneity and paid search effectiveness: A large scale field experiment,”National Bureau of Economic Research, Meeting on the Economics of Digitization (March 8, 2013).
Randall A. Lewis, Justin M. Rao, and David H. Reiley: “Here, there, and everywhere: Correlated online behaviors can lead to overestimates of the effects of advertising,” Proceedings International World Wide Web Conference WWW 2011 (March 28–April 1, 2011, Hyderabad, India).
