SCENARIO
When designing web pages it is common to use what is called “A/B testing.” This is essentially a mini-experiment in which you create two versions of the same web page, and randomly assign visitors to see one version or the other. You then collect data on user behavior for each version, and analyze those data to decide which web page works better. For an e-commerce site, “better” might be defined as a higher rate of “conversions,” i.e., visitors to the site who end up making a purchase.
Imagine you are working for a company called Twisted Cedar Expeditions that makes gear for outdoor activities (packs, water bottles, climbing equipment, etc.). They have conducted an A/B test for their on-line store. In this test, the A version is the existing version of the web site, and the B version is a new version of the site that is intended to make more of an emotional connection with users through the extensive use of images and personal stories of customers.
The B version was the brainchild of one the VPs of the company, who personally oversaw the design process. About $30,000 has been invested so far in development of the B version. Initially the new version was going to be rolled out without doing A/B testing, but the head of the web design department insisted on it. At this point, it is already 3 weeks past the originally planned launch date for the new version.
Here are the details of the A/B test. For a two-week period, the company collected three kinds of data from visitors to the site:
- Duration of time spent on the site (in seconds)
- Whether a purchase was made
- Upon leaving the site (whether or not a purchase was made), customers were invited to type a few words into a text box to describe their experience on the site
208 visitors were shown the A version of the site, and 205 visitors saw the B version. A summary of the data that were collected is provided below. Your job is to write a brief report (1 page max) that…
- States the purpose of doing the A/B testing
- Briefly describes the methodology
- Describes the key results of the testing
- Makes a recommendation on how to proceed regarding the roll-out of the B version
ANALYSIS OF TIME SPENT
ANALYSIS OF PURCHASES
COMMENTS
Version A
| The gear is too expensive, you should have more sales. |
| The item I was looking for was out of stock, no indication of when it would be available again. Guess I’ll spend my money elsewhere. |
| The site is simple and easy to use. Find the gear, order it, done. |
| Looking forward to getting my new hiking poles, can’t wait! |
| I like the comparison feature, it really helps me decide which brand to buy. |
| Wish you would sell boots, it’s the only piece of gear I can’t buy on this site. |
| I like that the site loads quickly on my phone. |
| I wish people would stop always asking my opinion. |
| I often need an x-tra large size, can’t get it most places, glad I could find it here |
| Need extra cash? Earn money as a secret shopper. Up to $500/month. |
| I like how they have information about all the sizes. S/M/L/XL varies a lot across brands. |
| The site is functional but a little boring. |
| Don’t ever buy from this company. Their products are made by slave labor in Southeast Asia. |
| Links and buttons are big, good for a mobile device. |
| Found a sweet climbing helmet and it was cheaper than REI. |
| I guess the web site is fine. Looks like every other web site. |
Version B
| OMG the pictures were beautiful. I wanted to go to all those places. |
| I knew exactly what I wanted and it was a bit hard to get to it. |
| What’s with all the pictures? Is this National Geographic? Makes everything load slow. |
| The new Arc’Teryx pack is ill. Can’t wait for it to arrive. |
| I wanted to do a return but couldn’t find information on the web site. Have to call instead >:| |
| I had heard about a new model of harness that was coming out but it isn’t on the site yet. Will check back later. |
| Reading the stories is entertaining. Would come back and read more even if I wasn’t going to buy anything. |
| The web site is pretty. The links are really small on my phone. |
| I liked all the different colors of water bottles. |
| They won’t ship to a PO box. Wish they’d said that before I entered all my information. |
| I got lost in the stories – so interesting! Wasted 30 minutes on that. |
| Tried to view site on my phone. So slooooow to load. |
| Improve your love life! These pills are 100% natural and guaranteed to work. |
| Are the stories supposed to be inspirational? Mostly they made me sad about all the cool things I’m not doing as I sit at work L |
| Really liked the pictures. Wish they focused more on the products than the people. |
| Mostly I use the web site to look around and do research. I don’t buy things on-line because it’s hard to make sure you get the right size. |