You are in the experiment
Am I part of an experiment, and how would I tell?
Most large sites and apps test different versions of pages, prices, and offers on their visitors without saying so. This chapter explains what is being tested, how dynamic and personalised pricing differ, how tested pressure designs spread, and how to compare versions across windows and devices so that you choose from more than the one you were shown.
All stages
Experiments are routine, and the goal is what matters
Almost every large website and app runs controlled experiments. Visitors are split at random into groups, each group sees a different version of a page, a price, or an email subject line, and the version that produces more of the desired behaviour is kept. You are not told, and you cannot see the other versions. This is routine, and much of it is harmless or helpful: a clearer button label that wins a test is clearer for you too.
What matters is what the test is trying to increase. A test that measures whether people understand the price improves the page. A test that measures only how many people sign up will, over many rounds, favour whatever gets sign-ups, including confusion and pressure. A manipulative checkout need not be a deliberate plan by anyone; it can be what remains after a hundred tests scored on conversions alone.
Prices that differ
Two different things are called dynamic pricing. In the first, the price changes for everyone with time, demand, or stock: flights, hotels, ride fares. In the second, properly called personalised pricing or steering, the price or the order of results differs between people according to what is known about them: device, location, browsing history, whether they are logged in, whether they are a new or returning customer. Hannak and colleagues measured major retail and travel sites and found cases of both price differences and steering, linked to factors such as the device used and account status.
In the EU, traders selling online or otherwise at a distance must tell consumers before they buy when a price has been personalised by automated decision-making; prices that simply move with demand are not covered by that duty. Introductory discounts for new customers are the most visible form of differential pricing, and loyal customers are often the ones paying most.
When the experiment optimises pressure
Interface tricks are tested like anything else. In a controlled study by Luguri and Strahilevitz, adding dark patterns to a sign-up offer substantially increased the share of people who accepted a service they had little reason to want, and the milder versions did so without provoking much resentment. That is why such designs spread: they win tests.
From the outside, the tells of being in an experiment are small inconsistencies. A friend sees a different price or layout. The offer changes between your phone and your laptop. A page looks different after you clear cookies. A different plan is selected for you each time you arrive.
Compare versions before you choose
You cannot leave the experiment, but you can look at more than one arm of it. Before a purchase that matters, view the offer in a private or incognito window, on a second device, logged in and logged out, and if possible on a different network such as mobile data. Ask someone else to open the same page. Take screenshots with the date, because the version you saw is the offer you accepted and you may need to prove it later. Then choose the best terms you were shown, provided they are honestly open to you.
Keep two cautions in mind. A price difference is not proof of personal targeting: random test groups, stock changes, and currency or tax differences produce the same picture, so a single comparison shows that prices vary, not why. And the comparison is for information only. Do not misstate who you are, where you live, or whether you are a new customer in order to obtain a price; that crosses from defence into deception and may breach the terms you are agreeing to.
Techniques you will meet here, and the tell
Pressure designs persist because they win conversion tests, so the version you see has often been selected for its effect on people like you.
The tell: The layout, wording, or steps differ between your devices or from what a friend sees, and the differences are all in how hard the page pushes.
The order, grouping, and presentation of plans are test variables; the arrangement you see is the one that sold most.
The tell: The most expensive or longest plan is in the middle, highlighted, and labelled most popular.
Which plan, quantity, or add-on is preselected is tested, because most people keep whatever is already chosen.
The tell: A different option is preselected in a private window, or the preselected one costs more than the one you came for.
A plan that few are expected to buy is included to make a neighbouring plan look like good value.
The tell: One option is plainly worse value than the one beside it, and it makes the larger plan look like the obvious choice.
The first figure shown, whether a crossed-out price or a top-tier plan, is varied between versions to shift what seems reasonable.
The tell: The reference price differs between versions while the price you would actually pay stays the same.
Offers, wording, and timing can be selected automatically for each visitor from what is known or inferred about them.
The tell: The offer you see fits your recent behaviour unusually well and is not the one shown to a clean browser.
The advert or email shows one version of the offer and the checkout applies another.
The tell: The price or terms on the payment page do not match the advert or link that brought you there.
Worked example
Dana and Hearthbox are fictional. Before deciding, Dana looks at the Hearthbox offer in several ways to see whether everyone gets the page she got.
- 1In her usual laptop browser the offer is 60% off the first box. The plan preselected for her is four dinners for two, labelled most popular: eight servings, 95.92 dollars plus shipping, 105.91 dollars a week, not the 81.93 dollar plan she had been looking at.
- 2In a private window on the same laptop the offer reads 50% off the first box plus a free dessert, and the three-dinner plan is preselected. That first box would cost 35.97 dollars plus shipping, 45.96 dollars, which is worse for her than 38.77 dollars.
- 3On her phone, using mobile data, the offer is 60% off the first box and 20% off the next two. The second and third boxes would be 57.55 dollars plus shipping, 67.54 dollars each.
- 4Her partner opens the page on another phone and sees the laptop version, with the same countdown timer.
- 5She notes what never changes: 11.99 dollars a serving and 9.99 dollars shipping in every version. Only the introductory offer and the preselected plan vary. She cannot tell whether the variation is random or tied to her devices, and she does not need to.
- 6She screenshots the phone offer with the date and reads its terms: it is open to any new customer through that page, so she can take it without misstating anything.
Outcome. The best version honestly open to Dana makes three boxes cost 38.77 plus 67.54 plus 67.54 dollars, 173.85 dollars in total, against 202.63 dollars on the first version she saw, and inside her 200 dollar limit. The regular price is identical everywhere, so the long-run question from the earlier chapters is unchanged.
Testing worksheet: which version am I seeing?
Use this before a purchase that matters to you. Ten minutes of comparison is usually enough. Saved in this browser only.
Tick each view you checked; aim for at least two.
For each view, note the price, the introductory offer, and which option was preselected.
Write the terms that did not vary, such as the regular price and fees; these are the real offer.
Name the plan or quantity chosen for you and how its cost compares with the one you want.
Choose the description that best fits; one comparison rarely proves the cause.
Write the terms you intend to accept and confirm you qualify for them as you really are.
Record whether you have saved the version you intend to accept.
Decision check: have you seen more than one version?
0/6If any of these is not true: Hold the purchase and compare again. If versions differ, pick the best one that is honestly open to you and screenshot it. If the regular price itself differs between versions, ask the seller in writing which price applies to you and why. If the checkout does not match the advert, do not pay: keep both screenshots and treat it as a bait-and-switch for the report described in the final chapter.
Common mistakes
- ×Assuming the page you see is the page everyone sees. Offers, defaults, and layouts are commonly varied between visitors.
- ×Fixing on the variation in the introductory offer and forgetting the regular price, which is usually the same in every version and is what you will mostly pay.
- ×Misstating who you are to unlock a price. Comparing versions is fair; claiming to be a new customer or to live elsewhere when you do not is not.
- ×Over-suspicion about cause. A price difference between two windows may be a random test group, a stock change, or a tax difference; one comparison does not show that you personally were targeted.
- ×Treating experimentation itself as wrongdoing. Testing also fixes confusing pages; the fair objection is to what a test is optimised for, not to the fact of testing.
How do you find out what works without deceiving the people you test on, and who signs off?
Related
References
- Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014). Measuring Price Discrimination and Steering on E-commerce Web Sites. Proceedings of the 2014 Internet Measurement ConferenceMeasures price differences and steering of search results on retail and travel sites and links some to device type and account status.
- Luguri, J., & Strahilevitz, L. J. (2021). Shining a Light on Dark Patterns. Journal of Legal AnalysisReports experiments in which dark patterns substantially increased acceptance of a service, with mild versions drawing little backlash.
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B TestingDescribes how routinely online businesses run controlled experiments on their users and how the choice of success metric drives outcomes.
- Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving Decisions About Health, Wealth, and HappinessExplains choice architecture and the strong effect of defaults on what people end up choosing.
- European Parliament and Council of the European Union (2019). Directive (EU) 2019/2161 as regards the better enforcement and modernisation of Union consumer protection rulesAdds a duty, for distance and off-premises contracts, to inform consumers when a price has been personalised on the basis of automated decision-making.