I was in the middle of my podcast talk with Lukas Vermeer, Director of Experimentation at Booking.com, when it hit me. We were talking about how experimentation helps you build better travel digital products when Lukas said something really powerful.
Rolling out features is not the goal. Solving customer problems is the goal.
And we don’t know which of the features that we’re rolling out are actually solving customer problems unless we test them.
This was his counter to my observation that especially in times like this when we need to make new digital solutions fast, experimentation is often the odd man out. I often hear, “Yeah, experimentation is great, but we need to build new solutions now, and we need it fast. Running experiments slows us down.”
This idea of “I can be faster if I don’t run experiments” to me is a little bit the same as saying “I can run faster if I close my eyes.” That’s true. It may be true. If you close your eyes, you can go faster. That’s because you don’t see the wall. You’re still going to run into it. It’s still going to hurt, but you just won’t notice as much if you’re not testing.
So it’s super important, especially when you’re moving fast, especially when you’re taking increased risk, when you’re not certain that something’s going to work because the environment is different because you’re trying to act quickly, especially in those circumstances, you should double check to ensure that you’re having the impact you want.
If you work on new travel digital products, please read Lukas’s quote again. This is what experimentation is all about. It’s about how you can solve your customers’ problems, not about how fast you can roll out new features.
My latest Diggintravel Podcast talk with Lukas was all about how to leverage experimentation to be more customer-centric and to build better airline and travel digital products.
Listen to the new episode of the Diggintravel Podcast to learn about how they build, test, and optimize travel digital products at Booking.com, or read on for key highlights from our talk with Lukas:
And don’t forget to subscribe to the Diggintravel Podcast in your preferred podcast app to stay on top of travel digital, experimentation, and UX trends!
Booking.com is a leader in experimentation and in building great travel digital products. However, for most airlines and travel companies, it seems like Booking.com is two or three steps ahead. When we benchmark digital optimization maturity here at Diggintravel, we see most airlines starting to integrate experimentation in their digital product development process. But how can airlines scale up their experimentation and digital optimization process? How can you make it a core part of your digital product development?
I asked Lukas, based on his learnings so far, what would be his advice on how to democratize experimentation? How can it be scaled to enable airlines to do experimentation at a larger scale?
I’m sorry there’s not a single right answer for every individual company. I think it highly depends on where you’re at. Together with Aleksander Fabijan and Microsoft, I’ve tried to clarify our thinking. Mostly Aleksander, actually. I should give him 99% of the credit. I mostly just gave feedback on his draft.
He specified this idea of an experiment flywheel where the value of experimentation and the way that you embed this in an organization really should be an incremental flywheel style approach. If you start running your first experiment and you show value, the fact that you show value will make it easier to dedicate resources to scaling the experimentation infrastructure, which will make it easier to run more experiments, which will make it easier to show more value, which will make it easier to dedicate value.
You get into this loop where each of these steps is reinforcing this idea that experimentation should grow within an organization, and it should start from a few small experiments, but it should slowly scale out.
Source: Aleksander Fabijan
One of the key elements of the experimentation flywheel concept is that as you run more tests and measure their effect, it enables you to invest back in your experimentation program. Here is how Lukas sees the end-goal, or the ultimate effect of the flywheel:
One of the things that makes that model work is the idea that the investments that you make into the experimentation platform or into your processes should be directed towards lowering the cost of each individual experiment.
We want to get to a point where running an experiment is not just considered the right thing to do; it’s also considered the easy thing to do. You want to get to a place where product teams don’t even think about it anymore. It becomes so easy to run an experiment that they just naturally gravitate towards that. It becomes the easy default.
I also talked to Lukas about one other thing that is crucial if you want to build great travel digital products: you always need to challenge your assumptions. Often, digital marketing people get too caught up in trying to optimize existing features and products and not thinking about whether or not they matter in the first place. We try to optimize and run A/B tests on what color a button is, without considering whether the button matters at all.
I think it’s very natural for people to only challenge the surface of an idea. It’s very natural to jump into a product and say, how can I tweak the color of the button? How can I tweak the location of the logo?
And that’s only scratching the surface of the assumptions that you’re making about how this product works in the first place. Before you think about the optimal color of this button, you have already made the assumption that the button is important in the first place. You’re assuming that this button matters, even. And unless you have some pretty clear evidence that that’s the case, I would say it’s more informative to challenge that assumption first.
The reason I say that is because if the button doesn’t matter, that means the color also doesn’t matter. By implication, the color is only important if the button is important.
So you need to understand first what is the value of this feature in the first place before you start optimizing it.
Source: Luka Vermeer, Booking.com
That last thing that Lukas said in the excerpt above is really important. You need to understand the value of a feature in the first place before you start optimizing it.
It might just be a deep-seated assumption that the feature matters at all. The best way for you to see if the feature actually matters is to run an experiment:
Ironically, it’s often easier to challenge those deeper seated assumptions than it is to challenge the details of the implementation. Running an experiment where you remove the button entirely is often easier to interpret than changing the color of the button because changing the color of the button might have very subtle effects on how people behave, but they’re going to be so subtle that you need a pretty powerful experimentation device in order to be able to pick up those things.
But if this button is important to the user experience, if you remove it, you will notice. You will see it from space. That’s also why I say we have to distinguish between this idea of a controlled experiment and an experiment. Especially these deeper seated assumptions, you might actually be able to test without a controlled experiment. If the impacts are large enough, you could get away with what we call a regression discontinuity design, which is where I’m going to remove the button today, and if my sales drop by 50% today, then I’m going to put it back tomorrow. And if my sales go up by 50%, then I know that it’s probably the button.
One of the questions I get a lot when I talk to airline digital professionals is about what kinds of experiments they should run. Should you test big things and big changes, or do a series of smaller optimizations that will in the end have an incremental uplift? Here’s Lukas’s take on this:
Those are two questions. Let me first counter a little bit. I am not saying that you should run bigger experiments. I am saying that you should run experiments on the bigger things. There’s a subtle difference there. I’m going on the assumption that you are changing your product and that some of those changes are big and some of those changes are small, and I’m further going to assume that those big changes have important consequences for your business and how you think about your customers.
Now, I say you should run those through an experiment first because that will have the biggest impact on your long-term strategic direction. It will teach you more about your customers than all of those little things that you’re also concerned about. Now obviously, you can also run experiments on the little things. But the reason I say run experiments on the big things is not to say run bigger experiments, but run experiments on the big things.
I really liked how Lukas framed it: you should focus on running experiments on the big things – the things that matter, the things that will have the biggest impact on your long-term strategic direction..
Lukas circled back to my question about what kinds of experiments you should run and added this important advice:
The second part of your question is more around how you decide how you spend your time between doing big things and doing small things. I think that really depends on the maturity of the product space that you’re in.
For a company like Booking.com, we have been in the space of selling accommodations on the internet since 1996. There are some aspects of our business that we are super mature in. In those aspects of our business, we are tweaking the machine. We are making small iterative changes to see whether we can make it just a little bit easier. But we’re really looking at micro optimizations here, and the only reason we can really do this is, one, we’ve already spent 20 years optimizing the crap out of this thing, so it’s already very good. Two, we have so much traffic coming in that we can actually see the differences in user behavior even when we make small changes.
Now, at the same time, Booking has already started selling flights. It’s already started selling cars, taxis, attractions. All of these things, we are very new to, and we haven’t been doing that for a very long time. So there, when I look at the experiments these teams are running, they’re much bigger things because the questions are so much bigger. There’s many things we don’t know about how customers want to book a flight or how customers want to book a taxi. The questions are larger, and so by their very nature, the experiments or the research questions are larger. Also, it’s more exploratory in nature because there’s so much we don’t know.
One of the questions I had for Lukas concerned how the pandemic is changing the way we build new travel digital products. Airlines are under a lot of pressure to build new touchless, digital solutions. Lukas and I had talked about challenging our assumptions, so my question to him was, should we challenge all these assumptions even more?
I think so, yes. With an upheaval and a catastrophe this big, it’s almost inevitable that some consumer behavior or some dynamics have shifted forever. I don’t know what those things are, but I’m pretty sure that something has changed. Something of this magnitude will have an effect. So it’s even more important that we figure out, are the things we’re doing still relevant to our customers? They may have been relevant two years ago. Are they still relevant now, or has something changed about the way our customers interact with our product?
The world is changing very quickly. We should be adapting our product. If we’re adapting our product in such an uncertain environment as we have at the moment, then it’s all the more important that we actually test what the impact of the changes are. Like I said before, any change that we make to the product is essentially an experiment. The only choice is whether we measure what the impact is.
When talking about assumptions and the experiments we’re running during the pandemic, Lukas raised another good point. Will these learnings matter in the future?
Now, I know what people are saying. They’re going to say, “Well, the world is so much changing at the moment that if we run an experiment and we find that people enjoy this feature, that tells us very little about whether they will enjoy it one year from now.” This is the question of generalizability. The things that we learn now, are they going to be relevant one year from now?
That is a genuine concern. I understand that this is a concern and that an experiment might not be able to tell you very much about the future. But it will tell you something about the now. And because we’re so much in an uncertain environment, it’s all the more important that we pay attention to the impact that our changes are having on our users now, because we really do not know how people will behave in the current conditions.
The last thing I asked Lukas was whether he sees any specifics in terms of building travel digital products and experimentation.
I have to say, I love the travel space. It’s just such a nice product space to work in. I love to travel. I think a lot of people love to travel. Everyone wants to travel. So making that easier for everyone has always, for me, been a nice mission to work against, even in pandemic times. I think this is somewhat different from some other players in the ecommerce space where you’re selling products that I don’t believe have as much positive impact on people’s lives. I honestly believe that travel makes people better people.
My comment was that travel and booking travel is a combination of emotional plus rational decisions. The purchases are more expensive, but still, they are emotional decisions. Here is how Lukas sees this:
I love that comment. One of the things I bump into sometimes is people say, “Booking a flight or booking a car should be as relaxing as possible. It should be as frictionless and easy as possible.” I agree to some point; there shouldn’t be artificial friction in this process.
But I think some of the friction and stress is needed. Some of this friction and stress is just part and parcel of how emotionally important this decision is to our lives. If we remove that stress and friction, we’re essentially telling people, “Oh, no, this is not important to your life.” Which is not true. These are important life decisions.
In the airline industry, we talk a lot about removing friction and copying Amazon’s 1-click purchase model – that is, removing as much friction as possible for fast, seamless booking. However, as Lukas pointed out, sometimes some friction is needed. He introduced an interesting concept, the idea of purposeful friction:
It’s this idea – in the user experience field, they call this purposeful friction. Some friction is actually good. Some friction has purpose, and it’s important that we think about which friction in the process is useful and which friction in the process we can remove. We should not be dogmatic about removing friction and always removing it.
The example I gave earlier of our experimentation platform is a good example. This button that was green, if we want people to really think about this decision, then it should not be a giant green button that is right in their face because they’re going to click on it without thinking. We should purposefully add friction to this process in order to get better outcomes for everyone. In this case, for travelers, in some cases we must add friction. Even if that’s perceived as a negative, we must add friction to make better outcomes for everyone.
If you want to get insights from leaders like Lukas and from airline digital case studies, please:
No Comments