“Iztok, you have to talk to Lee.” This is what I said to myself as I sat in my office thinking about the key problem of airline conversion optimization – how to scale up airline AB testing programs and run more AB tests.
I was deep into the research for my Airline Digital Optimization Yearbook when I remembered Lee Barrett’s (Head of Ancillary Revenue at VivaAerobus) presentation at this year’s Aviation Festival.
You see, after every event, I wait for the organizers to upload the presentations. Then I turn my phone off, grab a huge cup of coffee and slowly browse through them to see if there are any new concepts I can use in my materials.
Unfortunately, most of them are not really useful. You know, they’re the typical presentations talking about one of the latest buzzwords, like personalization and why you should create personalized offers, followed by some fancy statistic to stress the point. So, a bunch of nice charts but not really helpful for understanding what to do or how to put it into practice.
Now, you’ll ask, why was Lee’s presentation different?
Because he was talking about what I think are some of the key airline industry trends for 2019 – but he wasn’t just talking about the trends.
How VivaAerobus is using artificial intelligence to offer the right ancillary product, at the right time, to the right person by propensity modeling. Or, how they use artificial intelligence to significantly scale up their airline AB testing and how this helps them grow their ancillary revenue.
Sound good to you? It sounded great to me. In particular, the last part of his presentation, where he talked about how they use artificial intelligence to run more AB tests, really made me curious. Why? Because I’m in the middle of the 2018 Airline Conversion Optimization survey, and I’m seeing one big pain point among all airline optimization pros. Here is a striking fact from the survey:
I surveyed more than 35 airline experts in charge of conversion optimization, and almost every single one of them would like to run more experiments and AB tests than they currently do.
2020 NOTE: There is a newer, bigger and better version of this research available >> The 2020 Airline Digital Optimization Yearbook
So, I called Lee hoping he can provide you with some ideas on how to solve this problem.
Lee Barret, Director of Ancillary Revenue at VivaAerobus
Before jumping into the meat of how VivaAerobus is scaling up their airline AB testing, I asked Lee about his background and how he ended up doing optimization for ancillary revenue:
I worked for a number of airlines through[out] New Zealand, Australia, and Asia and finally ended up in Mexico. Based on my background I was pretty well positioned to come into a young airline that was looking to take their ancillary business to the next level. I came here because ancillary revenue was not performing at the highest level. It wasn’t a sustainable long term business model. They had the basics in place, but they weren’t really sure how to continue to grow and deliver double digit growth year on year.
And this is what Lee found challenging:
I really like to pull things apart, really understand what’s driving the business and customers’ decisions, and then look into how we can capitalize on that and increase the revenue. This is really what I found fun. I always worked in transformation and start-ups.
Lee’s background and mindset definitely explain why he saw an opportunity in experimentation and optimization. However, to me, it was still unusual to find this approach in an airline ancillary revenue department. Lee’s response:
“What I discovered when I looked into VivaAerobus’ ancillary business model was that it was heavily focused on punitive fees. Change fees, flexibility costs, excess baggage, late fees… and it was unsustainable. When you look into benchmarks and the percentage of the ticket that is going into fees, you realize that if you want to grow double digit year on year, you simply can’t just keep increasing fees.”
So, Lee was looking for where to find an opportunity to grow the ancillary revenue and he was facing a familiar dilemma:
Everybody is facing the same issue of what the next big product is. We’ve done baggage, basic economy, food and beverage on board, excess baggage at the airports, travel insurance. So what’s the next big opportunity? We were looking for our new lever because we found that we were really ‘between a rock and a hard place’: bring more products that produce little more revenue, or increase prices and impact conversion.
Especially the product that has the greatest share of customer conversion; the more you increase prices, the more you risk damaging your core conversion, which is ultimately just moving money from your left pocket to your right pocket. So, we were desperately looking for a way. We knew there was a solution somewhere, but just didn’t know how to get the right offers to the right people at the right time.
And that’s how they got to optimization and testing. Together with their technology partner, they started to play around with propensity modeling and personalization. Or as Lee put it:
That’s where we created our sustainability.
Actually, VivaAerobus didn’t start with a big artificial intelligence-driven experimentation program and platform from the get-go. I asked Lee if they started by doing basic AB tests and simple experiments before calculating propensity and scaling up experimentation:
Oh, yeah! We’ve evolved from using basic airline AB testing. We would run two or three variants over a period of 28 days to get to statistical significance and then start implementing it.
However, they wanted to do more. They wanted to test more and learn faster:
We knew the opportunity was there, but we were so slow. We were crawling. We’re like ‘test, test, let’s go,’ but it was only possible to do a test every 28 days.
This is how they came up with a new approach – multi-armed bandit testing:
Now, we’ve moved on to multi-armed bandit AB testing, which is multiple variants, like 9 or 10 variants and using an algorithm to reallocate the weight of the test according to the highest performing variant.
Soruce: Lee Barrett’s presentation
Although I’m a huge analytics enthusiast, I’m not a data scientist or an artificial intelligence expert. So, I tried get to the bottom of what this actually means. I asked Lee if they are using the multi-armed bandit AB testing to speed up the time needed to come to a winning variant for their experiments:
Well, there are two sides to it. First, AB testing can be expensive and it’s painfully slow. For example, only 30% of our AB tests pass. Each variant you test that performs poorly vs. the current offer is costing you dollars until the test is complete.
You’re also limited to the number of variants that you can test, and it takes almost a month to complete each test once you consider the analytics and everything else that is required after the test. It’s just too slow and there is so much money on the table.
Whereas with the multi-armed bandit you have many more variants running at the same time. It’s reallocating test weight away from underperforming variants, so you’re saving money by quickly abandoning concepts that are not delivering and therefore reducing your test cost, and also speeding up the result with higher volumes running though the stronger variants. If you’re using a 5-6% test group, you can for example deliver the results in 7 to 10 days as opposed to 28 days. You can declare a winner much faster and move on to your next test.
Wow! Just think about this and imagine if you could do three times more experiments than you do currently, or if you could do your experiments three times faster. Not bad, huh?
The next thing I asked Lee about is how many experiments they’re running now with this new approach:
Now we are at around 35 to 40 tests in a year we can deliver with the current technology. The number of variants we can now include in a typical multi-armed bandit test is 8 to 10, so with all 40 tests running 8 to 10 variants (versus our old capability of 12 tests per year with 2 to 3 variants), we are seeing revenue results much faster. Today we are running the equivalent of 100 to 110 AB tests in the same amount of time.
However, Lee and the team are already working on the next step to learn even faster:
We are moving from the multi-armed bandit testing to contextual decisioning. Hopefully by the beginning of next year we’ll fully implement contextual decisioning, so the whole concept of traditional testing just goes out the window. You’re just adding more and more variants to your algorithm.
Source: Lee Barrett presentation
Basically, they are taking the next step to almost automated testing. As they see it, the more you test, the faster you learn:
Always testing, always learning, always improving. That’s our whole philosophy.
“This sounds great, but we could never do it with our organization, given our technology and development backlogs.” This is what you probably want to tell me, right?
First, let’s look at the organization part. This is how they do it at VivaAerobus, according to Lee:
Keeping ancillary revenue, revenue management and e-commerce as three separate units has actually been very good. It challenges us all the time to work in a way to not prioritize the importance of one area over another. We all get equal voice in it. When you elevate all three positions, everybody is learning from what we’re doing in ancillary revenue.
Of course, e-commerce generally speaking looks after the web and the customer experience, and they are impacting our decisioning in a positive way. They are giving us ideas about what customers want to convert on the core product. We’re exploring the customer’s willingness to spend as much as possible without impacting the core conversion. Revenue management’s key focus is the core conversion. We are all working in a symbiotic way. It forced us to develop frameworks and it forced us to understand each other.
The next challenge you’ll probably face is a lack of resources and development backlog. Based on our research, these are the two biggest challenges airline optimization pros often list when asked why they aren’t running more AB tests.
So, I asked Lee if his ancillary revenue department is “fighting” with the e-commerce department for the development resources:
No, we outsource our [development for ancillaries]. We knew it was going to be an issue from the beginning; we knew we were going to be fighting over how fast we can get our ideas to market. So, we outsourced all our data science and all the development.
We integrated with a company called Fusion in our booking funnel. From the fare availability page in the funnel and as far as the payment section, this is where Fusion and we [the ancillary revenue department) have total control and ownership. We’re looking to add Manage My Flight to it as well. Basically, ancillary revenue real estate begins at the availability page, where fares and bundles are displayed, and ends at the payment screen – and Fusion manage 90% of that. As a result, we don’t compete over the development resources because we made that strategic decision from the beginning.
In a way, the technology reflects their organization. There are independent units, each with their own priorities and their own development pipeline, but they’re aligned and sharing results and learnings.
The last thing I asked Lee is the hottest topic in our industry. Yep, you guessed it – personalization.
We’re still learning about it, to be honest, about how we do this together. E-commerce is responsible for personalization. We’re using it as our decision engine behind our post-purchase campaigns for ancillaries. The collaborative relationship we’ve built, the optimization we do in the funnel, has allowed us to expand the relationship with e-commerce.
How personalization and ancillary optimization and testing are connected:
Our algorithms are purely based on propensity. We talk about personalization in the context of very granual segments. Based on the data we have, we calculate how likely this person is to behave in a certain way. What’s the relevant information, product, price we want to present to this person during the interaction? Personalization in terms of what the customer has done before, his preferences, we don’t see it as relevant in the funnel as we do in the way we communicate with the customer in the post-purchase context.
That’s interesting. VivaAerobus looks at personalization during the booking funnel (and ancillary upsell) more in terms of what actions a person (or similar customers) took during the purchase process than his prior history.
When we send an email to a customer and say ‘It’s that time of year again – will you travel to Cancun?’, it makes perfect sense to use preferences and what he has done before. But when it comes to upselling baggage, we don’t see it as relevant. We more look into how other customers are behaving for this same flight.
To understand customer behavior when it comes to ancillary revenue, VivaAerobus uses artificial intelligence. They use it for propensity modeling – a method of predicting the product with the highest likelihood of purchase from a group of products. With historical data, they use Logistic Regression to produce estimates on how changes in each variable affect the odds of a purchase.
Based on different factors (channel, activity, product categories, product variations, content variations, customer groupings, sequencing variations), they calculated over 1.2 million potential combinations. While propensity modeling will allow them to calculate the best combinations for a specific customer, contextual testing will allow them to test combinations on a huge scale.
The next level for us is contextual decision making. It’s an unlimited number of different pairings of different products of different pricing. What the algorithm will do, as it builds sophistication, is it will allow us to look at what opportunities exist in variances added in order to create a truly personalized offer for the customer. In other words, instead of us defining what the offer should be, every customer will have the opportunity to receive any offer based on the data and his behavior.
No Comments