I met Andres Bucchi, Chief Digital Officer at LATAM, last year in September in Amsterdam. He reached out to me after listening to my podcast chat with Harvard Business School professor Stefan Thomke. He wanted to brainstorm ideas about experimentation. What surprised me once we started talking is that he had ideas about running experiments not only in the digital space, but in the real-world, physical airline environment.
Andres talked about data science and experimentation with such passion that the decision to extend our chat into a podcast talk about airline scientific decision-making came really easy.
Now, once we recorded the podcast I realized there were so many valuable insights and so many different ideas to think through that I decided to split our podcast talk into two parts.
In Part I, you’ll be able to learn about experimentation, running experiments in the real world (e.g., airline revenue management) and what Andres learned working for Uber.
In Part II, we touched on the topics of data science, Andres’ learnings from Big Tech and of course the hottest topic at the moment, ChatGPT.
Listen to the new episode of the Diggintravel Podcast to learn about Andres and his data science team work at LATAM, or read on for key highlights from our talk:
And don’t forget to subscribe to the Diggintravel Podcast in your preferred podcast app to stay on top of the latest airline digital optimization, data science and CRO trends!
Andres is relatively new to the airline industry, but has vast experience in business intelligence and data science.
I’m currently the CDO of LATAM. I joined about six months ago. Prior to that, I acted as a VP of Data and Analytics at Sodimac. Sodimac is probably the biggest home improvement retailer in Latin America with between $6 to $7 billion in sales, 260 stores, presence in seven markets in Latin America, you get the idea.
But it was his experience while working at Uber that sparked his passion for data science and experimentation.
Before that I was working at Uber HQ in San Francisco. I went through different areas and I ended up in the pricing team, which is a very scientific team, and I loved it. It was a very watershed moment for me, as they call it.
As mentioned in the intro, I’ve talked previously about the concepts of running business experiments with Stefan Thomke, but have never seen it done at a bigger scale in the airline world. This is why I was curious about how Andres came up with the idea to do this at LATAM.
First of all, it’s not my idea. Econometrics actually plays very well with data science in these physical digital environments. I came across this when I joined Uber. Most companies in most industries are physical companies, so customer journeys and experiences are physical mostly.
This is very similar to the airline world, where we have a fusion of the digital and physical customer journey.
Yes, like airlines, Uber, supermarkets, convenience stores, home improvement stores. What’s bound to happen is that these digital transformation journeys eventually will run into the bottleneck that experiences are physical, and those pose a lot of challenges. And if you look into big tech, this actually happened already. Big tech, even if it’s natively digital in the cases of Uber and companies like Uber that have physical services, did go through this transformational process, where they needed to understand how to roll out better products quickly. And this is mostly driven by experimentation.
I write a lot about airline digital optimization and experimentation here at Diggintravel. But this is experimentation on another level.
You already know a lot about that in the digital world, as it’s your area of expertise. But that can translate fairly well into the physical world. When I joined the Pricing team at Uber I realized that this (experimentation) was driving everything. You would see people go out of one experiment and start thinking immediately about the next experimentation time slot which could come a month from then. Everybody threw ideas, then they would do some back-of-the-envelope calculations and everything revolved around it. The organization was set so that this could be fast. You would see some teams competing (with different approaches to the same problem), it was crazy. Like you thought you were working for NASA or something.
Source: LATAM airlines website
I was really intrigued by this concept, so I wanted to better understand how this works in practice and with a case that could be replicated in the airline world. Andres shared an interesting case from Uber and explained the challenges of running experiments in a physical environment:
I’m going to talk about pricing first, but we can talk about how this translates to home improvement. We did a lot of experimentation there and some cases that might relate to aviation as well. So the main problem with changing prices is that you affect a specific set of Drivers or Riders (Uber drivers) if you want to do A/B testing, like in traditional randomized control trials.
The problem with that is that if you affect let’s say a bunch of Drivers with a more effective matching algorithm, they would be exposed to this more effective algorithm, and in theory what would happen is they are able to take more trips. That means they can satisfy more demand with the same supply. The problem is that on the other end, there are some drivers that have not been treated with this new algorithm. What you’ll see in the experiment is a big bias, because those drivers are not only affected by the effect that the algorithm itself would yield, but also by the fact that the treated group is being more effective and kind of taking demand from them.
So, it’s not like in the digital world, A versus B? It’s more like comparing two physical things?
Yes, it’s a physical thing, so they compete for the same resource. And you cannot really separate everything where everything is super entangled and it doesn’t matter if it’s one, two or three alternatives. You’re never going to run out of rendering your webpage or right buttons to click. But in this case it’s the physical world, and you actually run out of stock. And so this was a big problem, and it was solved with experimental setups.
Instead of using traditional A/B testing with random control trials, you would do time sampling. You have these windows where you will turn (the algorithm) on, then off, etc. Then you would get your pseudo-randomized sample, which was good enough for running these experiments. There were other alternatives, where you sample in clusters that do not interact with each other – this is something Facebook does a lot – and what you end up having is groups that are comparable in time. There is a technique that’s called difference in differences which is very widely used for this.
If we translate this to the airline world, could the same logic be applied to airline prices? So you could say different groups on the same route, or even on the same plane?
Yeah, totally. You got it. This is actually the same problem that the airlines have and the problem with rolling out new pricing algorithms, or new revenue strategies. It’s that you share the same routes. It’s not like you can “treat” a plane and then “un-treat” it for some customers, right? You would get into big trouble, probably. Some people would end up hating you, others loving you. But probably what would happen is that the ones that love you would get all the tickets first. So then you would bias everything.
One way to go about this is using what traditional (big-tech) companies use when they compete against a limited stock of physical goods. Amazon has been doing this for a while now. I think eBay started doing this way back when they realized that their network had a limited capacity, so treatment and controls would compete for a limited resource. And they started doing either switchbacks or comparing routes that do not interact with each other and setting up these difference-in-differences A/B testing type of scenarios.
If you want to learn more about the difference-in-differences method, please listen to the full podcast chat where Andres explained more.
Personally, I think airlines doing scientific decision-making by expanding experimentation from digital to other areas makes a lot of sense. This gives airlines a data-driven approach to test and measure the effect of their business decisions. The logical next question I had for Andres was how to convince the top-level management to go on this journey of experimentation and scientific decision-making.
The approach I took in Sodimac is that I went for problems that were unattended but had potentially big impacts. You have the main problems everybody agrees on that are very difficult and everybody’s focusing on, but others are fairly unattended. And when you tackle those and you demonstrate that there is a technique that can accelerate everything, everybody will agree with this concept.
If you are fairly good at estimating and building a hypothesis you invest in that and then you go tackle it and show that the results are reflected in the bottom line, you get the right people listening. “Okay, I’m listening, explain this to me so I understand it and I can give you permission to step forward.”
Again, I tried to translate this to the airline world and to revenue management as Andres was talking about pricing. Would this give you a method to try different approaches to pricing and really test them and see what’s better?
Yes, and they [revenue management] understand it right away. This is something that they probably look for and they just don’t know that it already exists. Probably a decade ago it was already implemented in some big tech businesses.
The problem is incredibly complex, so I guess some other airlines are probably realizing this, but going through the whole process is very difficult.
What exactly did Andres mean by the process? Is it setting up this practice and framework to measure and experiment?
Exactly. It’s not like you have this experimental setup and you just use it. There’s a lot of research you have to do to understand which are the upstreas factors that influence things, how to calibrate the tools. It’s like, let’s say I give you a rifle with a laser sight and you just plug it in and try to shoot. You will never hit the target unless you calibrate it. And you need to calibrate it for how far you’re shooting, for the environment conditions, the wind, etc. So, there are a lot of things that we need to do. And the experience is actually very important.
In Andres’ role as a CDO, he is responsible for not only experimentation but also data and data science at large. My question was, do you need to have data science maturity at a high level, a data-driven culture and the ability to measure, as a prerequisite step to run the experiments he explained? Or can you learn and build both in parallel?
I think that’s a very good question. It’s probably a chicken and the egg thing. How I would go about it is how I’ve seen it play out before. So, you probably want to grow both at the same time. But what you need to get sorted out for sure before you start thinking about this is where you’re going to get the talent, because this is something very specific and very technical. You need to find the right people in leadership and in technical teams to push this through. It’s a long road map. It probably would never end, and you want to definitely build the strength to do it. So yes, talent is probably going to be a significant bottleneck.
However, the upside is huge. If you manage to implement a scientific approach to decision-making, you can roll out new ideas, new processes or new products with confidence that you’ll be able to measure their effect.
This (approach) works very well in strategies that are trying to iterate products very quickly.
As I mentioned, once people realize the power of experiments, once people see this, then it’s fairly clear. Talent is the main driver, so you need to check if you’ve got the right people and they understand the problem. It’s a very technical problem, and if you have the people then you just need to start trying.
Finally, what are some other applications that Andres sees for airlines?
There are some things I can talk about and some secret sauce that I’m going to steer away from. But getting the technology to compare routes probably untaps a lot of things that have to do with maintenance, pricing, and customer experience, so that’s actually a huge opportunity. If you are interested in understanding a bit more, look into comparisons of non-randomized groups of things.
If you want to learn from leaders like Andres about how to build innovative airline digital products or want to be the first to know when our next Airline Digital Talk will be published, please:
No Comments