Thinking in Bets: Annie Duke
Most people know Annie Duke as a phenomenally successful professional poker player with a World Series of Poker gold bracelet. But for over a decade, Duke has been consulting with executives of major companies on how they can accelerate their learning to improve their decision making. In her new book, Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts, Duke explains the dangers of connecting the quality of your decision to the outcome, why life is more like poker than chess, and how this all applies to whether your team is an expert organization or not learning at all.
What inspired you to write a book about decision making and how you can accelerate learning?
I started off as a cognitive scientist. I was getting my PhD at U Penn, and was planning to be an academic. But when you study cognition, you’re really thinking about how you are processing information out in the world. And I was specifically studying language acquisition, which may seem pretty far afield, but that is really getting down into the core of learning. You have these little tiny beings that can do it super fast. How are they doing that when everything is incredibly uncertain to them. So I carried that into poker, which turned out to be a really natural lab for understanding these kinds of decisions.
When you’re studying decision making in the natural lab of poker, there are these real monetary pressures on you. Your money is a limited resource, so you need to speed up the learning curve in order to be able to increase your advantage at the table. And there’s pressure on you to make money to be able to move up in stakes. When you move up stakes, you’re risking more money, but you can also win more money. But every time you move up stakes, the people you’re playing against are better. So if you wanted to relate this to professional athletes, there’s a lot of pressure to be able to try to move up in levels and solve these kinds of learning problems, because your finite resource as an athlete is the time that you are at your peak.
Why did you retire from poker to become an author and public speaker?
In 2002, I got asked to speak to a group of options traders about what goes into poker decision making. I had been thinking about it a little bit, but this was the moment—it was eight years into my career—where I started thinking very explicitly about this merging of what I’ve been doing in cognitive science with what I was learning at the poker table. It was about understanding these problems, learning under conditions of uncertainty, and how much uncertainty really gives us this cognitive rope that gets us to where we’re not learning from the feedback that we’re getting. This was a huge problem that I was seeing as I was watching people play poker. So I started developing talks and consulting and doing corporate retreats. People knew me as a poker player, and initially the speaking was much less than the poker. But then it started to become a bigger part of part of my life. By 2012, I retired completely from poker. So that’s the point at which I was really thinking about a book. I really saw how this is not only a learning problem in poker, but that it’s really a problem everywhere. It’s the biggest thing that stops us from achieving excellence. It’s the uncertainty of how you process your own outcome.
One topic you delve into in great detail is “resulting.” Can you explain what resulting is and why it hampers learning?
I opened the book with one of the best cases of resulting ever. In the 2015 Super Bowl, the Seahawks have the ball, it’s second and one and there are 26 seconds left in the game. The Seahawks have one timeout left. They are losing by four points. Head coach Pete Carroll calls a pass play, Russell Wilson passes the ball, and as we know, Malcolm Butler of the Patriots intercept the ball. On the telecast, Cris Collinsworth is so brutal about this call. I mean, it’s the worst call ever. The next day, the pundits pretty much agreed. The disagreement seems to be about whether it was the worst call in Super Bowl history or the worst call in NFL history. So that seems to be where the argument lay. So this is really where we get down to the core of resulting, because we can stop and do this thought experiment. Imagine if Pete Carroll calls the pass play, and the ball is caught in the end zone for a touchdown. Obviously, it doesn’t leave very much time on the clock. So let’s just make the assumption that New England doesn’t come back and score there. Just close your eyes and envision this and imagine what Cris Collinsworth is saying there. He’s probably talking about what a genius Pete Carroll is. He’s not saying, “I can’t believe that was the worst call in Super Bowl history, but it worked out. The next day, the pundits are saying things like, He out Belichicked Belichick.”
So the result informed our opinion of whether it was the correct decision?
This is what got Pete Carroll here. This is why he’s one of the best coaches in the NFL. What I think is interesting about this, is that we know that this is true now, because of the Philly Special call that Doug Peterson made in this year’s Super Bowl. There are two unconventional things that happened there. One is that the Eagles don’t go for the field goal when they are up by three points, which seems surprising. And then obviously, their quarterback Nick Foles, is, surprise, catching the ball in the end zone. After that play, it was all about how bold Doug Peterson is and what a great decision maker he is, and this is why the Eagles won. But you can easily imagine that if the play hadn’tworked out, and the Eagles go on to lose that game, people would be pointing at that play and giving it the Pete Carroll treatment. So this is where we can see resulting. It’s basically this idea that we have this very strong bias to equate the quality of an outcome with the quality of a decision. In other words, we think that we can work backwards from the quality of the outcome to the quality of the decision making. But hopefully you agree with me that whether Malcolm Butler intercepts that ball, or whether the ball is caught for a touchdown, or whether the ball is actually just dropped in the end zone as an incomplete pass, that doesn’t change the quality of the decision itself, except that, cognitively, it does.
In the book, you mention that out of the other 66 passes made from the opponent’s one-yard-line that season, none were intercepted.
Exactly. That’s resulting. And essentially, the reason why we result is that the quality of the decision is actually quite opaque to us. What part of it is luck and what is skill is hard to see. But we do know that in places where there isn’t hidden information, and there isn’t a lot of the influence of luck, that this is actually a really good strategy. If you take a game like chess. If I lose the game to you, it’s because you made better decisions than I did. We can see that as a matter of fact. And if you lose to me, you made worse decisions than I did. So there is this very strong connection that when you take the luck and hidden information out of it, it is actually a reasonable strategy to look at the quality of the outcome because I know the quality of the decision. But that’s not true in anything where there’s any amount of luck involved, and certainly not where any hidden information is involved. When people are looking at outcomes in sports, they treat it like they’re watching chess, which they’re not. They’re watching something that’s more poker, where there are cards face down. That’s the hidden information. And there are also cards that are yet to come that you have no control over. That’s the luck element of how the future turns out. And this is really problematic because, what are we learning from? We tend to be learning from the outcomes that come our way. And so if we’re misinterpreting those outcomes, what’s going to happen is that we’re going to get really weird reactions to the decision quality that are going to distort our ability to learn from those decisions. And even if we’re trying not to, I hope that from that thought experiment, you can see that there’s such a cognitive shadow cast by the outcome that even if you thought you were trying to think about the decision rationally, if you know the outcome, it’s just too big a weight. There’s so much gravitational pull from the outcome, it makes it very, very hard to step back and see the decision quality with clear eyes.
One play like that will probably determine whether or not a coach makes that decision again.
That’s why the NFL has been slow to go for it on fourth down, because the math has been pretty clear on those situations for quite a long time. And the bigger the stage, the more likely you are to choose the status quo and not go for it on fourth down. So coaches are much more willing to go for it on fourth down in a game at the beginning of the season than they are in the playoffs. And it has to do with resulting. The status quo choice is much less likely to get criticism on a bad outcome. That bad outcome is much more likely to get attributed to the luck element in the game because people are doing what everybody else does. There’s already consensus around it. So once there’s consensus around that choice, people are generally in agreement that the choice itself was probably correct. So that’s where the outcome has to be due to luck.
This is where we get into what is known in sports as “conventional wisdom.”
Let’s do a different thought experiment with Pete Carroll. What if he had handed it off to Marshawn Lynch and Marshawn Lynch fails to get into the end zone? So Pete Carroll uses his one time out, they run a second play, they hand it off to Marshawn Lynch again, and he fails to get into the end zone. What are the pundits saying the next day? I think they would say, “New England’s defensive line held. They have a really good defensive line, and even Marshawn Lynch couldn’t get through that line.” Nobody is questioning that decision now, because the decision is really the consensus decision. It’s what everybody was expecting them to do. So if you think about this resulting problem, and the fact that we know that people really connect the quality of the outcome to the quality of the result, if I choose the innovative line, and it doesn’t work out, you’re going to get the Pete Carroll treatment. This is even true, about your personal life.
Let’s say you are going to the movies with your spouse, and there’s a specific time that you need to be there. If you’re late, you’re going to miss the movie. So let’s say that you take the normal route that you always take to go to the movies, and there happens to a lot of traffic or an accident that makes you late for the movie. Nobody’s yelling at each other. There was bad traffic. how could we know? But what if you offered to your spouse, “There’s a shortcut to the theater. I know a shortcut that we can take.” So now, it’s as though you’re passing the ball, and it’s second and one on the goal line. We agree that the traffic is not in your control, or there just happens to be a lot of traffic because there’s a car accident along the way. Because you chose a different way, your spouse is freaking out at you. “Why did you make me take the short cut? I can’t believe that we’re stuck in traffic.” Now you’re in a big fight.
We think a lot in terms of decision making and experience. This person has made this decision a thousand times, so they are an expert. But in your book, you say firmly that repetition isn’t sufficient enough to make you an expert.
It is absolutely not sufficient. First of all, on the poker tour, there are plenty of people that are not experts, and they have had lots of repetition. So I think that that was absolutely the main thing that really made me want to start going into this. I was taught, like all the people in psychology programs are, that learning occurs when you have lots of feedback tied closely in time to decisions and actions. The rat presses the lever, and they get a pellet. So the rat learns to press the lever. They connect that they do an action, and they have an outcome. You haven’t injected any uncertainty into the problem. Now I sit down at the poker table. When I first started playing with people who have been playing forever, I mean I had really good mentors, but obviously, the number of hands they’ve played that they have on me is basically infinite. And people made mistakes that were relatively clear to me, particularly given the mentorship that I had. But here they are, they’ve been playing for 20 years, they’ve had thousands upon thousands upon thousands of hands that they have actually seen in their life. And yet they’re not learning anything about their game.
Is this problem more or less apparent the higher the skill level gets?
There was just a really interesting study that was done, looking at when NBA coaches would change their starting lineup. What they were specifically looking at was what happens after a very close win or a very close loss. They defined a close win or loss as if an NBA team wins by a point or two or loses by a point or two. I think we can agree that those results are relatively the same. And we can assume that given the skill elements, the game is close enough that it may be purely luck whether you won by the point or two or lost by the point or two, because the results are just too close. So what happened is you will see an NBA coach change their lineup after a very close loss, and not after a very close win, even though their reaction to those two situations should be really the same. They weren’t able to embrace the uncertainty of the outcome of the game.
You mention the willingness to embrace uncertainty. Is it harder to do this in sports, because for male athletes, you have to have such unwavering confidence that it’s really hard to say, “I don’t know what will happen.”
Once you accept that outcomes and decisions are only loosely linked, once you accept that given any decision that you make, there’s going to be a set of possible outcomes that are going to occur, and each of those outcomes can occur with some likelihood, but you can’t guarantee which outcomes it will be, all you can do is make the decision that will get you the highest probability of the desired outcome. That’s all you can do. You can never guarantee what the outcome is. Let’s say that I think that 70 percent of the time, a strategy is going to have a good result, and 30 percent of the time the strategy is going to have a bad result. Now I’ve memorialized that, and I’ve thought about that in advance when that bad result happened. I’m less likely to overreact to it, because it was already baked into the decision in advance. It means that you have to have a mindset shift where you’re saying that outcomes are uncertain, so they can’t teach me that much in the short term—at least not when I don’t have enough data. I’m worried about whether my decision is going to get me to the best result, but I have to be ok with not being certain about how it’s actually going to turn out. People really crave certainty. I think that for one thing, it certainty gets conflated with confidence. But it’s also just sort of the way that our brains are wired in an evolutionary way. We want to know if that rustle in the bushes is a lion coming to eat us on not.
How does fan reaction and criticism enter this?
I think about the history of our species. There’s actually a couple of really good examples out there of people who sort of flipped this need for certainty on it’s head. And we appealed to a different thing about human nature, which is tribalism. So sports teams are actually a great example of tribe anyway. With our tribe, it’s us versus them, and we’re much more forgiving about things that happened to our tribe. We’ll defend our tribe against other tribes. There are a few things that tribes provide us with. One of them is to feel special in a relationship to other people. So there are different ways that we can feel special. But one of them is that fans of a team can define themselves as “we’re the people who trust the process. My team is different than the other team, and I am part of this tribe that is special because we think about decisions in a different way.” That’s something that I think Bill Belichick has done really well. When you listen to him do interviews about an unusual play, he’ll just say that it was a really bad result, and then he just went on his merry way, making decisions as he did. And I think Doug Peterson, particularly after the Philly Special, Eagles fans are more like, “Oh, we’ve got this really cool coach that does different things than other coaches do.” So if you can appeal to the tribal nature, if you can start to make that part of the definition of who you are as a team, I think that you can allow much more uncertainty to get wrapped in and start worrying less about what the outcome of a given game or given play is.
Well, it’s funny you should mention “trust the process.” In your very own hometown of Philadelphia, the Sixers had a GM in Sam Hinkie who had been trying to do this. He would tell the fan base, “Hey, we do things differently, and your patience will be rewarded.” But the fans got tired of it. We’re sort of evolutionarily wired to decide things in a certain way. So this mind shift isn’t a natural thing that you’re trying to do, right?
For sure. And what I would say about the Philadelphia problem was that it just took too long, I mean obviously the fans should have trusted the process because its working out now. You can wire this tribal nature to get around this in the short term, but eventually, the results have to play out. So you have to have a balance. The fans are being told to trust the process for six seasons, back to back to back. At some point, they’re going to say, “Well, what process?” What Belichick has going for him is that he’s telling people to trust the process, and the process is actually working, which is helpful. You see this with Doug Peterson as well. It was trust the process, and then the process is actually working. What that means is that you are much more likely to get good results in the future, because you got people to buy in just long enough that you can get the process going. And now, because you’ve had good results, the fans are now going to continue to buy in and trust the process, and that allows you to make mathematically better plays like going for it on fourth and three, when the other team will not and the data is telling you that’s the correct way. So what will happen is the fans will have more tolerance for these bad outcome, non-status quo plays because you’ve had some results from trusting the process.
If there’s one thing that you hope that people will take away from the book, what would it be?
I hope that people come away realizing that saying, “I’m not sure,” not in the sense of “I have no idea,” but in the sense of “There are several possible outcomes, and I don’t know which one will happen. I just know that these are the possibilities,” I think it allows you to be a much better decision maker. It allows you to be much more forgiving of your own outcome and to dig down into your decision process so you can actually start doing some learning and become an expert. And if you happen to be an organization that doesn’t have that, let’s get some other people involved that will do this kind of thinking with you. You have to have cultural support. Who will poke at the status quo with you? It’s very hard to overcome these things on our own.
Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts (Portfolio) is on sale now.