The following excerpts are taken from the book: The Drunkard’s Walk by Leonard Mlodinow. Copyright © 2008 by Leonard Mlodinow. All rights reserved. Published by arrangement with Pantheon Books, a division of Random House, Inc. Read more about the author and the story behind his book here.

From “Peering through the Eyepiece of Randomness”

“Nobody knows anything,” the writer William Goldman famously remarked about Hollywood. Goldman was referring to the apparently inscrutable process by which a movie becomes a blockbuster or a bomb or something in between, often in defiance of all expectations. The film industry’s movers and shakers live and die by the numbers, but as Mlodinow explains in this chapter, the most meaningful numbers may have little or nothing to do with the box office.

We habitually underestimate the effects of randomness. Our stockbroker recommends that we invest in the Latin American mutual fund that “beat the pants off the domestic funds” five years running. Our doctor attributes that increase in our triglycerides to our new habit of enjoying a Hostess Ding Dong with milk every morning after dutifully feeding the kids a breakfast of mangoes and nonfat yogurt. We may or may not take our stockbroker’s or doctor’s advice, but few of us question whether he or she has enough data to give it. In the political world, the economic world, the business world—even when careers and millions of dollars are at stake—chance events are often conspicuously misinterpreted as accomplishments or failures.

Hollywood provides a nice illustration. Are the rewards (and punishments) of the Hollywood game deserved, or does luck play a far more important role in box office success (and failure) than people imagine? We all understand that genius doesn’t guarantee success, but it’s seductive to assume that success must come from genius. Yet the idea that no one can know in advance whether a film will hit or miss has been an uncomfortable suspicion in Hollywood at least since the novelist and screenwriter William Goldman enunciated it in his classic 1983 book Adventures in the Screen Trade. In that book, Goldman quoted the former studio executive David Picker as saying, “If I had said yes to all the projects I turned down, and no to all the other ones I took, it would have worked out about the same.”

That’s not to say that a jittery homemade horror video could become a hit just as easily as, say, Exorcist: The Beginning, which cost an estimated $80 million. Well, actually, that is what happened some years back with The Blair Witch Project: it cost the filmmakers a mere $60,000 but brought in $140 million in domestic box office revenue—more than three times the business of Exorcist. Still, that’s not what Goldman was saying. He was referring only to professionally made Hollywood films with production values good enough to land the film a respectable distributor. And Goldman didn’t deny that there are reasons for a film’s box office performance. But he did say that those reasons are so complex and the path from green light to opening weekend so vulnerable to unforeseeable and uncontrollable influences that educated guesses about an unmade film’s potential aren’t much better than flips of a coin.

Examples of Hollywood’s unpredictability are easy to find. Movie buffs will remember the great expectations the studios had for the megaflops Ishtar (Warren Beatty + Dustin Hoffman + a $55 million budget = $14 million in box office revenue) and Last Action Hero (Arnold Schwarzenegger + $85 million = $50 million). On the other hand, you might recall the grave doubts that executives at Universal Studios had about the young director George Lucas’s film American Graffiti, shot for less than $1 million. Despite their skepticism, it took in $115 million, but still that didn’t stop them from having even graver doubts about Lucas’s next idea. He called the story Adventures of Luke Starkiller as taken from “The Journal of the Whills.” Universal called it unproducible. Ultimately 20th Century Fox made the film, but the studio’s faith in the project went only so far: it paid Lucas just $200,000 to write and direct it; in exchange, Lucas received the sequel and merchandising rights. In the end, Star Wars took in $461 million on a budget of $13 million, and Lucas had himself an empire.

Given the fact that green light decisions are made years before a film is completed and films are subject to many unpredictable factors that arise during those years of production and marketing, not to mention the inscrutable tastes of the audience, Goldman’s theory doesn’t seem at all far-fetched. (It is also one that is supported by much recent economic research.) Despite all this, studio executives are not judged by the bread-and-butter management skills that are as essential to the head of the United States Steel Corporation as they are to the head of Paramount Pictures. Instead, they are judged by their ability to pick hits. If Goldman is right, that ability is mere illusion, and in spite of his or her swagger no executive is worth that $25 million contract.

Deciding just how much of an outcome is due to skill and how much to luck is not a no-brainer. Random events often come like the raisins in a box of cereal—in groups, streaks, and clusters. And although Fortune is fair in potentialities, she is not fair in outcomes. That means that if each of 10 Hollywood executives tosses 10 coins, although each has an equal chance of being the winner or the loser, in the end there will be winners and losers. In this example, the chances are 2 out of 3 that at least 1 of the executives will score 8 or more heads or tails.

Imagine that George Lucas makes a new Star Wars film and in one test market decides to perform a crazy experiment. He releases the identical film under two titles: Star Wars: Episode A and Star Wars: Episode B. Each film has its own marketing campaign and distribution schedule, with the corresponding details identical except that the trailers and ads for one film say Episode A and those for the other, Episode B. Now we make a contest out of it. Which film will be more popular? Say we look at the first 20,000 moviegoers and record the film they choose to see (ignoring those die-hard fans who will go to both and then insist there were subtle but meaningful differences between the two). Since the films and their marketing campaigns are identical, we can mathematically model the game this way: Imagine lining up all the viewers in a row and flipping a coin for each viewer in turn. If the coin lands heads up, he or she sees Episode A; if the coin lands tails up, it’s Episode B. Because the coin has an equal chance of coming up either way, you might think that in this experimental box office war each film should be in the lead about half the time. But the mathematics of randomness says otherwise: the most probable number of changes in the lead is 0, and it is 88 times more probable that one of the two films will lead through all 20,000 customers than it is that, say, the lead continuously seesaws. The lesson is not that there is no difference between films but that some films will do better than others even if all the films are identical.

Identical films should do nearly identical business, right? Not in this galaxy, says Mlodinow.

Such issues are not discussed in corporate boardrooms, in Hollywood, or elsewhere, and so the typical patterns of randomness—apparent hot or cold streaks or the bunching of data into clusters—are routinely misinterpreted and, worse, acted on as if they represented a new trend.

One of the most high profile examples of anointment and regicide in modern Hollywood was the case of Sherry Lansing, who ran Paramount with great success for many years. Under Lansing, Paramount won Best Picture awards for Forrest Gump, Braveheart, and Titanic and posted its two highest-grossing years ever. Then Lansing’s reputation suddenly plunged, and she was dumped after Paramount experienced, as Variety put it, “a long stretch of underperformance at the box office.”

In mathematical terms there is both a short and a long explanation for Lansing’s fate. First, the short answer. Look at this series of percentages: 11.4, 10.6, 11.3, 7.4, 7.1, 6.7. Notice something? Lansing’s boss, Sumner Redstone, did too, and for him the trend was significant, for those six numbers represented the market share of Paramount’s Motion Picture Group for the final six years of Lansing’s tenure. The trend caused BusinessWeek to speculate that Lansing “may simply no longer have Hollywood’s hot hand.” Soon Lansing announced she was leaving, and a few months later a talent manager named Brad Grey was brought on board.

How can a surefire genius lead a company through seven great years and then fail practically overnight? There were plenty of theories explaining Lansing’s early success. While Paramount was doing well, Lansing was praised for making it one of Hollywood’s best-run studios and for her knack for turning conventional stories into $100 million hits. When her fortune changed, the revisionists took over. Her penchant for making successful remakes and sequels became a drawback. Most damning of all, perhaps, was the notion that her failure was due to her “middle-of-the-road tastes.” She was now blamed for green-lighting such box office dogs as Timeline and Lara Croft Tomb Raider: The Cradle of Life. Suddenly the conventional wisdom was that Lansing was risk averse, old-fashioned, and out of touch with the trends. But can she really be blamed for thinking that a Michael Crichton bestseller would be promising movie fodder? And where were all the Lara Croft critics when the first Tomb Raider film took in $131 million in box office revenue?

Even if the theories of Lansing’s shortcomings were plausible, consider how abruptly her demise occurred. Did she become risk averse and out of touch overnight? Because Paramount’s market share plunged that suddenly. One year Lansing was flying high; the next she was a punch line for late-night comedians. Her change of fortune might have been understandable if, like others in Hollywood, she had become depressed over a nasty divorce proceeding, had been charged with embezzlement, or had joined a religious cult. That was not the case. And she certainly hadn’t sustained any damage to her cerebral cortex. The only evidence of Lansing’s newly developed failings that her critics could offer was, in fact, her newly developed failings.

In hindsight it is clear that Lansing was fired because of the industry’s misunderstanding of randomness and not because of her flawed decision making: Paramount’s films for the following year were already in the pipeline when Lansing left the company. So if we want to know roughly how Lansing would have done in some parallel universe in which she remained in her job, all we need to do is look at the data in the year following her departure. With such films as War of the Worlds and The Longest Yard, Paramount had its best summer in a decade and saw its market share rebound to nearly 10 percent.

That isn’t merely ironic—it’s again that aspect of randomness called regression toward the mean. A Variety headline on the subject read, “Parting Gifts: Old Regime’s Pics Fuel Paramount Rebound,” but one can’t help but think that had Viacom (Paramount’s parent company) had more patience, the headline might have read, “Banner Year Puts Paramount and Lansing’s Career Back on Track.”

Sherry Lansing had good luck at the beginning and bad luck at the end, but it could have been worse. She could have had her bad luck at the beginning. That’s what happened to a Columbia Pictures chief named Mark Canton. Described as box office savvy and enthusiastic shortly after he was hired, he was fired after his first few years produced disappointing box office results. Criticized by one unnamed colleague for being “incapable of distinguishing the winners from the losers” and by another for being “too busy cheerleading,” this disgraced man left in the pipeline when he departed such films as Men in Black ($589 million in worldwide box office revenue), Air Force One ($315 million), The Fifth Element ($264 million), Jerry Maguire ($274 million), and Anaconda ($137 million). As Variety put it, Canton’s legacy pictures “hit and hit big.”

Well, that’s Hollywood, a town where Michael Ovitz works as Disney president for fifteen months and then leaves with a $140 million severance package and where the studio head David Begelman is fired by Columbia Pictures for forgery and embezzlement and then is hired a few years later as CEO of MGM. But as we’ll see in the following chapters, the same sort of misjudgments that plague Hollywood also plague people’s perceptions in all realms of life.

From “The Law of Truths and Half-Truths”

Why do we inevitably seem to end up in the slowest line in the supermarket? Here, Mlodinow argues that there’s nothing inevitable about it.

Which is greater: the number of six-letter English words having n as their fifth letter or the number of six-letter English words ending in ing? Most people choose the group of words ending in ing. Why? Because words ending in ing are easier to think of than generic six-letter words having n as their fifth letter. But you don’t have to survey the Oxford English Dictionary—or even know how to count—to prove that guess wrong: the group of six-letter words having n as their fifth letter includes all six-letter words ending in ing. Psychologists call that type of mistake the availability bias because in reconstructing the past, we give unwarranted importance to memories that are most vivid and hence most available for retrieval.

Looks like these shoppers should have plenty of time to assess the odds of ending up in the slow lane at the supermarket.

The nasty thing about the availability bias is that it insidiously distorts our view of the world by distorting our perception of past events and our environment. For example, people tend to overestimate the fraction of homeless people who are mentally ill because when they encounter a homeless person who is not behaving oddly, they don’t take notice and tell all their friends about that unremarkable homeless person they ran into. But when they encounter a homeless person stomping down the street and waving his arms at an imaginary companion while singing “When the Saints Go Marching In,” they do tend to remember the incident. How probable is it that of the five lines at the grocery-store checkout you will choose the one that takes the longest? Unless you’ve been cursed by a practitioner of the black arts, the answer is around 1 in 5. So why, when you look back, do you get the feeling you have a supernatural knack for choosing the longest line? Because you have more important things to focus on when things go right, but it makes an impression when the lady in front of you with a single item in her cart decides to argue about why her chicken is priced at $1.50 a pound when she is certain the sign at the meat counter said $1.49.

From “Measurement and the Law of Errors”

Wherein the physicist turned writer confronts the vagaries of his chosen trade.

One day not long ago my son Alexei came home and announced the grade on his most recent English essay. He had received a 93. Under normal circumstances I would have congratulated him on earning an A. And since it was a low A and I know him to be capable of better, I would have added that this grade was evidence that if he put in a little effort, he could score even higher next time. But these were not normal circumstances, and in this case I considered the grade of 93 to be a shocking underestimation of the quality of the essay. At this point you might think that the previous few sentences tell you more about me than about Alexei. If so, you’re right on target. In fact, the above episode is entirely about me, for it was I who wrote Alexei’s essay.

Okay, shame on me. In my defense I should point out that I would normally no sooner write Alexei’s essays than take a foot to the chin for him in his kung fu class. But Alexei had come to me for a critique of his work and as usual presented his request late on the night before the paper was due. I told him I’d get back to him. Proceeding to read it on the computer, I first made a couple of minor changes, nothing worth bothering to note. Then, being a relentless rewriter, I gradually found myself sucked in, rearranging this and rewriting that, and before I finished, not only had he fallen asleep, but I had made the essay my own. The next morning, sheepishly admitting that I had neglected to perform a “save as” on the original, I told him to just go ahead and turn in my version.

He handed me the graded paper with a few words of encouragement. “Not bad,” he told me. “A 93 is really more of an A− than an A, but it was late and I’m sure if you were more awake, you would have done better.” I was not happy. First of all, it is unpleasant when a fifteen-year-old says the very words to you that you have previously said to him, and nevertheless you find his words inane. But beyond that, how could my material—the work of a person whom my mother, at least, thinks of as a professional writer—not make the grade in a high school English class? Apparently I am not alone. Since then I have been told of another writer who had a similar experience, except his daughter received a B. Apparently the writer, with a PhD in English, writes well enough for Rolling Stone, Esquire, and The New York Times but not for English 101. Alexei tried to comfort me with another story: two of his friends, he said, once turned in identical essays. He thought that was stupid and they’d both be suspended, but not only did the overworked teacher not notice, she gave one of the essays a 90 (an A) and the other a 79 (a C). (Sounds odd unless, like me, you’ve had the experience of staying up all night grading a tall stack of papers with Star Trek reruns playing in the background to break the monotony.)

Numbers always seem to carry the weight of authority. The thinking, at least
subliminally, goes like this: if a teacher awards grades on a 100-point scale, those tiny distinctions must really mean something. But if ten publishers could deem the manuscript for the first Harry Potter book unworthy of publication, how could poor Mrs. Finnegan (not her real name) distinguish so finely between essays as to award one a 92 and another a 93? If we accept that the quality of an essay is somehow definable, we must still recognize that a grade is not a description of an essay’s degree of quality but rather a measurement of it, and one of the most important ways randomness affects us is through its influence on measurement. In the case of the essay the measurement apparatus was the teacher, and a teacher’s assessment, like any measurement, is susceptible to random variance and error.

“Numbers always seem to carry the weight of authority,” says Mlodinow. But do they?