Someone once tried to convince me that the IMDb Top 250 could not be an inaccurate portrayal of what Real People thought were the best movies. After all, if more people were willing to vote for a movie, didn’t that by definition make it better? And if that was just because more people had seen it, didn’t that by definition mean that it was more popular with Real People? And there’s something to be said for that argument, I guess, until you really look at what the Top 250 is.
They claim that the rankings are weighted somehow to prevent campaigns. It’s definitely true that they have a minimum number of votes before a movie will be considered, I suppose to prevent the list from being topped with movies that seven people have seen and given a ten to; current and I guess eternal list-topper The Shawshank Redemption is at 9.2. I suspect that the hefty 1.6% of voters who have given it one star—more than have given it 2, 3, 4, or 5 put together—are factored into the weighted rankings? IMDb won’t reveal how they do it, because they say that doing so would make it easier for people to beat their algorithm.
That said, I’ve definitely spotted campaigns. Some years ago, I noticed that the ratings of certain movies among women I believe it was 44 and up were heavily skewed from every other demographic. These were usually movies that were highly rated, often movies that fell just out of the Top 250. It seemed possible that, without these skewed ratings, the movies would have made it in. Similarly, there would be movies that were rated much higher in that demographic than in any other. And while it’s possible it just appealed to that demographic more, the difference between one age group of women and another usually isn’t that drastic.
Harder to deal with, though, are the Film Bros. I do not believe this to be a deliberate campaign. It’s just that, yes, there are certain movies that are going to get certain ratings regardless of their quality because of how they’re perceived in certain circles. There are many movies in this group that I go along with that consensus on, for good and bad. (I’ve seen roughly 210 of the current list.) Will Logan still be perceived in the same way in ten years? Who knows? What’s certain, though, is that it’s unlikely to have as many ratings in ten years as it did in the first six months after its release, and it may never have enough to balance out the high ratings from the people who were likely to love it—the people who were going to go see it in the first six months.
Because that’s just it, isn’t it? Unless you’re a film critic, you mostly see movies you like. Sure, there are movies that have disappointed you, but I’d suggest that one of the other problems with the list is that the bottom half of the bell curve is chopped off. It doesn’t matter how much my mother, for example, would or wouldn’t like Mad Max: Fury Road, because she’s never going to see it. This is a list without her opinion, and it always will be.
What’s more, my mother won’t vote. She doesn’t do much on the internet in the first place; I can barely get her to exchange e-mails. So it won’t have her opinion based on that. The list is self-selecting, and you don’t have to know much about how polls work to know the problem with self-selecting polls. It doesn’t matter that we’re looking at the entire width and breadth of film, ones my mother likes or ones my mother hates or ones my mother just doesn’t care about. My mother’s voice isn’t part of it, nor is that of enormous numbers of other people. That’s both people who don’t care about the internet much and people who don’t care about movies much.
Right now, to my surprise, 12 Angry Men is at number five on the list. But it’s worth noting that its position is less stable than either of the movies on either side of it. You see, as of right now (about eleven PM on 24 July 2018), 12 Angry Men has 551,068 votes. Right above it is The Dark Knight with 1,945,711 votes; right below it is Schindler’s List with 1,018,947 votes. It will take fewer votes to shift 12 Angry Men. (Shawshank is never moving because it has more votes than any other movie I’ve looked at!) Most of the movies with more than a hundred thousand votes are the aforementioned Film Bro movies.
They’re also mostly movies from within my lifetime. It’s part of why I’m surprised about 12 Angry Men. By my count, roughly a third of the movies are from before 1976. An even fifth of the list is from the last ten years. (Be aware these counts are me just scrolling and counting, so I may be off a bit.) That is recency bias. Are those movies better? No. More of the people who vote have seen them.
Which also leads into another problem; very few of these movies are anything but American. If you compare the English-language movies list and the regular list, it appears to be a difference of forty-five movies, and there aren’t a whole lot of British or Canadian movies on the list. I’m given to understand that Bollywood films were specifically exempted, and indeed there’s a separate list of the top-rated Indian movies—and the top-rated movie on that list isn’t on the main list. So i don’t know what that’s all about.
It’s a mess. That’s unfortunate, because a truly populist film rating system would be a great thing. However, I genuinely believe that to be impossible, for several reasons. The biggest two are recency bias and self-selection, and I’m just not sure how you get over that. I think it’s obvious that IMDb just doesn’t try.
This is another one of those articles I put a little actual research into; show that matters by supporting my Patreon!