We spend much of our days telling the folks at American Idol how to run their business. This week, it's time to worry about how we run our own.
AI8 marked WhatNotToSing.com's third season online, and our second in which the full database has been open to the public. It's been a long, strange trip for us, marked with many highs and a few lows here and there. The very first email we received upon publishing the database came from a rather irate Kellie Pickler fan who was indignant at his or her hero's low average rating. Fortunately that turned out to be the exception rather than the rule. Well over 95% of the letters we receive are positive, helpful, and thoughtful. In fact, without question, the best part of this job has been the many friends we've made around the Idolsphere: the correspondents, contributors, and constructive critics who keep us busy, honest, thinking, and laughing.
Below is what we at WNTS have planned for the future, both short-term and long-term...if there is a long-term; we'll eventually get to that, too. We'll break this down into four basic categories: The Rating System, The Database, The Encyclopedia, and The Community.
Our approval rating system hasn't changed much since the day we first devised it. Quite honestly, we assumed it would never need to change. It's all based on very basic statistical principles: collect reviews, tally individual rankings and ratings, and make a few minor adjustments (one based on stock indexing) so that its 0 to 100 scale is consistent from episode to episode and from season to season.
What we didn't count on, alas, is that the viewership for American Idol would change so dramatically over the years. As the show grew in popularity and offered more diverse music, it drew a wider and more fragmented audience, making consensus at a very high level far more difficult to attain. We've discussed this topic in detail on several occasions, including one editorial last year and one earlier this season. The bottom line is that the number of "showstoppers" (our arbitrary term for a performance that rates at 90 or higher) has been on a steady and precipitous decline: from 21 in the first four seasons to just nine in the last four, including one – Mad World, at 92 – in all of Season Eight.
In theory, this is not a big deal. The approval ratings within any given season are still 'correct' relative to one another. As long as one doesn't try to compare ratings across seasons, there's no problem. Except, everyone does. Including us. That's sort of the whole point of the website, you know.
After giving this matter a lot of thought and taking into consideration the opinions of many correspondents, we decided the right thing to do was to take action. Comparing performances across seasons will always be something of an apples-to-oranges affair, but at least we should stop weighing the apples in ounces and the oranges in grams. So, sometime later this year, and with great trepidation, we're going to recalculate the approval rating of every American Idol performance to date. This is a one-time-only readjustment (we hope), and we plan to adapt the system so that it will be truly self-adjusting as we move forward.
The gist of what we plan to do is as follows. Every season's approval ratings can be loosely expressed, like any bell curve, as a mean and a standard deviation. Now, be careful: this "standard deviation" is very different than the one you're used to seeing throughout the pages of WNTS. Our usual sigma is the variance of opinions for a given performance, ranging from the very low (the 9 of Kris Allen's original Ain't No Sunshine, indicating that almost everyone liked it) to the insanely high (the legendary 35 of Adam Lambert's no-middle-ground Ring Of Fire.) Forget about those for the moment. What we're concerned about is the more traditional usage of standard deviation; namely, the distribution of approval ratings within one given season.
The table at right shows these distributions for each of the show's first eight chapters. Note that in AI7 and AI8, the "real" standard deviation is lower than in past years, indicating that there were relatively few very high or very low rated performances. Our plan is to "normalize" the ratings so that this spread is roughly the same from one season to the next. When we're through, a performance rated (approximately) 20 points above its season average will represent roughly one standard deviation above average, each and every season. Confused? Write us for details if you'd like.
We're also going to take this opportunity to clean up a couple of loose ends. For example, the ratings for AI3 and AI4 have long been skewed a bit because they were the first two seasons we calculated, so their only cross-season reference points were to one another. We'll finally clean that up; ratings for both seasons should moderate a bit when we're through. Also, there have been performances throughout the years that we know need a second look due to exceptional circumstances, one of which happens to be "Mad World." (For that one, several early reviewers omitted it from their online posts because the show ran way long and their Tivos ran out. We were so delighted to have finally a 90+ rating this year that we failed to consider an important implication: most of the people who skipped that performance probably weren't Lambert's biggest fans, no? If they were, surely they'd have gone online and watched the video clip before they published their review. That might have skewed its rating slightly higher than it would have normally been, a glitch that has been nagging at us all season. We'll look into that matter, among others.)
Finally, as part of the Great Ratings Recalculation, we're going to make a significant nomenclature change here in WNTS-land. What we've been calling Standard Deviation will now be called (tentatively) Variance of Opinion, or VO. It will still be the same as always: the standard deviation of the individual opinions that make up an approval rating. But, VO seems a lot more descriptive, less threatening and more accessible term. (Yes, we know "variance" has a formal Statistics definition too, but shh-h-h. It's still a step forward)
In contrast to the approval rating system, changes to the WNTS database are both commonplace and desirable. We essentially have just two goals for it. One: to capture as much key information about every performance, contestant, episode, etc. as is humanly practical. Two: to allow you the reader to access whatever you want, whenever you want.
We're probably doing a better job of the former than the latter. We continually add fields to the database every year, though not all of them currently appear online. One problem is that making database changes are easy, but updating our web pages to reflect those changes is far more time-consuming. Figuring out how to arrange tables of data so they fit in the limited horizontal width of your browser is always a challenge.
Another feature we know we need to add is an Advanced Search page for every section of the database. For example, you should be able to go into Performance Search and specify that you're only interested in R&B performances by female contestants in Season Six, sorted by standard dev--...whoops, we mean "...by Variance of Opinion." Unfortunately, this feature would take a lot of work, so while we realize it's important it might not make it to your computer by Season Nine. (When it does, we'll make it possible for you to capture the results as an Excel or CSV file for your own research and benefit.)
One feature we're forever asked about are links to performance videos. We'd love to offer video clips on every performance page for our readers' convenience. Alas, they're copyrighted material, and Fremantle and Fox are understandably very aggressive about protecting their copyright. We'd need permission from them to offer clips, and we rather doubt they'd grant it to us. Links to YouTube clips often have a short shelf life, plus they're not legit either. To their credit, the producers made every Season Eight performance available online at AmericanIdol.com, and we're hoping they'll do the same retroactively for past seasons. If so, we'll gladly link to or embed them.
Other new database features either in the works or on the drawing board are:
The latter three, and particularly that scary final one, are ones on which we'd appreciate help from our readers. We certainly have no intention of going through 130-plus results shows to tally the group-sing songs, but we occasionally are asked for that information so it must be worth considering. If you'd like to see a new field in the DB, and you're willing to do the legwork to collect and contribute the information for it, we'd be happy to consider adding it as part of our ongoing Fill In The Blanks project. Just drop us a line with your suggestion.
Finally, sometime this summer, we'll undertake the massive database and scripting changes necessary to support $%^&* duets. (Stop snickering, Mr. Fuller.)
Our goal has been not only to have a database entry for every season, episode, contestant, performance, song, and original artist featured on American Idol, but also a short article on each. Collectively, we call this compendium of Idol wisdom the "encyclopedia". Unfortunately, this part of Project WNTS is lagging far beyond our goals. It takes a special effort for contributors to write articles and email them to us, but most of all, it takes a lot of effort for us to edit them and get them online. As such, we have a rather large backlog of contributions sitting on our hard drive, waiting for a weekend in which we have nothing else to do for the site but edit them. Which might be forever.
Long ago we wrote that we didn't want to open up the encyclopedia to be a wiki, fearing that we'd wind up mostly with biased fluff pieces, hatchet jobs, fanbase warfare, and other nonsense you find on some poorly-moderated forums.
We've since reconsidered. The only way to build out the sheer volume of articles we need (which will eclipse the 3,000 mark next season), and to ensure everything stays up to date and accurate, is to enlist the help of our readers. So, our next big announcement is that the WNTS Encyclopedia will soon be a wiki! At the moment we're standing it up, getting it ready for prime time, and figuring out how we're going to integrate its articles so that they also show up on the appropriate database page. The DB and the wiki will remain separate web applications.
Mind you, this will not be a fully open wiki. Anyone can read it, but to gain editor access, you'll need to apply for an account and possibly submit a sample article (we'll see.) Our goal is to let in anyone who aims to be a serious chronicler of American Idol history, while keeping the, uh, scarier denizens of the Idolsphere at bay. Stay tuned here at WNTS for details this summer, after Camp Should-A-Been concludes.
Last but not least, we have a few things in the works that we hope will let us tie in better with the Idolsphere community.
For the time being, we're going to stick to our peculiar but firm policy of not offering any public forums or comment features. Our apologies to all – It's not that we're uninterested in hearing what folks have to say about the site. Rather, because our goal is to measure what Idol fans think of each performance, we still feel it would be wrong to compete in any material way with the many excellent boards and blogs out there. Also, if we don't count reviews from AmericanIdol.com, because the website and moderators are under the control of the show, we certainly couldn't count opinions posted on our own site. So, setting up an opinion forum here would be counterproductive. We'll reevaluate the situation next summer.
Still, we need to do a better job of getting our story told and letting people connect with us. We have a few ideas in mind. One is really simple: an RSS feed of our announcements and editorials, which we should have set up a long time ago. Another is to offer a presence is the web social network so that our correspondents and researchers can check in and hook up with one another. Our junior editor is working on a Facebook page for WNTS for that very purpose.
Another area we'd like to improve upon is media relations. We have a great working relationship with Ken Barnes at USA Today, and participating in his Idol Meter feature was one of the highlights of the year for us. But, we need to do more. We still frequently encounter major media articles about American Idol in which the reporter got his or her historical facts completely wrong, or who expressed opinions that don't seem to have any basis in reality. (E.g., one journalist, and we won't say whom, wrote in an end-of-season recap that Danny Gokey's elimination was a shocker because he was coming off three straight very strong weeks. Uh, ma'am, dream on about that.) We know that a few media figures use our database constantly...even, ahem, if a few don't credit us...but we'd like to expand upon that as much as possible.
Earning money here at WNTS hasn't been terribly high on our list of priorities thus far, though we of course wouldn't mind if we could improve on that, too. For now at least, our clickthrough revenues cover most of the cost of keeping the site running, so we're OK. Still, the more we make, the more obscene amounts of our free time we can justify building out the site and so offering more to our readers. We've considered a syndicated newsfeed for websites and other media outlets, compiling information after each performance episode and packaging it for republication. Perhaps there are better ways. If you have any ideas of your own, you know where to reach us.
All of these community issues and more are areas in which we are very much open to suggestions from our readers.
This has been an interesting season in a lot of ways. Our readership nearly quadrupled over last year, and that was both gratifying and heartening. Still, the last few months took a lot out of us. We started WhatNotToSing.com with the quixotic idea of helping to improve Idol in the future by divining what worked and what didn't in the past. Everything we do is predicated around the assumption that the competition is fair, above-board, and legitimate, and that any semifinalist has a legitimate chance of winning if they make good decisions (and, of course, they sing well.) That's why we were and remain furious about the sham semifinals and the Wild Card fiasco of Season Eight. The producers betrayed that fundamental trust, and that to us is unforgivable.
Several times this season, we were ready to pack it in, believing that there was no reason to do all this work to support a show that treated its fans as if they were the most gullible imbeciles on earth. We stuck it out largely because many of our correspondents privately urged us to, and because we haven't quite...quite...given up on the promise of this simultaneously brilliant (in concept) and abominable (in execution) show. However, if Season Nine is as rigged and scripted as Season Eight, we can't guarantee we'll make it through another year. Perhaps the boys at VoteForTheWorst.com are correct: the show is beyond hope of salvation, so why not hasten its demise and have some fun at the same time? We'd prefer to think they're mistaken, even if we must admit the preponderance of evidence at the moment is on their side. It's up to Simon Fuller, Simon Cowell, and the rest of the 19E/Fremantle/Fox team to prove them, and us, wrong. We wish we had more faith that they were up to the task.
We're not sure what the future of Idol holds, or even if the show will still be around after 2010. (Our bet: it may or may not be called American Idol, but it will be X-Factor America for all intents and purposes. We hate that notion, but it appears to us that the producers are hellbent on that course.) So, it's impossible to say what the future of WNTS holds, either. Perhaps we're building a long-term research and analysis site for an American institution, as we set out to do, or maybe we're just erecting a fancy online mausoleum for a fad TV show that's nearing the end of its days. Ah well, it's the journey that matters. Here's hoping you stick with us as we see how this movie ends.
- The WNTS.com Team