Ever had a pinball-related question here before?

lemon curry · 11-03-2009, 09:48 AM

Hi, stats people, I've spent enough time futilely Googling, so I thought I'd look for a stats forum to ask the experts. Oh, look, here's one!

I have a question related to the compilation of user ratings, specifically on pinball games. The website ipdb.com has a top 300 pinballs list, where they sort by user ratings. Each member gets to rate as many machines as they want. I realize such ratings are highly subjective, but it bothers me the way their statistics are compiled.

Long story short, I've noticed that games produced by "less popular" manufacturers have fewer ratings, meaning each rating that is entered on these games can affect their overall ratings disproportionately.

The average game rating is in the ballpark of 7.5/10, so when one of these "weak," "unpopular" games (by virtue of having few ratings) sneaks up too high in the list, all it takes is one hater to give it a low score like 0 or 1 to push it wayyyy back down.

The site even displays for you a chart of the ratings on each game. Some of these bell curves are wacky to say the least.

What I'm trying not to ramble on about is, is there a term for what I think they should do, which is throwing out or adjusting individual ratings that lie outside two or three standard deviations? Are there any resources on the web that describe valid methods of doing so?

Thanks!

AtlasFrysmith · 11-03-2009, 12:12 PM

The problem is that there aren't enough observations for some of the machines. Excluding observations won't help, and you'll bias your results. A better solution might be to not publish the composite ratings until a certain number of individual ratings are received.

More generally, it's probably not a good idea to exclude perceived outliers until you really know what the true distribution is.

11-03-2009, 09:48 AM	#1
lemon curry Junior Member Join Date: Nov 2009 Posts: 1 Thanks: 0 Thanked 0 Times in 0 Posts	Ever had a pinball-related question here before? Hi, stats people, I've spent enough time futilely Googling, so I thought I'd look for a stats forum to ask the experts. Oh, look, here's one! I have a question related to the compilation of user ratings, specifically on pinball games. The website ipdb.com has a top 300 pinballs list, where they sort by user ratings. Each member gets to rate as many machines as they want. I realize such ratings are highly subjective, but it bothers me the way their statistics are compiled. Long story short, I've noticed that games produced by "less popular" manufacturers have fewer ratings, meaning each rating that is entered on these games can affect their overall ratings disproportionately. The average game rating is in the ballpark of 7.5/10, so when one of these "weak," "unpopular" games (by virtue of having few ratings) sneaks up too high in the list, all it takes is one hater to give it a low score like 0 or 1 to push it wayyyy back down. The site even displays for you a chart of the ratings on each game. Some of these bell curves are wacky to say the least. What I'm trying not to ramble on about is, is there a term for what I think they should do, which is throwing out or adjusting individual ratings that lie outside two or three standard deviations? Are there any resources on the web that describe valid methods of doing so? Thanks!

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

11-03-2009, 12:12 PM	#2
AtlasFrysmith Member Join Date: Dec 2008 Location: Washington, DC Posts: 61 Thanks: 2 Thanked 12 Times in 12 Posts	The problem is that there aren't enough observations for some of the machines. Excluding observations won't help, and you'll bias your results. A better solution might be to not publish the composite ratings until a certain number of individual ratings are received. More generally, it's probably not a good idea to exclude perceived outliers until you really know what the true distribution is.