+ Reply to Thread
Results 1 to 8 of 8

Thread: correcting for early user bias

  1. #1
    Points: 13, Level: 1
    Level completed: 25%, Points required for next Level: 37

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    correcting for early user bias




    Okay, forgive me if this is in the wrong place. Feel free to move it if it isn't. And please forgive me as I know VERY little about statistics and my math skills are very average.

    I'm creating a website where users can write posts and other users can add other people's posts to their "favorites" I also have a "best of" kind of page which shows the "best" posts in order from best to worst.

    I want every post that has at least 10 views to have a chance to make this list, but I know that the only people at first who will see their posts are their friends (who will get a personal notification when their post is published) and they are going to be MUCH more likely to favorite the post than a stranger.

    Obviously, ranking the posts strictly on the total number of "likes" (add to favorites) would say less about how good the post is and more how many people have seen it. So, I'm currently ranking them by the RATIO of likes / views.

    But, the problem remains. Using the ratio favors the posts with fewer likes because the early friends "likes" will have more weight. I need to counter this somehow without hurting a great post's opportunity to get high into the best of list if it deserves it.

    My "shot in the dark" solution is to just subtract 8.5 from the total likes before dividing it by likes to get the ratio. My stats-ignorant thinking is that this would negate the extra likes a post might get early on. But, the more I think about it the dumber I think that probably is and the more I'm convinced that I'm in way over my head here and need to consult some experts like you guys.

    What are your thoughts?

  2. #2
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: correcting for early user bias

    So the person with the worst post on your page will know it, that seems sad.


    Will this process be active and on-going -- will scores get updated every time a person comments?
    Stop cowardice, ban guns!

  3. #3
    Points: 13, Level: 1
    Level completed: 25%, Points required for next Level: 37

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: correcting for early user bias

    Quote Originally Posted by hlsmith View Post
    So the person with the worst post on your page will know it, that seems sad.


    Will this process be active and on-going -- will scores get updated every time a person comments?
    No, no one will know how their post is ranked. The only one who sees this list is me. But anyone who wants to read the best posts can click on the page and see the best posts (the user's own posts will not be included) in order.

    It will refresh the ratio every time someone views a post and/or adds it to their favorites.

  4. #4
    Omega Contributor
    Points: 38,432, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    7,006
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: correcting for early user bias

    I poked a contributor on this forum that loves the rankings of movies. We can see if he has any recommendations. Because as I tried to point out awhile ago, the IMDB Top 250 are not the best 250 movies of all time, but the most popular per the contributors of the page.




    Maybe this will give you some ideas. However many of those lists, actually ask users to rank movies and I am guessing you do not solicit feedback in the same way.
    Stop cowardice, ban guns!

  5. #5
    TS Contributor
    Points: 12,501, Level: 73
    Level completed: 13%, Points required for next Level: 349

    Posts
    951
    Thanks
    0
    Thanked 103 Times in 100 Posts

    Re: correcting for early user bias

    In spite of your stated stats/math mediocrity, you're making good thinking. At least to start, which is the hardest thing.

    Do you know who are friends on your website? If so, maybe you could just make a ranking based on how non-friends rate the post.
    All things are known because we want to believe in them.

  6. #6
    Points: 13, Level: 1
    Level completed: 25%, Points required for next Level: 37

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: correcting for early user bias

    Quote Originally Posted by Mean Joe View Post
    In spite of your stated stats/math mediocrity, you're making good thinking. At least to start, which is the hardest thing.

    Do you know who are friends on your website? If so, maybe you could just make a ranking based on how non-friends rate the post.
    Thanks for your kind words. Unfortunately, there's really only two ways a person would ever see a post. either they are friends with the poster or they see it on the "best of" page.

    So, if the likes from the friends were invalid the post would never be seen by anyone and no post would ever make the best of page, if that makes sense.

    I changed 8.5 to 2.5 which keeps a pretty good weight on "more views". I'd like anything from 50 views on to be fairly the same, but the more the views are under 50 the less weight is on the result (sorry I don't know the correct terminology).

    So, if you take a ratio of 50% and subtract 2.5 from the total likes you get the following:

    100 = .475
    50 = .45
    40 = .44
    30 = .42
    20 = .375

    I have no idea if this would be the best number or not. I'm pretty clueless as to the way to go about this lol

  7. #7
    Points: 13, Level: 1
    Level completed: 25%, Points required for next Level: 37

    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: correcting for early user bias

    total views = likes = ratio

    100 = 50 = 48

    30 = 18 = 52
    30 = 17 = 48
    30 = 16 = 45

    20 = 13 = 53
    20 = 12 = 48

    Using the current way, a post with 30 views and one with 20 views would both need 3 more likes than a post with 100 views in order to "beat it". Not sure if that's enough. A post with only 20 views but 13 likes will beat a post with 100 views and 50 likes? Hmm

  8. #8
    Pirate
    Points: 15,159, Level: 79
    Level completed: 62%, Points required for next Level: 191
    victorxstc's Avatar
    Posts
    875
    Thanks
    229
    Thanked 332 Times in 297 Posts

    Re: correcting for early user bias


    Quote Originally Posted by hlsmith View Post
    I poked a contributor on this forum that loves the rankings of movies. We can see if he has any recommendations. Because as I tried to point out awhile ago, the IMDB Top 250 are not the best 250 movies of all time, but the most popular per the contributors of the page.

    Maybe this will give you some ideas. However many of those lists, actually ask users to rank movies and I am guessing you do not solicit feedback in the same way.
    I for one can't really see what is the reference for determining good and bad. So, how can we say most popular movies are not the best movies? Maybe a solution is to clarify that by "the best movies", we mean those of higher artistic value etc. But to many people, many of such good films are just lame or stupid.

    But neither of groups can invalidate the other. So I guess if IMDB calls the list "the most popular 250", and clearly separates them from "good / best" movies, many will question its way of determining the best. Some might be offended when they see the movie they thought was the best movie ever is not even in the list of good movies (my friend used to think Davinci Code was the best movie ever!)

    Regarding the question of the OP, I guess (s)he can weighten the favoriting of different people. For example, if a friend of userX favorites his post, the userX gets 1 point. If a friend of friend of the userX favorites the post of userX, then userX gets 4 points. If a total stranger (who is beyond 3 levels [friend of friend of friend of userX]) favorites his post, userX gets 10 point.

    If userX is favorited by a same-sex person (and both of them are straight), the weight of the score should be slightly greater. Because opposite sexes tend to favorite each other more easily (and thus, their favoriting might have a factor of flirting, and thus less valuable compared to a like one receives only because of the post itself)...

    So you have a system of weightening that increases / decreases the base (raw) score of each like a user receives.

    Another factor for weightening is the number of friends of one user. For example a user has 5 friends. If all of them like (favorite) his post, he will have a score of 5 within a couple of days. Another person has 1000 friends. So if you give the same score to each like of a friend, this second user will have about 400 scores within a very few hours. Therefore, you should lower the power (weight) of each favoriting, as the number of friends increase.

    (2)

    I think you should also increase the threshold to something like 20 or 30 (instead of 10). This ensures you that the person really deserved being in the hall of fame.

    (3)

    How to chose the right weights? Trial and error. All social networking websites adjust their scoring algorithms dynamically. They have sophisticated formulas and algorithms for weight adjustment, which is beyond me. Perhaps you will need to hire experts for that purpose, if you want to do it on an international level. You can change weight and see what happens, but when the number of users gets too many, the weightening needs to be adjusted by computer itself. You can find more about it by searching for machine learning.
    "victor is the reviewer from hell" -Jake
    "victor is a machine! a publication machine!" -Vinux

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats