Jonathan Buchanan recently joined Ohloh and gave me a Kudo and I noticed something fascinating: my KudoRank jumped about 88% all at once. Needless to say, that struck me as a little … odd.
So I checked their documentation, and checked it against the situation. Their rules line up as follows:
The more Kudos you receive, the higher your KudoRank becomes.
This was my first Kudo, so clearly this didn’t come into play.
Your influence increases as your KudoRank improves.
This would refer to Jonathan’s KudoRank, since it would be his opinion that’s affecting my rank. He had no Kudos initially, but I gave him one fairly quickly, so that probably affected the importance of his Kudo for me. Still, I can’t imagine one Kudo would have much impact.
Giving away more Kudos dilutes your opinion.
This I can see. Given that I’m Jonathan’s only Kudo so far, his opinion obviously hasn’t been diluted.
Stacks improve KudoRank.
I admit, I do have a few applications listed on Ohloh, but Jonathan is the only one other than myself who has stacked any of them, and he only did so for one of them. Again, I can’t see this doing much.
Now, don’t get me wrong, I actually like the guidelines they’ve listed above. They obviously put a lot of thought into trying to approximate, using code, the human dynamics of personal recomendations. The opinions of important people generally are taken more seriously than those of the lesser-known, and someone who’s stringy with praise will find that praise to me more influential than that of others. But they neglected to take into account something else, and it illustrates a bigger problem.
This post confirms my suspicions, showing that it was indeed the third rule above that caused the sudden increase. Because Jonathan hadn’t thrown Kudos out all over the place, his opinion is considered of highest importance. The problem is, his account was brand new, so he hasn’t yet had a chance to spread Kudos around. It’d be like a new guy at the office pitching his fist idea, and the CEO immediately running with it, because he didn’t have a record of pitching a billion nonsense ideas.
The trick here is to take time into account as well. The longer someone is a member, the more important their lack of Kudo activity should become. That way, users have to prove that they’re careful about their opinions before those opinions are given added weight. One simple — yet somewhat naïve — approach would be to give everybody a base weight when they join, and increase that weight each day (hour, week, whatever) that goes by without any new kudos. This would reward those who are consisently careful, without giving undue weight to just joined and don’t have a proven track record.
Now, I know I mentioned Jonathan several times here, but it has nothing to do with him at all, really; he just happened to give me a Kudo, which exposed the problem. I regard his opinion highly, but I wouldn’t expect Ohloh to do the same; at least, not yet.
The bigger problem at work here, though, is what happens when people try to use code to approximate human behavior. It’s easy to look at a few things and assume it’ll do a decent job, but it’s really hard to get it right. I won’t pretend to have all the answers, but it seems to me like Ohloh missed a very simple point in their KudoRank calculations. They seem relatively open to suggestions though, so maybe my comment on that thread ont he topic will come to something.
This area is something I find very interesting, and I’ll probably be writing about it in the future.