Home « Forum « Author Hangout

Forum: Author Hangout

Problem with the scoring system?

HeatAndChills

Just by eyeballing the graph of the scores for my story, I had a feeling that something wasn't quite right. So I manually added up the scores and divided by the number of votes to get the average and guess what? My score is REALLY 7.4, not the 6.09 that the site claims I have!

Even stranger, my overall score when I had 29 votes was 6.11. When I jumped up to 30 votes, my score decreased to 6.09. This would have to mean that my latest vote was either a 6, or lower, for it to decrease the value. (The math suggests it is either a 5 or a 6).

The problem is that I remember the state of my score graph from when there was just 29 votes, and the amount of votes rating 6 or below hasn't changed! I had 1 vote for 4, 3 votes for 5, and 2 votes for 6 then; and I have 1 vote for 4, 3 votes for 5, and 2 votes for 6 now!

Clearly there would seem to be something off about the scoring system on this site.

Switch Blayde
Updated:

@HeatAndChills

There's not a problem. That's how it works.

There's a blog on SOL that explains it. The score seen is not the raw score. It's adjusted based on other scores during the same period. I was testing it with Lazeez once and a 10 lowered my score. Also, the top and bottom 5% of the scores are dropped. That was done at the request of authors here.

If you promise not to complain about the scoring system, Lazeez will set up your account so that you can see your raw scores.

Replies:   richardshagrin
Chris Podhola

@HeatAndChills

I don't know exactly why they do it this way and to be honest, I never quite fully understood the rating system here. For some reason, an 8 here on Sol, does not equal 8. It equals somewhere around 6.2 (or close to it). I don't know why, but that is why your numbers will never add up to what you think they should. Suffice it to know that all authors play under the same rules, so when you gauge your scores against another author's, it is still a fair reference.

Chris Podhola

@Chris Podhola

Ah. Switch's explanation makes more sense than mine. I didn't know the dropping top and bottom scores thing, but to be honest, I think that screws you more than it helps you.

Replies:   Switch Blayde
Switch Blayde

@Chris Podhola

so when you gauge your scores against another author's, it is still a fair reference.


I'd add to the end of that:

...for the same size story with similar story codes.

Switch Blayde

@Chris Podhola

I didn't know the dropping top and bottom scores thing, but to be honest, I think that screws you more than it helps you


It was done to minimize the impact of the crazies who constantly give 1s.

Replies:   Chris Podhola
Chris Podhola

@Switch Blayde

It was done to minimize the impact of the crazies who constantly give 1s.


Well, math isn't and never was my strong point, but whether that helps, seems like it would depend on how many 1's versus tens you get. If you have 100 ratings total and you have 5 1's or 2's to erase, it might help you ... maybe. (depending on what your other scores are). But if you only have 1 or 2 1's, I think you would fair worse by eliminating five 10's than you would had the top and bottom five percent not been eliminated.

Again. I am no mathematician, so I could be wrong, but that's my suspicion.

Replies:   Switch Blayde
richardshagrin

@Switch Blayde

As the computer guys say, its not a bug, its a feature. My theory, (I've said it before, here it is again) Scores are for readers. Its ok for writers to look at them, but don't take them seriously. The number of downloads is for authors. More is better. Some authors have taken scores so seriously they have asked not to have votes taken on their stories, or particular ones. What that does is make it harder for readers to decide to read that (or those) stories. Given a choice between reading a story with a reasonable score (say six or higher) or a story with no score at all, if the description of the story and all the tags and with little or no information about liking a particular author's stories before, and no reviews to consider, a lot of people, including me would chose to read a story with a score.

Lets talk about another kind of voting. The Halloween story contest. Some story and its writer are going to win. The rest are going to be "inferior" if the author decides to take it that way. I don't think of it that way, but the authors who get upset because their story scores are less than tens (or eights, or whatever they feel is adequate) are like the "losers" in the contest. Their stories are likely to be good, and will have scores to help show that. Even a six is an average story, well worth reading if the plot description and tags appeal to a given reader.

Don't get mad. Write and post something else. If lots of readers read it, it will be a success.

Replies:   Chris Podhola
Chris Podhola

@richardshagrin

Don't get mad. Write and post something else. If lots of readers read it, it will be a success.


Definitely agree. Don't get mad. For me, when I was posting here on a regular basis, I simply looked at whatever the scores was and moved on. Then when I posted a new story or series, compared the two scores, hoping for a little bit of an improvement. It was a motivational tool to me. Eventually, as I wrote more and more stories and worked harder at trying to make them better, I began to see increases in average score.

I think your statement is true. Don't take them too seriously, but I don't think it's wrong to use them as a guide much in the same way that you suggest readers can or do.

Switch Blayde

@Chris Podhola

seems like it would depend on how many 1's versus tens you get.


Not just 1s and 10s. Top and bottom 5%.

Let's say you get scores of all 7s, 8s, 9s, and 10s. Then you get the few 1s. I'd give up some 10s to get rid of the 1s. The difference between a 10 and a 9 isn't that much. The difference between a 1 and a 7 is.

Replies:   Chris Podhola
Ernest Bywater

@HeatAndChills

it's explained in the Help page - link and copy below. Essentially it evens out identified periods and normalises the scores to remove the effect of hate votes and fanboy votes.

http://storiesonline.net/help.php?cmd=getAnswer&id=8

How is the score calculated? And how does scoring work in general?

It may seem like a score should be a straight forward affair with all the votes tallied and divided by their count to arrive at the score. Unfortunately, that would only work for a small number of stories and a small number of voters for a short period of time. After that, many elements start affecting how scoring happens.
Factors that affect scores:

Voters' Self-selection:
Not all readers vote. So voters are self selective and not necessarily a real representation of the general readership.

Voters' Niceness:
People are nice in general (unless they have an agenda). They don't like to punish or discourage. The general tendency is to only vote when they like a story.

People are lazy: Readers don't vote on stories they don't finish. To finish a story, they have to like it. So the general tendency is to get votes from those who liked the story in general.

People are easily influenced:
If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it's fairly readable?

For all the above reasons, a simple, average-the-votes-and-display-the-results-as-a-score voting system doesn't work.

We needed something more sophisticated: A voting system that weighs raw scores and makes them into something more logical.
How does it work then?

The system calculates a story's raw average vote after dropping the top 5% and the bottom 5% of the votes to eliminate outliers.

The system knows all the stories raw scores and knows the median of these raw scores.

The score weighing formula figures where the story's raw score sits between this median and the extremes of 1 and 10. Then it calculates the same relative location for a median of 6.00. So if a story's raw score is equal to the raw scores median, it will end up with a score of 6. if it has a perfect 10 score, it remains at 10. If it has a raw score of 1, then it stays at 1. Stories with raw scores closest to the raw median are moved the most on the scale. Stories closer to the extremes are moved less. The raw scores median is calculated twice per day.

The weighed score gets used as the story's score for display on the site.

This weighing algorithm preserves the relative order of the stories. So if you sort the stories by raw score and by weighed score, you'll get the exact same line up of stories.

To put it simply, the score weighing algorithm shifts scores to create an artificial median of 6.00.

Switch Blayde

@Ernest Bywater

People are easily influenced:
If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it's fairly readable?


I think it has the opposite psychological effect. Let's say I liked a story with a score of 6. I might have given in an 8. But since I think the story is better than a 6 I might bump my score from an 8 to a 9 or 10. I don't know if that happens, but it could.

This weighing algorithm preserves the relative order of the stories. So if you sort the stories by raw score and by weighed score, you'll get the exact same line up of stories.


That's not true. When I sort my stories by the SOL score and then by the raw score, I get a different order. I asked Lazeez about it and he explained why (I don't remember the answer).

Chris Podhola

@Switch Blayde

Again. I am no mathematician. I can only gauge based upon my past experiences. Maybe yours are different. Typically, however, what I experienced back when I submitting stories on a regular basis, was pretty standard for me. I would usually get one or occasionally two 1's. I might get a three ... maybe. maybe one five, a few sevens, a bunch of eights a bunch of nines and a bunch of tens.

So, if we're talking about 100 ratings with a similar spread to what I just described, you would be eliminating 1 (1), 1 (3), 1 (5) and 2 (7) from the low end. From the top end you would be eliminating 5 (10's).

I agree. If the author in question has 5 (1's) and five (10's) it probably works in his favor, but to any author who produces quality work (I am guessing), it doesn't seem like a good deal to eliminate the top and bottom five percent. This method would favor lower quality authors over higher quality writers.

I'd be curious to have someone who is better with statistics weigh in. It is possible that I am way off base, but common sense tells me that I may be onto something here.

Dominions Son

@Switch Blayde

People are easily influenced:
If most scores are higher than 8, then voting less than 8 becomes psychologically impossible for most people. How can you give a story a vote lower than the average when it's fairly readable?



I think it has the opposite psychological effect. Let's say I liked a story with a score of 6. I might have given in an 8. But since I think the story is better than a 6 I might bump my score from an 8 to a 9 or 10. I don't know if that happens, but it could.


I see no reason to think that the two effects are mutually exclusive. Both described effects can be real and affect the same person.

Chris Podhola

@Ernest Bywater

The system knows all the stories raw scores and knows the median of these raw scores.

The score weighing formula figures where the story's raw score sits between this median and the extremes of 1 and 10. Then it calculates the same relative location for a median of 6.00. So if a story's raw score is equal to the raw scores median, it will end up with a score of 6. if it has a perfect 10 score, it remains at 10. If it has a raw score of 1, then it stays at 1. Stories with raw scores closest to the raw median are moved the most on the scale. Stories closer to the extremes are moved less. The raw scores median is calculated twice per day.

The weighed score gets used as the story's score for display on the site.

This weighing algorithm preserves the relative order of the stories. So if you sort the stories by raw score and by weighed score, you'll get the exact same line up of stories.


This is why I never bothered to try to figure out how the scoring system on SOL worked. Even after this explanation, I still don't understand it. lol

Ernest Bywater

@Switch Blayde

When I sort my stories by the SOL score and then by the raw score, I get a different order.


I don't remember the exact dates involved, but one of the old SOL blogs had them. Over the years the scoring methodology and system changed for various systems (remember the old TPA system) and each scoring period was normalised by itself for the scores in that period. For some of the older stories all the raw scores aren't available for each period and only the normalised average and number of votes was carried over into the calculations for the current score for some of the earliest periods.

The point to remember is that what's being referred to in the relative order of the stories is not the order of all the stories but the order of of all the stories in that period and only that period. At one point in the past a major exercise was done by a few people and the results posted in the old forum. In each case the relative relationships between the stories for a period were found to be the same be they done on the raw scores or the normalised scores.

Switch Blayde
Updated:

@Chris Podhola


I'd be curious to have someone who is better with statistics weigh in. It is possible that I am way off base,


Let's use the following for a story with 100 votes:

1 - 2 of them (1*2=2)

2 - 1 of them (2*1=2)

3-6 - 0 (#*0= 0)

7 - 5 of them (7*5=35)

8 - 40 of them (8*40=320)

9 - 30 of them (9*30=270)

10 - 22 of them (10*22=220)

The total score for the 100 votes = 849

Raw score = 849/100=8.49

The bottom 5% (bottom 5 votes) is 2 1s(2), 1 2(2), 2 7s(14) which = 18.

Top 5% is 5 10s(50) which = 50.

Raw score of 849 minus 18 (bottom 5%) minus 50 (top 5%) = 781. Divide that by 90 = 8.67777.

8.49 with all scores

8.67 with the top and bottom 5% dropped.

So even though you lost five 10s and only two 1s, one 2, and two 7s, you did better. That's the effect the 1s have on the score of a good story.

Replies:   Chris Podhola
Chris Podhola

@Switch Blayde

8.67 with the top and bottom 5% dropped.


Oh, okay. That's interesting. I learned something today! ;)

Ernest Bywater

@Chris Podhola


I'd be curious to have someone who is better with statistics weigh in. It is possible that I am way off base, but common sense tells me that I may be onto something here.

No scoring system is perfect, what you aim for is the best possible arrangement for your situation. Years ago the scoring system did not remove the top and bottom scores and the forum posts were full of author complaints about the scoring system being abused due to fanboys voting a favourite author's stories a 10 on the first chapter as soon as it came up and some hater's voting the story a 1 as soon as it was posted. There are people who look for stories with certain tags or by certain authors and they give them an automatic 1 without reading the story.

Anyway, everyone was greatly annoyed (ie. super pissed off) about the regular complaints about fan and hate votes, so I suggested stealing the the voting idea used in international gymnastics where they have a panel of judges with one from each major country. In that the top and bottom vote from the judges are discarded to remove fan and hate votes. After a long discussion on the forum a number of the regulars suggested 5% of votes off each end would be a good figure to use for the discard points, and that's what Lazeez ended up doing it.

At various times since then I've looked at how this affects the vote average on some of my stories posted since it came in and found the effect is so minor it's not worth worrying about. But then, most of my stories tend to score above average, so how it goes with the average stories I'm not sure.

With my best scoring story Always a Marine which has a score of 9.25 from 2,439 votes there are 1,600 10s with a smattering of hate votes. Taking out the 5% of each end removes every vote below 9 as well as a number of 10s. The other stories I've checked which were posted since the change to this system have a similar result in that all the votes below 7 or 8 are tossed.

Replies:   Chris Podhola
Chris Podhola

@Ernest Bywater

No scoring system is perfect, what you aim for is the best possible arrangement for your situation


I was never one to complain about scoring systems. I simply posted my stories, assumed that stories were rated using the same methods for every author and every story, and went with it. I kept an eye on the scores of my stories, but only gave them enough importance to gauge whether or not I was making any progress as a writer. Other than that, I never worried about them. Even using them this way, was subjective. Subject matter (I believe) had as much effect on the story score as quality of writing does (in my opinion)).

And congratulations on the 9.25. That's impressive, no matter how the scores are tallied.

Replies:   Ernest Bywater
KinkyWinks

@HeatAndChills

The scoring here is a lot like algebra, the only reason you would need to understand it is so you could teach it to someone else. If it ain't real, it's B.S. and you will never get anyone here to admit it. Give up and move on, keep writing, and don't worry about the synthetic scores. If you send a message to SOL and request them to give you the real numbers, you might feel better. I know I did.

Lumpy

I will add something here that I learned. Non-straight up scoring, scoring that isn't a straight average I mean, isn't that uncommon.

IMDB for instances uses something called bayesian statistics to weigh their score.

Ernest Bywater

@Chris Podhola

I simply posted my stories, assumed that stories were rated using the same methods for every author and every story, and went with it.


That's basically how I regard the scoring. All the authors get treated the same, so it doesn't matter to me how it's done. I do think the system is a lot like the Gaussian Statistical System you often see referred to as the Bell System because the analysis looks like a bell. It's aim is to compare various exams to each other via a common median. It works for a comparison of result spreads but fails to show the real level of knowledge demonstrated by the people doing the exam.

I only look at the scores when involved in discussions about them.

HeatAndChills

Wow! Thanks everybody! There's obviously been a lot of response to my original post, so I won't bother replying to you all individually or I'd be here all day! :)

Suffice to say that this was very informative and a lot of you brought up very valid points that I hadn't considered. Thank you for taking the time to explain this. As many of you have noted, we are all playing by the same rules here, so I suppose that's all that really matters.

Crumbly Writer

The other thing, not mentioned, is that the 'normalization' has varied widely over time. It used to be that all the highest scoring stories were for 'legacy' stories. The newer stories always scored several points lower than the older ones. In each iteration, my scores have jumped up and down, so there is no consistent scores. In fact, I'm not even sure ones ranking remains consistent between scoring calculation changes.

However, the issue is how you decide to cope with scores. Scores really don't tell you what readers think about a story, only how they get reported. Certain stories (think gay, ponyboy or feces related) get a disportionate amount of 1s, which cause entire genres of authors to abandon the site. That's not good for anyone besides the haters.

As Switch says, scoring is a story selection guide, it's not intended to be a reliable comparison between stories (which I don't even think is possible, given the differences in people's likes). Authors need to focus on either total downloads (which also has reliability issues) or reader feedback. Many of us get a lot more useful data from the change in scores for each chapter, as that highlights which chapters fall flat, or which are the most popular.

In short, if you get so upset you turn scoring off, you're entirely too sensitive to survive as an author. Author's have always needed thick skins, because personal rejection is part and parcel of our jobs. If you can't bounce back, you're going to be offended more often then you'll be happy. You need to pick your battles. You want to build readership, rather than crying of spilled scores. And the best way to build readership is to deliver a consistent quality of work.

Back to Top