Yahoo Forum Archive

This is an archive of the MEFA Yahoo Group, which was shut down by Yahoo in 2019. The archive can be sorted by month and by topic ID. You can use your browser to search by keyword within the month or topic you have open.

JanFebMarAprMayJunJulAugSepOctNovDec
2004---18210426558925263362316285
20051895610753834744697276194358565136
200623166277611713912756676615979
200720257-297299143318583103
2008561335424014127477516090106
2009283-39194101722722153624
201067-14103138129321316330
20111-172625906132758
201230---812276-----
2013------------
2014---------1-2
2015------------
2016------------
2017------------
2018------------
2019---------1--

Msg# 6655

Re: points and various voting matters (Rhapsody) Posted by Marta Layton January 07, 2006 - 10:07:14 Topic ID# 6655
> Message: 3
> Date: Tue, 03 Jan 2006 16:44:44 -0000
> From: "rhapsody_the_bard" <rhapsody74@gmail.com>
> Subject: Re: points and various voting matters
>
> --- In MEFAwards@yahoogroups.com, "Laura" <thunderalaura@j...> wrote:
>>
>> "Marta Layton" <melayton@g...> wrote:
>>>> First, I think some people find it hard to write long reviews and
>>>> so feel that even stories they really like they can't give them the
>>>> points they're worth. I suggest that we have each review cap off at
>>>> a lower level (say, five points instead of ten). This effectively
>>>> means that those 1- and 2-point reviews have more weight. One such
>>>> spread would be:
>>>>
>>>> 1-50 1 point
>>>> 51-250 2 point
>>>> 251-500 3 point
>>>> 501-1000 4 point
>>>> 1001+ 5 point
>>>>
>>>> I know that I tend to be pretty long-winded in my reviews, so I'm
>>>> very interested in hearing frm people who struggled to write longer
>>>> ones.
>>>> Would this point spread work better?
>>
>> *chiming in once more as the voice of dissent*
>>
>> I'm honestly not trying to play devil's advocate. Really, I'm not!
>>
> And I think we've had some excellent ideas out of the post-mortem. But
> I have to weigh in as _strongly_ opposed to a change in the point
> levels. At least, a change like the one proposed.

Fair enough. Is it the uneven point spread that you're opposed to, or
would you still be against it if the point spread was more even -- say
one point for every 250 characters up to 1000 characters (five points
for anything over 1,000)?

>> Granted, I'm probably among the long-winded reviewers, but be that
>> as it may, if I rambled off a review and hit as many aspects of the
>> story as I could, I only got up to about 5 points on average. On
>> rare occasions, I could hit 6. But that was if I rambled, and I
>> caught most of those when I went back through and edited the
>> reviews. That being said, I know I gave out several 10-point
>> reviews, quite a few 9-point reviews, and even more 8-pointers and
>> 7-pointers. For all of them, I went back and put in enough effort to
>> get the story that high because I felt that strongly about it. I
>> think there should be a difference between a 2-point story and a
>> 10-point story, and I think the margin between points should reflect
>> that. Furthermore, if I go to the effort to get a 1001-character
>> review, I want it to count. I want the author to get those ten
>> points.
>
> I am a rambler as well, but when I leave reviews in general, they
> *are* long. No matter where I leave them. Story archives, MEFA's..
> While for the MEFA's, I only finalised them after I checked spelling,
> I really didn't care for the points, I was merely aiming at leaving a
> nice review that would make the author in question smile or for an ego
> boost. So if I really loved a story, wanted to pass back to the author
> how good it felt to have read the story, and I left a glowing review
> (since I did leave long glowing reviews to stories with just one
> review for the MEFAs I guess).. it feels like being ticked on my
> fingers for doing so. Weren't these the Feel good awards? So why not
> leave a story a glowing review (a glowing review can be either short
> or long) if my aim is to make a writer feel good about their work? Is
> that so wrong?
>
> *scratches her head* Just please, don't loose sight of the aim to make
> author's feel good about their work before it boils down to a points
> debate.
>

You know, it's funny -- I can be quite long-winded, too (not rambly, I
don't think - there's a difference) but only when I'm in the mood to
be. Sometimes when I just don't have the energy I will get to the point
and just not say loads. And yes, this is true for my reviewing at
archives as well.

I don't want to lose sight of the aim, either; like I said to RSF a
minute ago I don't want the points to make reviewing not fun, and I
don't want them to get in the way of encouraging good reviews. But I
also don't want people to forget about them either completely - if you
really like a story and want to do your best to make sure it's
recognised in the form of an award, then a longer review will do more
good than a shorter one. This doesn't mean you have to write 10 points,
but it might be a good idea to try to make it longer than most of your
reviews.

How do you do this without rambling? I found the simplest way was to
get into more specific aspects of the story. Maybe the story had some
simply beautiful turns of phrase, so you want to talk a little bit
about some simply beautiful turns of phrase. Then after you've talked
on that for 200 characters or so (which is what naturally came to me -
other people will be comfortable with different amounts) you move on to
how the author created a whole world with details that hinted at events
not told in the story, and that made it feel real. Maybe then go on to
how the pacing was just right, never too fast or too slow, and how the
story used a convincing real world historical model that made it that
much more believable. by going itno these different areas you have more
to talk about and the review will by default be longer - yet you
haven't spent that many more characters than you would have in a
shorter review. It's just that in a shorter review I wouldn't have gone
into as many specific areas.

>> Competition-wise, it doesn't make that much of a difference. But as
>> far as the quality of the reviews is concerned, I think there is a
>> difference. happen to think that the 1000+ reviews are good ego
>> boosters. I like receiving them, and I like giving them. And I think
> < there's more incentive to give them if it actually makes a
>> difference in the competition.
>
> I agree completely, changing the points system won't stop me from
> leaving long reviews since I didn't pay much attention to the number
> of points a story got in the first place.
>
> As for quoting from stories... isn't it an idea to add a button to the
> system, review form where a reviewer can click on (telling or putting
> it in the faq that you have to put the <blockquote> code around it
> will not work...). Let's say you want to quote from a story, a quick
> click, in which it automatically adds the quote-code (or maybe a java
> script kind of thing where you can paste the part in, click on ok and
> it gets inserted)...
>

Anthony, is this possible? I wouldn't mind having a piece of code that
automatically inserted the blockquotes, but it's up to you whether you
want to do this.

Rhapsody, would it work if we put a note *on the form where you insert
the vote* that said something like:

All quotes must be surrounded by tags so that they are not counted when
determining points. Please place "<blockquote>" before the beginning of
any quote you include in your review and "</blockquote>" at the end of
the quote. If you have any questions about this do not hesitate to
email the administrators at <link>mefasupport@gmail.com</link>.

Cheers,
Marta

Msg# 6677

Comparison of 2004 and 2005 points systems Posted by Anthony Holder January 08, 2006 - 2:51:16 Topic ID# 6655
Someone asked about checking the scoring system that we have against
the other possible methods.

Here is a breakdown of how the reviews would have scored with the 2004
system vs. how they were scored in 2005.

2005 2004 2004 system
1 810 40 0 to 20 char
2 1390 504 21 to 80
3 1179 2332 81 to 250
4 821
5 532 1856 251 to 500
6 495
7 270 697 501 to 750
8 155
9 104 327 751 to 1100
10 267 267 1101+

As you can see, the 2004 system was more heavily weighted toward the
smaller reviews. The vast majority of the reviews were 3, 5, or 7
pointers.

Above 5 points, the two systems are very similar, but they differ quite
a bit on the lower end. Essentially the change in 2005 made it much
harder to get a 3+ point review.

If one were to go back to essentially the 2004 system, but with a
continuous system with no discontinuities, the equation would be:

0 to 20: 100% of chars
21 to 80: 33.3% of chars
81 to 250: 11.76% of chars
251 to 750: 8% of chars
751 to 1100: 5.71% of chars

This is all relative to those first 20 characters getting one point in
the old system, so they had the most impact. After that point, it went
down pretty rapidly.

The 2005 system breaks down to:

0 to 500: 100% of chars
501 to 1100: 66.6% of chars

From this, you can see that in 2005, the longer reviews were more
heavily weighted, as compared to the very short reviews. Compared to
the medium-sized reviews, though, it was more similar. If you use the
'81 to 250' range as the standard, you get:

0 to 20: 850% of chars
21 to 80: 283% of chars
81 to 250: 100% of chars
251 to 750: 68% of chars
751 to 1100: 49% of chars

So the relative 'worth' of the 250 to 500 group was substantially
increased, while the 501 to 750 group was almost unchanged, relative to
the 3 point review.

I also split the reviews into groups of 20, and calculated point totals
via the two methods. In doing so, these groups showed that for the
lower point totals, the 2004 scoring system would have increased the
scores by about 45-50%, but for the higher point totals, the increase
was 6 to 15%. Everybody would have gotten higher points, but the
stories with lots of smaller reviews would have fared better than those
stories with a few longer reviews.

I think the basic idea behind the 2004 point system is that if you get
lots of reviews that should count somewhat better than someone that got
a few long reviews, if the total character count is similar between the
two. Thus, having the 'minimum score' (essentially) be 3 points, except
for those ultra-short <20-character or pretty darn short <80 character
reviews.

Assuming that this behavior is what you want, I propose the following:

Do a more continuous points scale, as follows:

characters percentage calculated score
0 to 20: 20% of characters 4 pts for a 20 character review
21 to 100: 10% of characters 4+8 = 12 pts for an 80 char review
100 to 300: 5% of characters 12 + 10 = 22 pts for a 300 char
review
300 to 700: 4% of characters 22 + 16 = 38 pts for a 700 char
review
700 to 1100: 3% of characters 38 + 12 = 50 pts for a 1100 char
review

Round up to the nearest 1 pt.

Here's how the reviews would have broken down this year using my
suggested system.

Pts num reviews at that score
1 0
2 4
3 9
4 27
5 44
6 49
7 76
8 109
9 114
10 112
11 127
12 139
13 295
14 285
15 313
16 235
17 262
18 301
19 252
20 241
21 204
22 181
23 246
24 228
25 189
26 158
27 145
28 154
29 117
30 116
31 113
32 93
33 78
34 83
35 67
36 61
37 70
38 44
39 58
40 59
41 39
42 34
43 33
44 49
45 27
46 19
47 34
48 18
49 11
50 301

This is similar to the 2004 system, but weights a bit higher for longer
reviews. It does not weight nearly as high as the 2005 system, though.

The maximum review is a nice round number, 50 points.

Total scores would be much higher (stroke those egos, now!).

year/total points awarded using that system and the 2005 reviews.
2004 27,816
2005 22,777
2006 138,403

Now that I've given you this data, I suspect that you'll need to decide
which sort of system you prefer (weight the shorter reviews or weight
the longer reviews).

I've put my spreadsheet on the Yahoo Group! files section. I haven't
documented it very well, but I think you can understand what I did. I
even graphed the proposed 2006 scores, and they look like a nice bell
curve with a long tail to the right. and a big bump at 50.

Didn't I say something about deciding things soon? I'm not helping that
along, I guess!

Later,
Anthony

Msg# 6679

Re: Comparison of 2004 and 2005 points systems Posted by rabidsamfan January 08, 2006 - 22:29:53 Topic ID# 6655
Oooh, someone who actually "gets" math! It took me three readings to
understand this all, but I'm impressed.

I didn't participate in the 2004 awards, so it looks to me like you almost
had to be deliberately terse to give a one point review. And I take it some
numbers were skipped entirely? I kind of like the idea, but would prefer a
slightly higher number as the "base" 100% zone (like 75 characters). That
way you could give a one point review without having to say nothing more
than "Nice story."

I like the distribution curve of the 50 point system, although I'll admit to
wondering what would happen to that tail if you took it to 100. It would be
very hard to get ties in that too.

Would it be easy to check Marta's suggestion of a 5 point scale in the same
way? And would there be more "ties" with a five point scale? (If this is
more math than you need to do right now, please don't worry about it.)

On 1/8/06, Anthony Holder <aaholder@swbell.net> wrote:
>
> Someone asked about checking the scoring system that we have against
> the other possible methods.
>
> Here is a breakdown of how the reviews would have scored with the 2004
> system vs. how they were scored in 2005.
>
> 2005 2004 2004 system
> 1 810 40 0 to 20 char
> 2 1390 504 21 to 80
> 3 1179 2332 81 to 250
> 4 821
> 5 532 1856 251 to 500
> 6 495
> 7 270 697 501 to 750
> 8 155
> 9 104 327 751 to 1100
> 10 267 267 1101+
>
> As you can see, the 2004 system was more heavily weighted toward the
> smaller reviews. The vast majority of the reviews were 3, 5, or 7
> pointers.
>
> Above 5 points, the two systems are very similar, but they differ quite
> a bit on the lower end. Essentially the change in 2005 made it much
> harder to get a 3+ point review.
>
> If one were to go back to essentially the 2004 system, but with a
> continuous system with no discontinuities, the equation would be:
>
> 0 to 20: 100% of chars
> 21 to 80: 33.3% of chars
> 81 to 250: 11.76% of chars
> 251 to 750: 8% of chars
> 751 to 1100: 5.71% of chars
>
> This is all relative to those first 20 characters getting one point in
> the old system, so they had the most impact. After that point, it went
> down pretty rapidly.
>
> The 2005 system breaks down to:
>
> 0 to 500: 100% of chars
> 501 to 1100: 66.6% of chars
>
> From this, you can see that in 2005, the longer reviews were more
> heavily weighted, as compared to the very short reviews. Compared to
> the medium-sized reviews, though, it was more similar. If you use the
> '81 to 250' range as the standard, you get:
>
> 0 to 20: 850% of chars
> 21 to 80: 283% of chars
> 81 to 250: 100% of chars
> 251 to 750: 68% of chars
> 751 to 1100: 49% of chars
>
> So the relative 'worth' of the 250 to 500 group was substantially
> increased, while the 501 to 750 group was almost unchanged, relative to
> the 3 point review.
>
> I also split the reviews into groups of 20, and calculated point totals
> via the two methods. In doing so, these groups showed that for the
> lower point totals, the 2004 scoring system would have increased the
> scores by about 45-50%, but for the higher point totals, the increase
> was 6 to 15%. Everybody would have gotten higher points, but the
> stories with lots of smaller reviews would have fared better than those
> stories with a few longer reviews.
>
> I think the basic idea behind the 2004 point system is that if you get
> lots of reviews that should count somewhat better than someone that got
> a few long reviews, if the total character count is similar between the
> two. Thus, having the 'minimum score' (essentially) be 3 points, except
> for those ultra-short <20-character or pretty darn short <80 character
> reviews.
>
> Assuming that this behavior is what you want, I propose the following:
>
> Do a more continuous points scale, as follows:
>
> characters percentage calculated score
> 0 to 20: 20% of characters 4 pts for a 20 character review
> 21 to 100: 10% of characters 4+8 = 12 pts for an 80 char review
> 100 to 300: 5% of characters 12 + 10 = 22 pts for a 300 char
> review
> 300 to 700: 4% of characters 22 + 16 = 38 pts for a 700 char
> review
> 700 to 1100: 3% of characters 38 + 12 = 50 pts for a 1100 char
> review
>
> Round up to the nearest 1 pt.
>
> Here's how the reviews would have broken down this year using my
> suggested system.
>
> Pts num reviews at that score
> 1 0
> 2 4
> 3 9
> 4 27
> 5 44
> 6 49
> 7 76
> 8 109
> 9 114
> 10 112
> 11 127
> 12 139
> 13 295
> 14 285
> 15 313
> 16 235
> 17 262
> 18 301
> 19 252
> 20 241
> 21 204
> 22 181
> 23 246
> 24 228
> 25 189
> 26 158
> 27 145
> 28 154
> 29 117
> 30 116
> 31 113
> 32 93
> 33 78
> 34 83
> 35 67
> 36 61
> 37 70
> 38 44
> 39 58
> 40 59
> 41 39
> 42 34
> 43 33
> 44 49
> 45 27
> 46 19
> 47 34
> 48 18
> 49 11
> 50 301
>
> This is similar to the 2004 system, but weights a bit higher for longer
> reviews. It does not weight nearly as high as the 2005 system, though.
>
> The maximum review is a nice round number, 50 points.
>
> Total scores would be much higher (stroke those egos, now!).
>
> year/total points awarded using that system and the 2005 reviews.
> 2004 27,816
> 2005 22,777
> 2006 138,403
>
> Now that I've given you this data, I suspect that you'll need to decide
> which sort of system you prefer (weight the shorter reviews or weight
> the longer reviews).
>
> I've put my spreadsheet on the Yahoo Group! files section. I haven't
> documented it very well, but I think you can understand what I did. I
> even graphed the proposed 2006 scores, and they look like a nice bell
> curve with a long tail to the right. and a big bump at 50.
>
> Didn't I say something about deciding things soon? I'm not helping that
> along, I guess!
>
> Later,
> Anthony
>
>
>
> SPONSORED LINKS
> Writing and publishing a book<http://groups.yahoo.com/gads?t=ms&k=Writing+and+publishing+a+book&w1=Writing+and+publishing+a+book&w2=Writing+book&w3=Writing+child+book&w4=Writing+a+book+report&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=OWB7vxgAbRkTahBRXFLnXw> Writing
> book<http://groups.yahoo.com/gads?t=ms&k=Writing+book&w1=Writing+and+publishing+a+book&w2=Writing+book&w3=Writing+child+book&w4=Writing+a+book+report&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=6Z1Zky8D0GhX4r1EuyACDA> Writing
> child book<http://groups.yahoo.com/gads?t=ms&k=Writing+child+book&w1=Writing+and+publishing+a+book&w2=Writing+book&w3=Writing+child+book&w4=Writing+a+book+report&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=uadXxz6rd6Zn-sk2kaB6cg> Writing
> a book report<http://groups.yahoo.com/gads?t=ms&k=Writing+a+book+report&w1=Writing+and+publishing+a+book&w2=Writing+book&w3=Writing+child+book&w4=Writing+a+book+report&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=TdlXhRsskXLKJt7xiPw7-w> Business
> writing book<http://groups.yahoo.com/gads?t=ms&k=Business+writing+book&w1=Writing+and+publishing+a+book&w2=Writing+book&w3=Writing+child+book&w4=Writing+a+book+report&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=7jDWw1GqY7lYHvBM5Y6uUA> Creative
> writing book<http://groups.yahoo.com/gads?t=ms&k=Creative+writing+book&w1=Writing+and+publishing+a+book&w2=Writing+book&w3=Writing+child+book&w4=Writing+a+book+report&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=3YxX5Su32OpxFDrWNs7PCg>
> ------------------------------
> YAHOO! GROUPS LINKS
>
>
> - Visit your group "MEFAwards<http://groups.yahoo.com/group/MEFAwards>"
> on the web.
>
> - To unsubscribe from this group, send an email to:
> MEFAwards-unsubscribe@yahoogroups.com<MEFAwards-unsubscribe@yahoogroups.com?subject=Unsubscribe>
>
> - Your use of Yahoo! Groups is subject to the Yahoo! Terms of
> Service <http://docs.yahoo.com/info/terms/>.
>
>
> ------------------------------
>


[Non-text portions of this message have been removed]

Msg# 6685

Re: Comparison of 2004 and 2005 points systems Posted by Anthony Holder January 10, 2006 - 1:09:21 Topic ID# 6655
I can see that the discussion is trending more to the 'why' than the
'what', and that you all have more clearly identified what the
discussion is. I've got some more statistics here that may help you see
what the impact of certain scoring systems could be.

RSF, Here's the breakdown you've asked for, with the 2004 and 2005, and
Thundera's suggestion for comparison.

2005 2004 1 to 5 5 to 10 Thundera
pts scale scale scale scale scale
1 809 40 209 0 809
2 1389 504 2664 0 0
3 1178 2329 1856 0 2064
4 821 0 961 0 0
5 532 1856 330 809 1623
6 495 0 0 2064 0
7 270 697 0 1623 842
8 155 0 0 842 0
9 104 327 0 352 352
10 267 267 0 330 330

Dividing the reviews into groups of 20 more or less at random, here are
some statistics:

2005 2004 AH/5 5-10 1-5
Min 35 51 48 112 34
Max 136 144 146.8 168 78
Range 101 93 98.8 56 44
Mean 75.7 92.4 91.9 136.2 55.1
StdDev 19.8 18.1 19.0 11.1 8.1

Remembering that these are all assuming that each group (story) gets 20
votes. It's not very realistic, but gives some indication of how the
scores might fall.

What I see here is that the 2005 and 2004 systems had similar standard
deviations (a measure of how much variation between groupings). This
indicates that the two systems, while weighted much differently, will
provide a range of scores that are similar. From last time, the lower
totals increased about 40-50%, and the higher totals increased 10-15%,
showing the influence of the higher weighting on the lower scores.

The two 5 point systems, however, have much smaller standard
deviations, indicating that there would be much less variation between
the top and bottom scorers.

To compare the 'AH' (my initials) system to the others, I divided the
numbers by 5, since the max score was 50 instead of 10. The system
compares fairly well to the 2004 and 2005 systems in variability,
minimum and maximum, which makes sense, because it doesn't really
change much except multiply the scores by 5 and increase the number of
categories.

The AH system does weight more heavily toward shorter reviews than the
2005 system (a 91 character review is 12 points, about 25% of the max,
and a 341 character review is 50% of the max), but there isn't an
immediate jump from 0 to any number. It is possible to write a 1 point
review ("Nice!" would be 1 pt.). If I was writing a review like that, I
would expect it not to count as much as a 1000 character review in this
sort of system. It does count, though, and it could make the difference
between first and second place for the story, but with the 5-10 system,
that review would count half as much as a 1001 character review.

Everyone seems to agree that there should be a weighted system. The
question is how to weight it.

Very little weighting (like 2005) gives more 'power' to those who
can/do write longer reviews, as theirs count proportionally more than
the shorter reviews. With both the 2004 and 2005 systems, about half of
these reviews are 3 points or less, with more being less in 2005. Over
3/4 were 5 or less with both systems.

So 1/4 of the reviews are > 5 pts with both systems.

Here's a calculation of the percentage of the total points given vs.
the point level.

2005 2004
1 4% 0%
2 12% 4%
3 16% 25%
4 14% 0%
5 12% 33%
6 13% 0%
7 8% 18%
8 5% 0%
9 4% 11%
10 12% 10%

From this, you can see that the 2004 and 2005 systems give 29 and 32%
of the 'power' to the 1-3 point reviews, the 33% and 26% to the 4-5
point reviews, 18% and 21% to the 6 and 7 point reviews, 11% and 9% to
the 8 and 9 point reviews and 10 and 12% to the 10 point reviews. Based
on this analysis, it appears that in both systems, the fact that there
are many more shorter reviews plays a significant role in the voting
outcome. In other words, don't worry that your review is shorter than
someone else's, because several short reviews can beat out a few long
reviews.

In fact, of the about 408 awards over all the categories, comparing 1st
to 2nd and 2nd to 3rd, only 61 stories with fewer reviews beat stories
with more reviews. In fact, in 97 of the 408, stories with lower
average length reviews beat stories with longer average length reviews.
In 95 cases, stories with lower average review scores beat stories with
higher average review scores.

For most stories, of course, more and longer reviews beat fewer and
shorter reviews.

I can not do the same statistics assuming the 2004 system had been in
place, but from earlier calculations, I can assure you that there would
have been far fewer stories winning with fewer/longer reviews over
more/shorter reviews, because the shorter ones got proportionally more
points in 2004. I would expect to see the 61 down to about 10-20, and
the 97 up to 150-200 of the 408 stories.

Anthony

Msg# 6690

Re: Comparison of 2004 and 2005 points systems Posted by rabidsamfan January 10, 2006 - 10:04:03 Topic ID# 6655
Thank you, Anthony, for providing the actual numbers. It makes it much
easier to look at the question dispassionately.

While I really like your fifty point system, I doubt we'll go that way this
year, as the 2005 system seems to have worked pretty well -- long reviews
had some good weight, but enough short reviews could lift a story too. That
seems like a good balance to me, as favoring short reviews too much would
make it seem like there wasn't any point in writing the longer ones, and as
an author I did like getting the longer ones. They were a lot more likely
to give me insights into why the story was liked and what I did right.

For that matter, I liked your idea of continuing to accrue points at a slow
rate for very very long reviews. I didn't see any reviews that felt "puffed
up" to me this year, and I don't really think it's that much of a problem,
especially since we're going to block all the quotes. But I know that one's
not going to fly. *grin*

And just think of all the things I've learned about statistics from working
my way through your posts!


On 1/10/06, Anthony Holder <aaholder@swbell.net> wrote:
>
> I can see that the discussion is trending more to the 'why' than the
> 'what', and that you all have more clearly identified what the
> discussion is. I've got some more statistics here that may help you see
> what the impact of certain scoring systems could be.
>
> RSF, Here's the breakdown you've asked for, with the 2004 and 2005, and
> Thundera's suggestion for comparison.
>
> 2005 2004 1 to 5 5 to 10 Thundera
> pts scale scale scale scale scale
> 1 809 40 209 0 809
> 2 1389 504 2664 0 0
> 3 1178 2329 1856 0 2064
> 4 821 0 961 0 0
> 5 532 1856 330 809 1623
> 6 495 0 0 2064 0
> 7 270 697 0 1623 842
> 8 155 0 0 842 0
> 9 104 327 0 352 352
> 10 267 267 0 330 330
>
> Dividing the reviews into groups of 20 more or less at random, here are
> some statistics:
>
> 2005 2004 AH/5 5-10 1-5
> Min 35 51 48 112 34
> Max 136 144 146.8 168 78
> Range 101 93 98.8 56 44
> Mean 75.7 92.4 91.9 136.2 55.1
> StdDev 19.8 18.1 19.0 11.1 8.1
>
> Remembering that these are all assuming that each group (story) gets 20
> votes. It's not very realistic, but gives some indication of how the
> scores might fall.
>
> What I see here is that the 2005 and 2004 systems had similar standard
> deviations (a measure of how much variation between groupings). This
> indicates that the two systems, while weighted much differently, will
> provide a range of scores that are similar. From last time, the lower
> totals increased about 40-50%, and the higher totals increased 10-15%,
> showing the influence of the higher weighting on the lower scores.
>
> The two 5 point systems, however, have much smaller standard
> deviations, indicating that there would be much less variation between
> the top and bottom scorers.
>
> To compare the 'AH' (my initials) system to the others, I divided the
> numbers by 5, since the max score was 50 instead of 10. The system
> compares fairly well to the 2004 and 2005 systems in variability,
> minimum and maximum, which makes sense, because it doesn't really
> change much except multiply the scores by 5 and increase the number of
> categories.
>
> The AH system does weight more heavily toward shorter reviews than the
> 2005 system (a 91 character review is 12 points, about 25% of the max,
> and a 341 character review is 50% of the max), but there isn't an
> immediate jump from 0 to any number. It is possible to write a 1 point
> review ("Nice!" would be 1 pt.). If I was writing a review like that, I
> would expect it not to count as much as a 1000 character review in this
> sort of system. It does count, though, and it could make the difference
> between first and second place for the story, but with the 5-10 system,
> that review would count half as much as a 1001 character review.
>
> Everyone seems to agree that there should be a weighted system. The
> question is how to weight it.
>
> Very little weighting (like 2005) gives more 'power' to those who
> can/do write longer reviews, as theirs count proportionally more than
> the shorter reviews. With both the 2004 and 2005 systems, about half of
> these reviews are 3 points or less, with more being less in 2005. Over
> 3/4 were 5 or less with both systems.
>
> So 1/4 of the reviews are > 5 pts with both systems.
>
> Here's a calculation of the percentage of the total points given vs.
> the point level.
>
> 2005 2004
> 1 4% 0%
> 2 12% 4%
> 3 16% 25%
> 4 14% 0%
> 5 12% 33%
> 6 13% 0%
> 7 8% 18%
> 8 5% 0%
> 9 4% 11%
> 10 12% 10%
>
> From this, you can see that the 2004 and 2005 systems give 29 and 32%
> of the 'power' to the 1-3 point reviews, the 33% and 26% to the 4-5
> point reviews, 18% and 21% to the 6 and 7 point reviews, 11% and 9% to
> the 8 and 9 point reviews and 10 and 12% to the 10 point reviews. Based
> on this analysis, it appears that in both systems, the fact that there
> are many more shorter reviews plays a significant role in the voting
> outcome. In other words, don't worry that your review is shorter than
> someone else's, because several short reviews can beat out a few long
> reviews.
>
> In fact, of the about 408 awards over all the categories, comparing 1st
> to 2nd and 2nd to 3rd, only 61 stories with fewer reviews beat stories
> with more reviews. In fact, in 97 of the 408, stories with lower
> average length reviews beat stories with longer average length reviews.
> In 95 cases, stories with lower average review scores beat stories with
> higher average review scores.
>
> For most stories, of course, more and longer reviews beat fewer and
> shorter reviews.
>
> I can not do the same statistics assuming the 2004 system had been in
> place, but from earlier calculations, I can assure you that there would
> have been far fewer stories winning with fewer/longer reviews over
> more/shorter reviews, because the shorter ones got proportionally more
> points in 2004. I would expect to see the 61 down to about 10-20, and
> the 97 up to 150-200 of the 408 stories.
>
> Anthony
>
>
>
> SPONSORED LINKS
> Writing and publishing a book<http://groups.yahoo.com/gads?t=ms&k=Writing+and+publishing+a+book&w1=Writing+and+publishing+a+book&w2=Writing+a+book+report&w3=Writing+book&w4=Writing+child+book&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=4NswWMNbpCcBYRSEhv4Qew> Writing
> a book report<http://groups.yahoo.com/gads?t=ms&k=Writing+a+book+report&w1=Writing+and+publishing+a+book&w2=Writing+a+book+report&w3=Writing+book&w4=Writing+child+book&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=HvbdbSVG-3TFLjgIeR2dig> Writing
> book<http://groups.yahoo.com/gads?t=ms&k=Writing+book&w1=Writing+and+publishing+a+book&w2=Writing+a+book+report&w3=Writing+book&w4=Writing+child+book&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=vL959fD5kQ6WHFSwZR3lzg> Writing
> child book<http://groups.yahoo.com/gads?t=ms&k=Writing+child+book&w1=Writing+and+publishing+a+book&w2=Writing+a+book+report&w3=Writing+book&w4=Writing+child+book&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=E4mE-duQvC3zQmLeREHSJw> Business
> writing book<http://groups.yahoo.com/gads?t=ms&k=Business+writing+book&w1=Writing+and+publishing+a+book&w2=Writing+a+book+report&w3=Writing+book&w4=Writing+child+book&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=pnD3Q6WpqansAHR90qpGmA> Creative
> writing book<http://groups.yahoo.com/gads?t=ms&k=Creative+writing+book&w1=Writing+and+publishing+a+book&w2=Writing+a+book+report&w3=Writing+book&w4=Writing+child+book&w5=Business+writing+book&w6=Creative+writing+book&c=6&s=158&.sig=INaED3DuAQYgIxqefxN98g>
> ------------------------------
> YAHOO! GROUPS LINKS
>
>
> - Visit your group "MEFAwards<http://groups.yahoo.com/group/MEFAwards>"
> on the web.
>
> - To unsubscribe from this group, send an email to:
> MEFAwards-unsubscribe@yahoogroups.com<MEFAwards-unsubscribe@yahoogroups.com?subject=Unsubscribe>
>
> - Your use of Yahoo! Groups is subject to the Yahoo! Terms of
> Service <http://docs.yahoo.com/info/terms/>.
>
>
> ------------------------------
>


[Non-text portions of this message have been removed]

Msg# 6692

Re: Comparison of 2004 and 2005 points systems Posted by dwimmer\_laik January 10, 2006 - 11:44:29 Topic ID# 6655
<snip first batch of stats>

Thanks, Anthony. This batch of statistics and the accompanying
explanation makes more sense to me than the previous one, and also
answers a key question: namely, are short reviews competitive?

> Everyone seems to agree that there should be a weighted system. The
> question is how to weight it.
>

<paste>

> For most stories, of course, more and longer reviews beat fewer and
> shorter reviews.

That makes sense, but it also seems the least controversial case. I
doubt anyone is going to complain in that scenario. Do you know off
hand what percentage of stories placed first on this basis?

> Very little weighting (like 2005) gives more 'power' to those who
> can/do write longer reviews, as theirs count proportionally more than
> the shorter reviews. With both the 2004 and 2005 systems, about half of
> these reviews are 3 points or less, with more being less in 2005. Over
> 3/4 were 5 or less with both systems.
>
> So 1/4 of the reviews are > 5 pts with both systems.
>
> Here's a calculation of the percentage of the total points given vs.
> the point level.
>
> 2005 2004
> 1 4% 0%
> 2 12% 4%
> 3 16% 25%
> 4 14% 0%
> 5 12% 33%
> 6 13% 0%
> 7 8% 18%
> 8 5% 0%
> 9 4% 11%
> 10 12% 10%
>
> From this, you can see that the 2004 and 2005 systems give 29 and 32%
> of the 'power' to the 1-3 point reviews, the 33% and 26% to the 4-5
> point reviews, 18% and 21% to the 6 and 7 point reviews, 11% and 9% to
> the 8 and 9 point reviews and 10 and 12% to the 10 point reviews. Based
> on this analysis, it appears that in both systems, the fact that there
> are many more shorter reviews plays a significant role in the voting
> outcome. In other words, don't worry that your review is shorter than
> someone else's, because several short reviews can beat out a few long
> reviews.

>In fact, of the about 408 awards over all the categories, comparing
1st to 2nd and 2nd to 3rd, only 61 stories with fewer reviews beat
stories with more reviews.

In other words, if you vote, even if you don't give a lengthy review,
your vote has a substantial impact; in only 15% of cases (rounding to
400 and 60) do fewer but longer reviews trump more plentiful short
reviews. So what we're dealing with when there's fear that a short
review doesn't count is just that: fear.

In which case, I'm inclined to say, if it isn't broken, don't "fix"
the points system. I'd go with Marta, given this analysis, and say
let's put this discussion aside until next year. Let's see how things
fall out, get a set of numbers that is clearly and immediately
comparable to the numbers from 2005, with no need for adjustment, and
then see what happens. If the results are similar, I think we don't
need to fuss with how points are distributed, even by curving the
system, although if people really were in favor of changing the points
system at that point, I'd still hold out for a curve over a reduced
scale range.

Dwim

Msg# 6715

Re: Comparison of 2004 and 2005 points systems Posted by Ainaechoiriel January 12, 2006 - 18:32:21 Topic ID# 6655
Thank you for the spread! That helped. So the folks at ASC were on to
something with thir spread that I basically stole (with permission).

However, I do not want to go over 10 points. So definitely, I prefer
weighting but not quite as you suggested.

--Ainaechoiriel
MEFA Admin and Founder

"This evil cannot be concealed by the power of the Elves," Elrond said, "for
it is Windows-compatible, and freeware at that." --H.F.

http://gabrielle.sytes.net/mefa The Middle-Earth Fanfiction Awards

Blog: http://www.ainaechoiriel.blogspot.com



> -----Original Message-----
> From: MEFAwards@yahoogroups.com
> [mailto:MEFAwards@yahoogroups.com] On Behalf Of Anthony Holder
> Sent: Sunday, January 08, 2006 2:51 AM
> To: MEFAwards@yahoogroups.com
> Subject: [MEFAwards] Comparison of 2004 and 2005 points systems
>
> Someone asked about checking the scoring system that we have
> against the other possible methods.
>
> Here is a breakdown of how the reviews would have scored with
> the 2004 system vs. how they were scored in 2005.
>
> 2005 2004 2004 system
> 1 810 40 0 to 20 char
> 2 1390 504 21 to 80
> 3 1179 2332 81 to 250
> 4 821
> 5 532 1856 251 to 500
> 6 495
> 7 270 697 501 to 750
> 8 155
> 9 104 327 751 to 1100
> 10 267 267 1101+
>
> As you can see, the 2004 system was more heavily weighted
> toward the smaller reviews. The vast majority of the reviews
> were 3, 5, or 7 pointers.
>
> Above 5 points, the two systems are very similar, but they
> differ quite a bit on the lower end. Essentially the change
> in 2005 made it much harder to get a 3+ point review.
>
> If one were to go back to essentially the 2004 system, but
> with a continuous system with no discontinuities, the
> equation would be:
>
> 0 to 20: 100% of chars
> 21 to 80: 33.3% of chars
> 81 to 250: 11.76% of chars
> 251 to 750: 8% of chars
> 751 to 1100: 5.71% of chars
>
> This is all relative to those first 20 characters getting one
> point in the old system, so they had the most impact. After
> that point, it went down pretty rapidly.
>
> The 2005 system breaks down to:
>
> 0 to 500: 100% of chars
> 501 to 1100: 66.6% of chars
>
> From this, you can see that in 2005, the longer reviews were
> more heavily weighted, as compared to the very short reviews.
> Compared to the medium-sized reviews, though, it was more
> similar. If you use the
> '81 to 250' range as the standard, you get:
>
> 0 to 20: 850% of chars
> 21 to 80: 283% of chars
> 81 to 250: 100% of chars
> 251 to 750: 68% of chars
> 751 to 1100: 49% of chars
>
> So the relative 'worth' of the 250 to 500 group was
> substantially increased, while the 501 to 750 group was
> almost unchanged, relative to the 3 point review.
>
> I also split the reviews into groups of 20, and calculated
> point totals via the two methods. In doing so, these groups
> showed that for the lower point totals, the 2004 scoring
> system would have increased the scores by about 45-50%, but
> for the higher point totals, the increase was 6 to 15%.
> Everybody would have gotten higher points, but the stories
> with lots of smaller reviews would have fared better than
> those stories with a few longer reviews.
>
> I think the basic idea behind the 2004 point system is that
> if you get lots of reviews that should count somewhat better
> than someone that got a few long reviews, if the total
> character count is similar between the two. Thus, having the
> 'minimum score' (essentially) be 3 points, except for those
> ultra-short <20-character or pretty darn short <80 character reviews.
>
> Assuming that this behavior is what you want, I propose the following:
>
> Do a more continuous points scale, as follows:
>
> characters percentage calculated score
> 0 to 20: 20% of characters 4 pts for a 20 character review
> 21 to 100: 10% of characters 4+8 = 12 pts for an 80 char review
> 100 to 300: 5% of characters 12 + 10 = 22 pts for a 300 char
> review
> 300 to 700: 4% of characters 22 + 16 = 38 pts for a 700 char
> review
> 700 to 1100: 3% of characters 38 + 12 = 50 pts for a 1100 char
> review
>
> Round up to the nearest 1 pt.
>
> Here's how the reviews would have broken down this year using
> my suggested system.
>
> Pts num reviews at that score
> 1 0
> 2 4
> 3 9
> 4 27
> 5 44
> 6 49
> 7 76
> 8 109
> 9 114
> 10 112
> 11 127
> 12 139
> 13 295
> 14 285
> 15 313
> 16 235
> 17 262
> 18 301
> 19 252
> 20 241
> 21 204
> 22 181
> 23 246
> 24 228
> 25 189
> 26 158
> 27 145
> 28 154
> 29 117
> 30 116
> 31 113
> 32 93
> 33 78
> 34 83
> 35 67
> 36 61
> 37 70
> 38 44
> 39 58
> 40 59
> 41 39
> 42 34
> 43 33
> 44 49
> 45 27
> 46 19
> 47 34
> 48 18
> 49 11
> 50 301
>
> This is similar to the 2004 system, but weights a bit higher
> for longer reviews. It does not weight nearly as high as the
> 2005 system, though.
>
> The maximum review is a nice round number, 50 points.
>
> Total scores would be much higher (stroke those egos, now!).
>
> year/total points awarded using that system and the 2005 reviews.
> 2004 27,816
> 2005 22,777
> 2006 138,403
>
> Now that I've given you this data, I suspect that you'll need
> to decide which sort of system you prefer (weight the shorter
> reviews or weight the longer reviews).
>
> I've put my spreadsheet on the Yahoo Group! files section. I
> haven't documented it very well, but I think you can
> understand what I did. I even graphed the proposed 2006
> scores, and they look like a nice bell curve with a long tail
> to the right. and a big bump at 50.
>
> Didn't I say something about deciding things soon? I'm not
> helping that along, I guess!
>
> Later,
> Anthony
>
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>