Yahoo Forum Archive

> Message: 6
> Date: Mon, 02 Jan 2006 17:28:02 -0000
> From: "dwimmer_laik" <dwimmer_laik@yahoo.com>
> Subject: Re: points and various voting matters
>
>> First, I think some people find it hard to write long reviews and so
>> feel that even stories they really like they can't give them the
>> points
>> they're worth. I suggest that we have each review cap off at a lower
>> level (say, five points instead of ten). This effectively means that
>> those 1- and 2-point reviews have more weight. One such spread would
>> be:
>>
>> 1-50 1 point
>> 51-250 2 point
>> 251-500 3 point
>> 501-1000 4 point
>> 1001+ 5 point
>
>
> Someone mentioned that this would help people write reviews more
> quickly (a good thing!) instead of having to worry over a review to
> make it the points value they thought the story was worth. I think for
> me that's probably the strongest argument for making it easier to
> reach certain points values, but capping it at five to me is less
> appealing. A 1000 characters is significantly more than 501, yet it
> gives the same points value.

You're absolutely right about there being a pretty big difference
between the two. The way I broke down the points probably isn't the
best way, and we could certainly rework that maybe a point every 250
characters or some such thing. Besides wanting to simplify things and
make reviewing easier, I heard from a few reviewer who said they felt
bad writing reviews when they could only do short ones because they
were "only" worth one point or two points, and this shocked me quite a
bit. Every point can make a difference, and I don't want people to feel
like short reviews aren't worth much. I thought if there were less
points per review, then those shorter reviews would feel like they were
wroth more - hence the push to cut back on the number of points.

> Also, I'm not sure why the 'step' has to be so short between points
> levels one and two, 2 and a half times as much as the first step in
> the next two categories, and then jump up by nearly the same amount
> between levels four and five as the move from level 1 to level 2 in
> the third. One of the nice things about this year's awards was that
> there were no such skips in the points structure--nice even,
> predictable intervals were, I thought, a good improvement on the first
> year's table, which was missing some points levels actually.
>
> Is the idea that the scoring levels are based on the average char
> counts people turned in? What's the logic behind this? Because what
> this seems to me to do is to make it easier to move out of level one,
> but then it becomes harder to move out of any given level at any point
> after that.
>
>

I'm not particularly attached to those points I proposed above. I'd be
just as happy with, say:

0-250 chars 1 pt
251-500 chars 2 pts
501-750 chars 3 pts
751-1000 chars 4 pts
1001+ chars 5 pts

There really *wasn't* any rhyme or reason to the point levels, except
that they were round numbers.

<snip>

> *At a guess* (please keep in mind that math is not my strong point,
> here), there will be a lot more ties to break based on absolute
> character counts because it'll be harder to vote proportionally--there
> just won't be enough differentiation within the higher points
> categories to make that possible.
>

I think you're right about this (and this is just my guess, too). Would
having a more equal distribution of points like what I laid out above
be any better? I think it would likely still be lower point totals
overall so more ties on the number of points, so you'll have more ties
if each vote is worth less point anyway.

A good way to avoid all of this would be to go with Rabidsamfan's
suggestion to do away with points and just go with character counts up
to a certain cap. 1000 characters or whatever. The more I think about
it, the more I like that idea - though I'm not so sold on it as to
insist on it.

>> Another way to address this is to assign honourable mentions based on
>> the number of entries per category. For example, let's say we want to
>> have the top half of stories receive a place award or an honourable
>> mention. (Not that out-of-line when you consider 60% of the stories in
>> a five-story category get an award.) Then we could just give
>> honourable
>> mentions to the top stories below the places until we reach this
>> point.
>
>
> I rather like this more proportional system. Depending on how things
> go in fandom, the further we get from movie years, we may have fewer
> entries or else more reviewers as word gets around and (ideally) more
> people participate as reviewers. So this would avoid us being in a
> situation where maybe very few stories make it to the threshold points
> level, or else where nearly every story makes it that far. There'd be
> no need to try and guess ahead of time or make a controversial
> retroactive decision about that points threshold after the voting is
> over. It'd be a simple decision, made once and for all, about a
> percentage of stories that would get awards, and then a mathematical
> formula would determine the actual number of honorable mentions.
>

Thanks for articulating this so well. These were a lot of the reasons I
like this option so much, but you've presented them more clearly than
I've been able to. I whole-heartedly agree with what you've said.

>> One more thing: the author awards. This one may be more strongly
>> dictated by technical concerns. Would it be a good idea to have an
>> easy
>> way to enter the same author vote for all the categories where the
>> author has works entered. For instance, this year I had to enter votes
>> for Dreamflower for every one of the categories she had stories
>> entered
>> in, but the votes were the same. It would have been much easier to be
>> able to go to vote for any one category, click a check-box of some
>> sort
>> and have this vote automatically entered in the other categories.
>
> I'd support anything that makes this an easier task. I ended up having
> to find all the author's stories that I had read, and then divide them
> into stories I thought could be mentioned together as contributing
> towards a coherent review of the author. So I might mention stories in
> Horror, Drama, and Action/Adventure to write an author review, then
> enter that review in three different categories. I'd then have a
> different review written out for the author using her/his stories from
> LOTR, Silm, and Humor.
>
> So for me, having a single author review, where I could use all the
> stories at once, without having to figure out how to carve them up and
> write two reviews without being repetitive, would be a significant
> improvement.
>

Well, technically you can enter the same author review for every
category an author is entered in. But this does get repetitive when
trying to read through author reviews, I agree. It would be nice to not
display duplicate reviews; I'm not sure how we'd program this.

Short of that, the only way I can see this suggestion working would be
to have all the authors in a single category - and I'm not crazy about
that idea.

>> Barring that, can you guys think of any ways that we could make author
>> voting run more smoothly?
>
> Being able to see all the author's stories and categories on a single
> page whenever one goes to write an author review would also be a huge
> help. I got very tired of having to go dig through my review files to
> find the person's stories, and determine categories and subcategories.
> The filters were very problematic, because they weren't independent of
> each other, but one governed the other. So:
>
> Author: Name
> Category: Romance
>
> Would give you Romance, plus subcats, iirc, but then *because* it
> *was* Romance as the main category, when you tried to select a
> different one, you could only get the Romance subcats. And that would
> result in an impossible filter combination like:
>
> Author: Name
> Category: Action/Adventure
> Subcat options: Only romance subcats
>
> You'd get a blank page, have to clear all filters, and then start
> again instead of being able to use the filters to help you search out
> the author's stories.
>

I hadn't notice that but agree that it would be problematic. For me
personally I always hit the "reset" button and dealt with the delay.
But then I have DSL and never had problems with the server; I realise
this would be much harder for some people.

Anthony, would it be possible for the drop-down menu of the
subcategories to include a list like:

*****
Diplay all applicable sub-categories.
---
Display all Books/Time: Gap-Filler sub-categories.
Drama
Drabble
Hobbits
War of the Ring
---
Display all Books/Time: Post-Ring War Sub-Categories.
Aman

[etc.]

*****

So for example if the category was Romance and...
a. You selected "Display all applicable sub-categories." --> The page
would display all of the stories in Romance.
b. You selected "Display all Genres: Romance sub-categories." --> The
page would display all of the stories in Romance.
c. You selected a particular subcategory under "Display all Genres:
Romance subcategories." --> The page would display those stories in
that sub-category of romance.
d. You selected something else. --> The page would be blank.

I think this would address Dwim's concerns (I could be wrong, of
course).

>
> Topic we possibly have forgotten:
>
> How do we treat quotes in reviews? What about quotes from sources that
> are not from the fanfic under review, but which are either from
> Tolkien's own writing or else from a different published author? Did
> we decide how to treat these already? I'm afraid I can't recall.
>

I think we decided that all reviews, regardless of source or length,
had to be within blockquote tags. You can include them but they won't
count for scoring purposes. Am I misremembering this?

> Thanks for all your work, Marta. This has definitely dragged out a
> bit, but you've persevered and kept pushing it forward.
>

Thanks. I don't mind things going on if there's still stuff to discuss,
though we do need to wrap up things that might require coding fairly
soon. I just don't want this to become a burden to anyone.

Marta

> > Someone mentioned that this would help people write reviews more
> > quickly (a good thing!) instead of having to worry over a review to
> > make it the points value they thought the story was worth. I think for
> > me that's probably the strongest argument for making it easier to
> > reach certain points values, but capping it at five to me is less
> > appealing. A 1000 characters is significantly more than 501, yet it
> > gives the same points value.
>
>
> You're absolutely right about there being a pretty big difference
> between the two. The way I broke down the points probably isn't the
> best way, and we could certainly rework that maybe a point every 250
> characters or some such thing. Besides wanting to simplify things and
> make reviewing easier, I heard from a few reviewer who said they felt
> bad writing reviews when they could only do short ones because they
> were "only" worth one point or two points, and this shocked me quite a
> bit. Every point can make a difference, and I don't want people to feel
> like short reviews aren't worth much.

I think we're too used to watching our votes shrivel in ballot boxes
to be happy with writing 1-2 pointers for stories we like is the
problem. :-/ Waiting on a solution to that sort of thing, however, may
be problematic...

I could see revising the thresholds to make it easier to score
someone's story more points with fewer words--it might look like an
exponentional curve, where the points on the graph start off closer
together and then get further and further apart on the curve. So it'd
be easiest to get from points level one to points level two, a little
harder to move from two to three, a little harder to go to four, up
through around six. Then maybe you'd start seeing bigger gaps, where
it's harder to write a seven pointer, harder to move from a seven
point review to an eight point review, and so on.

> There really *wasn't* any rhyme or reason to the point levels, except
> that they were round numbers.

Ah. Ok, makes sense.

> A good way to avoid all of this would be to go with Rabidsamfan's
> suggestion to do away with points and just go with character counts up
> to a certain cap. 1000 characters or whatever. The more I think about
> it, the more I like that idea - though I'm not so sold on it as to
> insist on it.

I'd go for this. It'd make things easier, I think, so long as there
was a cap in place. I'm happy with 1000 being the cap--that seems
reasonable to me and appears to have worked well as a cap the past two
years.

> >
> > I rather like this more proportional system. <snip>
>
> Thanks for articulating this so well. These were a lot of the reasons I
> like this option so much, but you've presented them more clearly than
> I've been able to. I whole-heartedly agree with what you've said.

You're welcome.

<snip>

> > So for me, having a single author review, where I could use all the
> > stories at once, without having to figure out how to carve them up and
> > write two reviews without being repetitive, would be a significant
> > improvement.
> >
>
> Well, technically you can enter the same author review for every
> category an author is entered in. But this does get repetitive when
> trying to read through author reviews, I agree. It would be nice to not
> display duplicate reviews; I'm not sure how we'd program this.
>
> Short of that, the only way I can see this suggestion working would be
> to have all the authors in a single category - and I'm not crazy about
> that idea.

I just posted on Author Awards, so you can see how I'd try to modify
them to suit the concerns raised on the list. The Author Awards are
undoubtedly the part of the MEFAs that works least well, since it's
just plain confusing and results in weird mismatches between the
category in which a person wins an Author Awards and the content of
the actual review.

> > Topic we possibly have forgotten:
> >
> > How do we treat quotes in reviews? What about quotes from sources that
> > are not from the fanfic under review, but which are either from
> > Tolkien's own writing or else from a different published author? Did
> > we decide how to treat these already? I'm afraid I can't recall.
> >
>
> I think we decided that all reviews, regardless of source or length,
> had to be within blockquote tags. You can include them but they won't
> count for scoring purposes. Am I misremembering this?

I'm not sure. That's why I asked! I'm happy if all quotes, regardless
of their source, get treated the same, so long as we *can* quote
freely, and so long as it's made clear in the rules how to handle them.

Dwim

> Message: 1
> Date: Tue, 03 Jan 2006 10:08:35 -0000
> From: "dwimmer_laik" <dwimmer_laik@yahoo.com>
> Subject: Re: points and various voting matters
>
> --- In MEFAwards@yahoogroups.com, "Kathy" <inkling-tcbs@s...> wrote:
>>
>> --- In MEFAwards@yahoogroups.com, "Laura" <thunderalaura@j...> wrote:
>>> <snip>
>>> In the end, I suppose it comes down to what our priorities are. In
>> my opinion, a higher point cap (eg: the current 10 points) encourages
>> longer reviews and rewards stories that deserve said reviews. A lower
>> point cap (eg: the proposed 5 points) might encourage more reviews,
>> but on average, they would be shorter as the competitive advantage
>> gained by long reviews would be lost.
>> latter.
>
>> I heartily agree that review quality is more important than quantity.
>> But are short reviews necessarily of lesser quality than
>> long ones?
>
> Two things seem to be going on here:
>
> Firstly:
>
> On the one hand, against Thundera's position, the question is raised:
> on what basis do we think that longer reviews are qualitatively
> superior to shorter ones, such that they merit being recognized by a
> wider points spread? Valid question, but I'd point out that just
> raising that as a question does not entitle us to conclude that in
> fact, shorter reviews are either superior to longer ones *or* that
> they are equal in value, and so it's a good idea to alter the points
> scale to give them more weight.
>

Very good question. As I thought about it, it occurred to me that while
longer reviews may not be better than shorter reviews, they at least
have the potential to cover more. Let's say that in 150 points I can do
a good job explaining why I like a certain element of a story - the
plot, canonicity, pacing, characterisation, theme, whatever. So in a
2-3 point review i can cover one or two of these areas. In a 7-point
review I can probably manage 4-5 if I want to. It's entirely possible
that I will have a lot of padding, but the *potential* is at least
there. Shorter reviews just can't cover as much material as a longer
review can (note I say can, not will).

Also I think for most people, it will take longer to write a longer
review than a shorter one. I may be able to write a 7-point review
quicker than you can write a 3-pointer, but I will spend longer on the
average 7-point review than I would to write a 3-point review. Simply
because there's less to write in the latter. So whether the longer
review is intrinsically better, we give more points to recognise the
greater investment of time.

> On the other hand, against the position that we should change the
> points scale to privilege shorter reviews, the underlying assumption
> that seems to be at work is that a long review is likely to be padded
> and so 'fake' in some sense, based on the fact that the questioner
> felt like s/he was padding his/her reviews when s/he tried to do
> longer ones. That may be very true sometimes and the rules in fact
> encourage this, or at least I recall that when questions came up about
> not having anything more to say but wanting to give more points to a
> story, the advice was: Stretch it out. Find a way to say more, even if
> it is fluffifying the review.
>
> However, that isn't true of every long review, and personal experience
> doesn't seem terribly helpful here. Personally, I found that taking
> the time to analyze a piece and saying to myself, "I think this is a
> ten-pointer and need to write about that much," made me see things
> about the story I wouldn't have been able to articulate on a first
> reaction. I've also found that in general, a more thought-out response
> is nicer than one that seems to be an immediate outpouring of raw
> reaction.
>

For the record, I know I proposed a point system that might favor
shorter reviews. But this isn't because I think longer reviews are
inflated. It's a very pragmatic concern - I've never heard any
complaints that longer reviews are too much work ffor the number of
points, but I have heard lots of complaints that people don't feel like
short reviews are worthwhile. If there's something in the point system
that can be changed to help solve this problem that's a change I'm
willing to consider - because that's where I saw the problem this year.

> Secondly, and I think more importantly, the issue has been almost
> immediately transformed from "What would be the advantages and
> disadvantages of changing the points thresholds *and* lowering the
> points cap by half?" into "How long does a review have to be before it
> is good?" We can't answer the second question (other than that they
> are as long as they need to be to say what you think you need to say
> about the story's good qualities, whether that's 1 word or 2,000+
> characters), but the first one strikes me as answerable without having
> to go through the quality argument.
>

I agree here - I'm a bit of a pragmatist here, and which type is
"better" is really immaterial to what the point limits would be isn't
really that material. I want people who tend to write reviews of all
lengths to feel like their reviewing is valued and important at the
MEFAs.

Marta

I think there's a simple question that if we can answer it, will solve the
point issue:

How can we change the points (yes points, not straight character count) to
give more advantage to smaller reviews and less to longer ones? Ah this
brings me back to some history. Help me with the memory. In 2004, I
borrowed the scale directly from ASC. I don't think the point spread was
evenly spaced. Maybe they did it on purpose. We found it confusing and
straightened it out in 2005, right?

Maybe it was weighted when it was uneven. Maybe the spaces between 1, 2, and
3 points should be lower (closer together) than the spacing between 8, 9,
and 10.

Straight character count has it's advantages but it doesn't give any weight
anywhere. Which could be good, too. But if we want to let people of
smaller reviews feel they have an impact (which I think they do
mathmatically anyway because there are a lot more short reviews written than
long ones, thus they add volume.) weighting them would do that better.

I think for people who write long reviews (which if I can manage I will for
a story I love even if it goes beyond the cap), we'll write them because
they are what we want to say not just because we're trying to reach a point.
If we feel a story is good enough to get a 10, we're going to expound on it
because we love it, not just because we're thinking, 10 more characters will
bring it to the cap! So if I write a long one, I won't feel bad that
pointing weights to the smaller.

Thoughts?

--Ainaechoiriel
MEFA Admin and Founder

"This evil cannot be concealed by the power of the Elves," Elrond said, "for
it is Windows-compatible, and freeware at that." --H.F.

http://gabrielle.sytes.net/mefa The Middle-Earth Fanfiction Awards

Blog: http://www.ainaechoiriel.blogspot.com

> -----Original Message-----
> From: MEFAwards@yahoogroups.com
> [mailto:MEFAwards@yahoogroups.com] On Behalf Of Marta Layton
> Sent: Friday, January 06, 2006 11:26 PM
> To: MEFAwards@yahoogroups.com
> Subject: Re: [MEFAwards] Re: points and various voting matters (Dwim)
>
> > Message: 1
> > Date: Tue, 03 Jan 2006 10:08:35 -0000
> > From: "dwimmer_laik" <dwimmer_laik@yahoo.com>
> > Subject: Re: points and various voting matters
> >
> > --- In MEFAwards@yahoogroups.com, "Kathy" <inkling-tcbs@s...> wrote:
> >>
> >> --- In MEFAwards@yahoogroups.com, "Laura"
> <thunderalaura@j...> wrote:
> >>> <snip>
> >>> In the end, I suppose it comes down to what our priorities are. In
> >> my opinion, a higher point cap (eg: the current 10 points)
> encourages
> >> longer reviews and rewards stories that deserve said
> reviews. A lower
> >> point cap (eg: the proposed 5 points) might encourage more
> reviews,
> >> but on average, they would be shorter as the competitive advantage
> >> gained by long reviews would be lost.
> >> latter.
> >
> >> I heartily agree that review quality is more important
> than quantity.
> >> But are short reviews necessarily of lesser quality than long ones?
> >
> > Two things seem to be going on here:
> >
> > Firstly:
> >
> > On the one hand, against Thundera's position, the question
> is raised:
> > on what basis do we think that longer reviews are qualitatively
> > superior to shorter ones, such that they merit being
> recognized by a
> > wider points spread? Valid question, but I'd point out that just
> > raising that as a question does not entitle us to conclude that in
> > fact, shorter reviews are either superior to longer ones *or* that
> > they are equal in value, and so it's a good idea to alter
> the points
> > scale to give them more weight.
> >
>
> Very good question. As I thought about it, it occurred to me
> that while longer reviews may not be better than shorter
> reviews, they at least have the potential to cover more.
> Let's say that in 150 points I can do a good job explaining
> why I like a certain element of a story - the plot,
> canonicity, pacing, characterisation, theme, whatever. So in a
> 2-3 point review i can cover one or two of these areas. In a
> 7-point review I can probably manage 4-5 if I want to. It's
> entirely possible that I will have a lot of padding, but the
> *potential* is at least there. Shorter reviews just can't
> cover as much material as a longer review can (note I say
> can, not will).
>
> Also I think for most people, it will take longer to write a
> longer review than a shorter one. I may be able to write a
> 7-point review quicker than you can write a 3-pointer, but I
> will spend longer on the average 7-point review than I would
> to write a 3-point review. Simply because there's less to
> write in the latter. So whether the longer review is
> intrinsically better, we give more points to recognise the
> greater investment of time.
>
> > On the other hand, against the position that we should change the
> > points scale to privilege shorter reviews, the underlying
> assumption
> > that seems to be at work is that a long review is likely to
> be padded
> > and so 'fake' in some sense, based on the fact that the questioner
> > felt like s/he was padding his/her reviews when s/he tried to do
> > longer ones. That may be very true sometimes and the rules in fact
> > encourage this, or at least I recall that when questions
> came up about
> > not having anything more to say but wanting to give more
> points to a
> > story, the advice was: Stretch it out. Find a way to say
> more, even if
> > it is fluffifying the review.
> >
> > However, that isn't true of every long review, and personal
> experience
> > doesn't seem terribly helpful here. Personally, I found that taking
> > the time to analyze a piece and saying to myself, "I think
> this is a
> > ten-pointer and need to write about that much," made me see things
> > about the story I wouldn't have been able to articulate on a first
> > reaction. I've also found that in general, a more
> thought-out response
> > is nicer than one that seems to be an immediate outpouring of raw
> > reaction.
> >
>
> For the record, I know I proposed a point system that might
> favor shorter reviews. But this isn't because I think longer
> reviews are inflated. It's a very pragmatic concern - I've
> never heard any complaints that longer reviews are too much
> work ffor the number of points, but I have heard lots of
> complaints that people don't feel like short reviews are
> worthwhile. If there's something in the point system that can
> be changed to help solve this problem that's a change I'm
> willing to consider - because that's where I saw the problem
> this year.
>
> > Secondly, and I think more importantly, the issue has been almost
> > immediately transformed from "What would be the advantages and
> > disadvantages of changing the points thresholds *and* lowering the
> > points cap by half?" into "How long does a review have to
> be before it
> > is good?" We can't answer the second question (other than that they
> > are as long as they need to be to say what you think you
> need to say
> > about the story's good qualities, whether that's 1 word or 2,000+
> > characters), but the first one strikes me as answerable
> without having
> > to go through the quality argument.
> >
>
> I agree here - I'm a bit of a pragmatist here, and which type
> is "better" is really immaterial to what the point limits
> would be isn't really that material. I want people who tend
> to write reviews of all lengths to feel like their reviewing
> is valued and important at the MEFAs.
>
> Marta
>
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>

On 1/7/06, Ainaechoiriel <mefaadmin@earthlink.net> wrote:

>
> I think there's a simple question that if we can answer it, will solve the
> point issue:
>
> How can we change the points (yes points, not straight character count) to
> give more advantage to smaller reviews and less to longer ones? Ah this
> brings me back to some history. Help me with the memory. In 2004, I
> borrowed the scale directly from ASC. I don't think the point spread was
> evenly spaced. Maybe they did it on purpose. We found it confusing and
> straightened it out in 2005, right?
>
> Maybe it was weighted when it was uneven. Maybe the spaces between 1, 2,
> and
> 3 points should be lower (closer together) than the spacing between 8, 9,
> and 10.
>
> Straight character count has it's advantages but it doesn't give any
> weight
> anywhere. Which could be good, too. But if we want to let people of
> smaller reviews feel they have an impact (which I think they do
> mathmatically anyway because there are a lot more short reviews written
> than
> long ones, thus they add volume.) weighting them would do that better.
>
> I think for people who write long reviews (which if I can manage I will
> for
> a story I love even if it goes beyond the cap), we'll write them because
> they are what we want to say not just because we're trying to reach a
> point.
> If we feel a story is good enough to get a 10, we're going to expound on
> it
> because we love it, not just because we're thinking, 10 more characters
> will
> bring it to the cap! So if I write a long one, I won't feel bad that
> pointing weights to the smaller.
>
> Thoughts?
> --Ainaechoiriel
> MEFA Admin and Founder

This comes close to Anthony's suggestion that we use weighted character
counts, where the first 100 characters count 100%, the next batch count 75%,
etc. and over 1000 you continue with 5% ad infinitum.

I very much agree that the solution to making shorter reviews more
significant is to make it easier at that end of the scale to give different
levels of approval.

i.e. a scale which looks like

1 point -- 1 to 75 characters
2 points 76 to 150 characters
3 points 151 to 250 characters
4 points 251 to 350 characters
5 points 351 to 475 characters...etc.

is going to be easier for succinct reviewers to give reviews that have a
range of point values than one which spaces the points at 100 or 200
character intervals.

I also still very much prefer a ten point spread over a 5 point spread, if
we don't use weighted character counts. I know I'm in the minority there,
but as an author it was very satisfying to get a review over five points
value in among the others, and as a reviewer, I liked having the broad range
to express my levels of satisfaction.


Is it possible to use the character counts from this year and apply the
different scale proposals to them, just to see what happens to the results?


[Non-text portions of this message have been removed]