| Pages in topic: < [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16] > | KudoZ: should "rubbish" help be appreciated? Thread poster: IanW (X)
|
|---|
| A "radioactive" ranking formula based on points AND ratios | Aug 2, 2006 |
Nikki is right: It is increasingly difficult for newcomers, especially in crowded pairs, to ever reach the first pages in a search.
Food for serious thought (also and especially for Site Staff):
Good old chaps may decide to return to Kudoz, and quality and participation would increase if ranking would be done on the basis of a mixed formula of points and ratios (after a minimum number of answers and/or points), and if the weight of the factors in such a formula would be... See more Nikki is right: It is increasingly difficult for newcomers, especially in crowded pairs, to ever reach the first pages in a search.
Food for serious thought (also and especially for Site Staff):
Good old chaps may decide to return to Kudoz, and quality and participation would increase if ranking would be done on the basis of a mixed formula of points and ratios (after a minimum number of answers and/or points), and if the weight of the factors in such a formula would be "radioactive", i.e. decaying in value as time passes.
Maybe this is just a 40-ºC-ambient-temp idea -:)
Pablo ▲ Collapse
| | | | Kirill Semenov 
Ukraine
Local time: 11:20
Member (2004)
English to Russian
+ ...
| No personal goals here | Aug 2, 2006 |
Charlie Bavington wrote: And in case you're wondering, while the fact that I have 1900-odd points is publicly available information, I've got an RR of over 40% in Fr-Eng in general and between 50 and 72% in my specialty fields, so please don't think I'm pointing these things out because I fear I may suffer somehow if the RR does get used 
Sure, I don't think many of those who support or object the idea cares much about their own position at the Top-X. Myself, I know perfectly, that I will drop out of the list and fall down to a much lower position if the idea is (ever) implemented. It's not about my own benefits, it's just an idea to make the system work fair and just for everyone not for me personally or you or anyone in particular, that's all. It may be discussed and polished and revised many times, still I think the ProZ' future are ideas like `Good for everyone', not `Good for me, personally'.
| | | | df49f (X)
France
Local time: 10:20
| reliability ratio and biases | Aug 2, 2006 |
I also like the idea of a reliability ratio, showing also the total counts in addition to a %...
... but let me also play devil's advocate assistant (if Charlie allows me...)
Based on current rules and practices, the ratio would end up being biased and therefore misleading due to two separate reasons but compounding each other:
1) First bias: How many times have we seen correct answers (ours or not ... See more I also like the idea of a reliability ratio, showing also the total counts in addition to a %...
... but let me also play devil's advocate assistant (if Charlie allows me...)
Based on current rules and practices, the ratio would end up being biased and therefore misleading due to two separate reasons but compounding each other:
1) First bias: How many times have we seen correct answers (ours or not - with several agrees or even without any) be discarded in favor of perfectly ludicrous ones? The record however shows an answer/points awarded to the wrong answerer and NOT awarded to the right one (even if we manage to get a Mod's attention to amend the gloss entry, that record isn't modified) - this gives a minus and a plus that add up not to zero but to a double problem: one unjustified gain (add one to numerator/chosen answer count) for wrong answerer and one unfair loss for right answerer (add one to denominator/total answer count).
2) Second bias: if our answer is NOT chosen (whether rightly or wrongly), we (us NOT point grabbers) leave our answer be, it remains in the archives for the benefit of future discussion, but it counts as a NOT CHOSEN answer (ADD ONE to denominator/total count-> worsen the ratio). Yet, I have noticed that more and more ridiculous, wrong and not chosen answers (particularly if they've received several peer disagrees shining like red lights) suddendly disappear from the screens... and from what I've been able to check, hidden answers also disappear from answerer's list of answers, and therefore are NOT counted as part of the total answer count (DEDUCT ONE from denominator/total answer count -> and of course improve the ratio...).
And both types of bias coexist and co-occur.
Question is: how to offset or eliminate them? several ways probably, I've only thought of two for the time being:
1. Bias 1: when Glossary is amended by Mods to modify the inappropriate answer and re-enter the appropriate one (or by others who might in the future become authorized to clean up the KOG - and if staff is serious about clean-up then such corrections would happen more often hopefully) , this could automatically trigger a switch of award in the record to re-select the right answerer and un-select the wrong answerer - maybe??
2. Bias 2: make it impossible for people to delete their answers if they're received at least/more than 2 (or 3?) disagrees - maybe?
Not sure if I've managed to make myself clear...
Though I personally don't care about points or ranking in the directory (have plenty of work and clients in the real world & absolutely no need to "look good" on the site and don't even want it known that I visit this site!), a lot of people seem to care a lot and their concern shouldn't be neglected. So this great idea of reliability ratio should be carefully thought out to make sure it is truly reliable and therefore truly helpful to those who need it, and cannot be fooled with or twisted around.
dominique
PS: one last thought: to make everybody happy or make sure noboby is unhappy, this RR should probably be statistics displayed IN ADDITION to and NOT in replacement of existing point/answer counts
...OK one more thought: those who cannot be made happy will of course be the serial askers who never answer/help and the serial answerers quick with "rubbish" answers... but it seems to be the price to pay if the site staff is truly sincere and serious about improving all-around quality and reputation (maybe then I wouldn't even mind if it became known that I visit it )
hmmm... one more: one incidental benefit would be to improve the quality of the KOG (in addition to other clean-up measures) - in fact, initial quality of entries being improved, there would be less need for subsequent clean-up)
[Edited at 2006-08-02 17:04] ▲ Collapse
| | | | Kirill Semenov
Ukraine
Local time: 11:20
Member (2004)
English to Russian
+ ...
| Think large numbers | Aug 2, 2006 |
df49f wrote:
1) First bias: How many times have we seen correct answers (ours or not - with several agrees or even without any) be discarded in favor of perfectly ludicrous ones?
1) Any our opinion of a `correct answer' is only ours. The KudoZ system proposes to select the most helpful answer, not the correct one (which, by the way, may never exist). So, the reliability ratio is in fact `being most helpful ratio'. Also, in the long run - think big numbers! - the percentage of wrong decisions is neglectable. Last, the chance to be not selected even with the great and obviously good answer is small - and equal for all of us. Meaning everybody will suffer the natural bias, both you or me or anyone. And everyone will suffer in equal degree. Sometimes you fail, sometimes you gain. For many answers, it does not affect the final result much. My observations show that after 1000 answers every answerer tends to reach his or her own upper ratio, with every new answer changing the 4th, 3rd or 2nd digit after after point, for percentage ratio. It's neglectable.
2) I can't say anything about your second point. I'm not sure I've understood what you're talking about.
[Edited at 2006-08-02 17:18]
[Edited at 2006-08-02 17:44]
| | |
|
|
|
Nikki Graham
United Kingdom
Local time: 09:20
Spanish to English
| Some thoughts | Aug 2, 2006 |
df49f wrote: 1. Bias 1: when Glossary is amended by Mods to modify the inappropriate answer and re-enter the appropriate one (or by others who might in the future become authorized to clean up the KOG - and if staff is serious about clean-up then such corrections would happen more often hopefully) , this could automatically trigger a switch of award in the record to re-select the right answerer and un-select the wrong answerer - maybe?? 2. Bias 2: make it impossible for people to delete their answers if they're received at least/more than 2 (or 3?) disagrees - maybe? dominique PS: one last thought: to make everybody happy or make sure noboby is unhappy , this RR should probably be statistics displayed IN ADDITION to and NOT in replacement of existing point/answer counts
Hi Dominique,
I think you've made some excellent points. In general, however, we already have the situation of incorrect answers being chosen, etc., which affect our points totals, and as Kirill pointed out (he seems to know lots more about all this statistical stuff than me), it wouldn't really affect the issue after loads of questions have been answered.
1. I agree. Any glossary clean-up should involve re-awarding of the points to the answer finally chosen as correct, with first correct answer in chosen if there are two or more the same.
2. I think people should be allowed to hide their answers if they want to (we can all make embarrassing mistakes), but the negative effect on the reliability ratio should still remain. In other words, answer not chosen and/or answer not chosen + X disagrees.
P.S. OK. Both the statistics and the points could be displayed, so we can all compare ourselves in different ways. But I think the ranking in the freelancer search should be based on reliability ratios. Otherwise, there wouldn't really be much change.
| | | | | regarding df49f´s comments | Aug 2, 2006 |
Kirill is right regarding 1st bias, it affects all in the same way, so not unjust.
As to the 2nd Bias, hiding is good for KOG, but only answers hidden before grading should not count for ratio.
Pablo
| | | | df49f (X)
France
Local time: 10:20
| statistics... | Aug 2, 2006 |
Kirill Semenov wrote: df49f wrote:
1) First bias: How many times have we seen correct answers (ours or not - with several agrees or even without any) be discarded in favor of perfectly ludicrous ones?
1) Any our opinion of a `correct answer' is only ours. The KudoZ system proposes to select the most helpful answer, not the correct one (which, by the way, may never exist). So, the reliability ratio is in fact `being most helpful ratio'. Also, in the long run - think big numbers! - the percentage of wrong decisions is neglactable. Last, the chance to be not selected even with the great and obviously good answer is small - and equal for all of us. Meaning everybody will suffer the natural bias, bot only you or me or anyone. And everyone will suffer in equal degree. Sometimes you fail, sometimes you gain. For many answers, it does not affect the final result much. My observations show that after 1000 answers every answerer tends to reach his or her own upper ratio, with every new answer changing the 4th, 3rd or 2nd digit after after point, for percentage ratio. It's neglectable. 2) I can't say anything about your second point. I'm not sure I've understood what you're talking about.
1) you're right, Kirill, that with large numbers, things tend to even themselves out statistically.
2) also true that the occurrences of "not chosen" yet correct/most helpful answers discarded to the benefit of "incorrect" ones end up statistically been spread out and shared by all of us, and thus offset each other in our personal counts.
3) I was talking indeed of correct in the sense of appropriate and "most helpful" as you rightly say. However, in contrast, when I speak of "incorrect" answers, I mean really that: i.e. the "rubbish" kind of answers that started this thread - yet still get chosen and entered as-is in the glossary and turn the glossary into a ridiculously laughable and useless tool.
Lastly on my item 2: sorry if I wasn't very clear...this is about people hiding their answer after it has received several peer disagrees and another has been chosen (a practice I had not really noticed before but which seems to be spreading lately in my language pairs FR/EN&EN/FR - or at least that I have been noticing more)- these hidden answers are no longer counted in the "total questions answered", thereby reducing the total answer count, therefore reducing the denominator of the fraction and artificially improving the ratio we are discussing - which would perhaps also even itself out, except that most of us DO NOT hide (or very rarely if we suddenly realize we were totally out in left field, which can happen) our not-chosen answers which are therefore counted in our own total answer count - thereby creating another bias to that RR.
df
[Edited at 2006-08-02 17:54]
| | | | Nikki Graham
United Kingdom
Local time: 09:20
Spanish to English
| Not in favour of no. 2 | Aug 2, 2006 |
Pablo Grosschmid wrote:
As to the 2nd Bias, hiding is good for KOG, but only answers hidden before grading should not count for ratio.
Interesting, Pablo. We differ slightly here. The problem with allowing people to post an answer, which collects a few disagrees and is then hidden, is that this will not stop the point grabbers from continuing their work. They can try an answer out for size, and if it doesn't fit, just get rid of it with no effect on their ratios at all.
| | |
|
|
|
| ranking, points, RR .... | Aug 2, 2006 |
Nikki Graham wrote:
P.S. OK. Both the statistics and the points could be displayed, so we can all compare ourselves in different ways. But I think the ranking in the freelancer search should be based on reliability ratios. Otherwise, there wouldn't really be much change.
IMHO, a single ranking should reflect quality (RR) and quantity (nr. of answers), equally weighted. And also the time factor should be incorporated, with progressive "decay", for example a half-life of one year, with monthly recalculation (see my former post).
Kirill, could you devise a formula for that?
Best
Pablo
| | | | df49f (X)
France
Local time: 10:20
| hidden answers and total count | Aug 2, 2006 |
Nikki Graham wrote: df49f wrote: 1. Bias 1: when Glossary is amended by Mods to modify the inappropriate answer and re-enter the appropriate one (or by others who might in the future become authorized to clean up the KOG - and if staff is serious about clean-up then such corrections would happen more often hopefully) , this could automatically trigger a switch of award in the record to re-select the right answerer and un-select the wrong answerer - maybe?? 2. Bias 2: make it impossible for people to delete their answers if they're received at least/more than 2 (or 3?) disagrees - maybe? dominique PS: one last thought: to make everybody happy or make sure noboby is unhappy , this RR should probably be statistics displayed IN ADDITION to and NOT in replacement of existing point/answer counts Hi Dominique, I think you've made some excellent points. In general, however, we already have the situation of incorrect answers being chosen, etc., which affect our points totals, and as Kirill pointed out (he seems to know lots more about all this statistical stuff than me), it wouldn't really affect the issue after loads of questions have been answered. 1. I agree. Any glossary clean-up should involve re-awarding of the points to the answer finally chosen as correct, with first correct answer in chosen if there are two or more the same. 2. I think people should be allowed to hide their answers if they want to (we can all make embarrassing mistakes), but the negative effect on the reliability ratio should still remain. In other words, answer not chosen and/or answer not chosen + X disagrees. P.S. OK. Both the statistics and the points could be displayed, so we can all compare ourselves in different ways. But I think the ranking in the freelancer search should be based on reliability ratios. Otherwise, there wouldn't really be much change.
hi Nikki
On statistics, you're right and so is Kirill (I already answered in a previous post)
On Point 2): you're absolutely right that the "hide answer" option needs to remain (I've used it also after realizing I had a typo in my answer or that I had entirely misread the question - which does happen! no, no, we are NOT perfect) - Your solution combined with Pablo's might solve the dilemna: i.e. hidden answers hidden AFTER grading should continue to be included in the total answer count.
On your/PS on using a RR as a basis for directory ranking: I don't really have an opinion since that ranking is basically meaningless for me - but your comment makes a lot of sense.
dominique
[Edited at 2006-08-02 17:53]
| | | | Kirill Semenov
Ukraine
Local time: 11:20
Member (2004)
English to Russian
+ ...
| Hiding answers | Aug 2, 2006 |
df49f wrote:
2) Second bias:
Now I probably see - no, an answer is an answer, and hiding your answer should not help. I mean it still must be counted. Everyone is supposed to think twice before posting an answer, otherwise be responsible for posting it. It should be included in the ratio, always.
| | | | df49f (X)
France
Local time: 10:20
Kirill Semenov wrote: df49f wrote:
2) Second bias: Now I probably see - no, an answer is an answer, and hiding your answer should not help. I mean it still must be counted. Everyone is supposed to think twice before posting an answer, otherwise be responsible for posting it. It should be included in the ratio, always.
Yes Kirill, that's precisely what I was trying to say!
(see Nikki's and Pablo's comments on that plus my own reply to them for more discussion on this issue).
I brought up the subject because I've been noticing more and more such answers "disappearing" after grading and/or after receiving several peer disagrees - I ended up purposely checking the list of "questions answered" from the profile of that "disappeared" answerer and discovered that the hidden question was no longer there - and deduced that it is no longer counted in the total (or is it??)
dominique
[Edited at 2006-08-02 18:22]
| | |
|
|
|
Kirill Semenov
Ukraine
Local time: 11:20
Member (2004)
English to Russian
+ ...
| A couple of misunderstandood concepts, probably? | Aug 2, 2006 |
df49f wrote:
3) I was talking indeed of correct in the sense of appropriate and "most helpful" as you rightly say. However, in contrast, when I speak of "incorrect" answers, I mean really that: i.e. the "rubbish" kind of answers that started this thread - yet still get chosen and entered as-is in the glossary and turn the glossary into a ridiculously laughable and useless tool.
The problem here is that it's impossible to give any *formal* definition of a `rubbish' answer. Thanks G-d, this is also why they still need human translators instead of robots. When you take a look at a question, you often can tell from your first glance that it's a crap. But it escapes any formal rule, right? So we have to face it and take it as the only criterion that the asker has selected the answer the most helpful, so it's a `winning answer'.
Item 2: Simple, let's count EVERY answer, hidden or not, as an answer, and add it to the stats. So anyone may hide it, it won't really help.
[Edited at 2006-08-02 18:55]
| | | | | Point grabbers | Aug 2, 2006 |
Nikki Graham wrote:
As I see it, anyone joining ProZ now is going to find it extremely difficult to "stand out" in the lists unless they dedicate vasts quantities of time and effort to answering any question vaguely within their grasp in the hope that by getting there fast and first (albeit inaccurately), their answer will be chosen with points in the bag.
So would it be possible to have some reliability ratio ranking instead? ...And wouldn't it help to decrease the rubbish answers Ian posted this thread about?
That's a good definition of a point grabber, Nikki. I think we're mainly talking about dishonest folks whose sole aim in KudoZ is to accumulate points and the asker and glossary be damned. They make a habit of answering anything that moves whether or not they know anything about the subject and whether or not they are competent in the languages involved. They are major contributors to the rubbish that will one day have to be cleaned out.
Currently, when a point grabber answers a question we see under his name in bold, blue letters "specializes in field" or "works in field" if the point grabber has decided to add dozens of fields he claims expertise in. This is one more marketing tool the point grabber can use to grab those points.
I'd like to see the reliability ratio right under "specializes in field"
| | | | Kirill Semenov
Ukraine
Local time: 11:20
Member (2004)
English to Russian
+ ...
| A magic formula | Aug 2, 2006 |
Pablo Grosschmid wrote:
IMHO, a single ranking should reflect quality (RR) and quantity (nr. of answers), equally weighted. And also the time factor should be incorporated, with progressive "decay", for example a half-life of one year, with monthly recalculation (see my former post).
First of all, I don't really see any need in weighting any parameter (statistically). We already have Points Total, and Answers Won, and Answers Provided. There are other things which were not mentioned in this thread so far as `Items', but they are worth to be considered:
1) Not-for-Points answers. We may or may not include these into the statistics, but it may change the numbers.
2) Non-Pro vs Pro points - How to treat them? Are we to include only Pro pts (or won answers) into the stats or also take into account the `non-pro' questions?
These two factors may matter, still I don't think we have to develop a complex formula, albeit weighting, for the reliability ratio. We should also take into account the ProZ server load which may be undermined by a lot of calculations.
I think, a simple `Points Total / Answer Provided' or `Answer Won / Answer Provided' would do for now. And it may be used as a basis for an alternative (second) Leader List.
Time factor may be included as naturally as it now is for Points Total Leader list.
| | | | | Pages in topic: < [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16] > | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » KudoZ: should "rubbish" help be appreciated? | Trados Studio 2022 Freelance | The leading translation software used by over 270,000 translators.
Designed with your feedback in mind, Trados Studio 2022 delivers an unrivalled, powerful desktop
and cloud solution, empowering you to work in the most efficient and cost-effective way.
More info » |
| | Trados Business Manager Lite | Create customer quotes and invoices from within Trados Studio
Trados Business Manager Lite helps to simplify and speed up some of the daily tasks, such as invoicing and reporting, associated with running your freelance translation business.
More info » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
