The Student Room Group

Moderation statistics :)

Scroll to see replies

Original post by geoking
high success rate on what is ultimately a subjective job? A success rate that high for volunteers who have minimal training is not "incredible", it's total bull**** - no system is that perfect when it's based on judgement calls. Maybe if they had all had many years of experience in busy HR departments of different companies, sure, but they don't.

Of course the process "works" because there is no accountability or transparency, there is no scope for it to go wrong :facepalm: If it wasn't working, how would you know? :confused:

How the hell is what I'm suggesting reflective of not understanding that the forum is owned by a private company? :colonhash: Who said do things by committee? I hope your meetings aren't ran like this - someone makes a suggestion, and then you widly extrapolate what they mean :tongue:


Let's put it this way. Take a random company. I dunno, Nike. You buy Nike products, for arguments sake. Why is Nike accountable to you, and not their regulators?

Why are Nike employees accountable to you, but not the Nike managers? etc.. etc....

(Just to make things perfectly clear Nike as a company, or their employees are not accountable to their customers)

Similarly, neither is TSR, however you're defining the word "customer"
(edited 9 years ago)
Reply 41
Original post by Kittiara
That's the thing, though. The error rate need not indicate bias. It may just be a genuine case of error or misunderstanding. Also, fewer cards issued may not be an indication of a productive community, nor is a higher amount of cards issued an indication of questionable activity on the part of the mods. Over the last couple of months I have reported so many spammers. I'm sure you've seen them around. Kitchens, "love remedies". One time chat was flooded with them. Those are the kind of posters that definitely need banning, but of course they will always return with new accounts, and there's nothing the mods, or TSR, or the community can do about it.

What would be helpful, I agree, is improved interaction with the community. The rules can be ambiguous, and that can lead to misunderstanding. It is known that we get trolls here from the Misc (amongst others), and yes, it would be helpful for people to know to avoid them, because the kinds of threads they create can cause upset and get people worked up and into trouble. Of course, that's assuming that TSR knows who is who. I don't know if they do.


Actually it does. It's a subjective job with minimal training. Rationally, there should be a high error rate but as Mr Vlad pointed out, they somehow, magically get it right 99% of the time. That simply isn't possible.

Cards issued may not, but they could do. Also you've shot yourself in the foot here. Your report on spammers actually proves something - if there was a table showing this and if there was an increase in spammers, it would actually show that they need to improve the registration and spam capture process. So there we go, the table would prove something useful to the community and company.

Original post by The_Internet
OK let's say you ARE the customer, but as Gofre said, why do you need to see this? Where's the demand? What's the benefit to TSR? Any business would need to understand the benefits of doing well... any thing, even if it's customers want x

Let's say it IS only 5 minutes of work (which it won't be... ), but then let's also say that that TSR would make way more money, by putting those efforts elsewhere ie: this would be a low priority, compared to something else, and again, you'd still need to say to TSR what exactly are the benefits?

Working in a "multi million pound business" myself last year, I know how bloody freaking slow it is to get things done, because of bureaucracy(ie: Generally the bigger the organisation, the slower things get done)... - it took several months to move some kit from one part of the data centre, to another part of the data centre, in the same damn room...

And TSR is in effect a "live production" environment - you'd need to go through testing elsewhere, and then see if it will work properly when "live" on the site.


****ing hell, do you want me to write a business case for this? For the millionth time, accountability and transparency are inherently good things for a moderation system, and would cost almost nothing to implement. It would show strengths and weaknesses of the moderation, areas to improve, trends and pretty well information business would consider important. Or maybe they can **** around with the UX and CSS a bit more for a considerable cost :facepalm:

And yes it would only be only 5 minutes of work because it'd be one SQL statement ran monthly, which could easily be automated :colonhash:

You don't need test environments to make a database query even if it's connected to the live db :curious: This is something that could be pushed through in an afternoon. Cost? Negligible. Benefits? Lets actually see how the moderation is actually functioning! :lol:
Original post by geoking

****ing hell, do you want me to write a business case for this? For the millionth time, accountability and transparency are inherently good things for a moderation system, and would cost almost nothing to implement. It would show strengths and weaknesses of the moderation, areas to improve, trends and pretty well information business would consider important. Or maybe they can **** around with the UX and CSS a bit more for a considerable cost :facepalm:

And yes it would only be only 5 minutes of work because it'd be one SQL statement ran monthly, which could easily be automated :colonhash:

You don't need test environments to make a database query even if it's connected to the live db :curious: This is something that could be pushed through in an afternoon. Cost? Negligible. Benefits? Lets actually see how the moderation is actually functioning! :lol:


See my edit RE: Facebook

Lots more info here as well:

https://www.google.co.uk/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=facebook%20users%20are%20customers

And again, even IF you're the customer, why should TSR be accountable to you? And not it's parent company? Why should TSR staff be accountable to you, and not the managers? Or section leaders?

What are the benefits to TSR specifically. What is the demand for this? Just becuase YOU want to see how the "moderation is actually functioning" that doesn't mean others do

And besides, let's say we do have a system in place, andwe give say 100 yellow cards, 100 blue cards etc... You won't know what that's for and if you want the transparency ie: to see exactly what was warned, when, then you'd be able to identify who got warned, because of what they said - destroying user confidentiality.

What they got warned for may also be pornographic as well. There's far too many risks to this, for "full transparency" to users. BEsides, do any other forums do this?

OK, those two are the biggest risks there, but they're way too big a risk - Let's say that in the past, we've had to remove some absolutely horrible content. Fortunately it doesn't happen that often...

And then if we did have this transparency you speak of, what would be the point of the recycle bin?
(edited 9 years ago)
Original post by geoking
From the business model I work in, I can tell you that categorically isn't true. You're a customer if you use a product provided by the company e.g. the forum. More so if you generate revenue for the business.


Speaking as an past TSR employee, I can tell you your business model is not the same as TSR's in that case. The customers are the universities and businesses paying for ad campaigns that keep the company afloat. Yes growing and retaining the userbase is important, but the money they actually make off said userbase is a very small percentage of the site's total income. The forum isn't a product, it's a means of delivering their actual products of advertising space and engagement opportunities. You as a forum user are here voluntarily, on the condition that you agree to the rules set forth by the site's parent company. They, and by extension the moderators that represent them, have no obligations to you, it's simply in the site's best interests to keep the users happy and the community thriving.
Reply 44
Mods - you're embarrassing yourselves. Again.

Stop answering.

Some of the CT have previously come out and said that transparency in modding is a good thing.

You're simply trying to defend the indefensible because it by extension defends your occasional poor decision making. What should really defend your poor decision making is that it's allowable under the circumstances - not that making mistakes is what you intended to do all along and the best way to sort that out is to keep it secret.
Reply 45
Original post by The_Internet
See my edit RE: Facebook

Lots more info here as well:

https://www.google.co.uk/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=facebook%20users%20are%20customers

And again, even IF you're the customer, why should TSR be accountable to you? And not it's parent company? Why should TSR staff be accountable to you, and not the managers? Or section leaders?

What are the benefits to TSR specifically. What is the demand for this? Just becuase YOU want to see how the "moderation is actually functioning" that doesn't mean others do

And besides, let's say we do have a system in place, andwe give say 100 yellow cards, 100 blue cards etc... You won't know what that's for and if you want the transparency ie: to see exactly what was warned, when, then you'd be able to identify who got warned, because of what they said

What they got warned for may also be pornographic as well. There's far too many risks to this, for "full transparency" to users. BEsides, do any other forums do this?


DEFINITION OF 'CUSTOMER'
An individual or business that purchases the goods or services produced by a business. The customer is the end goal of businesses, since it is the customer who pays for supply and creates demand. Businesses will often compete through advertisements or sales in order to attract a larger customer base.

Users are using the service provided by TSR, the forum. Happy customers keep on coming back. If there are no users of the forum, the whole company collapses. Better moderation = better community = better service for everyone.

Seeing the stats surely will show if the card system works, if there are problems that need sorting, if spam is an issue, the list goes on. To actually want to be blind to moderation statistics makes no sense at all. Yes other forums do this, and even if they didn't, why does it matter? It's not like TSR couldn't set a precedence is it?



Tell me, where would TSR be without the forum users? Seems that the problem is that the company doesn't seem to care about it's actual base customer.

Doesn't matter if we are here voluntarily or not, without the users, TSR wouldn't be able to advertise its market reach.

Nevertheless this is all distracting away from one of my main points that the success rate of the moderation is impossibly high. To argue against moderation transparency simply sounds like you don't want to be shown to have a flawed system in place and in this case, you couldn't care less about the customer as long as the boss is happy.
(edited 9 years ago)
Original post by geoking
DEFINITION OF 'CUSTOMER'
An individual or business that purchases the goods or services produced by a business. The customer is the end goal of businesses, since it is the customer who pays for supply and creates demand. Businesses will often compete through advertisements or sales in order to attract a larger customer base.

Users are using the service provided by TSR, the forum as search. Happy customers keep on coming back. If there are no users of the forum, the whole company collapses. Better moderation = better community = better service for everyone.

Seeing the stats surely will show if the card system works, if there are problems that need sorting, if spam is an issue, the list goes on. To actually want to be blind to moderation statistics makes no sense at all. Yes other forums do this, and even if they didn't, why does it matter? It's not like TSR couldn't set a precedence is it?


So what exactly have you paid for?

Besides let's say such a system is implemented, you're ignoring the knock on costs and effects

You can't put peoples opinions in to a pretty graph. You can put things like how many people have been banned, carded etc.. In to a graph but what if we have an influx of trolls in x month? So the system thats been implemented is pointless because the data is meaningless without context

And because it is about context and what people have said or done then it means extra admin work for TSR , to sift through the data and try and make sense of it

Again what's the benefit for tsr itself?

Posted from TSR Mobile
Original post by geoking
Actually it does. It's a subjective job with minimal training. Rationally, there should be a high error rate but as Mr Vlad pointed out, they somehow, magically get it right 99% of the time. That simply isn't possible.

Cards issued may not, but they could do. Also you've shot yourself in the foot here. Your report on spammers actually proves something - if there was a table showing this and if there was an increase in spammers, it would actually show that they need to improve the registration and spam capture process. So there we go, the table would prove something useful to the community and company.



That still doesn't indicate bias, though. I see bias as the mods discriminating between members, favouring certain opinions over others etc. Perhaps mods err on the side of caution, and that would be understandable. And perhaps the CT agrees with this. I don't know. I've only seen my own end of the moderation system, with my reporting posts (the vast majority of which are spammers, and therefore clear cases), so I can't know what goes on behind the scenes. But my not knowing what goes on behind the scenes also means that if they were to release figures, I wouldn't know how to interpret them. And anyone could read into them what they wanted. Those convinced there's a bias, would surely see it. Those feeling that the mods are lax, would think the figures are too low. Those feeling that the mods are too strict, would think that the figures are too high. And so on, and so forth. So, aside from running a program, someone would need to sit down and explain every figure, which would take ages.

Also, I am not a TSR representative, so I am not shooting myself in the foot. I'm merely offering my own opinion. I don't know what can be done about spammers. When someone signs up for an account, how can one tell if they're a genuine new member or a spammer? I figure that that's impossible, but I'm no expert. And anyway, I'm sure that the company already knows that there's a spamming issue.
Original post by geoking
It's good to know the health of the community - other forums do it, and explain if an account is a dupe or not, helps other users stay away from interacting with such accounts, plus it would be good for everyone to see how many cards are overturned.


What other forums do this? I browse a few as large as TSR semi-regularly, and have never seen this.
Reply 49
Original post by The_Internet
So what exactly have you paid for?

Besides let's say such a system is implemented, you're ignoring the knock on costs and effects

You can't put peoples opinions in to a pretty graph. You can put things like how many people have been banned, carded etc.. In to a graph but what if we have an influx of trolls in x month? So the system thats been implemented is pointless because the data is meaningless without context

And because it is about context and what people have said or done then it means extra admin work for TSR , to sift through the data and try and make sense of it

Again what's the benefit for tsr itself?

Posted from TSR Mobile


You don't have to pay for something to purchase it - do you not purchase free trials of services like netflix? You're a service customer.

There are no knock on costs other than the cost of a cron job which is probably in the thousandths of a pence.

Well then graph the influx of trolls and state it, because in following years it could prove that there are trends in trolls and spam, and if you were behind the scenes to do it daily or weekly you could even predict when an influx was going to occur, or if it was occurring. Arguing that you should be blind to statistics is insanity, but again I'm very certain that a large part of it is that it would show that the success rate of moderation is impossibly high.

What sifting through data?? Look, here is how I'm going to presume the database works. You either have a separate table for the users and their moderation history, or a giant moderation table - either way, it doesn't matter. What you do is you make a query on that table (or all the mod tables, doesn't change much other than about 6 or so words in the statement) by date and output it to a pretty CSV. This is a statement that could be ran once and either ran via a cron job or written into a PHP page for a moderator to create dynamic tables from. This is not a lot of work. This would take a half competent developer an hour to do at the very most, and that's the shiney PHP version, not the 5 minute cron job version.

If you seriously can't see the benefit of this figures now, then this has nothing to do with the positives of it, but everything to do with the negatives e.g. the nonsensically high "conviction rate".

Original post by Hype en Ecosse
What other forums do this? I browse a few as large as TSR semi-regularly, and have never seen this.

If you can tell me what relevance that actually has other than a fallacial tangent, I'll be glad to answer.

Original post by Kittiara
X.


So which is more likely, bias, or that a group of volunteers with minimal training are right 99% of the time? Bias as in when a mod makes a decision, others don't objectively consider both viewpoints, but they will inevitably side with the moderator.
(edited 9 years ago)
Original post by geoking
If you can tell me what relevance that actually has other than a fallacial tangent, I'll be glad to answer.


Because I'd like to know! It's useful to see how other sites moderate their site and what effect that has, so we can better reflect on how TSRG does things. If you want us to implement transparent moderation statistics akin to what other sites do; it'd be useful to have an example to know how they do it and what TSR can learn from it.
Original post by Clip
Mods - you're embarrassing yourselves. Again.

Stop answering.

Some of the CT have previously come out and said that transparency in modding is a good thing.

You're simply trying to defend the indefensible because it by extension defends your occasional poor decision making. What should really defend your poor decision making is that it's allowable under the circumstances - not that making mistakes is what you intended to do all along and the best way to sort that out is to keep it secret.


Hear. TSR - The Student Room or The Soviet Republic?
Original post by geoking
Tell me, where would TSR be without the forum users? Seems that the problem is that the company doesn't seem to care about it's actual base customer.

Doesn't matter if we are here voluntarily or not, without the users, TSR wouldn't be able to advertise its market reach.


Hence why I mentioned that it's in their best interests to keep forum users happy. From what I have seen behind the scenes TSR do care about their users, but as a privately owned profit driven business the onus is always going to be on signing their employee's paycheques.

Nevertheless this is all distracting away from one of my main points that the success rate of the moderation is impossibly high. To argue against moderation transparency simply sounds like you don't want to be shown to have a flawed system in place and in this case, you couldn't care less about the customer as long as the boss is happy.


As I mentioned in my post on the previous page, I'm arguing that the sort of information you're calling for is utterly meaningless without the information we cannot share to use as context. We have to keep details confidential, and the leftover top-level stuff is useless at gauging forum wellbeing in any tangible way- it's not a figure used internally for anything either.

Mods wouldn't be here if we didn't care, we don't devote large chunks of our free time to unpaid work for the fun of it. The moderation system has been through a decade of refinement and only works because of the level of input and support we get from each other and moving up the hierarchy. If things were screwed up the right people would notice.
Not really sure how this is helpful to be honest. All it would potentially serve to do is show up inefficiency on the mods part and give certain members further ammunition against them when they are doing what I think is a bloody difficult job already. This is not like a normal business, in a actual business you dont walk up to your boss and say " I think your doing a **** job " yet this is the kind of abuse I suspect the tsr mods have to deal with day to day.

Below is basically what would happen each month when the figures were revealed :

High figures
Members - " Mods are so harsh, not promoting free speech, facist bastards etc "

Low figures
Members - " Mods arent doing their job properly. Too many trolls. Bring back neg rep etc "

I'm not going to claim I know that much about business but I am going to apply one A level econmics term here . Allocative efficiency is maximising welfare for everyone concerned. With greater transparency all youll get are angry members and a mass exodus as mods quit. Weve already seen 3 of the longest standing section leaders go in the last few months. Increased transparency =/= increased efficiency.

Also the forum in question are ones like Smeggy's ( I think) which are much more informal and care less about "the product". If a new member signs up and the first thing they see is "200 members banned in the last month", they're just going to think its a cesspit and leave straight away. I have no idea why the ct would want to do this, but it would be interesting to here their views when they're back on Monday morning :smile:
(edited 9 years ago)
Original post by geoking


So which is more likely, bias, or that a group of volunteers with minimal training are right 99% of the time? Bias as in when a mod makes a decision, others don't objectively consider both viewpoints, but they will inevitably side with the moderator.


I don't know. Neither do you, nor anyone who doesn't know what goes on behind the scenes. You expect bias. I expect caution. In order to know for certain, we'd need an independent third party to come in (as in, not a TSR member, nor a mod or TSR employee) and go over every single decision made by the mods along with any objections to said decisions, for a period of, say, three months so that there is a decent amount of data available. I don't think that's going to happen.
Reply 55
Original post by Kittiara
I don't know. Neither do you, nor anyone who doesn't know what goes on behind the scenes. You expect bias. I expect caution. In order to know for certain, we'd need an independent third party to come in (as in, not a TSR member, nor a mod or TSR employee) and go over every single decision made by the mods along with any objections to said decisions, for a period of, say, three months so that there is a decent amount of data available. I don't think that's going to happen.


It was rhetorical :colonhash: It's obvious that they aren't going to be right 99% of the time, but the system in place allows for them to be ignorant of this fact.
Original post by geoking
In the interest in moderator transparency, can you release statistics to do with moderation on a monthly basis such as amount of cards issued, bans issued, and cards overturned.

A healthy community is one where the moderation is transparent and accountable, and it would help everyone to know of pitfalls that can be avoided, and trend in the moderation :smile:


I have no problems with transparency when it is beneficial to have input from the community, but the thing I dislike most strongly about your suggestion is that your proposed statistics give pretty much no information about what you want to know i.e. the bias/quality of moderation. Knowing the number of cards and banned issued tells you only how many cards or bans were issued, it tells you nothing about the correctness of the decision to issue them.

To even begin interpreting the data you would need to take other factors into account. For example you would expect some correlation between number of posts made and cards issued, or between reports made and cards issued. There are many more factors to account for than just the quality of moderation e.g. number of posts, number of reports, time of year, unpredictable waves of rule breaking activity (e.g. persistent dupes or attacks by other sites/organisations), level of card, user history, the forum where the rule breaking occurs, type of rule breaking, availability of moderators etc.

I also think it is a myth that moderation is necessarily subjective. Most of the time it is pretty clear cut. Advertising, swearing and adult content are almost always black and white cases. There certainly are grey areas, but mostly around offensiveness and trolling. In those grey areas the moderation team will frequently seek a second opinion and so decisions are made by committee. The fact that a lot of rule breaking is black and white, and that there are processes in place to standardise and peer review contribute to a high success rate for cards that are issued. Actually I think that somewhat misses the point though, because I guarantee and the mods will freely admit that a lot of stuff which should be carded is not (primarily because it isn't seen/reported) and this is where the biggest error rates occur over a perfect moderation system. The apparent inconsistency that results is what frequently frustrates users and causes resentment towards moderation.
(edited 9 years ago)
The moderators do work really hard and whilst these may be interesting to see, it seems like you are asking volunteers to do a lot of extra legwork for no real reason.
Original post by Clip
Mods - you're embarrassing yourselves. Again.

Stop answering.

Some of the CT have previously come out and said that transparency in modding is a good thing.

You're simply trying to defend the indefensible because it by extension defends your occasional poor decision making. What should really defend your poor decision making is that it's allowable under the circumstances - not that making mistakes is what you intended to do all along and the best way to sort that out is to keep it secret.

The circumstances don't change whether you get access to meaningless data or not. Ergo it doesn't change anything. I'd even go as far as saying that this would have a detrimental effect on the mod team.

So what's the point? You're throwing your toys out the pram because you're not getting your own way again.
Original post by F1 fanatic
I have no problems with transparency when it is beneficial to have input from the community, but the thing I dislike most strongly about your suggestion is that your proposed statistics give pretty much no information about what you want to know i.e. the bias/quality of moderation. Knowing the number of cards and banned issued tells you only how many cards or bans were issued, it tells you nothing about the correctness of the decision to issue them.

To even begin interpreting the data you would need to take other factors into account. For example you would expect some correlation between number of posts made and cards issued, or between reports made and cards issued. There are many more factors to account for than just the quality of moderation e.g. number of posts, number of reports, time of year, unpredictable waves of rule breaking activity (e.g. persistent dupes or attacks by other sites/organisations), level of card, user history, the forum where the rule breaking occurs, type of rule breaking, availability of moderators etc.

I also think it is a myth that moderation is necessarily subjective. Most of the time it is pretty clear cut. Advertising, swearing and adult content are almost always black and white cases. There certainly are grey areas, but mostly around offensiveness and trolling. In those grey areas the moderation team will frequently seek a second opinion and so decisions are made by committee. The fact that a lot of rule breaking is black and white, and that there are processes in place to standardise and peer review contribute to a high success rate for cards that are issued. Actually I think that somewhat misses the point though, because I guarantee and the mods will freely admit that a lot of stuff which should be carded is not (primarily because it isn't seen/reported) and this is where the biggest error rates occur over a perfect moderation system. The apparent inconsistency that results is what frequently frustrates users and causes resentment towards moderation.

Entirely this.

Quick Reply

Latest

Trending

Trending