May 12th, 2011 by rzr
In this Behind the Scenes, we’ll take a look at how FTS decided to handle forfeits. This issue hit us broadly: bringing up potential problems with our algorithm, how we visualize forfeits, and where and if we would deviate from WFTDA’s policy.
FTS initially found out about a forfeit by ROSI from Bleeding Heartland’s Facebook page.
Tom asks:
Possibly because of injuries, ROSI forfeited to Bleeding Heartland when the score was 184-5
Do any of ya'll know how that get's handled as far as sanctioning goes? Unfortunately there's no information about how far into the game they made it. I'm putting it in as unsanctioned for right now, but...
Aaron replies:
http://wftda.com/rules/wftda-rules.pdf
There's a bit in there about how a forfeit is declared, but it doesn't say anything about the sanctioned status. We can ping them in a few days to see what happened.
A little over a week later...
Pete asks:
Do we account for a forfeit? This feedback just came in about an hour ago.
"You [are] apparently unaware of Roller Girls of Southern Indiana 4/9 loss to Bleeding Heartland. The score reported to WFTDA was the half-time score of 117-2, but a check of BHRG facebook page shows score of 184-5 when ROSI forfeited the bout...”
What would be the score differential for a forfeit? MLB records forfeit as 9-0, 1 point for each inning. If we did 5 points per jam with an average of 60 jams per game, that would be 300 points. Some forfeits might occur because the travel team bus broke down, then they could get royally screwed in the rankings. If we did say 150-0, then some teams could benefit in ranking by forfeiting, which seems odd.
Donna replies:
In looking at the ROSI page.. it looks like the bout was labeled "Unsanctioned". Like Thomas said, if the paperwork has it that way, then it's unsanctioned... but if it's labeled sanctioned but forfeit.. then..
I think we have two problems; how do we visualize it? (strike through on the bout scores? replace scores with an "FFFF"? "winner/Forfeit"? If it's sanctioned, we color/track it as sanctioned.. if it's unsanctioned, likewise..
.. and how to model it. Regardless of however many jams we pick, the DOS would equal 1 (since team B was held to 0). I'm not sure if that would break our system, though.. ?
Tom comments:
It looks like it was sanctioned, but we're still waiting on the final word. More when I'm not supposed to be working.
David says:
So...from the Algorithmic perspective...if it's considered a sanctioned bout, the best thing to do would be to include the score at the time that the bout was terminated. The DoS doesn't care how many jams/minutes were included since it's a ratio of the scores that matters. Doing something like Peter or Donna suggested would have a worse effect because it would max-out the correction.
Tom adds:
The feedback we've gotten so far [from BHRG]:
Possibly because of injuries, ROSI forfeited to Bleeding Heartland when the score was 184-5
Do any of ya'll know how that get's handled as far as sanctioning goes? Unfortunately there's no information about how far into the game they made it. I'm putting it in as unsanctioned for right now, but...
Aaron replies:
http://wftda.com/rules/wftda-rules.pdf
There's a bit in there about how a forfeit is declared, but it doesn't say anything about the sanctioned status. We can ping them in a few days to see what happened.
A little over a week later...
Pete asks:
Do we account for a forfeit? This feedback just came in about an hour ago.
"You [are] apparently unaware of Roller Girls of Southern Indiana 4/9 loss to Bleeding Heartland. The score reported to WFTDA was the half-time score of 117-2, but a check of BHRG facebook page shows score of 184-5 when ROSI forfeited the bout...”
What would be the score differential for a forfeit? MLB records forfeit as 9-0, 1 point for each inning. If we did 5 points per jam with an average of 60 jams per game, that would be 300 points. Some forfeits might occur because the travel team bus broke down, then they could get royally screwed in the rankings. If we did say 150-0, then some teams could benefit in ranking by forfeiting, which seems odd.
Donna replies:
In looking at the ROSI page.. it looks like the bout was labeled "Unsanctioned". Like Thomas said, if the paperwork has it that way, then it's unsanctioned... but if it's labeled sanctioned but forfeit.. then..
I think we have two problems; how do we visualize it? (strike through on the bout scores? replace scores with an "FFFF"? "winner/Forfeit"? If it's sanctioned, we color/track it as sanctioned.. if it's unsanctioned, likewise..
.. and how to model it. Regardless of however many jams we pick, the DOS would equal 1 (since team B was held to 0). I'm not sure if that would break our system, though.. ?
Tom comments:
It looks like it was sanctioned, but we're still waiting on the final word. More when I'm not supposed to be working.
David says:
So...from the Algorithmic perspective...if it's considered a sanctioned bout, the best thing to do would be to include the score at the time that the bout was terminated. The DoS doesn't care how many jams/minutes were included since it's a ratio of the scores that matters. Doing something like Peter or Donna suggested would have a worse effect because it would max-out the correction.
Tom adds:
The feedback we've gotten so far [from BHRG]:
“ROSI forfeited with 16:50 left on the clock in Period 2. The score at that time was 184 (BHRG) - 5 (ROSI). Last I heard, WFTDA was going to accept the game as a sanctioned bout that would count towards rankings. I'm also CC'ing one of our WFTDA reps, who might have more information than I do on how the sanctioning has progressed.”
and
“... It will count as a sanctioned bout by WFTDA. I will have to check the sanctioning spreadsheet for exact scores that were entered. They advised us to turn in a 100-0 score, but we turned in what was on the scoreboard at the time that they forfeited.”
Aaron comments:
Yes we defer to WFTDA for whether a bout is official or unofficial. Their rules are deliberated, tested, and revised, and they have a process in place for reporting and approving bouts. I think it's better to stick with the big picture of roller derby, than forging our own opinions on the way it should be.
As far as visualizing that this was a forfeit, that's a little tricky. If this were an Unofficial bout (doesn't sound like it will be), we have an Unofficial Note field for tracking the nature of it's unofficial-ness (something that should be made public when I get around to it).
But that note doesn't exist for Official bouts, and it doesn't make sense to introduce one because barring this bout, there's never anything extra to say about an official bout. So if that were the end of it, I would be inclined to just let this pass as the one time it's ever happened (Lindsay and I couldn't think of another Official WFTDA bout that was forfeited).
The extra to piece to this though is I was playing around with a badging system for bout tables. Since more columns are sneaking into these tables (and even more contenders once Unofficial bouts are fully realized), I was thinking of killing some columns and have little badges line up in one column.
... so if forfeited bouts were a thing, this would work great for making a little forfeit icon and attaching it. But it's not a thing, so it's a separate debate whether it's worth tracking something that has only happened once (or a few times?)
Donna replies:
I love the badges idea.
I also think we should track forfeits. I don't expect it to be a major thing, but I would be surprised to not see more happen in the future.
David says:
… and as mentioned before, I'd use the 184 - 5 score, not the 100-0 score, for the algorithm.
Tom votes:
I think we should use the [reported score] so that our records match up with WFTDA records even if it means an oddball DoS. It'll wash out pretty quickly … .
Aaron says:
Yeah I agree, we can't make judgment calls on stuff like this even if we're left with the ranking unfriendly xx to 0. Doesn't look like that'll happen though.
Pete strategizes:
Also, as Donna mentioned if we follow what WFTDA says and use 100-0 for forfeits, that is actually worse than 185-5 because of how DoS is calculated. You are always better off scoring 1 point than 0.
Seriously, if I was coach and saw my team down 5-90, I would forfeit. That would result in a .89 DoS, which doesn't sound too bad especially if you playing against Gotham at the time. You would probably move up in ranking as a result.
Aaron retorts:
Well if teams were actually doing this they would receive mass social shame followed by WFTDA regulation that would make it unfavorable. Regardless how FTS was rating it of course :)
Point is, no one would game the FTS ranks. And since the voters of the ranking system that actually matters would be highly influenced by funny business like this, I don't think this would be a very good strategy.
Donna replies:
Honestly, I'd be flattered if they did do that. ... Just to David's point.. scoring 0 points does "break" things, so if we can avoid that..
Derby's young in terms of sports … I give them another 20 years before they start to game our website.. :)
David corrects:
um...just to be clear...scoring 0 points doesn't actually "break" anything. I was just saying that the most "accurate" value was the actual score, regardless of whether it was a full bout or not. In this case, the difference between 184-5 (DoS =0.95) and 100-0 (DoS=1) is pretty minimal and would only result in 3.5 ratings points difference.
Ultimately, FTS implemented the WFTDA policy and set the score to 100 - 0 in favor of BHRG. Other then this, we do not show that a forfeit occurred; however, we are working on implementing badges that will remedy this in the future.
Comments
While I don't have any insight into how this is being handled on the back-end, I did notice a couple things in the post that probably should be revisited.
First, the following statement:
"They advised us to turn in a 100-0 score, but we turned in what was on the scoreboard at the time that they forfeited."
Note that it was advice and not policy, and that BHRG turned in an IBRF with the 184-5 score. If you want to go off the official record, you'd follow what's on the IBRF.
As for the 100-0 suggestion, my guess is that's the advice for forfeits without a score, i.e. when the forfeiting team does so before the start of the first jam. Since this forfeit happened with about 75% of the game played, the confidence would be rather high that if the game were to continue that the DoS of 0.95 would be more accurate than 1.00.
Second, and more generally:
"FTS initially found out about a forfeit by ROSI from Bleeding Heartland’s Facebook page."
Y'all realize how broken that sounds, right? I realize that the accuracy of the dataset isn't protecting the free world from nuclear war or zombies, but what's your inherent level of confidence of the dataset? I can't imagine it's 100% because of cases like this one.
> If you want to go off the official record, you'd follow what's on the IBRF.
Not really following this. We don't want to go off the official record. WFTDA always makes the final call here as far as we're concerned. That call just may not be known right away, so we work with what we can until an IBRF can be confirmed. Are you saying that the IBRF says 184-5? If so, that's news to us.
And similarly to the second point. We'll hunt down a score unofficially in this way in order to get results up sooner. But we almost always verify these against DNN or a team's page later on. A big confusion like ROSI/BHRG is extremely rare.
And if we do miss something, there's enough eyes on these scores that people send in corrections. Given that people are always sending in corrections even for bouts that happened years ago gives me a very high level of confidence. Also we're not talking about a very large dataset. With just a dozen or so bouts per week, compared with the number of eyes that see it and have a chance to correct it ... the chances of something slipping by seems slim.
It's actually really great to know how on it people are -- when ambiguities do happen it's often a visitor that figures it out before we do.
Like I said, I don't have any inside information; My comment is based solely on the original post. The way I read the first quote is that BHRG contacted a WFTDA rep (league/games/rules?) and was advised (in other words, suggested) to submit the IBRF with a score of 100 to 0. The phrase "but we turned in what was on the scoreboard" leads me to believe that the IBRF has the cited 184-5 score on it. I seriously doubt The WFTDA would alter either the IBRF or the collected results considering both teams signed off on the document as submitted.
So, like I said before - if you want to go off the official record, you'd follow what's on the IBRF. The best way to verify that would be to re-check with the original BHRG contact to see what was submitted. There is no more canonical source that's publicly available.
I see, yes the quotes above are just taken a bit out of context (or before everything had been worked out). There was actually a lot more back and forth between a few officials/players from both teams. Ultimately, they waited until the IBRF was accepted and forwarded that info to us. 100-0 is the official score on the IBRF.