Anatomy of a Glitch

On Sunday, we had one of those million to one events that snuck through all the protections we have in place and led to some scoring total errors in Fantasy Basketball leagues. We strive for perfection, and are very disappointed when we don't achieve it. The events of Sunday were pretty interesting so in the spirit of full disclosure, I thought I would give a behind the scenes look at a glitch (really more of a generic breakdown) in our fantasy scoring system on Sunday.

It all started with an error in our statistical feed game schedule. These are pretty rare, but they have happened before. They probably happen at least once a year, but I can't honestly remember the last time it happened. There was an additional LAC vs. PHI game in the Sunday schedule. Both teams were playing other teams on Sunday. This was the first breakdown. A significant error in the live statistical feed. Many competitive websites have automatic game schedulers that can get into trouble when something like this happens. We manually schedule fantasy leagues for many reasons, but one is to let a human see what is about to be scheduled and catch things that are obviously wrong. We often check the game schedule in our system against an independent source, or at least check for duplication in teams in the existing schedule. This time the extra game slipped through the cracks and was scheduled in to all fantasy basketball leagues on Sunday. This was a human error, and the second error in a stack-up of errors that led to the glitch. We did not notice the problem in the Sunday league schedules and no users reported the schedule problem prior to the start of the leagues. Had this happened, the bad game would have been removed prior to the start of the leagues and the glitch would not have happened. With leagues scheduled the day before, there is a lot of time for an error like this to be spotted prior to the league start time, but it was not spotted (the bad game in every contest lobby, and one extra game in the game count) by anyone. This was the third breakdown.

When the fantasy basketball leagues started on Sunday, there were some issues reported with the scoring of Elton Brand in particular. This was looked into at that time, and his initial score of 27 could not be confirmed in the in-game scoring for the scheduled game. Again, it was not noticed that there was an extra LAC/PHI game on the schedule that should not have been there. It was assumed at the time that this was a live feed issue that would be resolved with the next update, or at least by the final box score. The scoring that appeared immediately at the start of the league was because of another significant failure in the live feed. A completely bogus box score was delivered against this non-existent game, and these bogus scores were displaying for the PHI and LAC players instead of their normally scheduled game scoring. There is no reason for this to have happened, and it pushed the issue on the feed side to a million to one or higher stack-up of errors. This was the fourth and most unusual breakdown in the system. We have never seen a bad game that should not have been on the schedule and was not actually played get scored in our system before Sunday.

More reports of scoring issues with Brand and other PHI and LAC players were reported towards the end of the night. At that point we discovered the problem with the bad game in the system. The bad game had not received a final box, so it was holding up the completion of Sundays leagues. The scoring displayed for all PHI and LAC players was obviously wrong in our fantasy leagues. It was displaying the statistics for the bogus game instead. We found that the in-game scoring could be corrected by adjusting this box to the stats from the actual game that was played by PHI and LAC. This corrected the scores in the contest leader boards. We though we had it. The bad box score had held things up as it is supposed to, and allowed us to correct the bad scoring for PHI and LAC players. The system worked. Well except for one last breakdown. This was another human error. There were ultimately two games on the schedule for the PHI and LAC players in our system. The real game boxes were in, and were we left with the bad box score. The last time something like this had happened before was during baseball in 2007. We had not yet learned how to deal with late scheduled doubleheaders, and had two games for one team on an actual league schedule. It was too long ago to fully remember how the system deals with a case like this, but it does what you logically think that it would do. For in game stats it shows the current game that is running only. Once the second game starts, it shows just the fantasy scores for that game. In the league totals, it includes the scores from both games. So it looks likes it is adding the scores incorrectly, but it is actually getting the overall score correct by including both games. So we really just had the doubleheader situation from 2007 again, but did not catch it. Adjusting the game two scores can make the scores for the players look right in the lobby, but you need to remember it is also totaling in the game 1 scores. Setting the bogus game scores to the actual scores had the effect of double counting each PHI and LAC player's score in the league totals. Had we realized this is what was happening, (we had no reason to think the totals were wrong, as they had never been before), we could have set the scores in the bogus game to zero which seems like the obvious solution in hindsight. This would have made the player scores for all PHI and LAC players zero in the contest lobbies, but would have scored the leagues perfectly correct by just counting the real games. So we had this last final breakdown that let it actually slip through even after the million to one stuff that got us to this point.

Once this happens and the leagues are settled, we have to go through them all by hand and see what leagues had the prize outcome changed. We had 92 daily fantasy basketball leagues on Sunday and about a dozen had the outcome changed by the bad scoring totals. We go through and credit the accounts who should have won prizes, and debit prizes that should not have been won. We are more interested in getting the people who were harmed whole, then clawing back bad prizes, so we often let the smaller bad prizes stick rather than debit accounts and contact the user. If you noticed this error, and we have not contacted you yet, we are done with the credits/debits at this point. If you think you should have won a prize and have not been credited yet let us know. Well that's it. The anatomy of a glitch in a nutshell.