BI Incorporated runs a Microsoft-based database of registered sex offenders in the US. They suffered a major outage when they hit more than 2.1 billion records. Apparently no one saw it coming.

An explanation is posted on their website:

“At 7:29 a.m. Mountain Time on Oct. 5, BI Incorporated experienced a problem with one of its offender monitoring servers that caused this server’s automatic notification system to be temporarily disabled, resulting in delayed notifications to customers. The issue was resolved approximately 12 hours later at 7:25 p.m. MT. The issue was confined to the BI TotalAccess Server when its database exceeded its 2.1 billion record threshold. The BI system notified administrators and technical staff of the issue immediately and a team was immediately assembled to diagnose and plan for recovery.

“Importantly, the monitoring system continued to operate and gather information, but transmissions were delayed until the system was restored. Offender activity logged while the server was being worked on was effectively processed at 7:25 p.m. MT when the system was restored. Alerts that may have occurred during this period were transmitted to our customers at that time.

The database ran “out of values in a column in a table”. It now has been expanded, they say, to 1 trillion records. They did not explain the rate of change to records over time. Was it getting exponentially larger lately or has it been slowly creeping? An expert is cited in their press release saying no one could have predicted running out of space.

The irony of the story, and that expert testimony, is that an alerting system for this alerting system is said to now be a priority for BI.

…we are working with Microsoft to develop a warning system on database thresholds so we can anticipate these issues in the future