In the mid-1980s, I was witness to an incident where an upgrade to the SS7 software used in AT&T's long distance network took most of North America's long distance service down hard for more than twenty-four hours. It was then that I began formulating what came to be called Pinkston's Law: MOST OUTAGES BEGIN AS UPGRADES

Over the years since, I have seen this happen so often that whenever I hear of a major telecom or data service outage, my first thought is, "Must have been an upgrade. Pinkston's Law." In the vast majority of cases it turns out that that's exactly what it was! So, at the urging of those closest to me, I've started this blog to chronicle the occurrences of Pinkston's law whenever I hear of them.

Friday, July 10, 2015

NYSE says 3.5 hour outage caused by software update

On a day filled with stories of hacks and outages, this one seemed to get the most attention.
On Tuesday evening, the NYSE began the rollout of a software release in preparation for the July 11 industry test of the upcoming SIP timestamp requirement. As is standard NYSE practice, the initial release was deployed on one trading unit. As customers began connecting after 7am on Wednesday morning, there were communication issues between customer gateways and the trading unit with the new release. It was determined that the NYSE and NYSE MKT customer gateways were not loaded with the proper configuration compatible with the new release.
The "SIP timestamp requirement" mentioned in the statement is an interesting topic in itself. Bloomberg has a bit more detail about this bit of esoterica here: http://www.bloombergview.com/articles/2015-07-09/market-complexity-broke-the-nyse-before-saving-it.

Source: https://www.nyse.com/market-status/history