United, NYSE and WSJ, Oh My!
“Everyone thinks of changing the world, but no one thinks of changing himself.” Leo Tolstoy
July 8, 2015 was certainly an interesting news day. United Airlines temporarily grounded flights, the NYSE halted trading for hours, and the Wall Street Journal (WSJ) home page was down for a while. All three were attributed to “technology fails,” ranging from a change gone wrong to capacity issues.
Every day our dependence on technology increases. We joke about digital addiction, but digital dependence is not funny. The now-cancelled television show “Revolution” depicted a world where power, and ergo technology, abruptly becomes unavailable. Mayhem predictably ensues. The “Terminator” series (now re-booted, which if you think about it is a pretty amusing term for a movie about technology) takes the opposite view, that technology takes over. Mayhem predictably ensues. (Frankly, I think the Stephen King short story “Trucks,” from the 1970s, is more frightening.)
“Technology fails” is not a full description of what happened on July 8. Technology usually doesn’t suddenly fail. The root cause of issues is often a result of a lack of investment, poor design, insufficient planning, or human error.
The NYSE issue is attributed to a software upgrade gone wrong. Upgrading technology, especially complex technology with high demands like NYSE’s, requires investment and discipline.
United claims a connectivity issue. A company’s myriad of software and hardware has to ‘talk’ to each other constantly to keep the lights on, and just like people misunderstanding each other, sometimes that communication breaks down.
WSJ indicated a capacity issue with its servers (ironically, perhaps due to intense traffic related to people wanting news on the NYSE problem). The timing and circumstance of high usage of web sites can be difficult to predict. There are certainly obvious peaks (retail sites before the holidays, cable television during the “Game of Thrones” season finale), but unpredictable spikes are becoming more common, and valleys few and far between.
CIOs and their teams are under pressure to keep the lights on and continually deliver innovation to enable the business. Sometimes, a chainsaw falls out of that juggling act in the form of a major outage. July 8th was a reminder to CXOs and Boards that investment in technology and the talent that designs, implements, and runs systems is critical. Investments such as…
- Redundancy and disaster recovery. When budgets get tight, one of the first things to get slimmed down is disaster recovery aka DR. There is often a misconception that everything is backed up, and if failures occur, there is instantaneous fail-over to a backup system. Clear and transparent communication and documentation about the DR (or not) between CXOs is paramount.
- Security. Though apparently the July 8th troika of issues was not due to cyber-attacks, don’t forget notable hacks like Sony and the U.S. government. Outdated and/or ill-maintained technology is a prime target.
- Change management and testing. Another victim of tight budgets can be attention to test systems that are identical or near-identical to production, so that changes can be put through their paces before implementation. Further, complexity can make complete test systems near impossible and or extremely costly.
- Talent. You can get the best technology money can buy, but if you don’t get the qualified personnel to go with it, likely the investment will be wasted.
July 8, 2015, was another wake-up call that change is needed in leadership understanding and attention to technology or July 8 will become the next Groundhog Day.
– Fascinating and sobering that Greece, when seeking to enter the eurozone in the early 2000s, was able to mislead other countries into believing it’s deficit was less than 3% of its GDP, when in fact it was far higher. Read more here.
– If you are one of the Microsoft employees being laid off, check out jobs in higher education, including those at #MSU.