Tokyo’s Stock Exchange (TSE) went offline for most of Thursday, its longest-ever outage and a very unwelcome one as it is the world’s third-largest bourse, when measured by market capitalisation.
The exchange yesterday morning posted news that “a technical glitch occurred to distribution of market data,” and the market therefore stopped all trading. Later in the day the bourse also took down its after-hours trading platform, ToSTNeT, and then issued warnings that some market data distributed to investors was invalid.
The exchange explained the cause of the outage in a statement that said it experienced “hardware failure,” followed by a failure-to-failover.
The statement continued: “the switchover from the failed device to the backup device did not work properly, and as a result, market information could not be distributed.”
Which sounds very like someone hasn’t run a disaster recovery simulation for a while.
While the exchange thought it could replace the hardware and resume trading, doing so would have required a reboot that it felt “would cause confusion for investors and market participants, which would make it difficult to execute smooth trading.”
After talks with stakeholders, it was decided to just give up on the day and resume on Friday. At the time of writing – a few minutes after Friday’s opening bell – that plan appears to have worked.
The exchange has apologised for the outage, and taken responsibility for the situation, and also made it plain that mess was the result of its own mistakes and key technology provider Fujitsu was not at fault.
Fujitsu promotes TSE’s use of “approximately 200” of its Primergy servers and the Primesoft in-memory data management software.
That combo can apparently handle 100 million orders a day, at a rate of 1.4 million order-per-minute, all with transaction time of 300 microseconds apiece. Well, sometimes.
Fujitsu has reportedly apologised for its role in the outage.
The exchange continues to do so at every opportunity, with its notification that it expects normal trading today ending with: “We would like to express our sincerest apologies for the inconvenience caused by the system failure of Tokyo Stock Exchange, and we would like to ask for your continued support and cooperation in the operation of the market.”
Robin Edgar
Organisational Structures | Technology and Science | Military, IT and Lifestyle consultancy | Social, Broadcast & Cross Media | Flying aircraft