Crisis Management: Operational Leadership

My biggest takeaway from this class: have plenty of specialists on board as suppliers, and have a plan ready in advance.

Case Summary

On June 27, 2017, the destructive worm (a self-propagating virus) NotPetya wiped out almost the entire digital system of TNT Express, which FedEx had just acquired. FedEx immediately activated its crisis playbook (BCDR), "cutting off + protecting itself" on one hand and "relaying by hand + rebuilding step by step" on the other — a live exercise in turning a "technical disaster" into "operational resilience."

That morning, the Americas headquarters received an alert from Europe. TNT's computers and servers went down like dominoes: "black screen + ransom screen." At first it looked like ransomware, but it was soon confirmed to be wiper malware, whose aim was "destruction," not "money." It spread extremely fast: within minutes "a whole site was down," within tens of minutes "clients worldwide were down." About 40,000 of TNT's computers + 10,000 servers were written off.

FedEx quickly set up a Cyber Incident Response Team (CIRT) and cut the connection between TNT and FedEx ("hit the big red button") to protect the "mothership." That night it decided to report a material event to the SEC (8-K) and briefly halted trading to explain. The share price stayed basically steady. The next day it issued a press release: other businesses were operating normally, TNT was affected, and the financial impact "could be material."

On the technical side, a war room was set up in Atherstone, UK. All TNT had left were scattered paper network diagrams and files on a few laptops that hadn't been networked. The phone system was down, so they communicated via WhatsApp. The team first rebuilt accounts/directory services (Active Directory) and "the five things needed to stay alive" (communications, payables/receivables, etc.).

On the operations side, they didn't shut down. Using the primary-secondary-tertiary (P-S-T) model: whatever could be switched to FedEx systems was switched; whatever couldn't went "fully manual." Trucks kept running, with routing done by "muscle memory." Meanwhile "FedEx Bank" injected working capital to make sure TNT stayed alive first.

The backup system was the pain point. TNT mainly used cold backups on tape, which couldn't be restored quickly in parallel. FedEx made an emergency purchase of hard drives, copied the tapes onto disks for parallel recovery, and created "three copies" of new backups — cloud, disk and offline; privileged accounts were cut from "thousands of people with access" to "just 8 people, like FedEx." At the same time, 900 applications were shed.

On the customer side, the CIO's team reassured 40 major customers "one-on-one." There were still week-long delays, the media reported "parcels piled to the ceiling" in warehouses, and competitors seized the chance to poach customers. FedEx had no cyber insurance at the time, and the losses were mainly lost business.

Tracing it back confirmed it was a supply-chain attack, infiltrating through the legitimate update channel of a Ukrainian accounting software package. Many global giants were hit, with total losses estimated at ≥ US$10 billion. The share price did not take much of a hit.

Class Notes

1. When a crisis hits, are you in Plan mode or Responding mode?

2. A crisis management company can first become a supplier and run crisis rehearsals for the company. Let the risk management side run "phishing stings" to check who is the weak point; anyone caught has to take an extra two-day weekend training.

3. Take Blame and Failure out of the discussion — Focus on MissionCEO NOT emotional is really important

4. Before bring a new company to your car, you need to understand the materials of this new company.

5. What is the last time you ask your people to put the company above them.

6. Are your people BURN OUT?

All My Crisis Response Advice

Stabilise the situation first

Confirm the real scope and impact of the crisis; avoid both excessive panic and overreaction.

Immediately set up a temporary crisis team to centralise communication and decisions.

Protect the core

First secure cash flow, key customers, core employees and brand reputation.

If it's a PR crisis, quickly align on one message to avoid confusion.

Communicate transparently

Internally: let employees know what's going on, to head off rumours and panic.

Externally: stay reasonably transparent with customers, partners and the media — acknowledge the problem and communicate the solution.

Draw up a short-term action plan

Set clear priorities within 72 hours (e.g. stopping the losses, stabilising relationships, compliance response).

Break the problem into a few actionable small steps and carry them out one by one.

Long-term repair and prevention

After the crisis, do a post-mortem: why did it happen, and which internal mechanisms failed?

Build an "early-warning system" and drill routines, turning the crisis into an opportunity to upgrade the organisation.