Cyber threats are becoming a cost of doing business for enterprises, as attacks surged last year, and the trend has only continued. Recent infrastructure failures have exposed these vulnerabilities, showing how a single provider issue can disrupt customers’ operations for hours. When systems fail, it’s not just resumption of service that matters for customers but whether their data has been safeguarded or exposed. As AI increasingly seeps into every aspect of work, engineering leaders must build products that, by default, collect evidence in the form of reliable, curated data on incidents.
Public post-mortems are a wake-up call.
In the global Cloudflare incident, a single data center power fault took down logging and analytics, and because critical control functions relied on that telemetry, a local failure became a global outage. If your telemetry fails and your observability is lost, you lose both control in the moment and the ability to explain what happened afterward.
I saw this up close during the Cloudflare outage, as head of engineering at a top fleet maintenance technology company. Our AWS application and database tiers stayed healthy, but customers couldn’t reach the service. We prioritized keeping customers online by flipping to a tested provider bypass, rerouting traffic, and restoring access in less than three hours. We also collected the logs and traces that formed our forensic bundle while updating our customers via status pages and direct outreach. The post-mortem led to lasting fixes, including automatic failover readiness and an automated bypass runbook. Mitigation and forensic collection are now repeatable parts of our response roadmap.
This incident reinforced my belief that it is table stakes for business resilience to continuously collect usable evidence of what happened during an outage. After an incident, procurement teams and insurers won’t accept “sorry.” They want proof – proof of what happened to their information during the downtime, and proof of a fix. We must be ready to deliver repeatable forensic bundles, like raw traces and logs, that customers can hand to auditors or insurers. There are crucial steps companies can take to achieve this, including embedding forensic data collection into their product architecture via modular logging frameworks, securing storage, and automating bundle generation, making forensic readiness seamless in the development process.
Designing a Forensic-First Engineering Roadmap
1. Build forensic-grade telemetry as a product.
Telemetry should be handled like any other core product data. Recorded events must remain unaltered once written, locking in the source hardware and software versions for later verification. Retention spans need to cover audit periods in full, with automatic packages generated per client that contain sequences, original records, and supplementary details, with retrieval possible at any time. Exports should follow fixed formats, building reliable timelines accepted by clients and underwriters alike.
2. Inform customers of the incident in real time.
Put the incident narrative in a customer-accessible portal that shows impacted accounts and a running timeline of what you’re doing to fix it. Provide a detailed forensic package for each customer, available for download, along with a concise overview for departments such as purchasing or compliance. This type of transparency strengthens confidence and reduces risk to existing contracts.
3. Translate resilience into dollars.
Redundancy and fallback modes directly impact business value. Translate controls into a simple “expected-loss” number: chance of failure × cost of failure. If staged releases reduce the risk of a significant outage from 2% to 0.5%, the expected loss falls by 75%, showing how resilience controls save money and help companies see their work as strategic and valuable.
Use incidents to generate artifacts that will hold up in an audit. Run cross-functional tabletop exercises (engineering, security, legal, sales, and support) that simulate corrupted data and supply-chain compromise. From each exercise, produce procurement-grade deliverables, such as an incident-disclosure policy, a standardized forensic bundle format, and a Service Level Agreement (SLA) blueprint. Then automate the process, making forensic package creation a single, repeatable operation rather than a month of ad hoc data pulls.
Adapting to a Continuously Shifting Threat Landscape
Continuity only goes so far if you can’t prove it with evidence. With cyberthreats and provider failures accelerating, the C-suite should focus on how many customers remained unaffected and how quickly we can prove it. Ask your engineering team to produce one forensic bundle within 30 days. This demonstrates your team’s readiness and commitment to accountability, fostering confidence across stakeholders.
Imagine you’re a customer whose platform has just failed. You’ve refreshed the page ten times, critical work has stopped, and you don’t know whether your data is safe. An uptime percentage tells you almost nothing. What builds confidence is a timestamped account of what happened: what failed, what continued working, what data remained protected, and what the company did in response. In moments of disruption, being able to hand a customer a transparent, timestamped account of events (what failed, what didn’t, and what data remained protected) builds better trust and resilience.
That standard will only tighten as supply chain attacks grow more frequent and more sophisticated. The teams that win in this environment will have clear, consistent evidence that withstands scrutiny. It lowers the risk of disputes and gives you a chance to build credibility. Over time, this level of transparency compounds into stronger customer relationships, smoother renewals, and a reputation for reliability that extends beyond performance metrics.
