Postmortem Report | Nov 15, 2022

Postmortem Report | Nov 15, 2022

This postmortem report is written for and provided to the community to provide transparency into a recent outage. This article will describe the event, its root cause, and its resolution while highlighting how improvements will be made in an effort to prevent a recurrence of the issue.

Event Date

Impact & Duration

  • Mainnet consensus halted
  • Duration: 4.5 Hours

Overview

What Occurred

Root Cause(s)

Resolution

Moving Forward

Additional Testing via Mainnet-Mock

  • Upgrades with impact potential will undergo additional rounds of testing on a transient network called “Mainnet-Mock”. This network will be an identical copy of Mainnet, including its data and validators, acting as a mirror of Findora’s production network.
  • Mainnet-Mock will be created and deconstructed as needed. As a mirror copy of Mainnet, testing against the Mainnet-Mock network will require the assistance and close communication of Findora community validators to reach > 66.67% consensus.
  • Upgrades that carry potential impact will be validated on the Mainnet-Mock network before being classified as “Mainnet-Ready”.

Change Management Process Improvements

  • Discreet Labs will continue to ensure that partners and CEX contacts are notified of upcoming changes using the schedule above. Technical support will continue to be provided when needed.
  • The details of changes that can impact production will be communicated extensively with the community well in advance of the scheduled change date. Discreet Labs will share information two weeks before the change, one week prior, three days prior, one day prior, the day of the change, and the day after.
  • Change information will be shared across social media, pinned to the official Discord and Telegram channels and Reddit as announcement posts. Discreet Labs will communicate to the validator community on validator-specific channels across Discord and Telegram.
  • Mainnet-Mock testing will be conducted as part of the change management process. See above for information on Mainnet-Mock.

About Findora