Unichain - Unichain Mainnet: Post-Fusaka Safe Head Stall – Incident details

All systems operational

Unichain Mainnet: Post-Fusaka Safe Head Stall

Resolved
Major outage
Started 3 days agoLasted about 4 hours

Affected

Unichain Mainnet

Major outage from 11:00 PM to 2:32 AM, Operational from 2:32 AM to 2:32 AM

Public RPC

Major outage from 11:00 PM to 2:32 AM, Operational from 2:32 AM to 2:32 AM

L2 Block Production

Major outage from 11:00 PM to 2:32 AM, Operational from 2:32 AM to 2:32 AM

L1 Batch Posting

Major outage from 11:00 PM to 2:32 AM, Operational from 2:32 AM to 2:32 AM

Node Syncing

Major outage from 11:00 PM to 2:32 AM, Operational from 2:32 AM to 2:32 AM

Flashblocks Websocket Stream

Major outage from 11:00 PM to 2:32 AM, Operational from 2:32 AM to 2:32 AM

Updates
  • Postmortem
    Postmortem

    Unichain Mainnet erroneously posted the same batch transaction at two different nonces. This triggered a corner case in the consensus layer client that made it impossible for the L2 Safe Head to progress. OP Labs has a fix available: us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:v1.16.3. Please upgrade your consensus layer clients on Unichain to use this image until OP Labs provides a newer release.

  • Resolved
    Resolved

    We have identified that the root cause of the safe head stall on Unichain Mainnet is due to a bug in how op-node handles fetching blobs. All nodes operators will need to update their op-node tous-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:e1db990da90a845e1943b1dc20f59e18099db538

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    After working with OP Labs, we have identified both an issue in our batch submission and in the consensus client handling of such a case. We are trying to roll out a new version of the consensus client that fixes this issue.

  • Investigating
    Investigating
    We are currently investigating this incident.