Duration: 8:27 AM – 10:08 AM (Europe/Berlin)

Summary:

In the early morning hours, we detected minor network instability on the IOTA validator “PANDABYTE II.” To restore stability, PANDABYTE II was restarted at 8:27 AM. As a result, the validator did not participate in the timely closure of Epoch 252 and became stuck in Epoch 252. Following an analysis together with the IOTA Foundation, we reset the current database, re-initialized it via an official formal snapshot, and completed re-synchronization.

Impact:

• Delayed participation in the Epoch 252 closure due to the restart and subsequent recovery actions
• Validator was temporarily not able to progress correctly / participate in consensus until snapshot re-initialization and sync completed
• Validator remained in the committee throughout; no lasting impact on validator status
• No impact on pool rewards for delegators

Root Cause:

Minor network instability prompted a restart at 8:27 AM. The restart coincided with the timing of the Epoch 252 closure, causing PANDABYTE II to miss the closure and remain stuck in that epoch.

Resolution:

• Restarted PANDABYTE II at 8:27 AM to stabilize operations
• Performed joint analysis with the IOTA Foundation
• Executed a database reset and re-initialization from a formal snapshot
• Completed full re-synchronization and restored normal operation

Next Steps / Preventative Measures:

• Implement measures to avoid collisions around epoch closures, particularly regarding restart timing
• Improve monitoring and automation to detect early network instability and enforce controlled recovery procedures
• Further harden operational processes to ensure a more stable validator operation

We apologize for any inconvenience caused and remain committed to providing reliable validator services.

Special thanks to Alexander Sporn (IOTA Foundation) for the fast and excellent support.