We’d like to share some context around a recent infrastructure issue that affected timestamps on our devnet, testnet, and mainnet environments, impacting services like Pyth and Swaylend.
What Happened?
Fuel recently migrated its core network infrastructure onto a new provider. During this transition, the new servers were provisioned with a basic OS setup by the provider and handed off to us. However, these machines did not have Network Time Protocol (NTP) synchronization configured.
As a result, many of the new servers were ~90 seconds out of sync with real clock time. This discrepancy led to issues with timestamp-sensitive components like Pyth, which began rejecting data due to out-of-date oracle timestamps.
Impact
- Block production was briefly disrupted (~2 minutes) in each environment.
- Time-sensitive protocols like Pyth were affected by invalid timestamps.
- Issue dates back to:
- May 13th for Testnet
- May 29th for Mainnet
The Fix
We have now:
- Enabled network time syncing across all hosts.
- Updated our infrastructure provisioning guide to ensure time sync setup is explicitly verified during every handoff going forward.
Following this update, all networks are back to normal, and we’ve confirmed that services dependent on accurate timestamps (like Pyth) are now functioning as expected.
If you have any questions or need more context, feel free to reach out to us. We appreciate your patience while we were ironing this out.