Mainnet Node: DA layer out of sync

General
1

Topic discussion checklist

  • Run $ fuelup show and add the output below
  • Detailed steps to recreate issue
  • Link full scope of the codebase pertaining to your issue
  • Any other relevant information

After running fuelup show

Default host: aarch64-apple-darwin
fuelup home: /Users/megabyte0x/.fuelup

Installed toolchains
--------------------
latest-aarch64-apple-darwin (default)

active toolchain
----------------
latest-aarch64-apple-darwin (default)
  forc : 0.66.5
    - forc-client
      - forc-deploy : 0.66.5
      - forc-run : 0.66.5
    - forc-crypto : 0.66.5
    - forc-debug : 0.66.5
    - forc-doc : 0.66.5
    - forc-fmt : 0.66.5
    - forc-lsp : 0.66.5
    - forc-tx : 0.66.5
    - forc-wallet : 0.11.1
  fuel-core : 0.40.0
  fuel-core-keygen : 0.40.0

fuels versions
--------------
forc : 0.66.10
forc-wallet : 0.66.9

Commands to recreate the issue

  1. Clone the repo: GitHub - FuelLabs/chain-configuration
  2. Since I am running local node (for now): ulimit -S -n 32768
  3. Run Command:
fuel-core run \
--enable-relayer \
--service-name fuel-mainnet-node \
--keypair {P2P_PRIVATE_KEY} \
--relayer {ETHEREUM_RPC_ENDPOINT} \
--ip=0.0.0.0 --port 4000 --peering-port 30333 \
--db-path ~/.fuel-mainnet \
--snapshot ./your/path/to/chain_config_folder \
--utxo-validation --poa-instant false --enable-p2p \
--bootstrap-nodes /dnsaddr/mainnet.fuel.network \
--sync-header-batch-size 100 \
--relayer-v2-listening-contracts=0xAEB0c00D0125A8a788956ade4f4F12Ead9f65DDf \
--relayer-da-deploy-height=20620434 \
--relayer-log-page-size=100 \
--sync-block-stream-buffer-size 30

Logs I am receiving

2024-12-25T09:39:31.068701Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-25T09:39:31.109144Z  INFO new{name=fuel-mainnet-node}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Processing(424887..=10000639) } height=10000640}: fuel_core_sync::state: 204: Status change from: Processing(424887..=10000639), to: Processing(424887..=10000640)

Another thing which I noticed was that my RPC provider (ANKR with Premium Sub) is not getting any request from the node.

1 Like
2

Hey @megabyte0x, how are you doing?
are you still experiencing the issue? Could you please share more logs?

3

hey man! I am doing good. What about you?

I was just trying to run the node and yes still facing the same issue.

The logs are mostly same, here they are:

..=10072437)
2024-12-26T05:46:13.350166Z  INFO new{name=fuel-mainnet-node}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Processing(424887..=10072437) } height=10072438}: fuel_core_sync::state: 204: Status change from: Processing(424887..=10072437), to: Processing(424887..=10072438)
2024-12-26T05:46:13.639830Z  INFO new{name=fuel-mainnet-node}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Processing(424887..=10072438) } height=10072439}: fuel_core_sync::state: 204: Status change from: Processing(424887..=10072438), to: Processing(424887..=10072439)
2024-12-26T05:46:13.724717Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:13.724592Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.016056Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.016366Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.016835Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.017055Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.022457Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.308685Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.310709Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.310798Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.311952Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.314800Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.315063Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.315407Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.315475Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.356901Z ERROR new{name=fuel-mainnet-node}:initialize_loop{service="Relayer"}: fuel_core_services::service: 373: e=Failed to check if DA layer is syncing No Quorum reached.
2024-12-26T05:46:14.604938Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.605188Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.605311Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.606325Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.693326Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:14.693460Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.285769Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.285802Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.296705Z  INFO new{name=fuel-mainnet-node}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Processing(424887..=10072439) } height=10072440}: fuel_core_sync::state: 204: Status change from: Processing(424887..=10072439), to: Processing(424887..=10072440)
2024-12-26T05:46:15.441089Z  INFO fuel_core_txpool::update_sender: 82: Transaction ae3205c28d2dc5887baa26d99689ab2c1a69fbcee7dbc7f047554290a0278c9c squeezed out because Transaction input validation failed: UTXO (id: 5e577b4112c14627412741c9c6375aef59838c5a8fcece029eb91f59cb1455270003) does not exist
2024-12-26T05:46:15.586141Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.586141Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.586206Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.628245Z  INFO new{name=fuel-mainnet-node}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Processing(424887..=10072440) } height=10072441}: fuel_core_sync::state: 204: Status change from: Processing(424887..=10072440), to: Processing(424887..=10072441)
2024-12-26T05:46:15.810788Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:15.879128Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:16.101456Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:16.174498Z ERROR fuel_core_services: 70: Failed to wait for DA layer to sync Relayer is too far out of sync
2024-12-26T05:46:16.628697Z  INFO new{name=fuel-mainnet-node}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Processing(424887..=10072441) } height=10072442}: fuel_core_sync::state: 204: Status change from: Processing(424887..=10072441), to: Processing(424887..=10072442)
2024-12-26T05:46:17.714245Z  WARN fuel_core_upgradable_executor::executor: 459: The block version(10) is different from the native executor version(16). The WASM executor will be used.
4

I am doing good too and thanks for sharing the logs. I will share the logs internally and will get back to you

1 Like
5

Hey @megabyte0x, this error means that the Ethereum RPC provider is either slow or doesn’t work at all. Maybe it is because you have hit the limit of requests on your key? To help you better we will need full logs.