Hit the end of buffer, expected more data

Hi, after the new update of Fuel Core to 0.31.0
I’ve encountered this weird behavior while the node is running as seen below.
Also the syncing is a bit slower as it always downloading many blocks and processing it at once instead of syncing latest block like last version.

2024-07-13T16:25:34.234808Z ERROR new{name=puckanode-sepolia-testnet}:initialize_loop{service="P2P"}: fuel_core_p2p::p2p_service: 626: RequestResponse outbound error for peer: PeerId("16Uiu2HAmDxoChB7AheKNvCVpD4PHJwuDGn8rifMBEHmEynGHvHrf") with id: OutboundRequestId(25) and error: Io(Custom { kind: Other, error: "Hit the end of buffer, expected more data" })
2024-07-13T16:25:34.234979Z ERROR fuel_core_services: 55: Failed to get headers Invalid response from peer P2P(Io(Custom { kind: Other, error: "Hit the end of buffer, expected more data" }))
2024-07-13T16:25:34.235093Z  INFO new{name=puckanode-sepolia-testnet}:initialize_loop{service="ImportTask"}:import:failed_to_process{self=State { status: Processing(4975845..=4975888) } range=4975845..=4975868}: fuel_core_sync::state: 204: Status change from: Processing(4975845..=4975888), to: Committed(4975844)
2024-07-13T16:25:34.235106Z ERROR new{name=puckanode-sepolia-testnet}:initialize_loop{service="ImportTask"}: fuel_core_sync::service: 178: error=Failed to import range of blocks: 4975845..=4975868
2024-07-13T16:25:34.235113Z ERROR new{name=puckanode-sepolia-testnet}:initialize_loop{service="ImportTask"}: fuel_core_services::service: 384: e=Failed to import range of blocks: 4975845..=4975868
2024-07-13T16:25:34.336447Z  INFO new{name=puckanode-sepolia-testnet}:initialize_loop{service="SyncTask"}:sync:observe{self=State { status: Committed(4975844) } height=4975889}: fuel_core_sync::state: 204: Status change from: Committed(4975844), to: Processing(4975845..=4975889)

Hey @puckapao, the slow syncing speed might be due to the SRs attacking the node and making many requests in a short time. Though, I will make sure that the team look at it and see if there’s any other issue

Can you make sure you are using the correct chainconfig for the snapshot?

Yes, I’m using Ignition chainconfig since previous version and it worked fine. But after the update, it’s broken and sync every 100 blocks not real time syncing.

I wonder, what’s the usecase here if the node need to download about 100 blocks behind and playing endless catching up. I prefer previous version of fuel-core that allow real time syncing. Is there any issue with block proproser or prover? Could it be possible to open that for public to try out and reduce workload on current proposer or prover?

Can you please share the output for the fuelup show from your terminal? it will help me to debug the issue

Installed toolchains
--------------------
latest-x86_64-unknown-linux-gnu (default)

active toolchain
----------------
latest-x86_64-unknown-linux-gnu (default)
  forc : 0.61.2
    - forc-client
      - forc-deploy : 0.61.2
      - forc-run : 0.61.2
    - forc-crypto : 0.61.2
    - forc-debug : 0.61.2
    - forc-doc : 0.61.2
    - forc-fmt : 0.61.2
    - forc-lsp : 0.61.2
    - forc-tx : 0.61.2
    - forc-wallet : 0.8.1
  fuel-core : 0.31.0
  fuel-core-keygen : 0.31.0

fuels versions
--------------
forc : 0.64.0
forc-wallet : 0.64.0

what command are you running to start the node and what chainconfig are you using for the snapshot?

fuel-core run \
--service-name=nodecattel-fuel-testnet \
--keypair {my-pv-key} \
--relayer {my-sepolia-rpc} \
--ip=0.0.0.0 --port=4000 --peering-port=30333 \
--db-path $HOME/.fuel-sepolia-testnet \
--snapshot $HOME/.fuel-sepolia-testnet \
--utxo-validation --poa-instant false --enable-p2p \
--reserved-nodes /dns4/p2p-testnet.fuel.network/tcp/30333/p2p/16Uiu2HAmDxoChB7AheKNvCVpD4PHJwuDGn8rifMBEHmEynGHvHrf \
--sync-header-batch-size 100 \
--enable-relayer \
--relayer-v2-listening-contracts=0x01855B78C1f8868DE70e84507ec735983bf262dA \
--relayer-da-deploy-height=5827607 \
--relayer-log-page-size=500 \
--sync-block-stream-buffer-size 30

Ignition chain config is moved into .fule-sepolia-testnet

I have the same issue here. I don’t know why the node sync 100 block per time and keep failing.

I start the node with

sudo tee /etc/systemd/system/fueld.service > /dev/null << EOF
[Unit]
Description=Fuel Node Beta-5
After=network.target

[Service]
User=$USER
Type=simple
ExecStart=$(which fuel-core run) \
--service-name fuel-sepolia-testnet \
--keypair <P2P key> \
--relayer <ETH RPC API> \
--ip=0.0.0.0 --port=4000 --peering-port=30333 \
--db-path $HOME/.fuel-sepolia-testnet \
--snapshot $HOME/.fuel-sepolia-testnet \
--utxo-validation --poa-instant false --enable-p2p \
--reserved-nodes /dns4/p2p-testnet.fuel.network/tcp/30333/p2p/16Uiu2HAmDxoChB7AheKNvCVpD4PHJwuDGn8rifMBEHmEynGHvHrf \
--sync-header-batch-size 100 \
--enable-relayer \
--relayer-v2-listening-contracts=0x01855B78C1f8868DE70e84507ec735983bf262dA \
--relayer-da-deploy-height=5827607 \
--relayer-log-page-size=500 \
--sync-block-stream-buffer-size 30

Restart=on-failure
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target 
EOF

The output from fuelup show below;

Default host: x86_64-unknown-linux-gnu
fuelup home: /root/.fuelup

Installed toolchains
--------------------
latest-x86_64-unknown-linux-gnu (default)

active toolchain
----------------
latest-x86_64-unknown-linux-gnu (default)
  forc : 0.61.2
    - forc-client
      - forc-deploy : 0.61.2
      - forc-run : 0.61.2
    - forc-crypto : 0.61.2
    - forc-debug : 0.61.2
    - forc-doc : 0.61.2
    - forc-fmt : 0.61.2
    - forc-lsp : 0.61.2
    - forc-tx : 0.61.2
    - forc-wallet : 0.8.1
  fuel-core : 0.31.0
  fuel-core-keygen : 0.31.0

fuels versions
--------------
forc : 0.64.0
forc-wallet : 0.64.0

This is the error that I encounter

I will forward your question to the team, but as the last try can you try using the nightly toolchain? You might need to run the command fuelup toolchain install nightly and then run fuelup default nightly . You can also follow the guide here.

1 Like

Thank you! I will try this

1 Like