Our brother team reported that the indexer in their project used Hyperindex, which often caused problems, and the purchased vps was destroyed due to excessive CPU usage. We also tried to use Hyperindex, but it was indeed too heavy. Is there an official indexer? Available?
We have quite a few options. You can choose among subsquid, envio, or superchain
Hi there, the co-founder of Envio. Sorry to hear about the issues you’re facing with HyperIndex. We’d love to hop on a call to better understand your experience and help you out.
Just to clarify, HyperIndex is a very light JavaScript process—on our hosted service, it typically idles at a tiny fraction of a CPU. Is there possibly something inside one of your handlers or a side process that is causing the CPU usage. For convenience, I’d recommend trying out our hosted service.
If HyperIndex is too heavy, you might find our lower-level API, HyperFuel, more flexible. We offer libraries in Rust, Python, and JavaScript depending on your preference.
Envio is the most widely used indexer for Fuel, and we’re committed to building the best tools for the Fuel ecosystem.
Looking forward to connecting!
3 Likes
Hi @Clara - I’d love to hear more about the problems experienced as it will help drive improvements. As Jason mentioned, its very unlikely due to excessive usage, but more likely due to the recent major reorgs that present a problem for all indexers.
I’d recommend trying out this tutorial: Indexing Sway Farm on the Fuel Network | Envio
and hopping into the Envio discord. We’d be more than happy to help you on your way and I’m sure you’ll be pleasantly surprised.
Also check out this Indexing on Fuel video which I am sure will help answer many of your questions: https://youtu.be/BqiCjLKorRs
Have a great day!
2 Likes
Hello @Nazeeh21 @JasoonS @jonjon
The Index still have some problems.
The development of the first version of our full-chain game is almost completed, but there has been a problem with the indexer. Due to time reasons, we chose the hyperindex we used before, but the new version could not run, so we tried to restore the version in the development environment, and it worked Running.
But unfortunately, we found the culprit of high CPU usage: Virus!
Yes, this virus is hidden in hyperindex’s docker. The following is the process of discovering the virus(named kdevtmpfsi).
root@02-sk01:~/indexjs# top
top - 04:09:49 up 22 min, 1 user, load average: 0.86, 0.53, 0.34
Tasks: 164 total, 1 running, 163 sleeping, 0 stopped, 0 zombie
%Cpu(s): 99.7 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 959.0 total, 68.0 free, 643.8 used, 247.2 buff/cache
MiB Swap: 2400.0 total, 2386.2 free, 13.8 used. 156.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10028 systemd+ 20 0 717932 270196 0 S 99.0 27.5 0:18.72 kdevtmpfsi
1388 linuxus+ 20 0 1024.6g 137964 6528 S 0.3 14.0 0:11.66 graphql-engine
1 root 20 0 171196 10712 6620 S 0.0 1.1 0:06.42 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
9 root 20 0 0 0 0 S 0.0 0.0 0:00.38 ksoftirqd/0
10 root 20 0 0 0 0 I 0.0 0.0 0:00.34 rcu_sched
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
12 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_inject/0
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
16 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_tasks_kthre
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
21 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
23 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
24 root 39 19 0 0 0 S 0.0 0.0 0:00.00 khugepaged
70 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kintegrityd
71 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockd
72 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 blkcg_punt_bio
73 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 tpm_dev_wq
74 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 ata_sff
75 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 md
First I try to delete the virus
root@02-sk01:~/indexjs# ps -aux | grep kinsing
systemd+ 5960 0.0 1.1 708612 10848 ? Sl 03:59 0:00 /tmp/kinsing
root 10690 0.0 0.0 6432 656 pts/0 S+ 04:11 0:00 grep --color=auto kinsing
root@02-sk01:~/indexjs# kill -9 5960
root@02-sk01:~/indexjs# ps -aux | grep kdevtmpfsi
systemd+ 10531 61.0 27.5 652356 270160 ? Ssl 04:10 0:41 /tmp/kdevtmpfsi
root 10796 0.0 0.0 6432 656 pts/0 S+ 04:11 0:00 grep --color=auto kdevtmpfsi
root@02-sk01:~/indexjs# kill -9 10531
root@02-sk01:~/indexjs# rm -rf /tmp/kdevtmpfsi
root@02-sk01:~/indexjs# rm -rf /var/tmp/kinsing
Next I want to find out which application the virus is hiding in.
root@02-sk01:~/indexjs# sudo crontab -l
no crontab for root
root@02-sk01:~/indexjs# find / -name kdevtmpfsi
/var/lib/docker/overlay2/2586c3fd0a48a576b763dd2797e9218516c528d41b32783e4f7498af8356b13c/diff/tmp/kdevtmpfsi
/var/lib/docker/overlay2/2586c3fd0a48a576b763dd2797e9218516c528d41b32783e4f7498af8356b13c/merged/tmp/kdevtmpfsi
root@02-sk01:~/indexjs# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bef8b7b8725f hasura/graphql-engine:v2.23.0 "/bin/sh -c '\"${HGE_…" 2 months ago Up 26 minutes (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp generated-graphql-engine-1
14f4d0681f7a postgres:16 "docker-entrypoint.s…" 2 months ago Up 26 minutes 0.0.0.0:5433->5432/tcp, :::5433->5432/tcp generated-envio-postgres-1
root@02-sk01:~/indexjs# docker inspect 14f4d0681f7a
[
{
"Id": "14f4d0681f7a7b013b3fe7f263f6239b65a99b0a0c334ad0a08991322ae6ee4e",
"Created": "2024-06-27T01:41:52.561252098Z",
"Path": "docker-entrypoint.sh",
"Args": [
"postgres"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1291,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-08-28T03:47:23.166257543Z",
"FinishedAt": "2024-08-28T03:45:55.945468057Z"
},
"Image": "sha256:74cc00b2e28f8b5cad42680cc425b261544eee3dfe70fbdf903015dac9b0fe4a",
"ResolvConfPath": "/var/lib/docker/containers/14f4d0681f7a7b013b3fe7f263f6239b65a99b0a0c334ad0a08991322ae6ee4e/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/14f4d0681f7a7b013b3fe7f263f6239b65a99b0a0c334ad0a08991322ae6ee4e/hostname",
"HostsPath": "/var/lib/docker/containers/14f4d0681f7a7b013b3fe7f263f6239b65a99b0a0c334ad0a08991322ae6ee4e/hosts",
"LogPath": "/var/lib/docker/containers/14f4d0681f7a7b013b3fe7f263f6239b65a99b0a0c334ad0a08991322ae6ee4e/14f4d0681f7a7b013b3fe7f263f6239b65a99b0a0c334ad0a08991322ae6ee4e-json.log",
"Name": "/generated-envio-postgres-1",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "local_test_network",
"PortBindings": {
"5432/tcp": [
{
"HostIp": "",
"HostPort": "5433"
}
]
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"Mounts": [
{
"Type": "volume",
"Source": "generated_db_data",
"Target": "/var/lib/postgresql/data",
"VolumeOptions": {}
}
],
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware",
"/sys/devices/virtual/powercap"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/*2586c3fd0a48a576b763dd2797e9218516c528d41b32783e4f7498af8356b13c-init/diff*:/var/lib/docker/overlay2/d068a31268cf774521eb2e5d624ad5b488be01dc18136ee508c14dd39d2bb85b/diff:/var/lib/docker/overlay2/296372ea0f5d9a7edadbfe0e6f5bf06d09755bdffaa51aedf8cb7dc3e3b10084/diff:/var/lib/docker/overlay2/83eeafde857b89fbb32e6ad5b679e5853d145c623d285a16a65c9222d3a2f491/diff:/var/lib/docker/overlay2/b4e155dfd337d667e83966cf11af910916849778bc9080dff5fdc74ff29a8754/diff:/var/lib/docker/overlay2/1e7d68abd2b138562d0e3cc8e7f56fa4d65b35437cfa6db799405c9c19788131/diff:/var/lib/docker/overlay2/f7907c9035dc54b8537a6b1779db3686d4175e3ce9caf7b07ccf509efbc87ee3/diff:/var/lib/docker/overlay2/6d65f49890d13f8a205d0b65d038164dc0659a016af2bdad48c2217b8ceba34f/diff:/var/lib/docker/overlay2/fdc417be3c2a42cd2f16efdec4044d31bca484f6b263c580de9fd86c35e87b7a/diff:/var/lib/docker/overlay2/cc6b24eab225161fd0470722a74218f71d813b93d8bc65e6e510e108aea95920/diff:/var/lib/docker/overlay2/60f5b93cfc171d7863d30b1c59b2286a68a4f8616b9256d4b97108a24fea3826/diff:/var/lib/docker/overlay2/d41492b38ce5f3284146a0202a9eb6013bba4e8bd36986a0042acfc445c3f61f/diff:/var/lib/docker/overlay2/e67eded62660fd584c6eac6a5c0c2ebfd8f7455157de845263298ddc7edb0666/diff:/var/lib/docker/overlay2/dfccd8df65deeec446b2dc80d489e2e422d14e53b95b73f2ad758589ac7c9188/diff:/var/lib/docker/overlay2/bf7a0bb8ea9f3e2c29566ca79bcdcad7102d7bc2ccee3d4cd03cf1725992073c/diff",
"MergedDir": "/var/lib/docker/overlay2/2586c3fd0a48a576b763dd2797e9218516c528d41b32783e4f7498af8356b13c/merged",
"UpperDir": "/var/lib/docker/overlay2/2586c3fd0a48a576b763dd2797e9218516c528d41b32783e4f7498af8356b13c/diff",
"WorkDir": "/var/lib/docker/overlay2/2586c3fd0a48a576b763dd2797e9218516c528d41b32783e4f7498af8356b13c/work"
},
"Name": "overlay2"
},
"Mounts": [
The virus hiding in the docker of 14f4d0681f7a postgres:16.
It’s the database docker of hyperIndex.
So What should we do? try another index or develop it for ourself, a little tired.
Hi Clara,
Trust you are well and thanks for reaching out!
I understand your concerns about a Virus. This has nothing to do with Envio. It has most likely come about as you have insecurely setup your docker on your cloud machine and exposed your postgres port to the outside world. Take a google of “kdevtmpfsi” and you will be able to learn more.
I encourage you to simply use envio’s hosted service if you would like to avoid having to set this up yourself. If you head over to envio.dev its simple to get started with your first deployment.
Let us know how else we can help.
1 Like
@Clara here is some more info on it: Reddit - Dive into anything
If you are self hosting you need to be very careful how you expose ports and configure your set up.
1 Like
@jonjon
Following your tips, I reinstalled it, opened the firewall, and processed the server’s external port, but I still couldn’t synchronize the data. The error was as follows:
[07:59:59.815] INFO (126184): Tracking tables in Hasura
Starting indexer
> game@0.1.0 start
> node generated/src/Index.bs.js
[08:00:09.219] ERROR (126253): Failed to import handler file
Contract Name: "Game"
Expected Handler Path: "src/EventHandlers.js"
Code: "EE500"
err: {
"type": "TypeError",
"message": "Cannot read properties of undefined (reading 'DeListEvent')",
"stack":
TypeError: Cannot read properties of undefined (reading 'DeListEvent')
at Object.<anonymous> (/root/game/src/EventHandlers.js:32:14)
at Module._compile (node:internal/modules/cjs/loader:1460:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1544:10)
at Module.load (node:internal/modules/cjs/loader:1275:32)
at Module._load (node:internal/modules/cjs/loader:1091:12)
at wrapModuleLoad (node:internal/modules/cjs/loader:212:19)
at Module.require (node:internal/modules/cjs/loader:1297:12)
at require (node:internal/modules/helpers:123:16)
at registerContractHandlers (/root/game/generated/src/RegisterHandlers.bs.js:10:5)
at Object.registerAllHandlers (/root/game/generated/src/RegisterHandlers.bs.js:28:3)
}
[08:00:09.225] ERROR (126253): Failed at initialization
err: {
"RE_EXN_ID": "ErrorHandling.JsExnError/1",
"_1": {}
}
our config file is on our github:
Please see where the configuration is wrong?
I’ll make a time to dig into it this afternoon.
I’d highly recommend using typescript for your indexer. Javascript will be challenging due to these kinds of runtime errors without clear idea of the cause.
This looks like a simple type error that would be easily detected via TypeScript.
You can also view the docker-compose file that is used in ./generated/docker-compose.yaml and validate that there is nothing malicious there. Definitely make sure you secure your server adequately 
@JasoonS Thank you for your reply, but we are not familiar with ts and have always used js.