erigon-pulse/cmd/downloader
2022-01-09 15:18:45 +07:00
..
debug Small step towards torrent downloader (#2929) 2021-11-08 20:40:56 +07:00
downloader Snapshot: add hash first byte to headers.seg, serve p2p blocks from snapshots (#3198) 2022-01-05 17:14:37 +07:00
downloadergrpc Snapshots download and seed (#3117) 2021-12-14 10:13:17 +00:00
tracker Small step towards torrent downloader (#2929) 2021-11-08 20:40:56 +07:00
trackers Snapshots download and seed (#3117) 2021-12-14 10:13:17 +00:00
main.go Snapshot: add hash first byte to headers.seg, serve p2p blocks from snapshots (#3198) 2022-01-05 17:14:37 +07:00
readme.md Snapshots: create .seg and huffman_codes.txt in tmpdir (#3223) 2022-01-09 15:18:45 +07:00

Downloader

Is a service which does download and seed historical data.

Historical data - is immutable, files have .seg extension.

Architecture

Downloader works based on <your_datadir>/snapshots/*.torrent files (etl-tmp and snapshots directories MUST be on same drive). Such files can be created 4 ways:

  • Erigon can do grpc call downloader.Download(list_of_hashes), it will trigger creation of .torrent files
  • Erigon can create new .seg file, Downloader will scan .seg file and create .torrent
  • operator can manually copy .torrent files (rsync from other server or restore from backup)
  • operator can manually copy .seg file, Downloader will scan .seg file and create .torrent

Erigon does:

Downloader does:

How to

Start

downloader --datadir=<your_datadir> --downloader.api.addr=127.0.0.1:9093
erigon --downloader.api.addr=127.0.0.1:9093 --experimental.snapshot

Limit download/upload speed

downloader --download.limit=10mb --upload.limit=10mb

Print info_hashes in format compatible with https://github.com/ledgerwatch/erigon-snapshot

downloader info_hashes --datadir=<your_datadir>

Force re-calculate info_hashes and print in format compatible with https://github.com/ledgerwatch/erigon-snapshot

// will re-read all .seg files (high disk IO)
// also does remove and create .torrent files
downloader info_hashes --datadir=<your_datadir> --rebuild

Create new snapshots

rm <your_datadir>/snapshots/*.torrent
erigon snapshots create --datadir=<your_datadir> --from=0 --segment.size=500_000

Download snapshots to new server

rsync server1:<your_datadir>/snapshots/*.torrent server2:<your_datadir>/snapshots/
// re-start downloader 

Re-create all .idx files (by re-read all .seg files)

// Disk-read-intense
erigon snapshots index --datadir=<your_datadir> --rebuild

Known Issues

  • RPCDaemon with --datadir option need restart to make new segments available