erigon-pulse/cmd/downloader
2022-03-12 17:26:11 +07:00
..
downloader Snapshots: retire testing tool (#3684) 2022-03-12 17:26:11 +07:00
downloadergrpc Embed rpcdaemon: prepared direct clients (#3492) 2022-02-12 19:47:19 +07:00
tracker Move cli root context to erigon-lib (#3294) 2022-01-19 10:49:07 +07:00
trackers Torrent less goroutines (#3423) 2022-02-04 12:42:55 +07:00
main.go Rename Datadir/dataDir to follow conventions. (#3576) 2022-02-22 17:39:48 +00:00
readme.md Update readme.md 2022-02-10 08:33:05 +07:00

Downloader

Service to seed/download historical data (immutable .seg files)

How to Start Erigon in snapshot sync mode

Only Mainnet, Goerli and BSC networks are supported now.

# 1. Downloader by default run inside Erigon, by `--experimental.snapshot` flag:
erigon --experimental.snapshot --datadir=<your_datadir> 
# 2. It's possible to start Downloader as independent process, by `--experimental.snapshot --downloader.api.addr=127.0.0.1:9093` flags:
make erigon downloader 

# Start downloader (can limit network usage by 512mb/sec: --download.rate=512mb --upload.rate=512mb)
downloader --downloader.api.addr=127.0.0.1:9093 --torrent.port=42068 --datadir=<your_datadir>
# --downloader.api.addr - is for internal communication with Erigon
# --torrent.port=42068  - is for public BitTorrent protocol listen 

# Erigon on startup does send list of .torrent files to Downloader and wait for 100% download accomplishment
erigon --experimental.snapshot --downloader.api.addr=127.0.0.1:9093 --datadir=<your_datadir> 

How to create new network or bootnode

# Need create new snapshots and start seeding them
 
# Create new snapshots (can change snapshot size by: --from=0 --to=1_000_000 --segment.size=500_000)
# It will dump blocks from Database to .seg files:
erigon snapshots create --datadir=<your_datadir> 

# Create .torrent files (Downloader will seed automatically all .torrent files)
# output format is compatible with https://github.com/ledgerwatch/erigon-snapshot
downloader torrent_hashes --rebuild --datadir=<your_datadir>

# Start downloader (seeds automatically)
downloader --downloader.api.addr=127.0.0.1:9093 --datadir=<your_datadir>

# Erigon is not required for snapshots seeding 

Architecture

Downloader works based on <your_datadir>/snapshots/*.torrent files. Such files can be created 4 ways:

  • Erigon can do grpc call downloader.Download(list_of_hashes), it will trigger creation of .torrent files
  • Erigon can create new .seg file, Downloader will scan .seg file and create .torrent
  • operator can manually copy .torrent files (rsync from other server or restore from backup)
  • operator can manually copy .seg file, Downloader will scan .seg file and create .torrent

Erigon does:

  • connect to Downloader
  • share list of hashes (see https://github.com/ledgerwatch/erigon-snapshot )
  • wait for download of all snapshots
  • when .seg available - automatically create .idx files - secondary indices, for example to find block by hash
  • then switch to normal staged sync (which doesn't require connection to Downloader)

Downloader does:

Technical details:

  • To prevent attack - .idx creation using random Seed - all nodes will have different .idx file (and same .seg files)

How to verify that .seg files have same checksum withch current .torrent files

# Use it if you see weird behavior, bugs, bans, hardware issues, etc...
downloader torrent_hashes --verify --datadir=<your_datadir>