2021-12-25 08:32:51 +00:00
|
|
|
# Downloader
|
|
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
Service to seed/download historical data (immutable .seg files)
|
2021-12-25 08:32:51 +00:00
|
|
|
|
2022-02-07 05:07:46 +00:00
|
|
|
## How to Start Erigon in snapshot sync mode
|
2022-01-31 07:04:16 +00:00
|
|
|
|
|
|
|
|
Only Mainnet, Goerli and BSC networks are supported now.
|
2022-01-26 09:11:22 +00:00
|
|
|
|
|
|
|
|
```shell
|
2022-02-10 01:33:05 +00:00
|
|
|
# 1. Downloader by default run inside Erigon, by `--experimental.snapshot` flag:
|
|
|
|
|
erigon --experimental.snapshot --datadir=<your_datadir>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
```shell
|
|
|
|
|
# 2. It's possible to start Downloader as independent process, by `--experimental.snapshot --downloader.api.addr=127.0.0.1:9093` flags:
|
2022-01-26 09:11:22 +00:00
|
|
|
make erigon downloader
|
|
|
|
|
|
2022-01-26 15:26:33 +00:00
|
|
|
# Start downloader (can limit network usage by 512mb/sec: --download.rate=512mb --upload.rate=512mb)
|
2022-01-26 09:11:22 +00:00
|
|
|
downloader --downloader.api.addr=127.0.0.1:9093 --torrent.port=42068 --datadir=<your_datadir>
|
|
|
|
|
# --downloader.api.addr - is for internal communication with Erigon
|
|
|
|
|
# --torrent.port=42068 - is for public BitTorrent protocol listen
|
|
|
|
|
|
|
|
|
|
# Erigon on startup does send list of .torrent files to Downloader and wait for 100% download accomplishment
|
|
|
|
|
erigon --experimental.snapshot --downloader.api.addr=127.0.0.1:9093 --datadir=<your_datadir>
|
|
|
|
|
```
|
|
|
|
|
|
2022-03-16 02:57:48 +00:00
|
|
|
Use `--experimental.snapshot.keepblocks=true` to don't delete retired blocks from DB
|
|
|
|
|
|
2022-02-07 05:07:46 +00:00
|
|
|
## How to create new network or bootnode
|
2022-01-26 09:11:22 +00:00
|
|
|
|
|
|
|
|
```shell
|
2022-02-07 05:07:46 +00:00
|
|
|
# Need create new snapshots and start seeding them
|
|
|
|
|
|
2022-01-26 09:11:22 +00:00
|
|
|
# Create new snapshots (can change snapshot size by: --from=0 --to=1_000_000 --segment.size=500_000)
|
|
|
|
|
# It will dump blocks from Database to .seg files:
|
|
|
|
|
erigon snapshots create --datadir=<your_datadir>
|
|
|
|
|
|
|
|
|
|
# Create .torrent files (Downloader will seed automatically all .torrent files)
|
|
|
|
|
# output format is compatible with https://github.com/ledgerwatch/erigon-snapshot
|
2022-02-07 05:07:46 +00:00
|
|
|
downloader torrent_hashes --rebuild --datadir=<your_datadir>
|
2022-01-26 09:11:22 +00:00
|
|
|
|
2022-02-07 05:07:46 +00:00
|
|
|
# Start downloader (seeds automatically)
|
2022-01-26 09:11:22 +00:00
|
|
|
downloader --downloader.api.addr=127.0.0.1:9093 --datadir=<your_datadir>
|
|
|
|
|
|
|
|
|
|
# Erigon is not required for snapshots seeding
|
|
|
|
|
```
|
|
|
|
|
|
2021-12-25 08:32:51 +00:00
|
|
|
## Architecture
|
|
|
|
|
|
2022-01-26 09:11:22 +00:00
|
|
|
Downloader works based on <your_datadir>/snapshots/*.torrent files. Such files can be created 4 ways:
|
2021-12-31 05:09:11 +00:00
|
|
|
|
|
|
|
|
- Erigon can do grpc call downloader.Download(list_of_hashes), it will trigger creation of .torrent files
|
|
|
|
|
- Erigon can create new .seg file, Downloader will scan .seg file and create .torrent
|
|
|
|
|
- operator can manually copy .torrent files (rsync from other server or restore from backup)
|
|
|
|
|
- operator can manually copy .seg file, Downloader will scan .seg file and create .torrent
|
|
|
|
|
|
2021-12-25 08:32:51 +00:00
|
|
|
Erigon does:
|
|
|
|
|
|
|
|
|
|
- connect to Downloader
|
|
|
|
|
- share list of hashes (see https://github.com/ledgerwatch/erigon-snapshot )
|
|
|
|
|
- wait for download of all snapshots
|
2022-01-26 09:11:22 +00:00
|
|
|
- when .seg available - automatically create .idx files - secondary indices, for example to find block by hash
|
2021-12-25 08:32:51 +00:00
|
|
|
- then switch to normal staged sync (which doesn't require connection to Downloader)
|
|
|
|
|
|
|
|
|
|
Downloader does:
|
|
|
|
|
|
2021-12-31 05:09:11 +00:00
|
|
|
- Read .torrent files, download everything described by .torrent files
|
|
|
|
|
- Use https://github.com/ngosang/trackerslist see [./trackers/embed.go](./trackers/embed.go)
|
2021-12-25 08:32:51 +00:00
|
|
|
- automatically seeding
|
|
|
|
|
|
2022-01-26 09:11:22 +00:00
|
|
|
Technical details:
|
2022-01-05 10:14:37 +00:00
|
|
|
|
2022-02-07 05:07:46 +00:00
|
|
|
- To prevent attack - .idx creation using random Seed - all nodes will have different .idx file (and same .seg files)
|
|
|
|
|
|
|
|
|
|
## How to verify that .seg files have same checksum withch current .torrent files
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
# Use it if you see weird behavior, bugs, bans, hardware issues, etc...
|
|
|
|
|
downloader torrent_hashes --verify --datadir=<your_datadir>
|
|
|
|
|
```
|
2022-03-21 02:39:48 +00:00
|
|
|
|
|
|
|
|
## Faster rsync
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
rsync -aP --delete -e "ssh -T -o Compression=no -x" <src> <dst>
|
|
|
|
|
```
|
