2021-12-25 08:32:51 +00:00
|
|
|
# Downloader
|
|
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
Service to seed/download historical data (immutable .seg files)
|
2021-12-25 08:32:51 +00:00
|
|
|
|
|
|
|
|
## Architecture
|
|
|
|
|
|
2022-01-09 08:18:45 +00:00
|
|
|
Downloader works based on <your_datadir>/snapshots/*.torrent files (`etl-tmp` and `snapshots` directories MUST be on
|
|
|
|
|
same drive). Such files can be created 4 ways:
|
2021-12-31 05:09:11 +00:00
|
|
|
|
|
|
|
|
- Erigon can do grpc call downloader.Download(list_of_hashes), it will trigger creation of .torrent files
|
|
|
|
|
- Erigon can create new .seg file, Downloader will scan .seg file and create .torrent
|
|
|
|
|
- operator can manually copy .torrent files (rsync from other server or restore from backup)
|
|
|
|
|
- operator can manually copy .seg file, Downloader will scan .seg file and create .torrent
|
|
|
|
|
|
2021-12-25 08:32:51 +00:00
|
|
|
Erigon does:
|
|
|
|
|
|
|
|
|
|
- connect to Downloader
|
|
|
|
|
- share list of hashes (see https://github.com/ledgerwatch/erigon-snapshot )
|
|
|
|
|
- wait for download of all snapshots
|
|
|
|
|
- then switch to normal staged sync (which doesn't require connection to Downloader)
|
|
|
|
|
|
|
|
|
|
Downloader does:
|
|
|
|
|
|
2021-12-31 05:09:11 +00:00
|
|
|
- Read .torrent files, download everything described by .torrent files
|
|
|
|
|
- Use https://github.com/ngosang/trackerslist see [./trackers/embed.go](./trackers/embed.go)
|
2021-12-25 08:32:51 +00:00
|
|
|
- automatically seeding
|
|
|
|
|
|
|
|
|
|
## How to
|
|
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
### Start erigon with snapshot sync
|
2021-12-25 08:32:51 +00:00
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
```shell
|
2021-12-25 08:32:51 +00:00
|
|
|
downloader --datadir=<your_datadir> --downloader.api.addr=127.0.0.1:9093
|
|
|
|
|
erigon --downloader.api.addr=127.0.0.1:9093 --experimental.snapshot
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Limit download/upload speed
|
|
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
```shell
|
2021-12-25 08:32:51 +00:00
|
|
|
downloader --download.limit=10mb --upload.limit=10mb
|
|
|
|
|
```
|
|
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
### Print info_hashes
|
2021-12-25 08:32:51 +00:00
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
```shell
|
|
|
|
|
# format compatible with https://github.com/ledgerwatch/erigon-snapshot
|
2022-01-05 10:14:37 +00:00
|
|
|
downloader info_hashes --datadir=<your_datadir>
|
|
|
|
|
```
|
|
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
### Create .torrent files
|
2022-01-05 10:14:37 +00:00
|
|
|
|
2022-01-24 06:47:05 +00:00
|
|
|
```shell
|
|
|
|
|
downloader info_hashes --rebuild --datadir=<your_datadir>
|
2021-12-25 08:32:51 +00:00
|
|
|
```
|
|
|
|
|
|
2021-12-31 04:53:15 +00:00
|
|
|
### Create new snapshots
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
rm <your_datadir>/snapshots/*.torrent
|
|
|
|
|
erigon snapshots create --datadir=<your_datadir> --from=0 --segment.size=500_000
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Download snapshots to new server
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
rsync server1:<your_datadir>/snapshots/*.torrent server2:<your_datadir>/snapshots/
|
2022-01-24 06:47:05 +00:00
|
|
|
# re-start downloader
|
2021-12-31 04:53:15 +00:00
|
|
|
```
|
|
|
|
|
|
2022-01-09 07:43:58 +00:00
|
|
|
### Re-create all .idx files (by re-read all .seg files)
|
2022-01-07 13:52:38 +00:00
|
|
|
|
|
|
|
|
```
|
2022-01-24 06:47:05 +00:00
|
|
|
# Disk-read-intense
|
2022-01-09 07:43:58 +00:00
|
|
|
erigon snapshots index --datadir=<your_datadir> --rebuild
|
2022-01-07 13:52:38 +00:00
|
|
|
```
|
|
|
|
|
|
2022-01-05 10:14:37 +00:00
|
|
|
## Known Issues
|
|
|
|
|
|
|
|
|
|
- RPCDaemon with --datadir option need restart to make new segments available
|
