erigon-pulse/erigon-lib/downloader
Alex Sharov 329d18ef6f
snapshots: reduce merge limit of blocks to 100K (#8614)
Reason: 
- produce and seed snapshots earlier on chain tip. reduce depnedency on
"good peers with history" at p2p-network.
Some networks have no much archive peers, also ConsensusLayer clients
are not-good(not-incentivised) at serving history.
- avoiding having too much files:
more files(shards) - means "more metadata", "more lookups for
non-indexed queries", "more dictionaries", "more bittorrent
connections", ...
less files - means small files will be removed after merge (no peers for
this files).


ToDo:
[x] Recent 500K - merge up to 100K 
[x] Older than 500K - merge up to 500K 
[x] Start seeding 100k files
[x] Stop seeding 100k files after merge (right before delete)

In next PR: 
[] Old version of Erigon must be able download recent hashes. To achieve
it - at first start erigon will download preverified hashes .toml from
s3 - if it's newer that what we have (build-in) - use it.
2023-11-01 23:22:35 +07:00
..
downloadercfg snapshots: reduce merge limit of blocks to 100K (#8614) 2023-11-01 23:22:35 +07:00
downloadergrpc Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
snaptype snapshots: reduce merge limit of blocks to 100K (#8614) 2023-11-01 23:22:35 +07:00
downloader_grpc_server.go snapshots: reduce merge limit of blocks to 100K (#8614) 2023-11-01 23:22:35 +07:00
downloader_test.go downloader: support token (#8507) 2023-10-18 14:24:09 +07:00
downloader.go Added beacon snapshots download (#8601) 2023-10-28 17:41:50 +02:00
mdbx_piece_completion_test.go Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
mdbx_piece_completion.go Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
path_plan9.go Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
path_unix.go Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
path_windows.go Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
path.go Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3' 2023-09-20 14:50:25 +02:00
util.go webseed: don't download .torrent files (#8629) 2023-11-01 15:50:35 +07:00
webseed.go webseed: don't download .torrent files (#8629) 2023-11-01 15:50:35 +07:00