This change introduces additional processes to manage snapshot uploading
for E2 snapshots:
## erigon snapshots upload
The `snapshots uploader` command starts a version of erigon customized
for uploading snapshot files to
a remote location.
It breaks the stage execution process after the senders stage and then
uses the snapshot stage to send
uploaded headers, bodies and (in the case of polygon) bor spans and
events to snapshot files. Because
this process avoids execution in run signifigantly faster than a
standard erigon configuration.
The uploader uses rclone to send seedable (100K or 500K blocks) to a
remote storage location specified
in the rclone config file.
The **uploader** is configured to minimize disk usage by doing the
following:
* It removes snapshots once they are loaded
* It aggressively prunes the database once entities are transferred to
snapshots
in addition to this it has the following performance related features:
* maximizes the workers allocated to snapshot processing to improve
throughput
* Can be started from scratch by downloading the latest snapshots from
the remote location to seed processing
## snapshots command
Is a stand alone command for managing remote snapshots it has the
following sub commands
* **cmp** - compare snapshots
* **copy** - copy snapshots
* **verify** - verify snapshots
* **manifest** - manage the manifest file in the root of remote snapshot
locations
* **torrent** - manage snapshot torrent files
During testing we run into a "span 7813 not found (db)" due to a very
large unwind (1 million blocks).
This is because the block reader's `LastFrozenSpanID` and
`LastFrozenEventID` returned results that are not consistent with
`FrozenBorBlocks`. The latter is taking into account the existence of
`.idx` files while the former 2 functions were not.
Note such a large unwind is not likely to happen normally unless there
is a bug in our unwind logic or an operator is manually unwinding very
far back due to reasons like chain halts (ie mumbai bug problem from few
months ago), devel testing or anything else along these lines.
Regardless, it exposed the above discrepancy which is best to be fixed.
