Commit Graph

15 Commits

Author SHA1 Message Date
Mark Holt
79ed8cad35
E2 snapshot uploading (#9056)
This change introduces additional processes to manage snapshot uploading
for E2 snapshots:

## erigon snapshots upload

The `snapshots uploader` command starts a version of erigon customized
for uploading snapshot files to
a remote location.  

It breaks the stage execution process after the senders stage and then
uses the snapshot stage to send
uploaded headers, bodies and (in the case of polygon) bor spans and
events to snapshot files. Because
this process avoids execution in run signifigantly faster than a
standard erigon configuration.

The uploader uses rclone to send seedable (100K or 500K blocks) to a
remote storage location specified
in the rclone config file.

The **uploader** is configured to minimize disk usage by doing the
following:

* It removes snapshots once they are loaded
* It aggressively prunes the database once entities are transferred to
snapshots

in addition to this it has the following performance related features:

* maximizes the workers allocated to snapshot processing to improve
throughput
* Can be started from scratch by downloading the latest snapshots from
the remote location to seed processing

## snapshots command

Is a stand alone command for managing remote snapshots it has the
following sub commands

* **cmp** - compare snapshots
* **copy** - copy snapshots
* **verify** - verify snapshots
* **manifest** - manage the manifest file in the root of remote snapshot
locations
* **torrent** - manage snapshot torrent files
2023-12-27 22:05:09 +00:00
Alex Sharov
d41d523050
Downloader: add ProhibitNewDownloads() (#8939)
"whitelisting" mechanism (list of files - stored in DB) - which
protecting us from downloading new files after upgrade/downgrade was
broken. And seems it became over-complicated with time.
I replacing it by 1 persistent flag inside downloader:
"prohibit_new_downloads.lock"
Erigon will turn downloader into this mode after
downloading/verification of first snapshots.


```
//Corner cases:
	// - Erigon generated file X with hash H1. User upgraded Erigon. New version has preverified file X with hash H2. Must ignore H2 (don't send to Downloader)
	// - Erigon "download once": means restart/upgrade/downgrade must not download files (and will be fast)
	// - After "download once" - Erigon will produce and seed new files
```

------
`downloader --seedbox` is never "prohibit new downloads"
2023-12-12 16:05:56 +07:00
Anshal Shukla
8d1758ceea
Add support for amoy testnet (#8674)
Co-authored-by: Mark Holt <mark@distributed.vision>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-11-30 08:19:52 +07:00
Pratik Patil
59909a7efe
Added TxDependency Metadata to ExtraData in Block Header in Bor for Block-STM (#8037)
This PR adds support to store the transaction dependency (generated by
the block producer) in the block header for bor. This transaction
dependency will then be used by the parallel processor
([Block-STM](https://github.com/ledgerwatch/erigon/pull/7812/)).

I have created another
[PR](https://github.com/ledgerwatch/erigon-lib/pull/1064) in the
erigon-lib repo which adds the `IsParallelUniverse()` function.
2023-11-24 10:26:33 +00:00
Anshal Shukla
83bbe0dd37
Fix Agra hf scheduling on Mumbai (#8620)
Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
2023-10-30 15:18:47 +01:00
Alex Sharov
b311da959f
downloader: webseed better error messages (#8611) 2023-10-30 12:13:45 +07:00
Andrew Ashikhmin
a54939633e
Numerical instead of lexicographic sorting in borKeyValueConfigHelper (#8560)
Corresponds to Item 3 of https://github.com/maticnetwork/bor/pull/1055
2023-10-23 14:11:32 +02:00
Andrew Ashikhmin
a226b6ca29
Fix wiring of AgraBlock into tx pool (#8555)
Fixes and simplifications to PR #8504
2023-10-23 11:03:46 +02:00
Andrew Ashikhmin
ee942473d9
Replace eip1559FeeCollector with burntContract (#8553)
burntContract, introduced in PR #8504, is more generic than
eip1559FeeCollector
2023-10-22 16:23:40 +02:00
Anshal Shukla
7dce1268ab
Agra HF (#8504)
Adds agra HF to the bor consensus
2023-10-21 01:16:19 +05:30
Alex Sharov
33d5399436
downloader: support token (#8507) 2023-10-18 14:24:09 +07:00
Alex Sharov
3ac9f493b6
move chainname and snapcfg packages to erigon-lib (#8508) 2023-10-18 13:37:39 +07:00
Andrew Ashikhmin
b60642fa5a
Configure EIP-4844 parameters for Gnosis (#8464)
See https://github.com/gnosischain/specs/pull/20 &
https://github.com/gnosischain/specs/pull/24
2023-10-13 11:43:16 +02:00
Andrew Ashikhmin
0bd6d77acd
Remove CalcuttaBlock in favour of BlockAlloc (#8371)
System contract upgrades for Polygon are already handled by the
`BlockAlloc` logic and there's no need to duplicate it with the
`CalcuttaBlock` logic (there's no Calcutta in
https://github.com/maticnetwork/bor).
2023-10-05 15:39:57 +02:00
battlmonstr
231e468e19 Add 'erigon-lib/' from commit '93d9c9d9fe4bd8a49f7a98a6bce0f0da7094c7d3'
git-subtree-dir: erigon-lib
git-subtree-mainline: 3c8cbda809
git-subtree-split: 93d9c9d9fe
2023-09-20 14:50:25 +02:00