"whitelisting" mechanism (list of files - stored in DB) - which
protecting us from downloading new files after upgrade/downgrade was
broken. And seems it became over-complicated with time.
I replacing it by 1 persistent flag inside downloader:
"prohibit_new_downloads.lock"
Erigon will turn downloader into this mode after
downloading/verification of first snapshots.
```
//Corner cases:
// - Erigon generated file X with hash H1. User upgraded Erigon. New version has preverified file X with hash H2. Must ignore H2 (don't send to Downloader)
// - Erigon "download once": means restart/upgrade/downgrade must not download files (and will be fast)
// - After "download once" - Erigon will produce and seed new files
```
------
`downloader --seedbox` is never "prohibit new downloads"
Small fix to the script for the scenario where more than one matching
library can be returned.
For example, the command `/sbin/ldconfig -p | grep libstdc++ | awk '{
print $NF }'` can result in
```
/lib/x86_64-linux-gnu/libstdc++.so.6
/lib32/libstdc++.so.6
```
which then fails the check `if [[ ! -L "$link_path" ]]`
* Most of the PR changed files are extra and slightly more complicated
unit tests.
* Fixed Eth1DataVotes not inheriting genesis
* Fixed Attestations simulation using wrong slot when reconstructing
partecipation
* Fixed Copy() operation on BeaconState on Eth1DataVotes
* Used correct ListSSZ type for Eth1DataVotes and HistoricalSummaries
* Fixed wrong []uint64 deltas on empty slots
Because access lists use maps with the `StorageKey` as the key, they are
subject to inconsistent ordering in the results of the `.accessList()`
method.
To get around this, an `accessListSorted` method has been added, and
exposed with the same name. The `equal` method has also been exposed to
allow for equality checks at this level outside of this module.
Co-authored-by: 3commascapital <8562488-3commascapital@users.noreply.gitlab.com>
### Context
**Websocket port flag**
Hive tests for RPC suite depend on the (geth) default 8546 port. So,
opening one more listener for this additional port if `ws.port` was
specified. This flag isn't used in Erigon, as it shares port with http
listener. Normally, one may not specify and it offers no other benefit.
Silkworm built on Ubuntu 22 depends on glibc 2.34. In order to run on an
older OS, Silkworm needs to be built and linked with an older glibc, but
to build on an older OS we need a compatible compiler. Silkworm requires
gcc 11+ that is not available on Ubuntu 20 or Debian 11.
To simplify the deployment disable Silkworm support on versions before
Ubuntu 22, Debian 12, and glibc prior to 2.34. The check for Ubuntu and
Debian is explicit, because some Ubuntu 16 installations report glibc
2.35 with ldd, but `go build` still uses an older system one and fails.
What does this PR do:
* Optional Backfilling and Caplin Archive Node
* Create antiquary for historical states
* Fixed gaps of chain gap related to the Head of the chain and anchor of
the chain.
* Added basic reader object to Read the Historical state
1. When running locally, found that these **TODO** json-rpc have been
implemented. Updating these outdated information can make the document
clearer.
```
curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_newBlockFilter","params":[],"id":1}' http://loca
lhost:8545
{"jsonrpc":"2.0","id":1,"result":"0x02000000000000009c49b60f431cabc7"}
```
2. [but it's easy to accomplish] Points to an obsolete message
(9b8cdc0f22/eth/downloader/downloader.go (L673))
adds a two indexes to the validators cache
creates beaconhttp package with many utilities for beacon http endpoint
(future support for ssz is baked in)
started on some validator endpoints
Correction for "Lets assume":
Incorrect: "Lets assume"
Correct: "Let's assume"
Explanation: In this phrase, the word "Lets" is incorrectly written. The
correct form should be "Let's," which is a contraction of "let us."
Correction for the word "tripple":
Incorrect: "tripple"
Correct: "triple"
Explanation: The word "tripple" is misspelled. The correct spelling is
"triple," which refers to something made up of three parts, or
multiplied by three.
This PR is to add the request rate limiter.
The solution is to count the request number for each peer for each
minute, if the peer exceeds the limit, block the requests for a
specified time.
Current limits:
- Request limited to `5000` requests per minute for each handler.
- Penalty blockage time `1-minute`
This PR has fixes for a number of instances in the bor heimdall stage
where nil headers are either ignored or inadvertently processed.
It also has a demotion of milestone related logging messages to debug
for missing blocks because the process is not at the head of the chain +
a general reduction in periodic logging to 30 secs rather than 20 to
reduce the log output on long runs.
In addition there is a refactor of persistValidatorSets to perform
validator set initiation in a seperate function. This is intended to
clarify the operation of persistValidatorSets - which is till performing
2 actions, persisting the snapshot and then using it to check the header
against synthesized validator set in the snapshot.
eventsource is required for the validator api. this implements the
eventsource sink/server handler
the implementation is based off of this document:
https://html.spec.whatwg.org/multipage/server-sent-events.html
note that this is a building block for the full eventsource server.
there still needs to be work done
prysm has their own custom solution based off of protobuf/grpc:
https://hackmd.io/@prysmaticlabs/eventstream-api using that would be not
good
existing eventsource implementations for golang are not good for our
situation. options are:
1. https://github.com/r3labs/sse - has most stars - this is the best
contender, since it uses []byte and not string, but it allocates and
copies extra times in the server (because of use of fprintf) and makes
an incorrect assumption about Last-Event-ID needing to be a number (i
can't find this in the specification).
2. https://github.com/antage/eventsource -requires full buffers, copies
many times, does not provide abstraction for headers. relatively
unmaintained
3. https://github.com/donovanhide/eventsource - missing functionality
around sending ids, requires full buffers, etc
4. https://github.com/bernerdschaefer/eventsource - 10 years old,
unmaintained.
additionally, implemetations other than r3labs/sse are very incorrect
because they do not split up the data field correctly when newlines are
sent. (parsers by specification will fail to encode messages sent by
most of these implementations that have newlines, as i understand it).
the implementation by r3labs/sse is also incorrect because it does not
respect \r
finally, all these implementations have very heavy implementation of the
server, which we do not need since we will use fixed sequence ids.
r3labs/sse for instance hijacks the entire handler and ties that to the
server, losing a lot of flexiblity in how we implement our server
for the beacon api, we need to stream:
```head, block, attestation, voluntary_exit, bls_to_execution_change, finalized_checkpoint, chain_reorg, contribution_and_proof, light_client_finality_update, light_client_optimistic_update, payload_attributes```
some of these are rather big json payloads, and the ability to simultaneously stream them from io.Readers instead of making a full copy of the payload every time we wish to rebroadcast it will save a lot of heap size for both resource constrained environments and serving at scale.
the protocol itself is relatively simple, there are just a few gotchas
