This fixes an issue where the mumbai testnet node struggle to find
peers. Before this fix in general test peer numbers are typically around
20 in total between eth66, eth67 and eth68. For new peers some can
struggle to find even a single peer after days of operation.
These are the numbers after 12 hours or running on a node which
previously could not find any peers: eth66=13, eth67=76, eth68=91.
The root cause of this issue is the following:
- A significant number of mumbai peers around the boot node return
network ids which are different from those currently available in the
DHT
- The available nodes are all consequently busy and return 'too many
peers' for long periods
These issues case a significant number of discovery timeouts, some of
the queries will never receive a response.
This causes the discovery read loop to enter a channel deadlock - which
means that no responses are processed, nor timeouts fired. This causes
the discovery process in the node to stop. From then on it just
re-requests handshakes from a relatively small number of peers.
This check in fixes this situation with the following changes:
- Remove the deadlock by running the timer in a separate go-routine so
it can run independently of the main request processing.
- Allow the discovery process matcher to match on port if no id match
can be established on initial ping. This allows subsequent node
validation to proceed and if the node proves to be valid via the
remainder of the look-up and handshake process it us used as a valid
peer.
- Completely unsolicited responses, i.e. those which come from a
completely unknown ip:port combination continue to be ignored.
-
rlp2 is a package that aims to replace the existing erigon-lib/rlp
package and the erigon/common/rlp
it is called rlp2 for now because it requires breaking changes to
erigon-lib/rlp and i do not have the time right now to test all current
uses of such functions
however, the encoder/decoder characteristics of rlp2 might be desirable
for caplin, and also for execution layer parsing blob txns, so im
putting it in a folder called rlp2 (note that it exports package rlp for
easier switching later)
importantly, rlp2 is designed for single-pass decoding with the ability
to skip elements one does not care about. it also is zero alloc.
Reason:
- produce and seed snapshots earlier on chain tip. reduce depnedency on
"good peers with history" at p2p-network.
Some networks have no much archive peers, also ConsensusLayer clients
are not-good(not-incentivised) at serving history.
- avoiding having too much files:
more files(shards) - means "more metadata", "more lookups for
non-indexed queries", "more dictionaries", "more bittorrent
connections", ...
less files - means small files will be removed after merge (no peers for
this files).
ToDo:
[x] Recent 500K - merge up to 100K
[x] Older than 500K - merge up to 500K
[x] Start seeding 100k files
[x] Stop seeding 100k files after merge (right before delete)
In next PR:
[] Old version of Erigon must be able download recent hashes. To achieve
it - at first start erigon will download preverified hashes .toml from
s3 - if it's newer that what we have (build-in) - use it.
read transaction was opened before stream.Recv(), but stream.Recv() is
blocking infinity loop. so, this read transaction never rollback -
causing unlimited db grow.
---------
Merge PR #8596 into `devel`
---------
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Calling vm to get a metric and then preregistering it to avoid
duplication is expensive.
To avoid this we can make a local lookup first which is a map this
quicker. This is a temporary fix
until vm is removed at which point we'll have a single prom_client
registry
Fixes: https://github.com/ledgerwatch/erigon/issues/8558
Should fix the [following
issue](https://discord.com/channels/687972960811745322/738982866670714901/1164291343745568858):
```
git_branch=devel git_tag=v2.52.0-50-g82f1e9f34-dirty git_commit=82f1e9f342ae46d38f41b0561f879f192cdd8fa0
[WARN] [10-18|19:40:19.892] newPayload failed err="rlp: element is larger than containing list"
[WARN] [10-18|19:40:19.892] fail to process block reason="rlp: element is larger than containing list" slot=7569470
```
This introduces _experimental_ RPC daemon run by embedded Silkworm
library. Same notes as in PR #8353 apply here plus the following ones:
- activated if `http` command-line option is enabled and `silkworm.path`
option is present, nothing more is required (i.e. currently, both block
execution and RPC daemon run by Silkworm when specifying
`silkworm.path`, just to keep things as simple as possible)
- only Execution API endpoints are implemented by Silkworm RPCDaemon,
whilst Engine API endpoints are still served by Erigon RPCDaemon
- some features are still missing, in particular:
- state change notification handling
- custom JSON RPC settings (i.e. Erigon RPC settings are not passed to
Silkworm yet)
db must not use "Durable" and "SyncPeriod" in same time
txpool and downloader must be: Durable
nodedb: SafeNoSync+SyncPeriod
because txpool and downloader already can call sync when they need
(txpool after regular flush, downloader after piece download+validation
completion).
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.15.0 to
0.17.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b225e7ca6d"><code>b225e7c</code></a>
http2: limit maximum handler goroutines to MaxConcurrentStreams</li>
<li><a
href="88194ad8ab"><code>88194ad</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="2b60a61f1e"><code>2b60a61</code></a>
quic: fix several bugs in flow control accounting</li>
<li><a
href="73d82efb96"><code>73d82ef</code></a>
quic: handle DATA_BLOCKED frames</li>
<li><a
href="5d5a036a50"><code>5d5a036</code></a>
quic: handle streams moving from the data queue to the meta queue</li>
<li><a
href="350aad2603"><code>350aad2</code></a>
quic: correctly extend peer's flow control window after MAX_DATA</li>
<li><a
href="21814e71db"><code>21814e7</code></a>
quic: validate connection id transport parameters</li>
<li><a
href="a600b3518e"><code>a600b35</code></a>
quic: avoid redundant MAX_DATA updates</li>
<li><a
href="ea633599b5"><code>ea63359</code></a>
http2: check stream body is present on read timeout</li>
<li><a
href="ddd8598e56"><code>ddd8598</code></a>
quic: version negotiation</li>
<li>Additional commits viewable in <a
href="https://github.com/golang/net/compare/v0.15.0...v0.17.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts).
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
This fixes 2 related issues:
* Now that the bor consensus engine is required for queries it can't be
created based on the pretense of a db directory, but must be based on
chain config read from the db. Using the DB presence causes Bor to get
instantiated for non bor chains which breaks.
* At the moment eth_calls on a remote daemon don't check Bor headers
prior to calling the EVM code as it was just using a fake ETHash
instance - which performs ETH header validation only.
The current version is mostly working but needs adapting to perform lazy
initialization of the engine.
System contract upgrades for Polygon are already handled by the
`BlockAlloc` logic and there's no need to duplicate it with the
`CalcuttaBlock` logic (there's no Calcutta in
https://github.com/maticnetwork/bor).
As per [EIP-1559](https://eips.ethereum.org/EIPS/eip-1559) , there is
only initial base fee after which it adjusts automatically. There are no
current references to a minimum protocol base fee on Ethereum, Gnosis or
Polygon.
The existing code would put a 7 wei minimum limit on transactions to be
included in the pool. This PR removes this check within txPool. Any
transaction can now be added to the `queued` sub-pool after which it
could get promoted to `pending` or `baseFee` subpools.
The`pricelimit` flag still exists and acts on non-local transactions as
a minimum feeCap for inclusion.
This introduces _experimental_ block execution run by embedded Silkworm
API library:
- new command-line option `silkworm.path` to enable the feature by
specifying the path to the Silkworm library
- the Silkworm API shared library is dynamically loaded on-demand
- currently requires to build Silkworm library on the target machine
- available only on Linux at the moment: macOS has issue with [stack
size](https://github.com/golang/go/issues/28024) and Windows would
require [TDM-GCC-64](https://jmeubank.github.io/tdm-gcc/), both need
dedicated effort for an assessment
Closes#8078
This change is primarily intended to support go 1.21, but as a
side-effect requires updating libp2p, which in turn triggers an update
of golang.org/x/exp which creates quite a bit of (simple) churn in the
slice sorting.
This change introduces a new `cmp.Compare` function which can be used to
return an integer satisfying the compare interface for slice sorting.
In order to continue to support mplex for libp2p, the change references
github.com/libp2p/go-libp2p-mplex instead. Please see the PR at
https://github.com/libp2p/go-libp2p/pull/2498 for the official usptream
comment that indicates official support for mplex being moved to this
location.
Co-authored-by: Jason Yellick <jason@enya.ai>
This is to fix an issue with resource usage if the db.size.limit is
increased from its default setting of 2TB. This is applied to the chain
DB, but should not be used on the consensus DB which has smaller data
requirements. Expanding both DBs results in excessive RAM being reserved
by the underlying OS.
