This PR separates ENGINE from Ethbackend. It makes it so:
1) EthBackend not a god class
2) We can abstract away engine API so that we can make it CL-like and
enable Consensus-Execution driven design
3) Objective is Json-RPC -> Engine Consensus Module -> Execution module.
- breaks dependency from staged_sync to package with block_reader
implementation
- breaks dependency from snap_sync to package with block_reader
implementation
- breaks dependency from mining to txpool implementation
- allow store non-canonical blocks/senders
- optimize re-org: don't update/delete most of data
- allow mark chain as `Bad` - will be not visible by eth_getBlockByHash,
but can read if have hash+num
It fixing 1 case: when erigon start from scratch
- it did skip stage_snapshots because of race-condition and start
printing `Waiting for Consensus Layer...` logs for long time - but
actually Erigon just did download snapshots at this time and printed
wrong logs.
we update observability in the p2p layer for handlers, and also properly
encode error codes, close streams.
---------
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
deduplicate logic
create more producer goroutines (torrent lib does limiting internally
amount of consumers/disk-readers/hashers by 2, and it's enough because
we can verify multiple files in parallel)
move flag from "downloader torrent_hashes --verify" to "downloader
--verify"
instead of converting from ssz -> struct -> ssz, it may be better to
just stay as ssz, then use methods to read the data.
this pr explores this concept, while maintaining compatiblity with the
existing codebase.
Update to latest erigon-lib which now has the kzg context and update
crate-crypto to latest.
---------
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
`excessDataGas` has been partially made eip-4844 ready, so instead of
passing nils to functions, now it actually assigned to some value (it is
expected to be nil until cancun update).
New transaction type added. This transaction differs from previous
transactions in a network encoding. Network encoding of a new
transaction wraps "regular" transaction with additional data: blobs and
kzgs. The network wrapper will be verified by execution layer using this
extra data.
This branch implements support of btree indexes instead of .kvi files.
Kvi also created, but not used during search/lookup, only for size
comparation.
Btree allows to make seek calls, while recsplit indexes does not support
that query.
During initial tests, btree indexes are ~1-3% smaller than recsplit but
avg lookup time bigger up to 1.5us (which compensates with ability to
fetch next key).
BtAlloc
```
337000 searches, last took 5.417µs avg=4.385µs next_access_last[of 10000 keys] 105ns
338000 searches, last took 3.792µs avg=4.385µs next_access_last[of 10000 keys] 103ns
339000 searches, last took 5.125µs avg=4.386µs next_access_last[of 10000 keys] 103ns
340000 searches, last took 4.75µs avg=4.386µs next_access_last[of 10000 keys] 103ns
341000 searches, last took 3.875µs avg=4.386µs next_access_last[of 10000 keys] 104ns
342000 searches, last took 6.042µs avg=4.387µs next_access_last[of 10000 keys] 107ns
343000 searches, last took 3.209µs avg=4.388µs next_access_last[of 10000 keys] 108ns
344000 searches, last took 2.292µs avg=4.393µs next_access_last[of 10000 keys] 113ns
345000 searches, last took 2.417µs avg=4.394µs next_access_last[of 10000 keys] 106ns
346000 searches, last took 2.625µs avg=4.394µs next_access_last[of 10000 keys] 103ns
347000 searches, last took 4.417µs avg=4.397µs next_access_last[of 10000 keys] 104ns
348000 searches, last took 5.458µs avg=4.398µs next_access_last[of 10000 keys] 103ns
349000 searches, last took 4.875µs avg=4.4µs next_access_last[of 10000 keys] 102ns
350000 searches, last took 4.25µs avg=4.4µs next_access_last[of 10000 keys] 104ns
351000 searches, last took 6.25µs avg=4.403µs next_access_last[of 10000 keys] 106ns
352000 searches, last took 3.125µs avg=4.404µs next_access_last[of 10000 keys] 106ns
```
Recsplit
```
354000 searches, last took 1.209µs avg=3.081µs
355000 searches, last took 1.333µs avg=3.076µs
356000 searches, last took 1.541µs avg=3.071µs
357000 searches, last took 1.333µs avg=3.066µs
358000 searches, last took 1.333µs avg=3.062µs
359000 searches, last took 1.167µs avg=3.057µs
360000 searches, last took 1.791µs avg=3.052µs
361000 searches, last took 1.209µs avg=3.048µs
362000 searches, last took 1.375µs avg=3.043µs
363000 searches, last took 1.166µs avg=3.038µs
364000 searches, last took 1.625µs avg=3.033µs
365000 searches, last took 1.125µs avg=3.029µs
366000 searches, last took 1.208µs avg=3.024µs
367000 searches, last took 1.333µs avg=3.019µs
368000 searches, last took 1.791µs avg=3.015µs
369000 searches, last took 1.333µs avg=3.01µs
370000 searches, last took 1.166µs avg=3.006µs
371000 searches, last took 1.125µs avg=3.001µs
372000 searches, last took 1.333µs avg=2.997µs
```
https://github.com/ledgerwatch/erigon-lib/pull/891/files
That's an initial PR mostly for code review, not ready for production
use
Got basic GraphQL working when querying a single block
---------
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.6.0 to
0.7.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="8e2b117aee"><code>8e2b117</code></a>
http2/hpack: avoid quadratic complexity in hpack decoding</li>
<li><a
href="547e7edf38"><code>547e7ed</code></a>
http2: avoid referencing ResponseWrite.Write parameter after
returning</li>
<li><a
href="39940adcaa"><code>39940ad</code></a>
html: parse comments per HTML spec</li>
<li>See full diff in <a
href="https://github.com/golang/net/compare/v0.6.0...v0.7.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The Read method of math package used in randomAddress function is not
thread safe, instead I've used crand package which is used by matic and
is a fork of thread safe crypto package.
