Alex Sharov
9a9db2d90b
downloader: whitelist only .seg.torrent files. ( #8729 )
2023-11-15 11:23:57 +07:00
Giulio rebuffo
8d8368091c
Add full support to beacon snapshots ( #8665 )
...
This PR adds beacon blocks snapshots for the following chains:
* Mainnet snapshots
* Sepolia snapshots
2023-11-13 14:10:57 +01:00
dependabot[bot]
97e94f6d2c
build(deps): bump github.com/consensys/gnark-crypto from 0.12.0 to 0.12.1 in /erigon-lib ( #8680 )
...
Bumps
[github.com/consensys/gnark-crypto](https://github.com/consensys/gnark-crypto )
from 0.12.0 to 0.12.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/consensys/gnark-crypto/releases ">github.com/consensys/gnark-crypto's
releases</a>.</em></p>
<blockquote>
<h2>v0.12.1</h2>
<h2>What's Changed</h2>
<ul>
<li>fix: GLV exponentiation in GT with large exponents by <a
href="https://github.com/ThomasPiellard "><code>@ThomasPiellard</code></a>
in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/451 ">Consensys/gnark-crypto#451</a>
<a
href="https://github.com/Consensys/gnark-crypto/security/advisories/GHSA-pffg-92cg-xf5c ">Security
Advisory</a></li>
<li>feat: kzg extended transcript by <a
href="https://github.com/ThomasPiellard "><code>@ThomasPiellard</code></a>
in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/452 ">Consensys/gnark-crypto#452</a></li>
<li>perf(fft): introduce cache efficient bit reverse shuffling by <a
href="https://github.com/gbotrel "><code>@gbotrel</code></a> in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/446 ">Consensys/gnark-crypto#446</a></li>
<li>perf: Improve performance of point equality checks by <a
href="https://github.com/jsign "><code>@jsign</code></a> in <a
href="https://redirect.github.com/Consensys/gnark-crypto/pull/450 ">Consensys/gnark-crypto#450</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/Consensys/gnark-crypto/compare/v0.12.0...v0.12.1 ">https://github.com/Consensys/gnark-crypto/compare/v0.12.0...v0.12.1 </a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="da0317fd01https://redirect.github.com/consensys/gnark-crypto/issues/453 ">#453</a>)</li>
<li><a
href="65cdb1d1c2https://redirect.github.com/consensys/gnark-crypto/issues/452 ">#452</a>)</li>
<li><a
href="95e674b3d6https://redirect.github.com/consensys/gnark-crypto/issues/446 ">#446</a>)</li>
<li><a
href="ec6be1a037https://redirect.github.com/consensys/gnark-crypto/issues/451 ">#451</a>
from Consensys/fix/glv</li>
<li><a
href="aef1eef48afeaf161b4dced3076bef4096ad9913ace5318b72https://redirect.github.com/consensys/gnark-crypto/issues/450 ">#450</a>
from jsign/jsign-eq-improv</li>
<li><a
href="dc4e2d4f77https://github.com/consensys/gnark-crypto/compare/v0.12.0...v0.12.1 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts ).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-13 10:27:26 +01:00
Alex Sharov
950bb12098
metrics: race fix ( #8701 )
2023-11-11 18:09:04 +03:00
Alex Sharov
35bfffd621
sys deps up ( #8695 )
2023-11-11 15:04:18 +03:00
Alex Sharov
6722334671
fix mdbx version in devel branch ( #8679 )
2023-11-10 19:44:15 +03:00
pwd123
ff88a9393e
fix-rules-typo ( #8681 )
2023-11-09 07:40:24 +03:00
pwd123
a0682088ab
Add readonly transaction rollback check rule ( #8682 )
2023-11-09 07:40:11 +03:00
Dmytro
9c7c758bda
added snapshot sync diagnostic information, updated diagnostic channel ( #8645 )
2023-11-07 12:50:36 +00:00
Mark Holt
509a7af26a
Discovery zero refresh timer ( #8661 )
...
This fixes an issue where the mumbai testnet node struggle to find
peers. Before this fix in general test peer numbers are typically around
20 in total between eth66, eth67 and eth68. For new peers some can
struggle to find even a single peer after days of operation.
These are the numbers after 12 hours or running on a node which
previously could not find any peers: eth66=13, eth67=76, eth68=91.
The root cause of this issue is the following:
- A significant number of mumbai peers around the boot node return
network ids which are different from those currently available in the
DHT
- The available nodes are all consequently busy and return 'too many
peers' for long periods
These issues case a significant number of discovery timeouts, some of
the queries will never receive a response.
This causes the discovery read loop to enter a channel deadlock - which
means that no responses are processed, nor timeouts fired. This causes
the discovery process in the node to stop. From then on it just
re-requests handshakes from a relatively small number of peers.
This check in fixes this situation with the following changes:
- Remove the deadlock by running the timer in a separate go-routine so
it can run independently of the main request processing.
- Allow the discovery process matcher to match on port if no id match
can be established on initial ping. This allows subsequent node
validation to proceed and if the node proves to be valid via the
remainder of the look-up and handshake process it us used as a valid
peer.
- Completely unsolicited responses, i.e. those which come from a
completely unknown ip:port combination continue to be ignored.
-
2023-11-07 08:48:58 +00:00
Giulio rebuffo
e08b031f74
Updated go.sum and go.mod for snapshots ( #8664 )
2023-11-06 23:39:44 +01:00
Giulio rebuffo
4b580dcc2f
update caplin snapshots hashes ( #8663 )
...
This PR also adds snippets to download caplin snapshots
2023-11-06 21:05:07 +01:00
Alex Sharov
bba91e90ec
downloader: demote webseed request errors ( #8662 )
2023-11-06 14:47:57 +01:00
ledgerwatch
6b09679774
Fix typo in make output ( #8649 )
2023-11-04 11:11:15 +00:00
a
2aab8f496c
rlp2 ( #8643 )
...
rlp2 is a package that aims to replace the existing erigon-lib/rlp
package and the erigon/common/rlp
it is called rlp2 for now because it requires breaking changes to
erigon-lib/rlp and i do not have the time right now to test all current
uses of such functions
however, the encoder/decoder characteristics of rlp2 might be desirable
for caplin, and also for execution layer parsing blob txns, so im
putting it in a folder called rlp2 (note that it exports package rlp for
easier switching later)
importantly, rlp2 is designed for single-pass decoding with the ability
to skip elements one does not care about. it also is zero alloc.
2023-11-04 09:22:11 +07:00
Andrew Ashikhmin
8288dcb5c2
erigon-lib: remove unused constants from protocol.go ( #8644 )
2023-11-03 10:20:36 +01:00
Alex Sharov
714fa0fdc7
mainnet 18.3M snaps ( #8639 )
2023-11-02 14:14:39 +07:00
Alex Sharov
329d18ef6f
snapshots: reduce merge limit of blocks to 100K ( #8614 )
...
Reason:
- produce and seed snapshots earlier on chain tip. reduce depnedency on
"good peers with history" at p2p-network.
Some networks have no much archive peers, also ConsensusLayer clients
are not-good(not-incentivised) at serving history.
- avoiding having too much files:
more files(shards) - means "more metadata", "more lookups for
non-indexed queries", "more dictionaries", "more bittorrent
connections", ...
less files - means small files will be removed after merge (no peers for
this files).
ToDo:
[x] Recent 500K - merge up to 100K
[x] Older than 500K - merge up to 500K
[x] Start seeding 100k files
[x] Stop seeding 100k files after merge (right before delete)
In next PR:
[] Old version of Erigon must be able download recent hashes. To achieve
it - at first start erigon will download preverified hashes .toml from
s3 - if it's newer that what we have (build-in) - use it.
2023-11-01 23:22:35 +07:00
Alex Sharov
8e54a965ca
webseed: don't download .torrent files ( #8629 )
2023-11-01 15:50:35 +07:00
Alex Sharov
b5ea87d8c5
mdbx_empty_kv_dup ( #8635 )
2023-11-01 15:50:27 +07:00
Alex Sharov
d4b19b3bb6
pool: limit broadcast tx size by 4kb ( #8634 )
...
similar to geth
2023-11-01 09:56:02 +07:00
yyjia
22332f72eb
fix: batch close confict with flush ( #8613 )
...
both batch.Flush() and batch.close() close channel, I think flush is not
necessary to close channel
2023-11-01 09:08:47 +07:00
battlmonstr
eb65ffbaa7
config: avoid OOM in docker using cgroups v2 limit ( #6646 ) ( #8632 )
...
Tested on Debian 11 by:
$ git revert HEAD
$ make docker
$ docker run --memory=8.8G <sha>
before the fix outputs:
15.6 GB
after the fix outputs:
8.8 GB
2023-11-01 09:02:34 +07:00
Mark Holt
c20af7d985
Add code to handle close/cancel semantics ( #8633 )
2023-10-31 18:43:23 +00:00
Alex Sharov
c90bff7e22
mdbx: use release tag ( #8628 )
2023-10-31 10:11:40 +07:00
Anshal Shukla
83bbe0dd37
Fix Agra hf scheduling on Mumbai ( #8620 )
...
Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
2023-10-30 15:18:47 +01:00
Alex Sharov
c23e5a1abf
downloader: preparations for reducing blocks merge limit ( #8612 )
2023-10-30 13:46:35 +07:00
Alex Sharov
b311da959f
downloader: webseed better error messages ( #8611 )
2023-10-30 12:13:45 +07:00
Andrew Ashikhmin
8f67a8c556
Txpool: db unlimited grow fix ( #8596 ) ( #8608 )
...
read transaction was opened before stream.Recv(), but stream.Recv() is
blocking infinity loop. so, this read transaction never rollback -
causing unlimited db grow.
---------
Merge PR #8596 into `devel`
---------
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-10-29 11:13:06 +01:00
Andrew Ashikhmin
8231cdaede
downloader: less webseed logs ( #8586 ) ( #8607 )
...
Merge PR #8586 into `devel`
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-10-29 10:37:50 +01:00
yyjia
216f20d5f8
batch flush can without read tx ( #8597 )
2023-10-29 12:33:47 +07:00
Giulio rebuffo
0e5af0a69c
Added beacon snapshots download ( #8601 )
2023-10-28 17:41:50 +02:00
Dmytro
9adf31b8eb
bytes transfet separated by capability and category ( #8568 )
...
Co-authored-by: Mark Holt <mark@distributed.vision>
2023-10-27 22:30:28 +03:00
Alex Sharov
f36d090f39
make db-tools: suspend enum-int linter error ( #8598 )
...
```
error: conflicting types for ‘mdbx_get_datacmp’ due to enum/integer mismatch; have ‘int (*(unsigned int))(const MDBX_val *, const MDBX_val *)’ {aka ‘int (*(unsigned int))(const struct iovec *, const struct iovec *)’} [-Werror=enum-int-mismatch]
26609 | __cold MDBX_cmp_func *mdbx_get_datacmp(unsigned flags) {
| ^~~~~~~~~~~~~~~~
mdbx.h:4932:1: note: previous declaration of ‘mdbx_get_datacmp’ with type ‘int (*(MDBX_db_flags_t))(const MDBX_val *, const MDBX_val *)’ {aka ‘int (*(MDBX_db_flags_t))(const struct iovec *, const struct iovec *)’}
```
can't upgrade mdbx version for now
2023-10-27 11:01:11 +07:00
Andrew Ashikhmin
38e91809f9
Revert "Move validator set snapshot computation to bor_heimdall stage… ( #8580 )
...
PR #8202 might cause Issue #8550 , so reverting it until Alexey's return.
This reverts commit 2ce98f8337
2023-10-25 14:02:31 +02:00
Alex Sharov
3d3c0bec18
downloader: log.Trace non-important log messages instead of skipping it ( #8574 )
2023-10-25 11:35:22 +07:00
Dmytro
ec59be2261
Dvovk/sentinel and sentry peers data collect ( #8533 )
2023-10-23 17:33:08 +03:00
Mark Holt
601b7e6870
Fix metrics get perf regression ( #8563 )
...
Calling vm to get a metric and then preregistering it to avoid
duplication is expensive.
To avoid this we can make a local lookup first which is a map this
quicker. This is a temporary fix
until vm is removed at which point we'll have a single prom_client
registry
Fixes: https://github.com/ledgerwatch/erigon/issues/8558
2023-10-23 15:19:52 +01:00
Mark Holt
047197e24c
Fix panic in isAgra check for txpool ( #8562 )
...
Fixes a nil pointer panic in the isagra check when running on an empty
chain db. In this case the block will return as nil with no error.
2023-10-23 15:17:55 +01:00
Andrew Ashikhmin
a54939633e
Numerical instead of lexicographic sorting in borKeyValueConfigHelper ( #8560 )
...
Corresponds to Item 3 of https://github.com/maticnetwork/bor/pull/1055
2023-10-23 14:11:32 +02:00
Andrew Ashikhmin
a226b6ca29
Fix wiring of AgraBlock into tx pool ( #8555 )
...
Fixes and simplifications to PR #8504
2023-10-23 11:03:46 +02:00
Giulio rebuffo
995009ac7b
Added cli too for Snapshots Generations for Caplin ( #8543 )
2023-10-22 19:21:37 +02:00
Andrew Ashikhmin
ee942473d9
Replace eip1559FeeCollector with burntContract ( #8553 )
...
burntContract, introduced in PR #8504 , is more generic than
eip1559FeeCollector
2023-10-22 16:23:40 +02:00
Andrew Ashikhmin
84a0b702b9
Update go-kzg-4844 to v0.7.0 ( #8551 )
2023-10-22 12:44:56 +02:00
a
436493350e
Sentinel refactor ( #8296 )
...
1. changes sentinel to use an http-like interface
2. moves hexutil, crypto/blake2b, metrics packages to erigon-lib
2023-10-22 01:17:18 +02:00
Anshal Shukla
7dce1268ab
Agra HF ( #8504 )
...
Adds agra HF to the bor consensus
2023-10-21 01:16:19 +05:30
ledgerwatch
2ce98f8337
Move validator set snapshot computation to bor_heimdall stage ( #8202 )
...
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-10-20 18:31:00 +01:00
Alex Sharov
f696c37183
etl: more tests ( #8534 )
2023-10-20 13:54:46 +07:00
Alex Sharov
783b2f39d6
downloader: fix e2 bucket name ( #8521 )
2023-10-20 13:54:38 +07:00
Giulio rebuffo
ec2207fc7f
Gnosis Snapshot updated ( #8532 )
2023-10-19 23:49:14 +02:00
