Commit Graph

20612 Commits

Author SHA1 Message Date
ledgerwatch
ec970ac6bc
Fixed duplicate bucket names, remove presigned URLs (#9060)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-12-23 04:11:12 +00:00
Dmytro
a36071e7ff
dvovk/snapidx (#9049) 2023-12-22 11:25:55 +00:00
Mark Holt
a3a61701e2
Allow proxy paths in Heimdall URL (#8940)
Add paths to the hiemdall config URL when creating calls so that extra
paths needs by, for example proxy servers are not stripped from the flag
value passed into the process.
2023-12-22 10:48:25 +00:00
milen
64072ce6c7
polygon/sync: implement header downloader (#9030)
Co-authored-by: battlmonstr <battlmonstr@users.noreply.github.com>
2023-12-22 10:44:55 +00:00
Alex Sharov
eb4685f3e2
mode to produce block snapshots (#9053)
`STAGES_ONLY_BLOCKS=true` may help to produce BlockSnaps by Erigon2 on
weak machines - it disabling all stages after StageSenders.
2023-12-22 05:43:30 +00:00
Delweng
a45b4c79af
txpool: the same logging style of announced and broadcasted tx (#9054)
Found the different log style for announced and broadcasted tx:

```
[INFO] [12-22|05:18:01.363] Local tx broadcasted                     txHash=ec6b1c87aafd7f8ead5794477be50bda696f2ce17271ad4f6022a756722fa0be to peer=10
[INFO] [12-22|05:18:01.363] local tx announced                       tx_hash=ec6b1c87aafd7f8ead5794477be50bda696f2ce17271ad4f6022a756722fa0be to peer=40 baseFee=1
```

adjust them to the same style

Signed-off-by: jsvisa <delweng@gmail.com>
2023-12-22 12:39:19 +07:00
Alex Sharov
657aafd5b7
allow erigon download .torrent from webseed by default (#9052) 2023-12-22 11:42:35 +07:00
Alex Sharov
95434d100e
webseed: remove dependency on db state (#9051)
erigon didn't download form webseeds if StageSnapshots progress > 0. but
sometime we want "just delete files and re-download them"
2023-12-22 03:35:48 +00:00
Giulio rebuffo
0e18866ed9
Added fill-up with pre-existing API (#9042) 2023-12-21 17:49:40 +01:00
Michelangelo Riccobene
393dd18cad
QA Test - Clean exit (#8981)
First "smoke test", to check Erigon clean exiting on ctrl-c
2023-12-21 16:16:57 +01:00
ledgerwatch
3b10ba95d5
Fix creation of torrent files (#9048)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-12-21 12:42:07 +00:00
battlmonstr
55d37b938c
bor: spanID calculation refactoring (#9040) 2023-12-21 09:52:00 +01:00
Alex Sharov
9eb9151be4
atomic CRUD for .torrent files (#9043) 2023-12-21 05:15:32 +00:00
Alex Sharov
1499fbb582
reduce mutex-contention in downloader.ReCalcStats (#9044) 2023-12-21 04:00:03 +00:00
Alex Sharov
56c5f656d0
downloader: handle race between adding magnet link and files from webseed (#9045) 2023-12-21 03:59:26 +00:00
Alex Sharov
8c822256db
remove bor mainnet files (#9033) 2023-12-21 08:09:54 +07:00
Alex Sharov
4eecd8c86c
print_stages: bor snaps info (#9008) 2023-12-21 08:08:55 +07:00
Alex Sharov
379a5f8ea1
harness test: to use Mock for genesis write - for e35 compatibility (#9012)
`e35` doesn't write genesis state by special func anymore - and Mock
using `m.InsertBlocks` to process genesis block (as any other block).
2023-12-21 08:07:29 +07:00
Alex Sharov
7107cfed0f
snaps: stop merge to 500K and enjoy immutability (#9034) 2023-12-21 08:04:46 +07:00
battlmonstr
2760eeb961
polygon: astrid sync heimdall wrapper (#9017) 2023-12-20 16:48:37 +01:00
Alex Sharov
1ae3f41189
db migration: if stage_snapshots > 0, then create prohibit_new_downloads.lock file (#9036)
it will protect existing nodes from downloading `v2` files - even if
they do upgrade
2023-12-20 18:04:43 +07:00
kaliubuntu0206
0c4bebbf82
Update Ethereum Mainnet storage requirements (#9037)
Sync from scratch took about 4 days 15 hours, with total disk used about
2.8TB

```
ubuntu@localhost:/erigon$ cast block-number
18825796
ubuntu@localhost:/erigon$ ls
chaindata  downloader  jwt.hex  LOCK  nodekey  nodes  snapshots  temp  txpool
ubuntu@localhost:/erigon$ du -hs chaindata
2.2T    chaindata
ubuntu@localhost:/erigon$ du -hs downloader
17M     downloader
ubuntu@localhost:/erigon$ du -hs snapshots
473G    snapshots
ubuntu@localhost:/erigon$ du -hs temp
79G     temp
```
2023-12-20 15:44:49 +07:00
Giulio rebuffo
e75dc25181
Disabled caplin phase2 for Gnosis temporairly. (#9006)
Gnosis phase2 will be disabled until we get good snapshot automation.

* Just run it in "phase1" mode.
* Create jwt secret accordingly
2023-12-19 13:10:34 +01:00
Giulio rebuffo
a0b9d60265
Update Consensus Specification tests (#9024) 2023-12-19 09:42:11 +01:00
dependabot[bot]
33fc53f83d
build(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0 (#9020)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from
0.16.0 to 0.17.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="9d2ee975ef"><code>9d2ee97</code></a>
ssh: implement strict KEX protocol changes</li>
<li><a
href="4e5a26183e"><code>4e5a261</code></a>
ssh: close net.Conn on all NewServerConn errors</li>
<li><a
href="152cdb1503"><code>152cdb1</code></a>
x509roots/fallback: update bundle</li>
<li><a
href="fdfe1f8531"><code>fdfe1f8</code></a>
ssh: defer channel window adjustment</li>
<li><a
href="b8ffc16e10"><code>b8ffc16</code></a>
blake2b: drop Go 1.6, Go 1.8 compatibility</li>
<li><a
href="7e6fbd82c8"><code>7e6fbd8</code></a>
ssh: wrap errors from client handshake</li>
<li><a
href="bda2f3f5cf"><code>bda2f3f</code></a>
argon2: avoid clobbering BP</li>
<li>See full diff in <a
href="https://github.com/golang/crypto/compare/v0.16.0...v0.17.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/crypto&package-manager=go_modules&previous-version=0.16.0&new-version=0.17.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/ledgerwatch/erigon/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-19 08:34:09 +07:00
milen
4f95342036
freezeblocks: fix blockreader last frozen bor span and event ids (#9018)
During testing we run into a "span 7813 not found (db)" due to a very
large unwind (1 million blocks).

This is because the block reader's `LastFrozenSpanID` and
`LastFrozenEventID` returned results that are not consistent with
`FrozenBorBlocks`. The latter is taking into account the existence of
`.idx` files while the former 2 functions were not.

Note such a large unwind is not likely to happen normally unless there
is a bug in our unwind logic or an operator is manually unwinding very
far back due to reasons like chain halts (ie mumbai bug problem from few
months ago), devel testing or anything else along these lines.
Regardless, it exposed the above discrepancy which is best to be fixed.
2023-12-18 19:13:21 +02:00
Andrew Ashikhmin
037754a177
rpc: add txHash to debug_traceBlock* results (#9016)
See https://github.com/ethereum/go-ethereum/pull/27183
2023-12-18 15:30:33 +01:00
Giulio rebuffo
a366629f8a
Cleanuped up request handling (#9007)
* Some requests may lead to undefined behaviour this will make sure to
call .Close once
2023-12-18 13:54:27 +01:00
Giulio rebuffo
ffc26b0f00
Added GET SyncCommittees (#9003) 2023-12-18 13:54:15 +01:00
Alex Sharov
1468317efd
erigon snapshots index: build bor indices (#9009) 2023-12-18 17:46:50 +07:00
Alex Sharov
47c5f16a8d
downloader: don't log complete files info (#9013) 2023-12-18 09:28:20 +01:00
Giulio rebuffo
6a1bb1dff1
Added Blinded blocks type and Beacon endpoints (#9002)
* Added blinded block for internal uses
* Added blinded block to the Beacon API
2023-12-17 00:22:47 +01:00
a
f934ecbd6a
Fix beacon api handler (#8999) 2023-12-16 19:12:40 +01:00
a
a53b1d8b29
flip this inequality (#8904)
this is funny
2023-12-15 21:52:55 -06:00
milen
d23f306fc1
borheimdall: add tests for state sync events persistence (#8989)
- adds a test for persisting state sync events at the beginning of every
sprint
2023-12-15 15:07:39 +02:00
Alex Sharov
9ccd8a1f91
snapshots: --metrics must not affect logs about peers rate (#8990) 2023-12-15 08:05:54 +01:00
Phillip LeBlanc
5d0627f148
Fix typo in log message (#8984) 2023-12-15 09:31:49 +07:00
Giulio rebuffo
eeb471d800
Added Finality_Checkpoints endpoint (#8979) 2023-12-15 02:27:27 +01:00
ddl
e018bb062a
cmd/evm:fix Env struct json tag (#8986)
use geth found this doc problem
https://github.com/ethereum/go-ethereum/pull/28635
2023-12-15 07:24:03 +07:00
Dmytro
e82147caf3
added collecting info about snapshot indexing, renamed downloading prop (#8987) 2023-12-15 07:23:26 +07:00
milen
1a6b83b82c
borheimdall: add test for span persistence (#8988)
1. Adds an eth/stagedsync/test package which provides a test Harness
object
2. Adds the first automated test to the bor-heimdall stage regarding
span persistence (more to come in subsequent PRs)
3. Fixes a bug in the bor-heimdall stage which was uncovered with the
test - we do not fetch span 0 when we sync straight from blockNum=0
without snapshots
4. Reorganises all mocks to be placed under ./mock sub-package within
their respective packages
2023-12-14 22:50:59 +02:00
battlmonstr
ce57b8f54f
silkworm: make install (#8985)
We've got a report from a user that erigon fails to run with this error:

```
/opt/erigon/releases/latest/bin/erigon: error while loading shared libraries: libsilkworm_capi.so: cannot open shared object file: No such file or directory
```

On this system erigon is executed using a service-account user which has
no permission to access the build user's $HOME where the
libsilkworm_capi.so resides (inside $GOPATH/pkg/mod).

This adds a support to silkworm-go to look for the library relative to
the executable path on Linux:

d4ec8a8bce

and a new `make DIST=<path> install` command which copies both the
library and erigon binary to any destination.
2023-12-14 21:45:18 +07:00
Andrew Ashikhmin
b26d0f201e
Engine API PR 498 (clarify payloadAttributes checks) (#8982)
Implements https://github.com/ethereum/execution-apis/pull/498
2023-12-14 13:59:46 +01:00
Dmytro
3b68d57c28
collect statistics even for download completed or not started files (#8956)
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-12-14 07:53:54 +01:00
ledgerwatch
58dabdc601
Fixup bittorrent/webseed download logs after refactoring (#8976)
After a refactoring, the detailed bittorrent vs webseeds log has
disappeared
2023-12-13 17:40:20 +01:00
Giulio rebuffo
4e83ef0ef5
Optimized beacon state retrieval (#8963)
* Reconstruct previous epoch without looking at DB: no hindrance to
performance -> removed 15GB
* Store inactivity scores and slashings in MDBX and do not store diffs
for them(they are tiny 700/400 bytes)
* Reduced dumps from every 2048 to 1024 -> Added 5 GB (maybe we should
down it to 768)
* Parallel processing of shuffled sets, 2x performance boost in reading
participation.
* Store balances diffs in a Btree diff matter, see:
https://github.com/ledgerwatch/erigon-documents/blob/master/caplin/design/data-model.md#uint64listuint64vector
2023-12-13 15:23:27 +01:00
Dmytro
ac1e42b68d
added grabbing info about downloaded metadata (#8972) 2023-12-13 21:04:14 +07:00
Alex Sharov
06e77d1705
don't log cancelation error at graceful shutdown (#8943) 2023-12-12 17:07:12 +07:00
battlmonstr
c1146bda49
p2p: skip TestUDPv4_smallNetConvergence on Linux (#8731) (#8962) 2023-12-12 17:06:48 +07:00
Andrew Ashikhmin
faaf5a0c86
Amend incarnation logic (#8954) 2023-12-12 11:03:41 +01:00