Commit Graph

1501 Commits

Author SHA1 Message Date
Alex Sharov
8850f3a76d
memstat: to use helper func (#8377) 2023-10-05 13:42:30 +07:00
Alex Sharov
c293883ec0
evm: no interface (#8376)
after removal of tevm experiment - we left interfaces everywhere 
removing it for performance and for geth compatibility
2023-10-05 12:23:08 +07:00
canepat
47690db676
Block execution using embedded Silkworm (#8353)
This introduces _experimental_ block execution run by embedded Silkworm
API library:

- new command-line option `silkworm.path` to enable the feature by
specifying the path to the Silkworm library
- the Silkworm API shared library is dynamically loaded on-demand
- currently requires to build Silkworm library on the target machine
- available only on Linux at the moment: macOS has issue with [stack
size](https://github.com/golang/go/issues/28024) and Windows would
require [TDM-GCC-64](https://jmeubank.github.io/tdm-gcc/), both need
dedicated effort for an assessment
2023-10-05 09:27:37 +07:00
Giulio rebuffo
cb9495cc01
Fixed excessive garbage in temporary directory (#8366)
Somebody forgot to close temp dbs
2023-10-04 16:37:44 +02:00
Srdjan
385e69cebc
Add support for additional block number values in eth_getLogs (#8281)
Closes #8199
2023-10-04 15:19:56 +07:00
Alex Sharov
0165d00ee6
downloader: don't drop torrents after download (performance problem there solved) (#8358) 2023-10-04 12:34:45 +07:00
Alex Sharov
fa3b8c23b2
Downloader: step towards more complex datadir (#8286)
migration included - no manual actions required
2023-10-04 11:01:02 +07:00
ledgerwatch
2521f1a696
Fix another case of header download hanging (non-POS) (#8356) 2023-10-03 17:34:03 +01:00
Dmytro
ae88a69511
diagnostics - added query for bootnodes (#8344) 2023-10-03 10:28:21 +07:00
Mark Holt
3d6d2a7c25
Added fix to allow getroothash to work with no api running (#8342)
Whitelisting calculation of the roothash should not be dependent on the
bor api running. This will not always be the case, for example when
erigon is configured with a separate rpc deamon.

To fix this the calculation has been moved to Bor.

Additionally the redundant Bor API code has been removed as this is not
called by any code and the functionality looks to have migrated to the
turbo/jsonrpc package.
2023-10-02 18:55:31 +01:00
Andrew Ashikhmin
0427b162c4
Optimize gas by default in eth_createAccessList (#8337)
Enable gas optimizations of PRs #3453 & #8261 unless the `optimizeGas`
argument is explicitly set to `false`.
2023-10-01 14:22:18 +02:00
Quentin Kniep
3775f4a5d4
Extend gas optimization for eth_createAccessList (#8261)
This builds upon #3453 and #3524, which previously implemented gas
optimizations for the access lists generated by Erigon's implementation
of the `eth_createAccessList` RPC call.

Erigon currently optimizes inclusion of the recipient address based on
how many storage keys are accessed, but does not perform the same
optimization for sender address and precompiled contract addresses.
These changes make the same optimization available for all of these
cases.

Additionally, this handles the cases of block producer address and
created smart contract addresses. If these cases were omitted on purpose
since they heavily rely on state, it may still make sense to offer them
to users but disable them by default.
2023-10-01 13:24:15 +02:00
Somnath
6dd7d8fe6a
Fix some hive tests (#8331)
Changes:
Reorder some Engine API invalid checks to be closer to specs/hive tests
Move the Engine API direct method names to `interfaces`
2023-10-01 12:42:27 +02:00
Jason Yellick
5654ba07c9
Upgrade libp2p (enables go 1.21 support) (#8288)
Closes #8078 

This change is primarily intended to support go 1.21, but as a
side-effect requires updating libp2p, which in turn triggers an update
of golang.org/x/exp which creates quite a bit of (simple) churn in the
slice sorting.

This change introduces a new `cmp.Compare` function which can be used to
return an integer satisfying the compare interface for slice sorting.

In order to continue to support mplex for libp2p, the change references
github.com/libp2p/go-libp2p-mplex instead. Please see the PR at
https://github.com/libp2p/go-libp2p/pull/2498 for the official usptream
comment that indicates official support for mplex being moved to this
location.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-09-29 22:11:13 +02:00
Alex Sharov
d0366ae34e
snapshots: remove concept of separated hist .toml file (#8326) 2023-09-29 11:30:02 +07:00
Alex Sharov
e993ee984b
stage loop: allow nil in hook (#8318) 2023-09-29 09:03:19 +07:00
Alex Sharov
664a58a1db
log skipped files (#8297) 2023-09-28 08:56:37 +07:00
Mark Holt
f26c7b389e
Bor break loop onrewind (#8302)
Add code to the headers state to break processing if a bor milestone
rewind is detected.

The rewind processing happens in the bor/heimdall stage - this change
just avoids unnecessary header loading
if a milestone fork is likely to be detected

---------

Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
2023-09-27 13:17:54 +01:00
Mark Holt
f794438335
Diag session routing (#8232)
Code to support react based UI for diagnostics:

* pprof, prometheus and diagnistics rationalized to use a single router
(i.e. they can all run in the same port)
* support_cmd updated to support node routing (was only first node)
* Multi content support in router tunnel (application/octet-stream &
appliaction/json)
* Routing requests changed from using http forms to rest + query params
* REST query requests can now be made against erigon base port and
diagnostics with the same url format/params

---------

Co-authored-by: dvovk <vovk.dimon@gmail.com>
Co-authored-by: Mark Holt <mark@disributed.vision>
2023-09-25 16:24:17 +01:00
ledgerwatch
ef1729214d
Fixes for bor snapshots (#8270) 2023-09-22 17:08:22 +07:00
lupin012
92446bfa66
Rpcdaemon: fixed few bugs in erigon_getLatestLogs (#8153)
doing some testing some bugs in the API implementation were found and
fixed

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-09-22 17:07:16 +07:00
Alex Sharov
75b0ba0c69
touch data when do warmup (#8268) 2023-09-22 14:36:54 +07:00
Andrew Ashikhmin
9da6b52dba
Add engine_forkchoiceUpdatedV3 to engine_exchangeCapabilities (#8259)
See
https://discord.com/channels/595666850260713488/892088344438255616/1154413019170869379
2023-09-21 16:23:48 +02:00
Anshal Shukla
3c8cbda809
fix finalized api for polygon chain (#8247)
Co-authored-by: Ubuntu <ubuntu@ip-10-0-0-234.eu-west-1.compute.internal>
2023-09-20 13:32:22 +01:00
Somnath Banerjee
f51d9b61a0
Txpool 4844 upgrades Part 2 (#8213)
Some peer-review changes from the last related PR. 
Addition of a flag for BlobSlots - for max allowed blobs per account in
txpool.
Use BlobFee from the block to validate txs in the pool.

See also https://github.com/ledgerwatch/erigon-lib/pull/1125
2023-09-20 17:29:30 +05:30
lmittmann
6985036070
turbo/adapter/ethapi: Add CallArgs.Input attribute (#8242)
This PR adds the ability to pass `input` instead of `data` in
`eth_call`.

Go-ethereum has support for the `input`-attribute (instead of the
`data`) in `eth_call` for some time. The recent go-ethereum release
v1.13.0 changed the ethclient to do a `eth_call` with the `input`
attribute rather than "data". This makes the latest go-ethereum
ethclient incompatible with erigon.

Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
2023-09-20 09:31:00 +07:00
Mark Holt
3b45f53f3d
Milestone stage processing (#8187)
This is the second part of the bor milestone release it contains the
following changes:

* Initialize services
* This is a change from the initial pull request I have moved all of the
initialization to the bor engine. To facilitate this I have just passed
in the heimdall client interface, rather than the whole engine
* Stage processing 
* This is also a change from the original PR - the code is contained in
the bor heimdall stage rather than in headers - the effect should be the
same, but this needs testing

---------

Co-authored-by: Mark Holt <mark@disributed.vision>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
2023-09-18 18:05:33 +01:00
Giulio rebuffo
2aa060f454
Fixed forkchoice nil exception (#8223) 2023-09-17 17:06:44 +02:00
Andrew Ashikhmin
17d6f86218
Don't mark blocks as bad on transient errors (#8197)
For example, erigon on devnet8 marked a block as bad due to
"mdbx_cursor_open: cannot allocate memory":
```
[INFO] [09-12|04:57:36.041] [NewPayload] Handling new payload        height=171035 hash=0x321dea00c4853ee354bebaf8aef3e63fbe06c4508271c0db4c92b0f087aedc3b
171034
[WARN] [09-12|04:57:36.069] Could not validate block                 err="[3/7 BlockHashes] table: Header, mdbx_cursor_open: cannot allocate memory, stack: [kv_mdbx.go:1057 kv_mdbx.
go:1069 kv_mdbx.go:1077 memory_mutation.go:473 memory_mutation.go:502 etl.go:123 etl.go:96 block_writer.go:40 stage_blockhashes.go:49 default_stages.go:457 sync.go:425 sync.go:258 s
tageloop.go:414 backend.go:476 fork_validator.go:250 fork_validator.go:156 ethereum_execution.go:151 execution_client.go:51 chain_reader.go:252 engine_server.go:741 engine_server.go
:235 engine_server.go:600 value.go:586 value.go:370 service.go:224 handler.go:494 handler.go:444 handler.go:392 handler.go:223 handler.go:316 asm_amd64.s:1598]"
[WARN] [09-12|04:57:36.069] ethereumExecutionModule.ValidateChain: chain is invalid hash=0x321dea00c4853ee354bebaf8aef3e63fbe06c4508271c0db4c92b0f087aedc3b
```
With this PR blocks are marked as bad only on genuine protocol errors.
2023-09-17 11:14:36 +02:00
Alex Sharov
eb747c042a
Downloader: add tests to protect from re-downloading (#8191) 2023-09-15 13:46:59 +07:00
Giulio rebuffo
2f8687fe08
Smart SQL transactions (#8185) 2023-09-14 02:56:29 +02:00
Alex Sharov
3813a5ea10
downloader: better logging (#8183) 2023-09-13 13:42:24 +07:00
Alex Sharov
2158b4916d
Downloader: correct logging when create .torrent files (#8182) 2023-09-13 11:49:45 +07:00
Alex Sharov
3cea1b9b9e
torrent: add --webseeds cli arg (#8176) 2023-09-12 12:18:47 +07:00
Somnath Banerjee
a699f64761
Txpool upgrades for EIP-4844 Blob Transactions (#8004)
See https://github.com/ledgerwatch/erigon-lib/pull/1075
2023-09-11 09:38:58 +07:00
Giulio rebuffo
346b278a3b
Caplin: Improved logging (#8169) 2023-09-10 22:10:21 +02:00
alex.sharov
6c1cf0a9b0 save 2023-09-10 15:49:13 +07:00
Alex Sharov
2318138c6c
Downloader: don't fail when see unusual file, skip it (backward/forward compatibility) (#8170) 2023-09-10 15:46:30 +07:00
alex.sharov
66c94d9760 retire snapshots - nil ptr fix 2023-09-09 08:33:42 +07:00
Somnath Banerjee
a9c3c90347
Fix pre cancun conditions (#8152) 2023-09-07 22:15:26 +05:30
Alex Sharov
d60940d7db
Avoid leaking more popped items (#8145) 2023-09-06 15:47:06 +07:00
Park Changwan
ef84972e7c
Add addPeer RPC (#7804)
This PR mirrors https://github.com/testinprod-io/op-erigon/pull/54.

Actual implementation for `admin_addPeer` method.
RPC Spec: Refer to
https://geth.ethereum.org/docs/interacting-with-geth/rpc/ns-admin.

> The addPeer administrative method requests adding a new remote node to
the list of tracked static nodes. The node will try to maintain
connectivity to these nodes at all times, reconnecting every once in a
while if the remote connection goes down.

Requires https://github.com/ledgerwatch/erigon-lib/pull/1033/

After https://github.com/ledgerwatch/erigon-lib/pull/1033 is merged,
will update erigon-lib version, removing replace at go.mod.

Note that even if RPC response returns `true`, it does not guarantee
that RLPx protocol is established between peers. It just adds node
entrypoint to its static peer list, and periodically tries and tests
connections.

## Testing

This RPC needs integration testing, so I made some scenario.

Use below command for testing:

Spin up two dev nodes which p2p enabled:

Start Node 1: RPC running at port 8545:
```sh
./build/bin/erigon --datadir=dev --chain=dev --port=30303 --http.port=8545 --authrpc.port=8551 --torrent.port=42069  --no-downloader --nodiscover --private.api.addr=127.0.0.1:9090 --http --ws --http.api=admin --p2p.allowed-ports=30306,30307,30308  --authrpc.jwtsecret=/tmp/jwt1 --p2p.protocol=67,68  --log.console.verbosity=5
```

Start Node 2: RPC running at port 8546:
```sh
./build/bin/erigon --datadir=dev2 --chain=dev --port=30304 --http.port=8546 --authrpc.port=8552 --torrent.port=42068 --no-downloader --nodiscover --private.api.addr=127.0.0.1:9091 --http --ws --http.api=admin --p2p.allowed-ports=30309,30310,30311  --authrpc.jwtsecret=/tmp/jwt2  --p2p.protocol=67,68  --log.console.verbosity=5
```

Get nodeInfo of node 1 using `admin_nodeInfo` RPC:
```sh
curl --location 'localhost:8545/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_nodeInfo",
	"params":[],
	"id":1
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "id": "b75e0c4d2113b6f144ea8fd356a8f90e612a2a5f48a13c78d7e0e176e5724eb2",
        "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
        "enode": "enode://05ab575d947f2d73065ea0f795dc2d96ed0ad603f3e730ab90dc881122d552c9f59ffcb148fe50546bec8b319daeb3c22ec02e7d12a7c4f2ac4cd26456a04a7c@127.0.0.1:30303?discport=0",
   ...
```

Get nodeInfo of node 2 using `admin_nodeInfo` RPC:
```sh
curl --location 'localhost:8546/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_nodeInfo",
	"params":[],
	"id":2
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 2,
    "result": {
        "id": "32d721e4d75219b021d7f83235f1f1eb8b705d6f85e634bccde564b8f7f94d78",
        "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
        "enode": "enode://1abb8579647779e13b7f68d18f9c776cbd29281841c7f950e9cf9afa996e31120a6f481cea8e90e0f42a0eb1aa00aeafee81c4bae6c31aa16810b795c6d6e069@127.0.0.1:30304?discport=0",
   ...
```

Call `admin_addPeer` RPC to node 2:
```sh
curl --location 'localhost:8546/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_addPeer",
	"params":["enode://05ab575d947f2d73065ea0f795dc2d96ed0ad603f3e730ab90dc881122d552c9f59ffcb148fe50546bec8b319daeb3c22ec02e7d12a7c4f2ac4cd26456a04a7c@127.0.0.1:30303"],
	"id":2
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 2,
    "result": true
}
```


Check peer info of node 1 using `admin_peers` RPC:
```sh
curl --location 'localhost:8545/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_peers",
	"params":[],
	"id":1
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
            "enode": "enode://1abb8579647779e13b7f68d18f9c776cbd29281841c7f950e9cf9afa996e31120a6f481cea8e90e0f42a0eb1aa00aeafee81c4bae6c31aa16810b795c6d6e069@127.0.0.1:55426",
            "id": "32d721e4d75219b021d7f83235f1f1eb8b705d6f85e634bccde564b8f7f94d78",
            "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
            "caps": [
                "eth/66",
                "eth/67"
            ],
            "network": {
                "localAddress": "127.0.0.1:30303",
                "remoteAddress": "127.0.0.1:55426",
                "inbound": true,
                "trusted": false,
                "static": false
            },
            "protocols": null
        }
    ]
}
```

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-09-06 15:31:02 +07:00
Alex Sharov
6c0c0ef75e exec server: run 1st cycle not in global txn. run exec in async tx, run prune in sync tx. partial progress loss fix (partial fix) (#8128) 2023-09-06 10:23:11 +07:00
Giulio rebuffo
544d4188af Fixed snapshots logging (#8132)
Problem was errgroup subroutines finished before their logs could
internally trigger
2023-09-06 10:03:54 +07:00
Giulio rebuffo
fccc563f69
Fixed snapshots logging (#8132)
Problem was errgroup subroutines finished before their logs could
internally trigger
2023-09-06 09:59:54 +07:00
Alex Sharov
13fc0f2a7e
exec server: run 1st cycle not in global txn. run exec in async tx, run prune in sync tx. partial progress loss fix (partial fix) (#8128) 2023-09-06 08:48:07 +07:00
Alex Sharov
8983128d24
Restore genesis reads code - remote rpc case. Remove ctx parameter from baseApi func (tx already has internal ctx) (#8122) 2023-09-04 12:42:08 +07:00
Mark Holt
8ea0096d56
moved metrics sub packages types to metrics (#8119)
This is a non functional change which consolidates the various packages
under metrics into the top level package now that the dead code is
removed.

It is a precursor to the removal of Victoria metrics after which all
erigon metrics code will be contained in this single package.
2023-09-03 08:09:27 +07:00
lupin012
5a9f430c5a
RPCdaemon: fix to work also with snapshot (#8111)
The FIX is necessary to permit to use API also in case block is in the
snapshot files
2023-09-02 09:55:48 +07:00
Giulio rebuffo
42101f529e
demote logs for fork validator (#8118) 2023-09-02 00:20:14 +02:00