Commit Graph

3123 Commits

Author SHA1 Message Date
alex.sharov
13f782cef7 TxLookup: don't fail if body not found 2023-09-12 11:02:28 +07:00
ledgerwatch
dd3e628d80
Fix heimdall incorrect sync data bug (#8174) 2023-09-11 12:17:41 +01:00
Somnath Banerjee
a699f64761
Txpool upgrades for EIP-4844 Blob Transactions (#8004)
See https://github.com/ledgerwatch/erigon-lib/pull/1075
2023-09-11 09:38:58 +07:00
maohuazhu
34c0df8bdd
config: fix CreateConsensusEngineBareBones (#8154)
Fix ``CreateConsensusEngineBareBones`` for bor and aura. Current codes
will pass a ``**chain.BorConfig`` or ``**chain.AuraConfig to``
``CreateConsensusEngine`` and cause a panic

Co-authored-by: Maohua Zhu <zhumaohua@cobo.com>
2023-09-11 09:21:18 +07:00
Andrew Ashikhmin
0ebc5fe24e
Don't mark block as bad if execution was cancelled (#8161)
A potential small fix that I made while looking at the Polygon bad
header issue (don't think it's the root cause though).
2023-09-10 11:25:02 +02:00
Alex Sharov
d60940d7db
Avoid leaking more popped items (#8145) 2023-09-06 15:47:06 +07:00
Park Changwan
ef84972e7c
Add addPeer RPC (#7804)
This PR mirrors https://github.com/testinprod-io/op-erigon/pull/54.

Actual implementation for `admin_addPeer` method.
RPC Spec: Refer to
https://geth.ethereum.org/docs/interacting-with-geth/rpc/ns-admin.

> The addPeer administrative method requests adding a new remote node to
the list of tracked static nodes. The node will try to maintain
connectivity to these nodes at all times, reconnecting every once in a
while if the remote connection goes down.

Requires https://github.com/ledgerwatch/erigon-lib/pull/1033/

After https://github.com/ledgerwatch/erigon-lib/pull/1033 is merged,
will update erigon-lib version, removing replace at go.mod.

Note that even if RPC response returns `true`, it does not guarantee
that RLPx protocol is established between peers. It just adds node
entrypoint to its static peer list, and periodically tries and tests
connections.

## Testing

This RPC needs integration testing, so I made some scenario.

Use below command for testing:

Spin up two dev nodes which p2p enabled:

Start Node 1: RPC running at port 8545:
```sh
./build/bin/erigon --datadir=dev --chain=dev --port=30303 --http.port=8545 --authrpc.port=8551 --torrent.port=42069  --no-downloader --nodiscover --private.api.addr=127.0.0.1:9090 --http --ws --http.api=admin --p2p.allowed-ports=30306,30307,30308  --authrpc.jwtsecret=/tmp/jwt1 --p2p.protocol=67,68  --log.console.verbosity=5
```

Start Node 2: RPC running at port 8546:
```sh
./build/bin/erigon --datadir=dev2 --chain=dev --port=30304 --http.port=8546 --authrpc.port=8552 --torrent.port=42068 --no-downloader --nodiscover --private.api.addr=127.0.0.1:9091 --http --ws --http.api=admin --p2p.allowed-ports=30309,30310,30311  --authrpc.jwtsecret=/tmp/jwt2  --p2p.protocol=67,68  --log.console.verbosity=5
```

Get nodeInfo of node 1 using `admin_nodeInfo` RPC:
```sh
curl --location 'localhost:8545/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_nodeInfo",
	"params":[],
	"id":1
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "id": "b75e0c4d2113b6f144ea8fd356a8f90e612a2a5f48a13c78d7e0e176e5724eb2",
        "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
        "enode": "enode://05ab575d947f2d73065ea0f795dc2d96ed0ad603f3e730ab90dc881122d552c9f59ffcb148fe50546bec8b319daeb3c22ec02e7d12a7c4f2ac4cd26456a04a7c@127.0.0.1:30303?discport=0",
   ...
```

Get nodeInfo of node 2 using `admin_nodeInfo` RPC:
```sh
curl --location 'localhost:8546/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_nodeInfo",
	"params":[],
	"id":2
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 2,
    "result": {
        "id": "32d721e4d75219b021d7f83235f1f1eb8b705d6f85e634bccde564b8f7f94d78",
        "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
        "enode": "enode://1abb8579647779e13b7f68d18f9c776cbd29281841c7f950e9cf9afa996e31120a6f481cea8e90e0f42a0eb1aa00aeafee81c4bae6c31aa16810b795c6d6e069@127.0.0.1:30304?discport=0",
   ...
```

Call `admin_addPeer` RPC to node 2:
```sh
curl --location 'localhost:8546/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_addPeer",
	"params":["enode://05ab575d947f2d73065ea0f795dc2d96ed0ad603f3e730ab90dc881122d552c9f59ffcb148fe50546bec8b319daeb3c22ec02e7d12a7c4f2ac4cd26456a04a7c@127.0.0.1:30303"],
	"id":2
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 2,
    "result": true
}
```


Check peer info of node 1 using `admin_peers` RPC:
```sh
curl --location 'localhost:8545/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_peers",
	"params":[],
	"id":1
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
            "enode": "enode://1abb8579647779e13b7f68d18f9c776cbd29281841c7f950e9cf9afa996e31120a6f481cea8e90e0f42a0eb1aa00aeafee81c4bae6c31aa16810b795c6d6e069@127.0.0.1:55426",
            "id": "32d721e4d75219b021d7f83235f1f1eb8b705d6f85e634bccde564b8f7f94d78",
            "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
            "caps": [
                "eth/66",
                "eth/67"
            ],
            "network": {
                "localAddress": "127.0.0.1:30303",
                "remoteAddress": "127.0.0.1:55426",
                "inbound": true,
                "trusted": false,
                "static": false
            },
            "protocols": null
        }
    ]
}
```

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-09-06 15:31:02 +07:00
alex.sharov
1160df12a3 save 2023-09-06 14:44:51 +07:00
Alex Sharov
608341b636
shutdown deadlock (#8135) 2023-09-06 10:23:59 +07:00
Mark Holt
8ea0096d56
moved metrics sub packages types to metrics (#8119)
This is a non functional change which consolidates the various packages
under metrics into the top level package now that the dead code is
removed.

It is a precursor to the removal of Victoria metrics after which all
erigon metrics code will be contained in this single package.
2023-09-03 08:09:27 +07:00
Mark Holt
e81852320c
Remove unused metrics and out of date versions (#8109)
ergon/metrics contains a lot of unused types - the pull is removing them
prior to rationalizing between Prometheus and Victoria Metrics types.

It also have upgrades of golang-lru + sets types to v2 when possible.
2023-09-01 07:13:13 +07:00
Mark Holt
a4cfbe0d56
Heimdall metrics + Metrics HTTP server rationalization (#8094)
This is an update of:

https://github.com/ledgerwatch/erigon/pull/7846

which uses a local fork of victoria metrics to include the changes that
https://github.com/anshalshukla added to the original for we where
using.

It also includes code to address the duplicate metrics issue identified
here:

https://github.com/ledgerwatch/erigon/issues/8053

It has one more associated fix which is to correctly add a metadata
label to counters, these where previously labelled as gauges.

e.g. 

```
# TYPE p2p_peers counter
p2p_peers 0
```
rather than

```
# TYPE p2p_peers gauge
p2p_peers 0
```

---------

Co-authored-by: Anshal Shukla <53994948+anshalshukla@users.noreply.github.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
2023-08-31 09:04:27 +01:00
Andrew Ashikhmin
9b63764b16
Move ApplyDAOHardFork & UpgradeBuildInSystemContract to engine.Initialize (#8095)
Now all protocol-stipulated changes at the beginning of the block (AuRa
stuff,
[DAO](https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
irregular state change, Calcutta system contract upgrade,
[EIP-4788](https://eips.ethereum.org/EIPS/eip-4788) beacon root) are
handled by consensus engine `Initialize()`.
2023-08-30 15:51:19 +02:00
Mark Holt
f05a6ab43e
Bor mining benchmark (#8096)
Replacement for: https://github.com/ledgerwatch/erigon/pull/7998 with
windows fixes

---------

Co-authored-by: SHIVAM SHARMA <shivam691999@gmail.com>
2023-08-30 10:25:02 +01:00
formatting
6728359e55
fix debug_traceTransaction get noce overflow (#8100)
fix: Prevent nonce overflow in debug_traceTransaction
fix #8099
2023-08-30 15:31:13 +07:00
Mark Holt
f2d0118a33
Bor snapshot block production (#8065)
I have added:

```go
{
	ID:          stages.BorHeimdall,
	Description: "Download Bor-specific data from Heimdall",
	Forward: func(firstCycle bool, badBlockUnwind bool, s *StageState, u Unwinder, tx kv.RwTx, logger log.Logger) error {
		if badBlockUnwind {
			return nil
		}
		return BorHeimdallForward(s, u, ctx, tx, borHeimdallCfg, true, logger)
	},
	Unwind: func(firstCycle bool, u *UnwindState, s *StageState, tx kv.RwTx, logger log.Logger) error {
		return BorHeimdallUnwind(u, ctx, s, tx, borHeimdallCfg)
	},
	Prune: func(firstCycle bool, p *PruneState, tx kv.RwTx, logger log.Logger) error {
		return BorHeimdallPrune(p, ctx, tx, borHeimdallCfg)
	},
},
```
To MiningStages as well as Default as otherwise bor events are not added
when the block producer creates new blocks.

There are a couple of questions I have around this implementation:

* Is this the right place to add this
* As the state is also executed when the default stage is processed ther
is some duplicate processing for the block producing node.
* There is a duplicated call to heimdall which could be removed if the
stages share state - but its not clear if we want to do this.
* I don't think the mining stage needs to prune as this will be
replicated in the default iteration

This can be tested using the devnet with the following arguments:

```
--chain bor-devnet --bor.localheimdall --scenarios state-sync
```

This will generate sync events via an ethereum devnet which are
transmitted to bor chain and will be executed at the end of the snapshot
delay, which results in events generated from the bor chain. This tests
the whole sync, block generation, event lifecycle. As it needs to wait
for sprints to end after a sufficient delay it is quite slow to run.
2023-08-30 08:06:09 +01:00
Alex Sharov
d1d348211f
add flag --force.partial.commit: to workaround problem "start from backup takes long time and can't save partial progress" (#8090) 2023-08-30 08:49:16 +07:00
Andrew Ashikhmin
63628618eb
Small fixes to Holesky config (#8092)
Small fixes to PR #8064:

- Don't set cancunTime for Holesky – otherwise we'll have to overwrite
it later using the `--override.cancun` flag
- We don't have block snapshots for Holesky yet
- Add Holesky to `isChainIDPoS`

See https://github.com/eth-clients/holesky/pull/68
2023-08-29 16:29:14 +02:00
racytech
f477348a50
Support for --chain=holesky (#8064) 2023-08-26 16:31:29 +06:00
Giulio rebuffo
8291aa2497
Caplin: Storing Execution Headers in Caplin (#8080)
This PR stores Execution Headers in Caplin when running without storing
fullblocks
2023-08-26 00:41:57 +02:00
Mark Holt
c51573f333
Bor eth event flow (#8068)
Implemented polygon->eth flow
2023-08-25 12:19:39 +01:00
battlmonstr
32ca0e5ab1
sync: revert flawed dropUselessPeers logic and alleviate its issues (#8062)
The current logic is flawed, because it drops all peers that are less
synced.
It is valid to return empty responses by the eth spec.
A proper logic should penalize from the context of the sync process,
where enough "reputation" data is collected about a peer.

In order to be able to connect to erigon 2.48 peers that have
--sentry.drop-useless-peers enabled,
this adds a check to not reply with an empty headers list.
If we reply with an empty list, we're going to be considered useless and
kicked.
Once enough of erigon nodes are updated in the network past this commit,
this check should be removed,
because it is totally acceptable to return an empty list by the eth
spec.
2023-08-25 11:42:54 +02:00
alex.sharov
5f7a84ff72 fix linter 2023-08-25 12:05:45 +07:00
alex.sharov
da275f35cc fix linter 2023-08-25 12:05:36 +07:00
Alex Sharov
2b6c21fddb
move mdbx to new org (#8061) 2023-08-24 18:00:24 +07:00
Andrew Ashikhmin
ece8010d0d
Remove unused transaction and node packets from eth/protocols (#8057)
Transactions are served by erigon-lib's txpool and we don't serve
NodeData.
2023-08-24 06:50:04 +02:00
battlmonstr
2e29ff33e1
bor: BroadcastNewBlock to all peers from validator nodes (#8030)
Currently PropagateNewBlockHashes and BroadcastNewBlock
selects a subset of all sentries by taking a `Sqrt(len(sentries))`,
and then for each sentry SendMessageToRandomPeers
selects a subset of its peers by taking `Sqrt(len(peerInfos))`.

This behaviour limits the broadcast scope with a lot of peers, e.g. 100
becomes 10,
but is not great with very few peers, or if the message is very
important
to broadcast to everyone, which is the case of bor validator/proposer
nodes.

* send to all sentries in both BroadcastNewBlock and PropagateNewBlockHashes
* remove peerCountConstrained sqrt logic in SendMessageToRandomPeers
* add maxPeers provider func as a parameter to MultiClient
* default it to 10 for eth and 0 (unlimited) for bor validators

---------

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-08-23 14:28:39 +02:00
battlmonstr
a83a478bae
bor: fix panic reading total difficulty for isChainPoS (#8032)
because bor doesn't have headers,
ReadCurrentHeader returns nil as expected

refactor isChainPoS to only call ReadCurrentHeader/ReadTd if needed and
use chain config IDs
2023-08-22 13:40:36 +02:00
Giulio rebuffo
bd81e15981
Caplin: Implemented SQL beacon indexer (#8043) 2023-08-22 01:24:26 +02:00
Andrew Ashikhmin
6bc0ca9e85
Correctly compute fork id when timestamp fork is activated in genesis (#8046)
See https://github.com/ethereum/go-ethereum/pull/27895
2023-08-21 15:35:13 +02:00
Alex Sharov
863a706858
caplin: use tmpdir inside datadir (#8041) 2023-08-20 15:26:34 +02:00
ledgerwatch
532cc13a03
Fix panic due to Borevents snapshots changes (#8042) 2023-08-20 10:42:09 +01:00
Giulio rebuffo
6d2a9aed2f
Fixed startup panic (#8040) 2023-08-18 22:16:30 +02:00
ledgerwatch
6b6c0caad0
Snapshots of Bor events (#7901)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp-2.home>
2023-08-18 17:10:35 +01:00
Giulio rebuffo
38db2ed2e7
[Caplin Phase 2]: Initial working implementation (#8033)
caplin just using execution.proto
2023-08-18 15:43:22 +02:00
Andrew Ashikhmin
2a4b1b3077
Pass proper timestamp in stage senders (#8013)
This fixes "blob tx is not supported by signer", which occurred in some
Hive tests.
2023-08-13 21:38:01 +02:00
Alex Sharov
c464d84334
tmpdb: move to tmpdir (#8003) 2023-08-12 01:06:15 +06:00
Andrew Ashikhmin
f854a73185
Add enough blob gas for block building (EIP-4844) (#8001) 2023-08-11 16:09:03 +02:00
Andrew Ashikhmin
03927d3e27
Call InitializeBlockExecution in SpawnMiningExecStage (EIP-4788) (#7999)
This fixes the trie state root issue that was occurring in the Hive
tests for Cancun.
2023-08-11 14:04:53 +02:00
a
db5b348673
Caplin block persistence (#7941)
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-08-09 01:21:19 +02:00
Mark Holt
6bd58f7c39
check difficulty for nil (#7978)
Fix checks `chainConfig.TerminalTotalDifficulty` for nil in
`isChainPoS`. Not doing so causes the Bor startup flow to panic.
2023-08-08 22:26:59 +01:00
Andrew Ashikhmin
ae76df1b8a
Update execution-spec-tests to v1.0.1 (#7980)
[v1.0.1](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.1)
"Cancun Devnet-8 Pre-Release" adds some tests for Cancun. This PR only
fixes a couple of issues; the majority of the new tests are still
failing and thus are skipped in `TestExecutionSpec`.
2023-08-08 15:01:35 +02:00
Alex Sharov
23270eae58
tests: fix nil genesis panic (#7979) 2023-08-08 15:19:08 +06:00
Andrew Ashikhmin
d69b20bc4e
Support engine_forkchoiceUpdatedV3 with ParentBeaconBlockRoot (EIP-4788) (#7969)
Prerequisites: https://github.com/ledgerwatch/interfaces/pull/187 &
https://github.com/ledgerwatch/erigon-lib/pull/1069. Also implement
https://github.com/ethereum/execution-apis/pull/426.
2023-08-06 11:54:14 +02:00
Giulio rebuffo
0e4e36b142
Replaced old version of Engine API with newer version (#7972) 2023-08-05 23:33:10 +02:00
Giulio rebuffo
9730cc796a
Cleanup of consensus separation (#7971) 2023-08-05 02:26:53 +02:00
Giulio rebuffo
b09d90ee0e
Added in-memory flushing to execution.proto (#7968) 2023-08-04 14:42:35 +02:00
Andrew Ashikhmin
d014da4dc0
Replace --override.shanghaiTime flag with --override.cancun (#7964)
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/1067
2023-08-03 16:05:35 +02:00
Giulio rebuffo
29935a65f4
Fixed hive tests in consensus separation (#7961) 2023-08-03 03:39:47 +02:00
오웬
3c62adcfe0
fix: prevent index out of range during call trace (#7951)
Previously the call tracer failed on this Ethereum transaction
[0xecfc724f75e39fa9e882ec44a4d0985e099b7a3ac9415466196128770def5bc2](https://etherscan.io/tx/0xecfc724f75e39fa9e882ec44a4d0985e099b7a3ac9415466196128770def5bc2)

Querying with this request
```shell
curl  http://localhost:8545  -X POST  -H "Content-Type: application/json"  --data '{"method":"debug_traceTransaction","params":["0xecfc724f75e39fa9e882ec44a4d0985e099b7a3ac9415466196128770def5bc2", {"tracer": "callTracer",  "tracerConfig": {"withLog": true}}],"id":1,"jsonrpc":"2.0"}
```

Resulted in this response
```json
{"jsonrpc":"2.0","id":1,"result":null,"error":{"code":-32000,"message":"method handler crashed"}}
```

With this error message in the console:
`RPC method debug_traceTransaction crashed: runtime error: index out of
range [-1]
[service.go:217 panic.go:884 panic.go:113 call.go:177 interpreter.go:234
interpreter.go:262 evm.go:55 evm.go:260 evm.go:303 instructions.go:800
interpreter.go:304 evm.go:55 evm.go:260 evm.go:283
state_transition.go:382 state_transition.go:186 tracing.go:206
tracing.go:263 value.go:584 value.go:368 service.go:222 handler.go:511
handler.go:444 handler.go:392 handler.go:223 handler.go:316
asm_arm64.s:1172]`
2023-08-02 08:09:28 +07:00