Commit Graph

4938 Commits

Author SHA1 Message Date
Mark Holt
ca3ad096e1
Bor fix rpcdeamon engine initialization (#8390)
This fixes 2 related issues:

* Now that the bor consensus engine is required for queries it can't be
created based on the pretense of a db directory, but must be based on
chain config read from the db. Using the DB presence causes Bor to get
instantiated for non bor chains which breaks.
* At the moment eth_calls on a remote daemon don't check Bor headers
prior to calling the EVM code as it was just using a fake ETHash
instance - which performs ETH header validation only.

The current version is mostly working but needs adapting to perform lazy
initialization of the engine.
2023-10-06 11:58:08 +01:00
Giulio rebuffo
0eed8e35dd
fixed --internalcl flag for gnosis (#8387)
little thing i forgot to fix up.
2023-10-05 22:48:06 +02:00
Anshal Shukla
da4e1c1ba4
fix bor.milestone flag (#8384)
- Made milestones enabled by default, since it has now been tested on
devnets
- Small bug fix which fixes nil panic while minning
2023-10-05 17:55:32 +01:00
Giulio rebuffo
2294c8c66c
EthereumExecutionService in MockSentry (#8373)
Now we use the ethereum execution service directly:

* Changed sig of InsertChain
* Use of the service in case of PoS
2023-10-05 18:30:19 +02:00
Alex Sharov
7dd678896a
downloader: move from snapshots/db to snapshots/downloader (#8375) 2023-10-05 14:25:00 +07:00
Alex Sharov
c293883ec0
evm: no interface (#8376)
after removal of tevm experiment - we left interfaces everywhere 
removing it for performance and for geth compatibility
2023-10-05 12:23:08 +07:00
canepat
47690db676
Block execution using embedded Silkworm (#8353)
This introduces _experimental_ block execution run by embedded Silkworm
API library:

- new command-line option `silkworm.path` to enable the feature by
specifying the path to the Silkworm library
- the Silkworm API shared library is dynamically loaded on-demand
- currently requires to build Silkworm library on the target machine
- available only on Linux at the moment: macOS has issue with [stack
size](https://github.com/golang/go/issues/28024) and Windows would
require [TDM-GCC-64](https://jmeubank.github.io/tdm-gcc/), both need
dedicated effort for an assessment
2023-10-05 09:27:37 +07:00
Alex Sharov
0165d00ee6
downloader: don't drop torrents after download (performance problem there solved) (#8358) 2023-10-04 12:34:45 +07:00
Alex Sharov
fa3b8c23b2
Downloader: step towards more complex datadir (#8286)
migration included - no manual actions required
2023-10-04 11:01:02 +07:00
Alex Sharov
ce47ad30e2
downloader: progress print better (#8350) 2023-10-04 09:57:37 +07:00
Alex Sharov
8dda88d866
downloader: check too big and too small content (#8349) 2023-10-03 14:24:08 +07:00
Alex Sharov
dbdb486dd3
downloader: download .torrent files from webseeds provider (#8346) 2023-10-03 11:21:41 +07:00
Mark Holt
3d6d2a7c25
Added fix to allow getroothash to work with no api running (#8342)
Whitelisting calculation of the roothash should not be dependent on the
bor api running. This will not always be the case, for example when
erigon is configured with a separate rpc deamon.

To fix this the calculation has been moved to Bor.

Additionally the redundant Bor API code has been removed as this is not
called by any code and the functionality looks to have migrated to the
turbo/jsonrpc package.
2023-10-02 18:55:31 +01:00
Mark Holt
0bdca6c457
Metrics label fixes (#8339)
Fixes for label discrepancies in collector for summaries etc which have
a template which includes a quantile.

Initial native Prometheus client implementation of metrics - which is
currently turned off except for local testing and interface exports.
2023-10-02 17:19:02 +01:00
Giulio rebuffo
10746eb376
Added BLS_TO_EXECUTION_CHANGES pool (#8332)
Adds BLS_TO_EXECUTION_CHANGES
2023-10-01 17:16:55 +02:00
lupin012
ecba93837a
rpctest: Add rpctest for eth_getBlockByHash() (#8323) 2023-09-30 13:06:14 +07:00
Giulio rebuffo
72ba18bd36
Beacon: Added basic operations pool (#8309)
Added operation pools for beacon chain. operations are the equivalent of
txs for eth2

Added operation pools for:

* Attester Slashings
* Proposer Slashings
* VoluntaryExits
* BLSExecutionToChange
* Postponed to later: Attestations (or maybe not)
2023-09-29 23:42:07 +02:00
Jason Yellick
5654ba07c9
Upgrade libp2p (enables go 1.21 support) (#8288)
Closes #8078 

This change is primarily intended to support go 1.21, but as a
side-effect requires updating libp2p, which in turn triggers an update
of golang.org/x/exp which creates quite a bit of (simple) churn in the
slice sorting.

This change introduces a new `cmp.Compare` function which can be used to
return an integer satisfying the compare interface for slice sorting.

In order to continue to support mplex for libp2p, the change references
github.com/libp2p/go-libp2p-mplex instead. Please see the PR at
https://github.com/libp2p/go-libp2p/pull/2498 for the official usptream
comment that indicates official support for mplex being moved to this
location.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-09-29 22:11:13 +02:00
Alex Sharov
d0366ae34e
snapshots: remove concept of separated hist .toml file (#8326) 2023-09-29 11:30:02 +07:00
sudeep
d39f03783f
fix t8n call to ExecuteBlockEphemerally by passing chainReader (#8298)
fixing crash on consensus/merge/merge.go#Initialize function, on this
line.

```
chain.Config().IsCancun(header.Time)
```
2023-09-29 09:04:13 +07:00
Alex Sharov
6cc2bd5751
downloader: non-readonly open db (so it can auto-recover if need) (#8312) 2023-09-28 11:38:29 +07:00
Mark Holt
f99f326363
Bor fix frozen snapshot load (#8305)
This is a fix for at least one cause of this isssue:
https://github.com/ledgerwatch/erigon/issues/8212.

It happens after the end of the snapshot load, because the snapshot
processing which was introduced a couple of month ago does not deal with
validation of the headers at the start of the start of the chain.

I have also added a fix to the persistence so that the last snapshot is
recorded so that subsequent runs are not forced to process the whole
snapshot run from start.

The relationship between this an memory usage is that the fact that
headers are not processed leads to a queue of pending headers with size
of around 5GB. I have not changed any header parameters - so likely a
prolonged stop to header processing will result in a similar level of
memory growth.
2023-09-27 13:45:09 +01:00
Giulio rebuffo
d62ef03cad
Added states/fork and states/root to beacon API (#8293)
Added /eth/v1/beacon/states/{state_id}/root and
/eth/v1/beacon/states/{state_id}/fork
2023-09-27 11:15:51 +02:00
Alex Sharov
b3f4520449
increase loginterval of txpool and p2p logs (#8291) 2023-09-26 13:29:19 +07:00
Mark Holt
f794438335
Diag session routing (#8232)
Code to support react based UI for diagnostics:

* pprof, prometheus and diagnistics rationalized to use a single router
(i.e. they can all run in the same port)
* support_cmd updated to support node routing (was only first node)
* Multi content support in router tunnel (application/octet-stream &
appliaction/json)
* Routing requests changed from using http forms to rest + query params
* REST query requests can now be made against erigon base port and
diagnostics with the same url format/params

---------

Co-authored-by: dvovk <vovk.dimon@gmail.com>
Co-authored-by: Mark Holt <mark@disributed.vision>
2023-09-25 16:24:17 +01:00
Giulio rebuffo
a95914df26
Added blocks API and attestations API (#8279) 2023-09-24 01:44:26 +02:00
a
de69dce19a
copy updates (#8275) 2023-09-22 21:09:26 +02:00
Srdjan
dedf04caaa
Pre-allocate method metric labels (#8243)
Closes #8085
2023-09-21 09:38:47 +02:00
Somnath Banerjee
f51d9b61a0
Txpool 4844 upgrades Part 2 (#8213)
Some peer-review changes from the last related PR. 
Addition of a flag for BlobSlots - for max allowed blobs per account in
txpool.
Use BlobFee from the block to validate txs in the pool.

See also https://github.com/ledgerwatch/erigon-lib/pull/1125
2023-09-20 17:29:30 +05:30
Michelangelo Riccobene
2146ff016c
Add util to test native recsplit index creation (#8238)
In order to assess Silkworm recsplit index creation capability we need
some utils to trigger native (go) index creation
2023-09-20 08:44:51 +07:00
Mark Holt
3b45f53f3d
Milestone stage processing (#8187)
This is the second part of the bor milestone release it contains the
following changes:

* Initialize services
* This is a change from the initial pull request I have moved all of the
initialization to the bor engine. To facilitate this I have just passed
in the heimdall client interface, rather than the whole engine
* Stage processing 
* This is also a change from the original PR - the code is contained in
the bor heimdall stage rather than in headers - the effect should be the
same, but this needs testing

---------

Co-authored-by: Mark Holt <mark@disributed.vision>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
2023-09-18 18:05:33 +01:00
Giulio rebuffo
c1961eff2b
Added the option to directly write blocks on io.Writer. (#8215) 2023-09-17 13:15:06 +02:00
Alex Sharov
eb747c042a
Downloader: add tests to protect from re-downloading (#8191) 2023-09-15 13:46:59 +07:00
Giulio rebuffo
c552f993ac
Removed Partial blocks and reduced pruneDepth to 1000 (#8198) 2023-09-14 22:14:05 +02:00
Giulio rebuffo
2f8687fe08
Smart SQL transactions (#8185) 2023-09-14 02:56:29 +02:00
Mark Holt
33d8c08c1c
Get vote on hash (#8172)
This is the initial merge for polygon milestones it implements an rpc
call used by heimdall but does not directly impact any chain processing
2023-09-13 11:49:49 +01:00
Alex Sharov
93fbda14a5
downloader: support --chain parameter (#8184) 2023-09-13 15:21:43 +07:00
Alex Sharov
2158b4916d
Downloader: correct logging when create .torrent files (#8182) 2023-09-13 11:49:45 +07:00
Alex Sharov
3cea1b9b9e
torrent: add --webseeds cli arg (#8176) 2023-09-12 12:18:47 +07:00
Somnath Banerjee
a699f64761
Txpool upgrades for EIP-4844 Blob Transactions (#8004)
See https://github.com/ledgerwatch/erigon-lib/pull/1075
2023-09-11 09:38:58 +07:00
Giulio rebuffo
22bd173ab3
Updating Libs (#8147)
* BLS
* Sqlite
2023-09-06 23:20:26 +02:00
Alex Sharov
d60940d7db
Avoid leaking more popped items (#8145) 2023-09-06 15:47:06 +07:00
Park Changwan
ef84972e7c
Add addPeer RPC (#7804)
This PR mirrors https://github.com/testinprod-io/op-erigon/pull/54.

Actual implementation for `admin_addPeer` method.
RPC Spec: Refer to
https://geth.ethereum.org/docs/interacting-with-geth/rpc/ns-admin.

> The addPeer administrative method requests adding a new remote node to
the list of tracked static nodes. The node will try to maintain
connectivity to these nodes at all times, reconnecting every once in a
while if the remote connection goes down.

Requires https://github.com/ledgerwatch/erigon-lib/pull/1033/

After https://github.com/ledgerwatch/erigon-lib/pull/1033 is merged,
will update erigon-lib version, removing replace at go.mod.

Note that even if RPC response returns `true`, it does not guarantee
that RLPx protocol is established between peers. It just adds node
entrypoint to its static peer list, and periodically tries and tests
connections.

## Testing

This RPC needs integration testing, so I made some scenario.

Use below command for testing:

Spin up two dev nodes which p2p enabled:

Start Node 1: RPC running at port 8545:
```sh
./build/bin/erigon --datadir=dev --chain=dev --port=30303 --http.port=8545 --authrpc.port=8551 --torrent.port=42069  --no-downloader --nodiscover --private.api.addr=127.0.0.1:9090 --http --ws --http.api=admin --p2p.allowed-ports=30306,30307,30308  --authrpc.jwtsecret=/tmp/jwt1 --p2p.protocol=67,68  --log.console.verbosity=5
```

Start Node 2: RPC running at port 8546:
```sh
./build/bin/erigon --datadir=dev2 --chain=dev --port=30304 --http.port=8546 --authrpc.port=8552 --torrent.port=42068 --no-downloader --nodiscover --private.api.addr=127.0.0.1:9091 --http --ws --http.api=admin --p2p.allowed-ports=30309,30310,30311  --authrpc.jwtsecret=/tmp/jwt2  --p2p.protocol=67,68  --log.console.verbosity=5
```

Get nodeInfo of node 1 using `admin_nodeInfo` RPC:
```sh
curl --location 'localhost:8545/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_nodeInfo",
	"params":[],
	"id":1
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "id": "b75e0c4d2113b6f144ea8fd356a8f90e612a2a5f48a13c78d7e0e176e5724eb2",
        "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
        "enode": "enode://05ab575d947f2d73065ea0f795dc2d96ed0ad603f3e730ab90dc881122d552c9f59ffcb148fe50546bec8b319daeb3c22ec02e7d12a7c4f2ac4cd26456a04a7c@127.0.0.1:30303?discport=0",
   ...
```

Get nodeInfo of node 2 using `admin_nodeInfo` RPC:
```sh
curl --location 'localhost:8546/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_nodeInfo",
	"params":[],
	"id":2
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 2,
    "result": {
        "id": "32d721e4d75219b021d7f83235f1f1eb8b705d6f85e634bccde564b8f7f94d78",
        "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
        "enode": "enode://1abb8579647779e13b7f68d18f9c776cbd29281841c7f950e9cf9afa996e31120a6f481cea8e90e0f42a0eb1aa00aeafee81c4bae6c31aa16810b795c6d6e069@127.0.0.1:30304?discport=0",
   ...
```

Call `admin_addPeer` RPC to node 2:
```sh
curl --location 'localhost:8546/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_addPeer",
	"params":["enode://05ab575d947f2d73065ea0f795dc2d96ed0ad603f3e730ab90dc881122d552c9f59ffcb148fe50546bec8b319daeb3c22ec02e7d12a7c4f2ac4cd26456a04a7c@127.0.0.1:30303"],
	"id":2
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 2,
    "result": true
}
```


Check peer info of node 1 using `admin_peers` RPC:
```sh
curl --location 'localhost:8545/' \
--header 'Content-Type: application/json' \
--data '{
	"jsonrpc":"2.0",
	"method":"admin_peers",
	"params":[],
	"id":1
}' 
```
Example response:
```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
            "enode": "enode://1abb8579647779e13b7f68d18f9c776cbd29281841c7f950e9cf9afa996e31120a6f481cea8e90e0f42a0eb1aa00aeafee81c4bae6c31aa16810b795c6d6e069@127.0.0.1:55426",
            "id": "32d721e4d75219b021d7f83235f1f1eb8b705d6f85e634bccde564b8f7f94d78",
            "name": "erigon/v2.47.0-dev-5d86cdb5/darwin-arm64/go1.19.6",
            "caps": [
                "eth/66",
                "eth/67"
            ],
            "network": {
                "localAddress": "127.0.0.1:30303",
                "remoteAddress": "127.0.0.1:55426",
                "inbound": true,
                "trusted": false,
                "static": false
            },
            "protocols": null
        }
    ]
}
```

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-09-06 15:31:02 +07:00
Alex Sharov
ff54df073e
etl: do sort and file flush in another goroutine (#7915) 2023-09-06 14:51:23 +07:00
Mark Holt
8ea0096d56
moved metrics sub packages types to metrics (#8119)
This is a non functional change which consolidates the various packages
under metrics into the top level package now that the dead code is
removed.

It is a precursor to the removal of Victoria metrics after which all
erigon metrics code will be contained in this single package.
2023-09-03 08:09:27 +07:00
Andrew Ashikhmin
9895b66b3d
Fix applyBlock in commands/state_domains (#8115)
Fix an issue introduced by PR #8095
2023-09-01 17:33:05 +01:00
alex.sharov
73616b7fd3 remove: grafana datasource id 2023-09-01 15:37:14 +07:00
alex.sharov
a72b7a9350 fix devel build 2023-09-01 14:31:18 +07:00
alex.sharov
e53df00849 split grafana board to: user-facing and dev-facing (internals) 2023-09-01 12:09:30 +07:00
battlmonstr
340b9811b0
p2p: refactor peer errors to propagate with a DiscReason (#8089)
Improve p2p error handling to propagate errors
from the origin up the call chain the Server peer removal code
using a new PeerError type containing a DiscReason and a more detailed
description.

The origin can be tracked down using PeerErrorCode (code) and DiscReason
(reason)
which looks like this in the log:

> [TRACE] [08-28|16:33:40.205] Removing p2p peer peercount=0
url=enode://d399f4b...@1.2.3.4:30303 duration=6.901ms
err="PeerError(code=remote disconnect reason, reason=too many peers,
err=<nil>, message=Peer.run got a remote DiscReason)"
2023-08-31 16:45:23 +01:00