Commit Graph

5084 Commits

Author SHA1 Message Date
Alex Sharov
d1d348211f
add flag --force.partial.commit: to workaround problem "start from backup takes long time and can't save partial progress" (#8090) 2023-08-30 08:49:16 +07:00
Giulio rebuffo
901edf9181
Accessors to Caplin indexed database (#8093) 2023-08-29 16:47:16 +02:00
Giulio rebuffo
93cdbae2e5
Caplin<->Erigon: Added Range methods (#8087) 2023-08-29 02:37:30 +02:00
Alex Sharov
42909ffb1b
grafana: hide low-level metrics. move all high-level metrics on top (#8088) 2023-08-27 19:12:27 +07:00
Giulio rebuffo
8291aa2497
Caplin: Storing Execution Headers in Caplin (#8080)
This PR stores Execution Headers in Caplin when running without storing
fullblocks
2023-08-26 00:41:57 +02:00
Giulio rebuffo
41185e2518
Added Online pruning to Caplin (#8058)
Basically, pruning is specified by the user, by default, 1 million (in
the PR set to 100 for pruning purposes). the pruning for the database is
stored inside the db
2023-08-25 14:58:35 +02:00
Mark Holt
c51573f333
Bor eth event flow (#8068)
Implemented polygon->eth flow
2023-08-25 12:19:39 +01:00
battlmonstr
32ca0e5ab1
sync: revert flawed dropUselessPeers logic and alleviate its issues (#8062)
The current logic is flawed, because it drops all peers that are less
synced.
It is valid to return empty responses by the eth spec.
A proper logic should penalize from the context of the sync process,
where enough "reputation" data is collected about a peer.

In order to be able to connect to erigon 2.48 peers that have
--sentry.drop-useless-peers enabled,
this adds a check to not reply with an empty headers list.
If we reply with an empty list, we're going to be considered useless and
kicked.
Once enough of erigon nodes are updated in the network past this commit,
this check should be removed,
because it is totally acceptable to return an empty list by the eth
spec.
2023-08-25 11:42:54 +02:00
battlmonstr
bb2c2adbb6
p2p: fix RLPx disconnect message decoding (#8056)
The disconnect message could either be a plain integer, or a list with
one integer element. We were encoding it as a plain integer, but
decoding as a list. Change this to be able to decode any format.
2023-08-24 13:49:19 +02:00
Alex Sharov
2b6c21fddb
move mdbx to new org (#8061) 2023-08-24 18:00:24 +07:00
Andrew Ashikhmin
ece8010d0d
Remove unused transaction and node packets from eth/protocols (#8057)
Transactions are served by erigon-lib's txpool and we don't serve
NodeData.
2023-08-24 06:50:04 +02:00
battlmonstr
2e29ff33e1
bor: BroadcastNewBlock to all peers from validator nodes (#8030)
Currently PropagateNewBlockHashes and BroadcastNewBlock
selects a subset of all sentries by taking a `Sqrt(len(sentries))`,
and then for each sentry SendMessageToRandomPeers
selects a subset of its peers by taking `Sqrt(len(peerInfos))`.

This behaviour limits the broadcast scope with a lot of peers, e.g. 100
becomes 10,
but is not great with very few peers, or if the message is very
important
to broadcast to everyone, which is the case of bor validator/proposer
nodes.

* send to all sentries in both BroadcastNewBlock and PropagateNewBlockHashes
* remove peerCountConstrained sqrt logic in SendMessageToRandomPeers
* add maxPeers provider func as a parameter to MultiClient
* default it to 10 for eth and 0 (unlimited) for bor validators

---------

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-08-23 14:28:39 +02:00
alex.sharov
02b7c28a94 bor: integration cmd support 2023-08-23 12:07:36 +07:00
battlmonstr
d2d261e560
cli: die if a config file is invalid (#8025) 2023-08-22 10:50:31 +02:00
battlmonstr
5b843fbddc
sentry: do not process empty responses (#8029)
if dropUselessPeers = false,
skip processing empty responses same as if it was true

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-08-22 10:16:12 +02:00
battlmonstr
61364e8a01
sentry: fix eth handshake sequence (#8031)
problem: it was possible to call startSync
and start sending messages before our Status is sent

solution: wait for the sender goroutine to finish
before calling startSync

refactor handShake parameters to not require peerID and a startSync
callback

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-08-22 10:15:35 +02:00
Giulio rebuffo
bd81e15981
Caplin: Implemented SQL beacon indexer (#8043) 2023-08-22 01:24:26 +02:00
Andrew Ashikhmin
6bc0ca9e85
Correctly compute fork id when timestamp fork is activated in genesis (#8046)
See https://github.com/ethereum/go-ethereum/pull/27895
2023-08-21 15:35:13 +02:00
Alex Sharov
863a706858
caplin: use tmpdir inside datadir (#8041) 2023-08-20 15:26:34 +02:00
ledgerwatch
6b6c0caad0
Snapshots of Bor events (#7901)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp-2.home>
2023-08-18 17:10:35 +01:00
Giulio rebuffo
38db2ed2e7
[Caplin Phase 2]: Initial working implementation (#8033)
caplin just using execution.proto
2023-08-18 15:43:22 +02:00
Giulio rebuffo
cab6d30322
Caplin's Persistence data format (#8028) 2023-08-16 23:03:25 +02:00
Giulio rebuffo
c8db11f150
Sentinel: improvements to block download (#8023) 2023-08-16 17:02:30 +02:00
a
521f0df55b
Historical block downloader (#8016) 2023-08-16 04:32:40 +02:00
Giulio rebuffo
de5706dbc6
Direct sentinel instead of over the network sentinel (#8006) 2023-08-12 01:54:45 +02:00
Giulio rebuffo
7c0d6e15bb
Added gnosis fields and removed body gap. (#7993) 2023-08-11 23:07:36 +02:00
Alex Sharov
c464d84334
tmpdb: move to tmpdir (#8003) 2023-08-12 01:06:15 +06:00
Andrew Ashikhmin
f854a73185
Add enough blob gas for block building (EIP-4844) (#8001) 2023-08-11 16:09:03 +02:00
Andrew Ashikhmin
03927d3e27
Call InitializeBlockExecution in SpawnMiningExecStage (EIP-4788) (#7999)
This fixes the trie state root issue that was occurring in the Hive
tests for Cancun.
2023-08-11 14:04:53 +02:00
Giulio rebuffo
e3a59ed902
Better caplin logging and logic (#7992) 2023-08-10 22:34:58 +02:00
a
db5b348673
Caplin block persistence (#7941)
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-08-09 01:21:19 +02:00
Andrew Ashikhmin
d69b20bc4e
Support engine_forkchoiceUpdatedV3 with ParentBeaconBlockRoot (EIP-4788) (#7969)
Prerequisites: https://github.com/ledgerwatch/interfaces/pull/187 &
https://github.com/ledgerwatch/erigon-lib/pull/1069. Also implement
https://github.com/ethereum/execution-apis/pull/426.
2023-08-06 11:54:14 +02:00
Giulio rebuffo
0e4e36b142
Replaced old version of Engine API with newer version (#7972) 2023-08-05 23:33:10 +02:00
Andrew Ashikhmin
d014da4dc0
Replace --override.shanghaiTime flag with --override.cancun (#7964)
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/1067
2023-08-03 16:05:35 +02:00
racytech
2c2ccb6e27
dencun-devnet: --trusted-setup-file flag support (#7963) 2023-08-03 14:36:47 +02:00
Andrew Ashikhmin
bd9e8b9b56
Fix eth_getTransactionByHash for EIP-4844 transactions (#7960)
Also pick https://github.com/ledgerwatch/erigon-lib/pull/1063 and
https://github.com/ledgerwatch/erigon-lib/pull/1065
2023-08-02 18:25:15 +02:00
Giulio rebuffo
1220ae659e
Most hive tests fixed in --experimental.modular (#7950)
Broken: 25/109
2023-08-01 02:08:15 +02:00
alex.sharov
b50f427641 save 2023-07-29 12:08:38 +07:00
Mark Holt
cc74e74a60
Devnet contract utils (#7928)
This request is extending the devnet functionality to more fully handle
contract processing by adding support for the following calls:
 * trace_call,
* trace_transaction
* debug_accountAt,
* eth_getCode
* eth_estimateGas
* eth_gasPrice

It also contains an initial rationalization of the devnet subscription
code to use the erigon client code directly rather than using its own
intermediate subscription management.

This is used to implement a general purpose block waiter - which can be
used in any scenario step - rather than being specific to transaction
processing.

This pull also contains an end to end tested sync processor for bor and
associated support services:
* Heimdall (supports sync event transfer)
* Faucet - allows the creation and funding of arbitary test specific
accounts (cross chain)

Notes and Caveats:
 
* Code generation for contracts requires `--evm-version paris`. For
chains which don't support push0 for solc over 0.8.19
* The bor log processing post the application of sync events causes a
panic - this will be the subject of a seperate smaller push as it is not
devnet specific
* The bor code seems to make repeated calls for the same sync events and
also reverts requests - this needs further investigation. This is the
behaviour of the current implementation and may be required - although
it does seem to generate repeat processing - which could be avoided.
2023-07-28 14:03:32 +01:00
Andrew Ashikhmin
7d35c6b737
EIP-4844: Rename "data gas" to "blob gas" (#7937)
See https://github.com/ethereum/EIPs/pull/7354 &
https://github.com/ethereum/consensus-specs/pull/3461. Prerequisite:
https://github.com/ledgerwatch/erigon-lib/pull/1058
2023-07-28 12:12:05 +02:00
Willian Mitsuda
d3f8b5861c
Max cap search results for ots API by default + cli flag to override it (#7924)
Otterscan API search methods allow the user to inform the page size.

This PR adds an internal max (default == 25 results) to cap the page
size, regardless of what the user asks.

It also adds a `--ots.search.max.pagesize` CLI args to override this max
(either in erigon and rpcdaemon binaries).
2023-07-28 08:29:17 +07:00
gladcow
7e3c6ea2f8
Adds clear_bad_blocks command (#7934)
Adds `clear_bad_blocks` command to integration tool. This command allows
to re-process blocks that were erroneously marked as bad.

Command just clears `BadHeaderNumber` table. It can be safer in some
cases than
```
./integration state_stages —unwind=<some_number>
./integration stage_headers —unwind=<some_number>
```
and can be used in the cases like this one
https://github.com/ledgerwatch/erigon/issues/7892%20

Command syntax:
```
./integration clear_bad_blocks --datadir=<datadir>
```
2023-07-28 08:28:47 +07:00
Giulio rebuffo
b4ecd7f524
Consensus Separation: Separated GetPayload using an execution service (#7933)
Miracoulously, hive tests pass first try. YIPPIE.

Also for the future, I added `--experimental.modular` which enables a
secondary engine API for consensus separation.

Now block building is responsibility of the execution module.
2023-07-27 14:37:49 +02:00
a
60a200fd46
[caplin] basic persistence layer (#7918) 2023-07-25 22:24:52 +02:00
Giulio rebuffo
9d3551f8cf
prototype of ValidataChain (#7921) 2023-07-23 19:10:24 +02:00
Giulio rebuffo
837ffb5b7e
Refactored ForkValidator to be Database-only (#7920) 2023-07-23 14:51:26 +02:00
Mark Holt
751f62615d
Devnet sync events (#7911)
This request implements an end to end Polygon state sync in the devnet.

It does this by deploying smart contracts ont the L2 & L2 chain which
follow the polygon fx portal model with security checks removed to
simplify the code. The sync events generated are routed through a local
mock heimdal - to avoid the consensus process for testing purposes.

The commit also includes support code to help the delivery of additional
contract based scenratios.
2023-07-20 23:10:18 +01:00
a
3ab373787e
[caplin] extracting beacon state interface (#7910)
we need to extract this interface from the struct. 

i need to also break down the interface more, to better show what parts
the caching is used, move some functions from the cache state to the
underlying.


don't merge
2023-07-20 00:20:33 +02:00
Mark Holt
529d359ca6
Bor span testing (#7897)
An update to the devnet to introduce a local heimdall to facilitate
multiple validators without the need for an external process, and hence
validator registration/staking etc.

In this initial release only span generation is supported.  

It has the following changes:

* Introduction of a local grpc heimdall interface
* Allocation of accounts via a devnet account generator ()
* Introduction on 'Services' for the network config

"--chain bor-devnet --bor.localheimdall" will run a 2 validator network
with a local service
"--chain bor-devnet --bor.withoutheimdall" will sun a single validator
with no heimdall service as before

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-18 09:47:04 +01:00
a
19bc41198e
[caplin] conn gater (#7900)
conn gater


so this is what prysm did to address the issue
https://github.com/prysmaticlabs/prysm/pull/8648/files
2023-07-16 08:31:06 +02:00
Giulio rebuffo
23f5377614
updated interfaces (#7891) 2023-07-14 02:51:47 +02:00
Giulio rebuffo
b2442a7b41
Removed unused stages (Cumulative index + Translation) (#7884) 2023-07-13 16:55:48 +02:00
Mark Holt
bd9896bf4b
added bor tx indexing with tests (#7826)
This request implements the insertion of Bor ephemeral transactions into
snapshot indexes.

I does this by taking the block hash from the header index and passing
it to the transaction indexer to add an additional index entry per block
into the transaction hash -> block index.

The passed entries are currently contained in an in memory array which
is (32 * number of blocks / sprint size) bytes.

In addition to the functional code there is also an update to the
`dump_test.go` so that it runs `DumpBlocks` to exercise the indexing
code. To facilitate this the `InsertChain` method in `mock_sentry` has
been modified so that it can process >128 blocks.

The code in this request also includes additional bor/consensus code
with the following functions:

`CalculateSprint`
`CalculateSprintCount`

The first function is a modification of the code in erigon-lib so that
the sprints are numerically rather than lexically ordered. This code
should be migrated to erigon-lib and should have its sprint set
calculated once from its underlying map rather than this process being
repeated every calculation.

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Enrique Jose  Avila Asapche <eavilaasapche@gmail.com>
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-07-12 23:31:38 +01:00
gladcow
8809208605
Expand home dir from tilda for integration tool settings (#7877)
Fixes https://github.com/ledgerwatch/erigon/issues/7814, it can be
really confusing when `erigon` and `integration` process `~` in path in
different ways.

I can't fix it for `integration` the same way it is done for `erigon`,
because in main app it is done with `urfave/cli` package that is not
used in `integration`, so I've used other simplest way to do it.
2023-07-13 00:45:38 +07:00
Giulio rebuffo
bd63cb7c8c
Removed GRPC layer from Engine API (#7878) 2023-07-12 18:11:41 +02:00
Andrew Ashikhmin
af504f675a
Use EIP-4844 constants from erigon-lib (#7874)
Small refactoring after
https://github.com/ledgerwatch/erigon-lib/pull/1037
2023-07-11 17:53:00 +02:00
Giulio rebuffo
6272559fb7
Separated PendingBlock behaviour to be chain agnostic (#7859)
Instead of getting the pending block with latest payload id, we just
store the latest block built and serve it in remote backend
2023-07-10 19:22:03 +02:00
ledgerwatch
dbb6d96dc5
Add FrozenBlocks function to chainReaders (#7866)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-10 17:35:27 +01:00
ledgerwatch
a9c86ab887
[devnet] Always select first node, fix configuration mess up between nodes (#7863)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-10 11:37:55 +01:00
ledgerwatch
e6c642b568
[devnet] Rename mining to block production (#7862)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-09 15:13:12 +01:00
Alex Sharov
bf8ac9a38f
mdbx_to_mdbx: to use logger (#7860) 2023-07-09 08:04:20 +01:00
Giulio rebuffo
20b8e156db
moved cmd/rpcdaemon/commands -> turbo/jsonrpc (#7858) 2023-07-08 19:01:26 +02:00
Enrique Jose Avila Asapche
bff1e0e901
[Caplin] Added json rpc cl (#7836)
Allows for Caplin to be used with other ELs from aside of Erigon

---------

Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-07-08 16:42:30 +02:00
Giulio rebuffo
84f31c873e
Separation of Engine from Ethbackend (#7821)
This PR separates ENGINE from Ethbackend. It makes it so:

1) EthBackend not a god class
2) We can abstract away engine API so that we can make it CL-like and
enable Consensus-Execution driven design
3) Objective is Json-RPC -> Engine Consensus Module -> Execution module.
2023-07-06 18:09:52 +02:00
Alex Sharov
62b8daca21
integration stage_exec: add flag --no-commit (#7851) 2023-07-06 11:31:59 +07:00
Alex Sharov
69ae999059
log file unmap/close errors (#7819) 2023-07-06 11:12:26 +07:00
Alex Sharov
4f308148ed
Grafana board: up version, clean filling (#7844) 2023-07-05 10:43:31 +07:00
alex.sharov
499680c5d2 docs 2023-07-05 09:35:04 +07:00
Alex Sharov
a7b9850ff4
mdbx_to_mdbx: docs (#7841) 2023-07-04 09:45:36 +07:00
a
9fd19338f5
reduce log level (#7835)
some people are getting confused by this log. 

it is not really an error anyways.
2023-07-02 22:37:35 +01:00
Mark Holt
5935b11b24
Devnet macos startup and windows committed memory startup fixes (#7832)
The fixes here fix a couple of issues related to devnet start-up

1. macos threading and syscall error return where causing multi node
start to both not wait and fail
2. On windows creating DB's with the default 2 TB mapsize causes the os
to reserve about 4GB of committed memory per DB. This may not be used -
but is reserved by the OS - so a default bor node reserves around 10GB
of storage. Starting many nodes causes the OS page file to become
exhausted.

To fix this the consensus DB's now use the node's OpenDatabase function
rather than their own, which means that the consensus DB's take notice
of the config.MdbxDBSizeLimit.

This fix leaves one 4GB committed memory allocation in the TX pool which
needs its own MapSize setting.

---------

Co-authored-by: Alex Sharp <akhounov@gmail.com>
2023-07-02 22:37:23 +01:00
sonicWhale
e18db89ced
RPCTxFeeCap flag for rpc daemon (#7824) 2023-06-30 10:02:27 +07:00
Andrew Ashikhmin
1f7de0e5cf
Rename StageLoopStep to StageLoopIteration (#7820)
Rename for clarity to highlight that not only one stage is executed, but
the entire loop.
2023-06-29 17:53:23 +02:00
ledgerwatch
0a1229bd76
Fixed for the devnet runner (#7808)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-06-28 18:21:27 +01:00
Mark Holt
bcc2a4a2f8
Cleaned up error handling in network and node start-up (#7811)
The check in catches errors in the node start-up code and makes sure
that the network is stopped if any node fails to start cleanly, and
that5 it returns an error - so that any calling code can take
appropriate action.
2023-06-28 18:21:15 +01:00
Manav Darji
6f0a02ac42
[bor] Implement bor_getSnapshotProposerSequence RPC method (#7770)
Implements `bor_getSnapshotProposerSequence` RPC method which returns an
ordered set of validators (selected to mine) for a given block number.

```
> curl http://localhost:8545 -X POST --data '{"jsonrpc":"2.0","method":"bor_getSnapshotProposerSequence","params":["0x235A310"],"id":1}' -H "Content-Type: application/json"

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "Signers": [
      {
        "Signer": "0xcfef2a3dc244ef7d0fb93c45e762d671445c4569",
        "Difficulty": 5
      },
      {
        "Signer": "0x3a22c8bc68e98b0faf40f349dd2b2890fae01484",
        "Difficulty": 4
      },
      {
        "Signer": "0xbe188d6641e8b680743a4815dfa0f6208038960f",
        "Difficulty": 3
      },
      {
        "Signer": "0xc26880a0af2ea0c7e8130e6ec47af756465452e8",
        "Difficulty": 2
      },
      {
        "Signer": "0xc275dc8be39f50d12f66b6a63629c39da5bae5bd",
        "Difficulty": 1
      }
    ],
    "Diff": 5,
    "Author": "0xcfef2a3dc244ef7d0fb93c45e762d671445c4569"
  }
}
```
2023-06-27 22:15:45 +07:00
Alex Sharov
1a1ca49ffa
change --db.pagesize default to 8KB (#7798)
reasons:
- mainnet: even nodes with small FreeList - still have millions of pages
there `GC: 46446830 5.8%`. Probability of getting into state where space
re-use will be slower than free-list grow is > 0% (we now using db
version which limiting freelist-overhead, but increasing such
probability)
- polygon: size is > 8Tb
- hardware slowly moving towards bigger pageSizes (because for
OS/Hardware) maintenance of pages metadata is also not free (metadata,
lists, LRU, etc...). Macbook's default pagesize now is 16Kb. Network
disks in cloud are also likely working with 16Kb pages.

pros:
- less db fragmentation (better FS-level compression)
- less overflow pages in DB (which also reducing free-list overhead)
- smaller free-list 
- bigger key-size-limit 
- no 8Tb db size limit
- can setup FS - to also use bigger pagesize - it will reduce FS
overhead also
- reducing amount of page-faults during batch-reads (if FS pagesize
match)
- less write syscalls during commit (when WriteMap disabled)

cons:
- ~10% more IO: because of more RAM waste and just because need
read/write bigger pages (not all updates are co-located).
2023-06-26 19:44:25 +07:00
Andrew Ashikhmin
a24eae8d6c
EIP-4844: Handle data gas in txpool (#7779)
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/1029
2023-06-23 11:10:23 +02:00
Alex Sharov
ded166d73d
e3: release chain maker changes (#7787) 2023-06-23 10:07:42 +07:00
Alex Sharov
7bad6d3fd9
erigon_info metric (#7769) 2023-06-21 11:13:50 +07:00
Mark Holt
051cad0fdd
Devnet diagnostics (#7762)
Added support tunnel to the devnet cmd. In order to get this to run I
made the following changes:

* Create a public function 
* Added non root logging

I have also added commentary to the readme to explain the additional
command line arguments needed to integrate with diagnostics. In summary,
if you set the --diagnostics.url the devenet will wait for diagnostic
requests rather than exiting

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-06-20 10:11:55 +01:00
Somnath Banerjee
71231140a5
Block Gas limit contract integration for Chiado (#7727)
Changes summary:
- Continue with the gasLimit check skip in ``verifyHeader`` of
``merge.go`` for unless pre-merge block and blockGasLimitContract
present
- Refactor ``aura.go`` a bit
- Have ``sysCall`` method customized to be able to call state (contract)
at a parent (or any other) header state
2023-06-19 22:06:51 +05:30
Alex Sharov
eae2d9a79c
Disable ci tests on another servers: step 2 (#7752) 2023-06-17 10:27:58 +07:00
Alex Sharov
d3c3be9c91
e2: optimize tests speed (#7738) 2023-06-15 16:09:11 +07:00
Andrew Ashikhmin
252e57b76f
Base BlobTx on DynamicFeeTransaction (#7736)
Continuation of PR #7715
2023-06-15 10:57:27 +02:00
Alex Sharov
e5023775aa
Enforce blockReader interface (#7737)
- breaks dependency from staged_sync to package with block_reader
implementation
- breaks dependency from snap_sync to package with block_reader
implementation
- breaks dependency from mining to txpool implementation
2023-06-15 13:11:51 +07:00
Mark Holt
f110102023
Devnet scenarios (#7723)
This is an update to the devnet code which introduces the concept of
configurable scenarios. This replaces the previous hard coded execution
function.

The intention is that now both the network and the operations to run on
the network can be described in a data structure which is configurable
and composable.

The operating model is to create a network and then ask it to run
scenarios:

```go
network.Run(
		runCtx,
		scenarios.Scenario{
			Name: "all",
			Steps: []*scenarios.Step{
				&scenarios.Step{Text: "InitSubscriptions", Args: []any{[]requests.SubMethod{requests.Methods.ETHNewHeads}}},
				&scenarios.Step{Text: "PingErigonRpc"},
				&scenarios.Step{Text: "CheckTxPoolContent", Args: []any{0, 0, 0}},
				&scenarios.Step{Text: "SendTxWithDynamicFee", Args: []any{recipientAddress, services.DevAddress, sendValue}},
				&scenarios.Step{Text: "AwaitBlocks", Args: []any{2 * time.Second}},
			},
		})
```
The steps here refer to step handlers which can be defined as follows:

```go
func init() {
	scenarios.MustRegisterStepHandlers(
		scenarios.StepHandler(GetBalance),
	)
}

func GetBalance(ctx context.Context, addr string, blockNum requests.BlockNumber, checkBal uint64) {
...
```
This commit is an initial implementation of the scenario running - which
is working, but will need to be enhanced to make it more usable &
developable.

The current version of the code is working and has been tested with the
dev network, and bor withoutheimdall. There is a multi miner bor
heimdall configuration but this is yet to be tested.

Note that by default the scenario runner picks nodes at random on the
network to send transactions to. this causes the dev network to run very
slowly as it seems to take a long time to include transactions where the
nonce is incremented across nodes. It seems to take a long time for the
nonce to catch up in the transaction pool processing. This is yet to be
investigated.
2023-06-14 12:35:22 +01:00
Giulio rebuffo
4f812efbc8
Better version of libp2p where losing peers bug fixed (#7726)
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-06-14 13:14:52 +02:00
Andrew Ashikhmin
862d7b6bc6
Move ChainID from CommonTx to DynamicFeeTransaction (#7732)
For legacy transactions ChainID is optional (missing in
pre-[EIP155](https://eips.ethereum.org/EIPS/eip-155) transactions) and
is derived from `V` anyway.

Also, cherry pick https://github.com/ethereum/go-ethereum/pull/27452.
2023-06-14 11:44:05 +02:00
Alex Sharov
1b14785d79
ws unfreeze (#7724)
attempt to address next issue: 

> when I'm having a lot of websocket connections the node is freezing
and then it needs like 10 mins to sync. Then if I keep pushing requests
it falls out of sync all the time
2023-06-14 16:25:05 +07:00
racytech
8b312d54e8
eip-4844: desszing blobtx: initial commit (#7715) 2023-06-14 11:01:03 +02:00
Alex Sharov
f4d8bd84b3
Crypto lib up (#7730) 2023-06-14 10:43:29 +07:00
Alex Sharov
5ff8246e2a
e4: release some tests (#7729) 2023-06-14 10:01:00 +07:00
Alex Sharov
ae1b193d6f
use 1 constant in tests to configure initialCycle variable (#7720) 2023-06-13 10:55:59 +07:00
omahs
c21d77aa20
Fix: typos (#7710)
Fix: typos
2023-06-12 14:39:52 +07:00
Giulio rebuffo
63006611ec
Added concurrent bls (#7709) 2023-06-12 00:40:03 +02:00
Enrique Jose Avila Asapche
e380ee4bdd
[Caplin] added configurations for beacon api (#7699)
Now you can configure:

Read timeout (default: 5s)
Write timeout  (default: 5s)
On/off (default: ON)
Port (default:5555)
Address (default:localhost)
Protocol (default: tcp)

#7684
2023-06-11 19:34:33 +02:00
Alex Sharov
afd8ea94d0
e3: move domains to tables.go (#7707) 2023-06-11 21:12:05 +07:00
Mark Holt
f9acbff96c
eth_sendRawTransaction: remove logging, because we have --txpool.trace.senders which enabling more verbose logging for given senders list (#7686)
When testing with `Bor` consensus turned on I discovered that
`SendRawTransaction` returns a 0x000... hash when transactions are
submitted during block transitions. This turns out to be spurious in the
sense that the transaction insertion is successful.

The cause is that `ReadCurrentBlockNumber` returns a nil block number.

This in turn is caused by the following: In `accessors_chain.go` there
are two methods: `WriteHeader` and `WriteHeadHeaderHash` when the first
is called the block number is written for the header. The second writes
the header has, but there is no guarantee when it does that the head
header will have been written yet. In fact it seems to happen sometime
later.

The problem for `SendRawTransation` is that it begins a transaction
after inserting into the txpool. And depending on timing this
transaction may see only the `WriteHeadHeaderHash` insertion, and hence
can't read the block number.

I have mitigated this by opening the db transaction before calling the
tx pool insertion, meaning that it is more likely to have a clean view
of the DB.

I have also moved the chain id check earlier in the code - as I think
that if this is invalid the method should not try to insert transactions
in the first place.

The `ReadCurrentBlockNumber` is only used to produce a log message - so
I've changed this to not fail the whole function but to just log an
unknown sender. Which means that the hash is still returned to the
sender after a successful txpool insertion
2023-06-11 13:18:04 +07:00
Alex Sharov
3b89ed81d6
e3: erigon_getBalanceChanges (#7704) 2023-06-11 11:49:53 +07:00
Giulio rebuffo
88410559c6
Forgot to configure flags (#7701) 2023-06-10 08:53:24 +02:00
Mark Holt
1e575ea172
rpc service and stageloop logger updates (#7696)
This is another update to logging to replace the root logger with a
contextual logger
2023-06-10 07:39:39 +01:00
Mark Holt
62b2375de9
7667 cli params vhost (#7669)
This change adds 'any' as an alternate wildcard to '*'.

I have updated all doc references in the main erigon repo - let me know
if there is anywhere else that needs changing.
2023-06-09 14:12:11 +01:00
Giulio rebuffo
614769f7ac
Prepare regression testings for kube (#7697) 2023-06-09 14:38:28 +02:00
Alex Sharov
1546a79771
lru lib version up (#7695) 2023-06-09 13:46:58 +07:00
Alex Sharov
afa4f53ae2
e3: use deserializeV3 for history (#7691) 2023-06-09 10:53:04 +07:00
Enrique Jose Avila Asapche
1cb1c303d4
added a listener for beacon api and added get genesis (#7659)
Added two new flags beacon.api.port and beacon.api.addr

Now we can listen for beacon api and get beacon genesis

---------

Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-06-08 09:43:27 +02:00
Manav Darji
421ee911d2
Indore hardfork for polygon mumbai and fixes in bor consensus (#7681)
This PR does the following things:
- Updates the hardfork number of the upcoming Indore hardfork schedule
at block 36877056.
- Refactoring to `CommitStates` method of bor consensus
- Fixes a bug in triggering mining
2023-06-08 09:20:40 +07:00
Giulio rebuffo
07e2100510
Added Regeression testing for Caplin (#7679) 2023-06-07 20:01:32 +02:00
Andrew Ashikhmin
fa0cfcbec7
Remove the Rinkeby testnet (#7677)
See https://github.com/ethereum/go-ethereum/pull/27406
2023-06-07 07:42:36 +02:00
Mark Holt
37a8450d5f
Added non root logger to bor.ValidatorSet (#7668)
I've added a non root logger to bor.ValidatorSet validator set. This
creates a signature change on a number of calling functions to propagate
the logger. This is mostly constrained to the bor package but impacts a
number of tests and utilities which call the validators set.
2023-06-06 10:55:59 +01:00
Alex Sharov
250ecfc4d1
Fix integration tests run3 (#7674) 2023-06-06 16:50:04 +07:00
Alex Sharov
bf9f5067f3
fix integration tests run (#7672) 2023-06-06 13:49:01 +07:00
Giulio rebuffo
3a8d5c4d0a
improved freezer for Caplin (#7670) 2023-06-06 03:24:59 +02:00
Alex Sharov
e14664d53b
Store Canonical/NonCanonical/Bad blocks/senders/txs in same tables. Make re-org cheaper (#7648)
- allow store non-canonical blocks/senders
- optimize re-org: don't update/delete most of data
- allow mark chain as `Bad` - will be not visible by eth_getBlockByHash,
but can read if have hash+num
2023-06-05 12:36:24 +07:00
Alex Sharov
f62b6fd722
e3: release switch (#7663) 2023-06-05 12:36:02 +07:00
Giulio rebuffo
e45ed7d139
Added record mode for Caplin (#7660) 2023-06-05 01:52:55 +02:00
Mark Holt
415cf86250
refactor to allow switchable consensus and multiple communicating nodes (#7646)
This branch is intended to allow the devnet to be used for testing
multiple consents types beyond the default clique. It is initially being
used to test Bor consensus for polygon.

It also has the following refactoring:

### 1.  Network configuration

The two node arg building functions miningNodeArgs and nonMiningNodeArgs
have been replaced with a configuration struct which is used to
configure:

```go
network := &node.Network{
		DataDir: dataDir,
		Chain:   networkname.DevChainName,
		//Chain:              networkname.BorDevnetChainName,
		Logger:             logger,
		BasePrivateApiAddr: "localhost:9090",
		BaseRPCAddr:        "localhost:8545",
		Nodes: []node.NetworkNode{
			&node.Miner{},
			&node.NonMiner{},
		},
	}
```
and start multiple nodes

```go
network.Start()
```
Network start will create a network of nodes ensuring that all nodes are
configured with non clashing network ports set via command line
arguments on start-up.

### 2. Request Routing

The `RequestRouter` has been updated to take a 'target' rather than
using a static dispatcher which routes to a single node on the network.
Each node in the network has its own request generator so command and
services have more flexibility in request routing and
`ExecuteAllMethods` currently takes the `node.Network` as an argument
and can pick which node (node 0 for the moment) to send requests to.
2023-06-04 20:53:05 +01:00
Alex Sharov
feea6b954c
simplify chain config read by rpcdaemon (#7656) 2023-06-04 11:49:10 +07:00
Alex Sharov
c8e717c957
rawdb methods to rw db schema version (#7653) 2023-06-04 09:20:22 +07:00
Alex Sharov
157b6de8e3
release small rename (#7647) 2023-06-03 17:26:20 +07:00
Alex Sharov
63c92010cd
remove txsV3 cli flag (#7644) 2023-06-03 15:54:27 +07:00
Alex Sharov
ad72b7178e
prune speedup. stage_senders: don't re-calc existing senders (#7643)
- stage_senders: don't re-calc existing senders
- stage_tx_lookup: prune less blocks per iteration - because
random-deletes are expensive. pruning must not slow-down sync.
- prune data even if --snap.stop is set
- "prune as-much-as-possible at startup" is not very good idea: at
initialCycle machine can be cold and prune will cause big downtime, no
reason to produce much freelist in 1 tx. People may also restart erigon
- because of some bug - and it will cause unexpected downtime (usually
Erigon startup very fast). So, I just remove all `initialSync`-related
logic in pruning.
- fix lost metrics about disk write byte/sec
2023-06-03 12:30:53 +07:00
Alex Sharov
d40317c905
preparation for --txs.v3, step 2 (#7636) 2023-06-03 07:20:22 +07:00
Andrew Ashikhmin
190bc9dfd8
EIP-4844: add data_gas_used (#7639)
Implement https://github.com/ethereum/EIPs/pull/7062 and
https://github.com/ethereum/EIPs/pull/7095. Pick up
https://github.com/ledgerwatch/erigon-lib/pull/1006.
2023-06-02 21:26:19 +01:00
Andrew Ashikhmin
b2a90e0ea9
Merge EngineGetPayloadWithBlobs into EngineGetPayload (#7637)
Pick up https://github.com/ledgerwatch/interfaces/pull/171 &
https://github.com/ledgerwatch/interfaces/pull/173
2023-06-02 13:02:13 +02:00
Alex Sharov
6439bdd835
txsv3: release small parts (#7632) 2023-06-02 11:38:20 +07:00
Alex Sharov
436656a40f
snapshots: DumpBodies - break dependency on body.BaseTxNum value in db (#7607)
it's step towards saving canonical and non-canonical bodies in same
table (and txs also in same own table). to reduce write amplification
(cheaper re-orgs)

PR change: reading BaseTxNum from existing snapshots instead of DB 
DB will store in field body.BaseTxNum - non-canonical TxnID  
Snapshots will store only canonical TxNum in field body.BaseTxNum
2023-06-02 11:01:57 +07:00
Alex Sharov
5fb31ae95f
e3: release some e4 parts (#7628) 2023-06-02 10:35:26 +07:00
Alex Sharov
2278bf5bfc
e3: move kvlist to erigon-lib (#7625) 2023-06-02 09:51:16 +07:00
Alex Sharov
7c89afcb42
e3: add kvlist.push method (#7624) 2023-06-02 09:24:22 +07:00
ledgerwatch
b35950441d
[sentry] initiate eth66 protocol under eth67 (#7612)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-31 20:29:57 +01:00
ledgerwatch
d66f9ce51f
[devnet] fix logging (#7611)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-31 20:29:36 +01:00
ledgerwatch
f5bd806b84
[devnet] Remove dependency on rpctest (#7608)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-31 19:47:32 +01:00
Alex Sharov
299c81e522
move "makeBodiesCanonical" method to blockWriter (#7605) 2023-05-31 15:44:01 +07:00
Alex Sharov
3cba3908dd
use BlockReader.CanonicalHash method instead of rawdb (#7604) 2023-05-31 13:41:10 +07:00
Alex Sharov
af83845f7e
cheaper isCanonicalHash func, less lookups of blockHash by blockNum in rpc (#7603) 2023-05-31 11:26:38 +07:00
ledgerwatch
708ea9225d
[devnet tool] single request generator (#7600)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-29 20:35:45 +01:00
Martin Holst Swende
c042afb4af
cmd/evm: support batched statetest-mode (#7598)
This implements batched state-test exectution, similar to
https://github.com/ethereum/go-ethereum/pull/27318 .
Some speedtests, executing a state-test twice on current master takes
~4-5 seconds, and scales linerarly.
```
Doing 2 execs old style

real    0m8.185s
user    0m8.081s
sys     0m0.110s
```
Doing `100` executions on this PR -- a few seconds of ramp-up time, but
very quick execution after that :
```
Doing 100 execs v2

real    0m5.009s
user    0m4.560s
sys     0m0.508s
```
I also tested a version where I moved the db instantiation into the top
callsite, with the `MustOpen` and `.Close` only performed once, instead
of `100` times -- however, I noticed no additional speed gains from
doing so (my branch `batched_evm_v2`).

Therefore, I suspect that the slowdowns comes not from the db, but the
kzg library initialization.
2023-05-29 18:28:24 +07:00
Giulio rebuffo
2d3b3c3b8d
Caplin: Proper validator set (#7591)
Adding proper flat buffer to Caplin
2023-05-28 17:11:18 +02:00
a
3c2b99e19c
[caplin] incremental improvements to p2p layer (#7582)
we update observability in the p2p layer for handlers, and also properly
encode error codes, close streams.

---------

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-05-28 07:54:04 +07:00
Alex Sharov
3ec7d9b010
receipts: less allocs in DeriveFields, use blockReader, remove ReadReceiptsByHash, gasPriceOracle don't read block twice and use blockLru (#7592)
preparation for adding BlockID
2023-05-27 16:39:14 +07:00
Andrew Ashikhmin
14653bbda1
Fix Gnosis rewards in trace_block and trace_filter (#7580)
Follow-up to PR #7473
2023-05-26 13:14:58 +02:00
Alex Sharov
ead45694ff
torrent-related goroutine leack (#7545)
can't merge - because `torrent` lib dropped go1.19 support
2023-05-26 17:12:47 +07:00
Alex Sharov
279e1bec33
use blockReader as service-provider of RoSnapshots (#7584) 2023-05-26 17:12:33 +07:00
Alex Sharov
111db5f655
blockReader in tests - step8 (#7578) 2023-05-25 12:46:11 +07:00
Alex Sharov
a79033cef8
blockReader in tests, step6 (#7573) 2023-05-24 18:34:36 +07:00
Alex Sharov
31687be599
blockReaders in tests, step4 (#7570) 2023-05-24 15:52:51 +07:00
Alex Sharov
179b138000
add blockReader to mining, blockchain_test (#7569) 2023-05-24 13:53:22 +07:00
Alex Sharov
23bd14744b
blockReader: use in ethstat (#7565) 2023-05-23 16:30:47 +07:00
Alex Sharov
63afe65686
Use BlockReader in ForkValidator, CliqueAPI (#7562) 2023-05-23 14:49:17 +07:00
Manav Darji
d279c43c81
eth/backend, cmd/erigon-el/backend: start mining immediately (#7424)
When mining is enabled, it waits for either a new block event, or a tx
notif or recommit interval before it starts mining the first block. This
PR achieves the following things:

- Start mining immediately (subsequent blocks will be mined via the new
head channel or miner.recommit timer).
- Modifies the conditions when it needs to look for new mining work
- Don't start mining on arrival of new transactions as it can be too
frequent (only for bor consensus as of now).
- Reset timer only if some work was done previously
2023-05-23 13:16:55 +07:00
Alex Sharov
18990ff91a
e2: reduce StageSyncStep interface size (#7561) 2023-05-23 12:28:31 +07:00
Alex Sharov
4d0dee6fb0
Introduce BlockWriter object (txsV3 step 0) (#7559) 2023-05-22 15:49:21 +07:00
ledgerwatch
a8ec9eb471
[devnet tool] Use logging instead of fmt.Printf (#7560)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 08:46:50 +01:00
Alex Sharov
2865b85888
move e2 snapshots management closer to e3: step 1 (#7543)
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 10:09:46 +07:00
ledgerwatch
cb04c203d8
[devnet] log devnet tool output into a file (#7557)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-21 15:33:11 +01:00
ledgerwatch
c408281d3b
[devnet] separate logging (#7556)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 21:57:32 +01:00
ledgerwatch
067f695fff
[devnet tool] Separate logging (#7553)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 14:48:16 +01:00
ledgerwatch
2a872b4d54
[devnet] separare logging - headers download (#7551)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 07:00:19 +01:00
ledgerwatch
c919283b0c
[devnet] separate logging p2p (#7549)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-19 23:08:45 +01:00
ledgerwatch
7272c00681
Update t8n_test.go (#7550) 2023-05-19 21:49:51 +01:00
racytech
65efc92af3
eip-4844: vm and engine_api extensions, minor fixes/changes (#7514) 2023-05-19 18:43:04 +01:00
ledgerwatch
b0117a7c30
[devnet] separate logging - p2p (#7547)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-19 18:41:53 +01:00
ledgerwatch
b382f96f6c
Introduce logger into etl (#7537)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 21:20:07 +01:00
ledgerwatch
685dee8490
Fix trace_filter regression due to gasBailout argument (#7539) (#7540)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 20:52:40 +01:00
a
761524a986
eth2 shuffle (#7522)
moves eth2shuffle package to erigon.
2023-05-18 01:06:25 +02:00
ledgerwatch
05597cb195
[devnet tool] separare logging (#7531)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 17:36:15 +01:00
Alex Sharov
c0096ee8c8
StageLoopStep: if node synced - then run initialCycle in 1 tx also (for data consistency) (#7532) 2023-05-17 13:39:21 +01:00
tripplegreen
02f2329c68
Fix internalcl flag description (#7530) 2023-05-17 11:25:29 +02:00
ledgerwatch
e75ea786c0
[devnet tool] separate logging (#7526)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 07:36:06 +01:00
Alex Sharov
a2ad68d634
downloader: speedup "--downloader.verify" mode (#7529)
deduplicate logic
create more producer goroutines (torrent lib does limiting internally
amount of consumers/disk-readers/hashers by 2, and it's enough because
we can verify multiple files in parallel)
move flag from "downloader torrent_hashes --verify" to "downloader
--verify"
2023-05-17 06:33:59 +01:00
ledgerwatch
3f9ae3ec77
[devnet tool] separate logging (#7525)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-16 10:53:50 +01:00
Giulio rebuffo
6f758fd0b9
added chiado and gnosis support to caplin (#7523) 2023-05-16 00:47:20 +02:00
ledgerwatch
90cb6be425
[devnet tool] fixing port conflicts (#7520)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-05-15 20:31:35 +01:00
Giulio rebuffo
4cbbe65f31
Resp/Req support for Deneb (#7519) 2023-05-15 20:07:27 +02:00
ledgerwatch
5d6b0ead8f
[devnet tool] separate logging (#7510)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-05-15 15:30:56 +01:00
Giulio rebuffo
dfadf495cf
Flat buffers for Attestations and Cached roots (#7512) 2023-05-15 00:12:24 +02:00
a
cda14447ad
[caplin] pkg refactor (#7507)
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-05-13 23:44:07 +02:00
ledgerwatch
bc169179b0
[devnet tool] Side-quest: logging step 7 (#7506)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-13 21:22:30 +01:00
Enrique Jose Avila Asapche
9cf2b42478
[Caplin] Block transition (#7485)
Added process of blob kzg commits for block transition following:

https://github.com/ethereum/consensus-specs/blob/dev/specs/deneb/beacon-chain.md#beacon-chain-state-transition-function

#7389
2023-05-13 21:52:49 +02:00
Giulio rebuffo
3216613892
Proper caching of attesting indicies (#7499) 2023-05-12 00:47:26 +02:00
ledgerwatch
5ae9cb7f39
[Devnet tool] side-quest logging step 6 (#7496)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-11 18:59:55 +01:00
a
fd6acd4b31
[Caplin] beginnings of instrumentation (#7486)
this pr is ready for review, but it is waiting on this PR 

https://github.com/VictoriaMetrics/metrics/pull/45

so that we do not need to use a replace directive.
2023-05-11 18:38:56 +02:00
Roberto Bayardo
8356f3a320
use ssz from updated erigon-lib (#7490)
ssz/clonable interfaces have been moved to erigon-lib so they can be
shared with the txpool code there
2023-05-11 12:54:20 +01:00
a
37da9ec1e3
[caplin] ssz byteobjects (#7454)
instead of converting from ssz -> struct -> ssz, it may be better to
just stay as ssz, then use methods to read the data.

this pr explores this concept, while maintaining compatiblity with the
existing codebase.
2023-05-10 21:37:50 +02:00
ledgerwatch
6ef3fc3a3c
[devnet tool] Side-quest logging, step 5 (#7484)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 19:36:27 +01:00
ledgerwatch
20437faaf5
[Devnet tool] side-quest logging step 4 (#7481)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 17:12:23 +01:00
Enrique Jose Avila Asapche
4f56247610
made KZGCommitment constant and SSZ compatiable (#7465)
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-05-10 14:40:19 +07:00
Alex Sharov
f23612bdfe
Enode logging broke when NAT Parameter set in 2.43.0 (#7480)
for https://github.com/ledgerwatch/erigon/issues/7472
2023-05-10 10:25:53 +07:00
Alex Sharov
dc11deed62
reduce default --db.size.limit from 7 to 3 Tb (to fit defaults of some systems) (#7478) 2023-05-10 09:41:22 +07:00
Andrew Ashikhmin
771c6fc202
Rename Serenity consensus engine to Merge (#7475)
[EIP-2982](https://eips.ethereum.org/EIPS/eip-2982) "Serenity Phase 0"
was superseded by [EIP-3675](https://eips.ethereum.org/EIPS/eip-3675)
"Upgrade consensus to Proof-of-Stake"
2023-05-09 18:45:33 +01:00
racytech
42e8db3958
eip-4844: modified DecodeTransaction version 1 (#7442)
Blob transactions are SSZ encoded, so it had to be added to decoding.
There are 2 encoding forms: `network` and `minimal` (usual). Network
encoded blob transactions include "wrapper data" which are `kzgs`,
`blobs` and `proofs`, and decoded by `DecodeWrappedTransaction`. For
previous types of transactions the network encoding is no different.
Execution-payloads / blocks use the minimal encoding of transactions. In
the transaction-pool and local transaction-journal the network encoding
is used.

Concerns: 
1. Possible performance reduction caused by these changes, not sure if
streams are better then slices. Go slices in this modifications are
read-only, so they should be referred to the same underlying array and
passed by a reference.
2. If `DecodeWrappedTransaction` and `DecodeTransaction` will create
confusion and should be merged into one function.
2023-05-09 18:44:53 +01:00
ledgerwatch
f38ec1e772
[devnet tool] side-quest: logging, step 3 (#7471)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-09 18:11:31 +01:00
Andrew Ashikhmin
f3144a6ed0
Proper Gnosis Chain rewards in trace_block (#7473) 2023-05-09 17:19:23 +02:00
Giulio rebuffo
3e44fe061a
fixed bls infinity bug (#7466) 2023-05-08 22:02:04 +02:00
ledgerwatch
3c1448afed
[devnet tool] Side-quest logging - replace quiet parameter (#7464)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-08 17:52:31 +01:00
Andrew Ashikhmin
cc11462860
Prioritize eth/68 by default (#7463)
Set `DefaultConfig.ProtocolVersion` to [68, 67, 66] instead of [67, 68].
See https://github.com/ethereum/hive/pull/776
2023-05-08 16:03:59 +02:00
Alex Sharov
d9abfd826f
e4: rename HistoryReaderV4 (#7461) 2023-05-08 15:44:19 +07:00
Alex Sharov
f28f97acc9
history_reader_v3: must always return accounts encoded as v3 (#7460) 2023-05-08 15:30:15 +07:00
Alex Sharov
231d128d91
e3: parallel exec docs (#7459) 2023-05-08 14:32:12 +07:00
racytech
36e70c545b
eip-4844: data gas fees & related check (#7449)
Logic to compute fees for data blobs as well as additional check that
verifies if user was willing to pay the current `data_gas` price.
Updated `FakeExponential` function to work with uint256.
2023-05-08 12:20:10 +07:00
Joe Netti
5a56b45b40
fix: erigon_getLatestLogs (#7450)
When calling erigon_getLatestLogs, I was getting a crash in
[erigon_receipts.go](beb97784d4/cmd/rpcdaemon/commands/erigon_receipts.go (L254)).
I think it is a simple indexing bug

```
[service.go:217 panic.go:884 panic.go:113 erigon_receipts.go:254 value.go:586 value.go:370 service.go:222 
handler.go:494 handler.go:444 handler.go:392 handler.go:223 handler.go:316 asm_amd64.s:1598]
[WARN] [05-05|21:13:59.749] Served                                   
conn=100.70.204.111:50141 method=erigon_getLatestLogs reqid=1 t=621.5936ms err="method handler crashed"
```
2023-05-08 11:54:17 +07:00
Giulio rebuffo
3b9636644d
fixed cache concurrency issue in BLS (#7455) 2023-05-07 14:38:44 +02:00
ledgerwatch
fdd385cef1
[Devnet tool] Side-quest to improve logging - part 1 (#7445)
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-07 07:28:15 +01:00
Giulio rebuffo
5b7643c57b
Added dencun networking (#7453) 2023-05-07 01:37:53 +02:00
Enrique Jose Avila Asapche
fbd18516ae
Deneb container changes beacon-chain (#7420)
Adding container changes for Deneb:
https://github.com/ethereum/consensus-specs/blob/dev/specs/deneb/beacon-chain.md

Changed to use pointers for BlobSideCar
 
 #7389
2023-05-06 18:50:58 +02:00
Andrew Ashikhmin
ac087d8ca5
[interfaces] Header mix_digest -> prev_randao (#7448)
Pick up https://github.com/ledgerwatch/interfaces/pull/169
2023-05-05 17:58:20 +02:00
Giulio rebuffo
9d443dd052
Caplin: fixed dereferencing (#7447) 2023-05-05 13:16:50 +02:00
Giulio rebuffo
43b09a7f55
Refactor of ssz interface (#7439)
Also added a better way to copy and the hasher
2023-05-05 11:19:24 +02:00
racytech
e751b9d219
eip-4844: minor additions (#7443) 2023-05-05 09:24:45 +07:00
0xMaxMa
9440126ddf
Add clique API, support "clique" option in --http.api flag (#6985)
* Enable the `clique` option in the `--http.api` flag.

* List of Clique commands:
`clique_getSnapshot(block number)`
`clique_getSnapshotAtHash(block hash)`
`clique_getSnapshotAtHash(block hash)`
`clique_getSigners(block number)`
`clique_getSignersAtHash(block hash)`
`clique_proposals()`
`clique_propose(signer address, bool)`
`clique_discard(signer address)`
`clique_status()`
Example:
`curl --data
'{"method":"clique_getSigners","params":[],"id":1,"jsonrpc":"2.0"}' -H
"Content-Type: application/json" -X POST http://localhost:8545`

* Please be careful while using the Clique API. Do not make the HTTP API
public on the Clique's signer node, as anyone can directly call a Clique
command. Instead, it should only be allowed in the localhost by using
the flag `--http.addr "127.0.0.1"`.
2023-05-05 09:20:40 +07:00
Roberto Bayardo
df124d4108
fix block rlp logging bug (#7441)
fixes a logging statement was logging the outer rpc proto instead of the
inner rlp of the block as intended
2023-05-05 09:16:43 +07:00
Alex Sharov
2a37cc8513
fix devel evm tests (#7444) 2023-05-05 09:16:13 +07:00
a
30430d585a
begin refactor of beacon state (#7433)
this first major move separates the transient beacon state cache from
the underlying tree.

leaf updates are enforced in the setters, which should make programming
easier.

all exported methods of the raw.BeaconState should be safe to call
(without disrupting internal state)

changes many functions to consume *raw.BeaconState in perparation for
interface


beyond refactor it also:

adds a pool for the leaves of the validator ssz hash 

adds a pool for the snappy writers
  
removed the parallel hash experiment (high memory use)
2023-05-04 15:18:42 +02:00
Alex Sharov
fe30cf8c6f
up urwafe and docker version (#7435) 2023-05-04 10:48:43 +07:00
Giulio rebuffo
0e0b12d68f
Added checkpoint states object to optimize memory consumption (#7429) 2023-05-03 10:51:39 +02:00
racytech
40947f6c98
eip-4844: adding data_gas to gaspool (#7428)
Adding `data_gas` to gas pool. EIP-4844 gas pool includes data_gas which
is used to fee blob transactions.
2023-05-03 09:02:30 +07:00
a
9644e6d220
Implement SpecTests in native go, add fork_choice handler (#7422)
a few TODO: remain to make this not a draft

---------

Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-05-02 16:19:22 +02:00
Giulio rebuffo
80d06bc385
Caplin: Fixed sentinel deadlock (#7419) 2023-05-01 23:32:00 +02:00
Andrew Ashikhmin
1533674dad
Implement EIP-1153 transient storage (#7405)
Port https://github.com/ethereum/go-ethereum/pull/26003

---------

Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
2023-05-01 18:26:24 +02:00
naviechan
d475bab15f
Add diagnostics endpoint for flags (#7417)
To expose context flags through diagnostic endpoint
`/debug/metrics/flags`.

The current `/debug/metrics/cmdline` only provides the command run by
user. In the case when user runs Erigon using config file, `cmdline`’s
info does not truly reflect the ‘launch setting' and flags set that
Erigon is running on.

## Example

### Command
```
erigon --datadir /tmp/data --config test.yaml
```
### Pseudo config file (in yaml)
```
datadir : '/tmp/data'
chain : "sepolia"
http : true
metrics: true
private.api.addr : "localhost:9090"

http.api : ["eth","debug","net"]
```
### Output
```
SUCCESS
chain=sepolia
config=test.yaml
datadir=/tmp/data
http=true
http.api=eth,debug,net
metrics=true
private.api.addr=localhost:9090
```
2023-05-01 15:38:00 +01:00
Giulio rebuffo
cab15c9f2e
Caplin: Remotion of bad peers on request (#7414) 2023-04-30 22:24:42 +02:00
racytech
726ce264ef
eip-4844: RPCTransactions to support BlobTx (#7407)
Small additions to support eip-4844 transaction marshalling
2023-04-30 14:03:38 +07:00
racytech
f31d2b097d
eip-4844: Unwrap() additional tx method (#7410)
Transaction extension to support BlobTxWrapper (or network
representation of BlobTx). BlobTxWrapper wraps BlobTx with additional
data: blobs, commitments and proofs. Unwrap() returns unwrapped tx if
its network tx, otherwise returns itself.
2023-04-30 14:03:15 +07:00
Alex Sharov
4344136973
e3: rethink meaning of kv_temporal.BeginRw() (#7413) 2023-04-30 10:08:41 +07:00
Giulio rebuffo
250a0a4a65
Caplin: Remotion of bad peers giving us bad blocks (#7411) 2023-04-29 21:32:33 +02:00
Giulio rebuffo
8a3b9e6d17
Added caching for BLS affines (#7408)
Improved CPU usage by 10x, we just really avoid re-compressing and
re-validating public keys
2023-04-29 02:36:53 +02:00
racytech
631681ccfd
eip-4844: receipts to include dataGasPrice and dataGasUsed (#7385)
Small additions to block receipts. Now it includes `dataGasPrice` and
`dataGasUsed`. `dataGas` is a new type of a gas and blobs are priced in
it.
2023-04-28 17:53:43 +01:00
ledgerwatch
9b5b24a0b3
Fix lint (#7399)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-04-27 13:39:27 +01:00
Alex Sharov
ded8283df7
erigon backup: v0 of sub-command (#7396) 2023-04-27 10:42:12 +07:00
Jason Yellick
4e9b378a5d
Enable negative Merkle proofs for eth_getProof (#7393)
This addresses the last known deficiency of the eth_getProof
implementation. The previous code would return an error in the event
that the element was not found in the trie. EIP-1186 allows for
'negative' proofs where a proof demonstrates that an element cannot be
in the trie, so this commit updates the logic to support that case.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-27 10:38:45 +07:00
Alex Sharov
7b7b2fca0a
e3: drain when no new tasks causing deadlock (#7395) 2023-04-27 09:10:24 +07:00
Giulio rebuffo
01ce968361
Fixed networking issue in Caplin (#7388) 2023-04-26 14:33:21 +01:00
Jason Yellick
434ac76f79
Add additional trie proof testing (#7382)
This PR extends the function merged in #7337 to cover the proof
generation paths as well.

The first commit simply migrates the internal proof checking code from
the rpc `eth_getProof` test to be available externally exported under
`turbo/trie`. In the process, it translates the in place testing
assertions to return more normally as errors and adapts to use the
pre-existing trie node types which are defined slightly differently
(although this code is still intended only for testing purposes).

The second commit refactors the existing trie fuzzing tests from the
previous PR and adds new fuzzing tests for the proof generation. In
order to validate the proofs, these fuzzing tests fundamentally do two
things. Firstly, after bootstrapping the test (by seeding, and modifying
the db), for each key we compute the 'naive' proof utilizing the
existing code in `proof.go` and the in memory `trie.Trie` structure. The
`trie.Trie` code actually had a couple small bugs which are fixed in
this PR (not handling value nodes, and not honoring the `NewTestRLPTrie`
contract of pre-RLP encoded nodes in proof generation). Secondly, we
re-compute the same proof for the flatDB production variant, and verify
that it is exactly the same proof as computed by the naive
implementation.

This fuzzing has been run for ~72 hours locally with no errors. Although
this code hasn't identified any new bugs in the proof generation path,
it improves coverage and should help to prevent regressions. Additional
extensions will be provided in a subsequent PR.

---------

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-26 09:33:46 +07:00
Alex Sharov
ca088bd68e
fix build of cli flags (#7377) 2023-04-24 17:47:23 +07:00
Alex Sharov
c5e7b07de5
custom cli flags - move to own package (#7376) 2023-04-24 13:31:52 +07:00
racytech
6588bca40b
eip-4844: NewMessage now expectes maxFeePerDataGas & GetPayloadV3 impl (#7365)
types.NewMessage now expects maxFeePerDataGas param, which will be used
in transaction verification (preCheck). GetPayloadV3 method added to
EngineAPI. Some cosmetic changes applied.
2023-04-23 18:27:05 +01:00
Alex Sharov
74e08e197d
e3: remove files when no readers (#7370) 2023-04-23 11:55:16 +07:00
alex.sharov
f3aefe2fc0 set default db.size.limit to 7gb (as it was before) 2023-04-23 11:28:09 +07:00
Alex Sharov
e1f549ff01
cli: use SplitAndTrim instead of strgings.Split(",") (#7369) 2023-04-23 10:54:55 +07:00
Alex Sharov
d7d7af3b86
p2p: reduce verbosity of NetworkIdMissmatchErr and EOF (because likely we can't do anything about it and user can't) (#7368) 2023-04-23 10:41:46 +07:00
rekyyang
d6986ad6d3
trace_filter: support Mode == intersection (#7345) 2023-04-21 09:28:42 +07:00
alexqrid
db51dd5f02
trace_block: add new gasBailOut parameter to request (default: false) (#7326)
**Problem**
While tracing the block with Parity tracer(`trace_block`) it returns
error for a few blocks.
<details><summary>Example of blocks that couldn't be traced</summary>

These are the example of payloads with the errors occured during tracing
the blocks:
1. `{"method":"trace_block","params":["0x24165bd"],"id":0}` - `first run
for txIndex 32 error: insufficient funds for gas * price + value:
address 0x000005D814d5abD6e0F9345c9b1f37C82Eaf1EBb have
547961731687102852 want 675024999764675175`
2. `{"method":"trace_block","params":["0x2658753"],"id":0}` - `first run
for txIndex 45 error: insufficient funds for gas * price + value:
address 0x000004BeDC012a5D043270AF67de24c20a3b8aeB have
211658993830905595 want 235579381558610848`
3. `{"method":"trace_block","params":["0x258b00c"],"id":0} -first run
for txIndex 274 error: insufficient funds for gas * price + value:
address 0xA3a762006D22806B35895f4C0599bAe3adF1B349 have
342048586356638524 want 386865441635818752`

</details>

After looking through the
[trace_filtering.go](https://github.com/ledgerwatch/erigon/blob/devel/cmd/rpcdaemon/commands/trace_filtering.go#L1006)
and
[trace_adhoc.go](https://github.com/ledgerwatch/erigon/blob/devel/cmd/rpcdaemon/commands/trace_adhoc.go#L1072)
noticed that `doCallMany` called with `gasBailOut = true` in
[one](https://github.com/ledgerwatch/erigon/blob/devel/cmd/rpcdaemon/commands/trace_adhoc.go#L1072)
place and `gasBailOut = false` in
[another](https://github.com/ledgerwatch/erigon/blob/devel/cmd/rpcdaemon/commands/trace_filtering.go#L1006)
(actually the part of the code that is called by `trace_block`.
Changing its value to `true` fixed the problem and traces are successful
with this change.
Attaching the partial result of the successful trace below.


<details><summary>{"method":"trace_block","params":["0x24165bd"],"id":0}</summary>

```json
{
  "jsonrpc": "2.0",
  "id": 0,
  "result": [
    {
      "action": {
        "from": "0x3eaf7de168a79c1d6a1aab8c106e42b6f4e0a7c8",
        "callType": "call",
        "gas": "0x30464",
        "input": "0x0000000100000000000000000000000000000000000000000000000000000001dcbde73f0000000000000000000000000000000000000000000002086b35cd47c9189dc02791bca1f2de4661ed88a30c99a7a9449aa841740d500b1d8e8ef31e21c99d1db9a6444d3adf12700001f4000000000000000000000f490d8e000000000000",
        "to": "0x1d36f9688cceafee9d7df45fe8e24884ed0d6730",
        "value": "0x0"
      },
      "blockHash": "0x96f0792349a67b7995dda853df79bd5fa9d2926eb2ac8a07ac46e4df429632af",
      "blockNumber": 37840317,
      "error": "Reverted",
      "result": {
        "gasUsed": "0x1d2d",
        "output": "0x08c379a0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000056572723332000000000000000000000000000000000000000000000000000000"
      },
      "subtraces": 1,
      "traceAddress": [],
      "transactionHash": "0x1417cab7ae635884117909cc828474df0121d4a2a0ad033f462e6fa84bfab176",
      "transactionPosition": 0,
      "type": "call"
    },
    {
      "action": {
        "from": "0x1d36f9688cceafee9d7df45fe8e24884ed0d6730",
        "callType": "staticcall",
        "gas": "0x2e9b7",
        "input": "0x3850c7bd",
        "to": "0xa374094527e1673a86de625aa59517c5de346d32",
        "value": "0x0"
      },
      "blockHash": "0x96f0792349a67b7995dda853df79bd5fa9d2926eb2ac8a07ac46e4df429632af",
      "blockNumber": 37840317,
      "result": {
        "gasUsed": "0xa88",
        "output": "0x000000000000000000000000000000000000000000000f49c33e3991750050fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffbc158000000000000000000000000000000000000000000000000000000000000096d00000000000000000000000000000000000000000000000000000000000009c400000000000000000000000000000000000000000000000000000000000009c400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001"
      },
      "subtraces": 0,
      "traceAddress": [
        0
      ],
.....
      },
      "blockHash": "0x96f0792349a67b7995dda853df79bd5fa9d2926eb2ac8a07ac46e4df429632af",
      "blockNumber": 37840317,
      "result": null,
      "subtraces": 0,
      "traceAddress": [],
      "type": "reward"
    }
  ]
}

```

</details>

---------

Co-authored-by: alexqrid <>
2023-04-21 09:28:04 +07:00
Giulio rebuffo
ec8cdd4701
Refactore fork choice to RR approach (#7359) 2023-04-20 22:47:58 +02:00
Jason Yellick
7f31b047f1
Fix eth_getProof element order (#7351)
According to EIP-1186 the `proof` parts of the response to eth_getProof
should be returned "starting with the stateRoot-Node, following the path
of the SHA3 (address) as key." Currently, the proof is returned in
traversal order, rather than from the root.

Although all of the proof elements are there and correct, this is
contrary to the EIP and will cause problems for some clients. The
existing rpc test uses a map to lookup proof elements by hash, rather
than by index, so this bug was not initially caught.

This commit fixes the behavior, updates the existing test, and adds
additional checks to the rpc test.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-20 10:06:19 +07:00
Giulio rebuffo
f8f1658f8d
added deep copy to BeaconState copy (#7352)
Before caches were recomputed, now caches are not recomputed anymore for
BeaconStates internals
2023-04-19 22:56:24 +02:00
Giulio rebuffo
2ce5e761bb
added --internalcl flag to Erigon (#7349) 2023-04-19 14:37:35 +02:00
Alex Sharov
b23a0267d3
e3: deadlock fix part2 (#7344) 2023-04-19 08:12:13 +00:00
mars
bcfaa84b35
add api debug_getRawHeader debug_getRawBlock (#7333)
Add  json rpc api :
debug_getRawHeader
debug_getRawBlock 

doc:
 https://ethereum.github.io/execution-apis/api-documentation/
2023-04-19 08:05:45 +00:00