Commit Graph

1654 Commits

Author SHA1 Message Date
Mark Holt
8ea0096d56
moved metrics sub packages types to metrics (#8119)
This is a non functional change which consolidates the various packages
under metrics into the top level package now that the dead code is
removed.

It is a precursor to the removal of Victoria metrics after which all
erigon metrics code will be contained in this single package.
2023-09-03 08:09:27 +07:00
lupin012
5a9f430c5a
RPCdaemon: fix to work also with snapshot (#8111)
The FIX is necessary to permit to use API also in case block is in the
snapshot files
2023-09-02 09:55:48 +07:00
Giulio rebuffo
42101f529e
demote logs for fork validator (#8118) 2023-09-02 00:20:14 +02:00
Giulio rebuffo
b4cf5f5be4
Fixed canonical chain truncation (#8117) 2023-09-01 23:44:18 +02:00
alex.sharov
253c232f77 pprof flags in erigon sub-commands 2023-09-01 12:22:24 +07:00
banteg
5fbb9e2f87
fix: correctly process extcodecopy in vmtrace (#8108)
vmtrace routine wasn't outputting the memory extcodecopy has written.
2023-09-01 07:10:34 +07:00
Mark Holt
a4cfbe0d56
Heimdall metrics + Metrics HTTP server rationalization (#8094)
This is an update of:

https://github.com/ledgerwatch/erigon/pull/7846

which uses a local fork of victoria metrics to include the changes that
https://github.com/anshalshukla added to the original for we where
using.

It also includes code to address the duplicate metrics issue identified
here:

https://github.com/ledgerwatch/erigon/issues/8053

It has one more associated fix which is to correctly add a metadata
label to counters, these where previously labelled as gauges.

e.g. 

```
# TYPE p2p_peers counter
p2p_peers 0
```
rather than

```
# TYPE p2p_peers gauge
p2p_peers 0
```

---------

Co-authored-by: Anshal Shukla <53994948+anshalshukla@users.noreply.github.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
2023-08-31 09:04:27 +01:00
Andrew Ashikhmin
9b63764b16
Move ApplyDAOHardFork & UpgradeBuildInSystemContract to engine.Initialize (#8095)
Now all protocol-stipulated changes at the beginning of the block (AuRa
stuff,
[DAO](https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
irregular state change, Calcutta system contract upgrade,
[EIP-4788](https://eips.ethereum.org/EIPS/eip-4788) beacon root) are
handled by consensus engine `Initialize()`.
2023-08-30 15:51:19 +02:00
Mark Holt
f2d0118a33
Bor snapshot block production (#8065)
I have added:

```go
{
	ID:          stages.BorHeimdall,
	Description: "Download Bor-specific data from Heimdall",
	Forward: func(firstCycle bool, badBlockUnwind bool, s *StageState, u Unwinder, tx kv.RwTx, logger log.Logger) error {
		if badBlockUnwind {
			return nil
		}
		return BorHeimdallForward(s, u, ctx, tx, borHeimdallCfg, true, logger)
	},
	Unwind: func(firstCycle bool, u *UnwindState, s *StageState, tx kv.RwTx, logger log.Logger) error {
		return BorHeimdallUnwind(u, ctx, s, tx, borHeimdallCfg)
	},
	Prune: func(firstCycle bool, p *PruneState, tx kv.RwTx, logger log.Logger) error {
		return BorHeimdallPrune(p, ctx, tx, borHeimdallCfg)
	},
},
```
To MiningStages as well as Default as otherwise bor events are not added
when the block producer creates new blocks.

There are a couple of questions I have around this implementation:

* Is this the right place to add this
* As the state is also executed when the default stage is processed ther
is some duplicate processing for the block producing node.
* There is a duplicated call to heimdall which could be removed if the
stages share state - but its not clear if we want to do this.
* I don't think the mining stage needs to prune as this will be
replicated in the default iteration

This can be tested using the devnet with the following arguments:

```
--chain bor-devnet --bor.localheimdall --scenarios state-sync
```

This will generate sync events via an ethereum devnet which are
transmitted to bor chain and will be executed at the end of the snapshot
delay, which results in events generated from the bor chain. This tests
the whole sync, block generation, event lifecycle. As it needs to wait
for sprints to end after a sufficient delay it is quite slow to run.
2023-08-30 08:06:09 +01:00
Alex Sharov
d1d348211f
add flag --force.partial.commit: to workaround problem "start from backup takes long time and can't save partial progress" (#8090) 2023-08-30 08:49:16 +07:00
racytech
f477348a50
Support for --chain=holesky (#8064) 2023-08-26 16:31:29 +06:00
Giulio rebuffo
f5cc67fe5b
Added ranges to the Execution.proto (#8081) 2023-08-26 02:25:48 +02:00
Giulio rebuffo
d223adf114
Better migrations and chain segmentation recovery (#8079) 2023-08-25 17:45:00 +02:00
battlmonstr
32ca0e5ab1
sync: revert flawed dropUselessPeers logic and alleviate its issues (#8062)
The current logic is flawed, because it drops all peers that are less
synced.
It is valid to return empty responses by the eth spec.
A proper logic should penalize from the context of the sync process,
where enough "reputation" data is collected about a peer.

In order to be able to connect to erigon 2.48 peers that have
--sentry.drop-useless-peers enabled,
this adds a check to not reply with an empty headers list.
If we reply with an empty list, we're going to be considered useless and
kicked.
Once enough of erigon nodes are updated in the network past this commit,
this check should be removed,
because it is totally acceptable to return an empty list by the eth
spec.
2023-08-25 11:42:54 +02:00
Andrew Ashikhmin
a6d9d26fe9
Fix opSelfdestruct6780 (#8066)
also upgrade execution-spec-tests to
[v1.0.2](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.2)
2023-08-25 08:06:59 +02:00
Alex Sharov
2b6c21fddb
move mdbx to new org (#8061) 2023-08-24 18:00:24 +07:00
battlmonstr
2e29ff33e1
bor: BroadcastNewBlock to all peers from validator nodes (#8030)
Currently PropagateNewBlockHashes and BroadcastNewBlock
selects a subset of all sentries by taking a `Sqrt(len(sentries))`,
and then for each sentry SendMessageToRandomPeers
selects a subset of its peers by taking `Sqrt(len(peerInfos))`.

This behaviour limits the broadcast scope with a lot of peers, e.g. 100
becomes 10,
but is not great with very few peers, or if the message is very
important
to broadcast to everyone, which is the case of bor validator/proposer
nodes.

* send to all sentries in both BroadcastNewBlock and PropagateNewBlockHashes
* remove peerCountConstrained sqrt logic in SendMessageToRandomPeers
* add maxPeers provider func as a parameter to MultiClient
* default it to 10 for eth and 0 (unlimited) for bor validators

---------

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-08-23 14:28:39 +02:00
Andrew Ashikhmin
6bc0ca9e85
Correctly compute fork id when timestamp fork is activated in genesis (#8046)
See https://github.com/ethereum/go-ethereum/pull/27895
2023-08-21 15:35:13 +02:00
ledgerwatch
6b6c0caad0
Snapshots of Bor events (#7901)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp-2.home>
2023-08-18 17:10:35 +01:00
Giulio rebuffo
38db2ed2e7
[Caplin Phase 2]: Initial working implementation (#8033)
caplin just using execution.proto
2023-08-18 15:43:22 +02:00
Andrew Ashikhmin
611145c385
updateForkChoice: send state changes to txpool after tx is committed (#8035)
[txpool](https://github.com/ledgerwatch/erigon-lib/blob/main/txpool/pool.go)
expects an `OnNewBlock` update only after the DB transaction is
committed.

This fixes, for example, a nonce gap mis-detection in Hive test
"engine-cancun/Blob Transactions On Block 1, Cancun Genesis".
2023-08-17 16:03:12 +02:00
Somnath Banerjee
1867d9393a
Fix duplicate commitments in GetPayloadV3 response (#8018) 2023-08-14 17:57:33 +05:30
Somnath Banerjee
143ff4695c
Fix 17 hive tests (#8014) 2023-08-14 09:24:59 +02:00
Andrew Ashikhmin
7947fc8ad1
Always encode EIP-4844 transactions unwrapped (without blobs) (#8012)
The only time we need to serialize blobs/commitments/proofs is when we
reply to `GetPooledTransactions` (and that's handled by the txpool).
2023-08-13 22:32:47 +05:30
Somnath Banerjee
b7070a9c08
Fix 12 hive tests (#8011) 2023-08-13 19:46:25 +05:30
Andrew Ashikhmin
9609ca9120
Fix JSON marshalling of BlobsBundleV1 (EIP-4844) (#8010)
According to the
[spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/cancun.md#blobsbundlev1),
commitments/proofs/blobs in `BlobsBundleV1` should have the
[DATA](https://github.com/ethereum/execution-apis/blob/main/src/engine/common.md#encoding)
encoding.
2023-08-13 15:43:55 +02:00
Giulio rebuffo
7c0d6e15bb
Added gnosis fields and removed body gap. (#7993) 2023-08-11 23:07:36 +02:00
Alex Sharov
c464d84334
tmpdb: move to tmpdir (#8003) 2023-08-12 01:06:15 +06:00
Andrew Ashikhmin
e377cea51d
Fix CopyTxs for BlobTxWrapper (EIP-4844) (#8002) 2023-08-11 20:30:02 +02:00
Andrew Ashikhmin
03927d3e27
Call InitializeBlockExecution in SpawnMiningExecStage (EIP-4788) (#7999)
This fixes the trie state root issue that was occurring in the Hive
tests for Cancun.
2023-08-11 14:04:53 +02:00
Andrew Ashikhmin
6adad0afde
Add parentBeaconBlockRoot to RPCMarshalHeader (EIP-4788) (#7997) 2023-08-11 10:26:44 +02:00
Alex Sharov
d2bde8c096
Recsplit: cancelable build (#7994) 2023-08-11 11:54:59 +06:00
alex.sharov
7ca6649e0c BlockReader: remove redundant check 2023-08-11 11:45:02 +06:00
racytech
6785ff28e4
hive: minor fixes (EIP-4788) (#7985) 2023-08-09 13:44:21 +06:00
a
db5b348673
Caplin block persistence (#7941)
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-08-09 01:21:19 +02:00
Giulio rebuffo
6081fe01b8
fixed forkchoices (#7983) 2023-08-08 22:56:16 +02:00
Andrew Ashikhmin
ae76df1b8a
Update execution-spec-tests to v1.0.1 (#7980)
[v1.0.1](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.1)
"Cancun Devnet-8 Pre-Release" adds some tests for Cancun. This PR only
fixes a couple of issues; the majority of the new tests are still
failing and thus are skipped in `TestExecutionSpec`.
2023-08-08 15:01:35 +02:00
Alex Sharov
23270eae58
tests: fix nil genesis panic (#7979) 2023-08-08 15:19:08 +06:00
Giulio rebuffo
e3d642d6a0
Fixed EL integration spectest (#7975) 2023-08-06 17:13:36 +02:00
Andrew Ashikhmin
d69b20bc4e
Support engine_forkchoiceUpdatedV3 with ParentBeaconBlockRoot (EIP-4788) (#7969)
Prerequisites: https://github.com/ledgerwatch/interfaces/pull/187 &
https://github.com/ledgerwatch/erigon-lib/pull/1069. Also implement
https://github.com/ethereum/execution-apis/pull/426.
2023-08-06 11:54:14 +02:00
Giulio rebuffo
b4057da8e6
removed missingHash (#7973) 2023-08-06 02:28:18 +02:00
banteg
63ecbf88ba
fix: show stack for push0 (#7970)
this fixes vmtrace after shanghai

before:
```python
VMOperation(pc=24, cost=2, ex=VMExecutedOperation(used=1942085, push=[], mem=None, store=None), sub=None, op='PUSH0', idx='14-644-13')
```

after:
```python
VMOperation(pc=24, cost=2, ex=VMExecutedOperation(used=1942085, push=[HexBytes('0x00')], mem=None, store=None), sub=None, op='PUSH0', idx='14-644-13')
```
2023-08-05 23:37:51 +02:00
Giulio rebuffo
0e4e36b142
Replaced old version of Engine API with newer version (#7972) 2023-08-05 23:33:10 +02:00
Giulio rebuffo
9730cc796a
Cleanup of consensus separation (#7971) 2023-08-05 02:26:53 +02:00
Giulio rebuffo
b09d90ee0e
Added in-memory flushing to execution.proto (#7968) 2023-08-04 14:42:35 +02:00
Giulio rebuffo
c08aa3ac03
fix undownloaded body (#7966) 2023-08-04 02:22:07 +02:00
Andrew Ashikhmin
d014da4dc0
Replace --override.shanghaiTime flag with --override.cancun (#7964)
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/1067
2023-08-03 16:05:35 +02:00
racytech
2c2ccb6e27
dencun-devnet: --trusted-setup-file flag support (#7963) 2023-08-03 14:36:47 +02:00
Giulio rebuffo
29935a65f4
Fixed hive tests in consensus separation (#7961) 2023-08-03 03:39:47 +02:00
Andrew Ashikhmin
bd9e8b9b56
Fix eth_getTransactionByHash for EIP-4844 transactions (#7960)
Also pick https://github.com/ledgerwatch/erigon-lib/pull/1063 and
https://github.com/ledgerwatch/erigon-lib/pull/1065
2023-08-02 18:25:15 +02:00
Somnath Banerjee
3e5b9d7490
Fix newPayload parentBeaconBlockRoot check (#7955) 2023-08-02 09:34:13 +02:00
Somnath Banerjee
d91f4263dc
EIP-4788: Beacon block root and stateful precompile (#7888) 2023-08-01 16:01:26 +05:30
Giulio rebuffo
1220ae659e
Most hive tests fixed in --experimental.modular (#7950)
Broken: 25/109
2023-08-01 02:08:15 +02:00
Andrew Ashikhmin
03ac80ad2d
EIP-4844: rename "data hash" to "blob hash" (#7947)
See https://github.com/ethereum/EIPs/pull/7001
2023-07-31 10:12:53 +02:00
Giulio rebuffo
443757edbd
Consensus separation for Engine API (Working on Sepolia) (#7945)
This makes the experimental consensus separation functional on sepolia.
2023-07-30 23:35:55 +02:00
Andrew Ashikhmin
5693439521
EIP-4844: Verify expectedBlobVersionedHashes (#7946)
See the
[engine_newpayloadv3](https://github.com/ethereum/execution-apis/blob/main/src/engine/cancun.md#engine_newpayloadv3)
spec
2023-07-30 17:10:54 +02:00
racytech
af884204cb
eip-4844: hive tests: RLP decode order and newPayloadV3 to expect versionedHashes (#7943) 2023-07-30 15:49:31 +02:00
Giulio rebuffo
2f2a1741d2
Protyped consensus separated engine api (untested) (#7942) 2023-07-29 00:22:38 +02:00
Andrew Ashikhmin
7d35c6b737
EIP-4844: Rename "data gas" to "blob gas" (#7937)
See https://github.com/ethereum/EIPs/pull/7354 &
https://github.com/ethereum/consensus-specs/pull/3461. Prerequisite:
https://github.com/ledgerwatch/erigon-lib/pull/1058
2023-07-28 12:12:05 +02:00
Willian Mitsuda
d3f8b5861c
Max cap search results for ots API by default + cli flag to override it (#7924)
Otterscan API search methods allow the user to inform the page size.

This PR adds an internal max (default == 25 results) to cap the page
size, regardless of what the user asks.

It also adds a `--ots.search.max.pagesize` CLI args to override this max
(either in erigon and rpcdaemon binaries).
2023-07-28 08:29:17 +07:00
Giulio rebuffo
a0865b489f
Prototyped experimental engine reverse downloader (#7936)
Erigon lib now good
2023-07-28 01:32:19 +02:00
Giulio rebuffo
b4ecd7f524
Consensus Separation: Separated GetPayload using an execution service (#7933)
Miracoulously, hive tests pass first try. YIPPIE.

Also for the future, I added `--experimental.modular` which enables a
secondary engine API for consensus separation.

Now block building is responsibility of the execution module.
2023-07-27 14:37:49 +02:00
Giulio rebuffo
f5f833fd70
Proper Error handling in direct communication with engine (#7929) 2023-07-25 14:15:50 +02:00
Giulio rebuffo
39853432be
added --experimental.modular (#7925) 2023-07-24 18:04:36 +02:00
Giulio rebuffo
9d3551f8cf
prototype of ValidataChain (#7921) 2023-07-23 19:10:24 +02:00
Giulio rebuffo
837ffb5b7e
Refactored ForkValidator to be Database-only (#7920) 2023-07-23 14:51:26 +02:00
Giulio rebuffo
9cf07f059d
Prototyped FCU for EL module (#7913)
save
2023-07-20 18:35:02 +02:00
Mark Holt
529d359ca6
Bor span testing (#7897)
An update to the devnet to introduce a local heimdall to facilitate
multiple validators without the need for an external process, and hence
validator registration/staking etc.

In this initial release only span generation is supported.  

It has the following changes:

* Introduction of a local grpc heimdall interface
* Allocation of accounts via a devnet account generator ()
* Introduction on 'Services' for the network config

"--chain bor-devnet --bor.localheimdall" will run a 2 validator network
with a local service
"--chain bor-devnet --bor.withoutheimdall" will sun a single validator
with no heimdall service as before

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-18 09:47:04 +01:00
Alex Sharov
368572f882
mdbx_to_mdbx: increase read-ahead threads amount (#7905) 2023-07-18 13:27:33 +07:00
Giulio rebuffo
735a800eb7
Added insertions and chainDb overlay (#7899) 2023-07-15 18:31:15 +02:00
Giulio rebuffo
814984dffb
Implementation of execution.proto: added getters (#7898) 2023-07-14 16:38:17 +02:00
Delweng
dec2f5a3b3
turbo/jsonrpc: eth_getBlockReceipts support by by hash (#7886)
Signed-off-by: jsvisa <delweng@gmail.com>
2023-07-14 08:57:09 +01:00
ledgerwatch
035a6f867c
Simplifications for bor tx code (#7889)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-14 08:49:10 +01:00
Giulio rebuffo
b2442a7b41
Removed unused stages (Cumulative index + Translation) (#7884) 2023-07-13 16:55:48 +02:00
Mark Holt
bd9896bf4b
added bor tx indexing with tests (#7826)
This request implements the insertion of Bor ephemeral transactions into
snapshot indexes.

I does this by taking the block hash from the header index and passing
it to the transaction indexer to add an additional index entry per block
into the transaction hash -> block index.

The passed entries are currently contained in an in memory array which
is (32 * number of blocks / sprint size) bytes.

In addition to the functional code there is also an update to the
`dump_test.go` so that it runs `DumpBlocks` to exercise the indexing
code. To facilitate this the `InsertChain` method in `mock_sentry` has
been modified so that it can process >128 blocks.

The code in this request also includes additional bor/consensus code
with the following functions:

`CalculateSprint`
`CalculateSprintCount`

The first function is a modification of the code in erigon-lib so that
the sprints are numerically rather than lexically ordered. This code
should be migrated to erigon-lib and should have its sprint set
calculated once from its underlying map rather than this process being
repeated every calculation.

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Enrique Jose  Avila Asapche <eavilaasapche@gmail.com>
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-07-12 23:31:38 +01:00
Giulio rebuffo
bd63cb7c8c
Removed GRPC layer from Engine API (#7878) 2023-07-12 18:11:41 +02:00
Giulio rebuffo
6272559fb7
Separated PendingBlock behaviour to be chain agnostic (#7859)
Instead of getting the pending block with latest payload id, we just
store the latest block built and serve it in remote backend
2023-07-10 19:22:03 +02:00
ledgerwatch
a9c86ab887
[devnet] Always select first node, fix configuration mess up between nodes (#7863)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-07-10 11:37:55 +01:00
Alex Sharov
bf8ac9a38f
mdbx_to_mdbx: to use logger (#7860) 2023-07-09 08:04:20 +01:00
Giulio rebuffo
20b8e156db
moved cmd/rpcdaemon/commands -> turbo/jsonrpc (#7858) 2023-07-08 19:01:26 +02:00
Enrique Jose Avila Asapche
bff1e0e901
[Caplin] Added json rpc cl (#7836)
Allows for Caplin to be used with other ELs from aside of Erigon

---------

Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-07-08 16:42:30 +02:00
Giulio rebuffo
84f31c873e
Separation of Engine from Ethbackend (#7821)
This PR separates ENGINE from Ethbackend. It makes it so:

1) EthBackend not a god class
2) We can abstract away engine API so that we can make it CL-like and
enable Consensus-Execution driven design
3) Objective is Json-RPC -> Engine Consensus Module -> Execution module.
2023-07-06 18:09:52 +02:00
Alex Sharov
69ae999059
log file unmap/close errors (#7819) 2023-07-06 11:12:26 +07:00
Mark Holt
5935b11b24
Devnet macos startup and windows committed memory startup fixes (#7832)
The fixes here fix a couple of issues related to devnet start-up

1. macos threading and syscall error return where causing multi node
start to both not wait and fail
2. On windows creating DB's with the default 2 TB mapsize causes the os
to reserve about 4GB of committed memory per DB. This may not be used -
but is reserved by the OS - so a default bor node reserves around 10GB
of storage. Starting many nodes causes the OS page file to become
exhausted.

To fix this the consensus DB's now use the node's OpenDatabase function
rather than their own, which means that the consensus DB's take notice
of the config.MdbxDBSizeLimit.

This fix leaves one 4GB committed memory allocation in the TX pool which
needs its own MapSize setting.

---------

Co-authored-by: Alex Sharp <akhounov@gmail.com>
2023-07-02 22:37:23 +01:00
Andrew Ashikhmin
1f7de0e5cf
Rename StageLoopStep to StageLoopIteration (#7820)
Rename for clarity to highlight that not only one stage is executed, but
the entire loop.
2023-06-29 17:53:23 +02:00
Andrew Ashikhmin
a2dd5da2a1
[erigon-lib] EIP-4844: parse transactions when not wrapped with blobs (#7805)
Pick up https://github.com/ledgerwatch/erigon-lib/pull/1034
2023-06-27 12:52:59 +02:00
Andrew Ashikhmin
a24eae8d6c
EIP-4844: Handle data gas in txpool (#7779)
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/1029
2023-06-23 11:10:23 +02:00
sonicWhale
1f9bec2557
enable cli RPCGlobalTxFeeCapFlag (#7775)
enable -rpc.txfeecap flag on the RPC daemon in the default sets of CLI
flags
2023-06-23 11:17:10 +07:00
Alex Sharov
f8cb4d6dbb
Fix subscription closing race (#7788)
`go test -p 1 -race -count=100 -run='TestFiltersDeadlock_Test'
./turbo/rpchelper/...`
2023-06-23 10:07:51 +07:00
Alex Sharov
ded166d73d
e3: release chain maker changes (#7787) 2023-06-23 10:07:42 +07:00
ledgerwatch
f8a5c8c6ce
Fix DumpTxs (#7782) 2023-06-22 12:08:16 +01:00
Mark Holt
051cad0fdd
Devnet diagnostics (#7762)
Added support tunnel to the devnet cmd. In order to get this to run I
made the following changes:

* Create a public function 
* Added non root logging

I have also added commentary to the readme to explain the additional
command line arguments needed to integrate with diagnostics. In summary,
if you set the --diagnostics.url the devenet will wait for diagnostic
requests rather than exiting

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-06-20 10:11:55 +01:00
Alex Sharov
a77edd7a32
faster opening of snapshots and indices (#7757)
- mostly by MADV_SEQUENTIAL during opening 
- see https://github.com/ledgerwatch/erigon-lib/pull/1025
2023-06-19 11:34:01 +07:00
Alex Sharov
d3c3be9c91
e2: optimize tests speed (#7738) 2023-06-15 16:09:11 +07:00
Andrew Ashikhmin
252e57b76f
Base BlobTx on DynamicFeeTransaction (#7736)
Continuation of PR #7715
2023-06-15 10:57:27 +02:00
Alex Sharov
e5023775aa
Enforce blockReader interface (#7737)
- breaks dependency from staged_sync to package with block_reader
implementation
- breaks dependency from snap_sync to package with block_reader
implementation
- breaks dependency from mining to txpool implementation
2023-06-15 13:11:51 +07:00
Mark Holt
f110102023
Devnet scenarios (#7723)
This is an update to the devnet code which introduces the concept of
configurable scenarios. This replaces the previous hard coded execution
function.

The intention is that now both the network and the operations to run on
the network can be described in a data structure which is configurable
and composable.

The operating model is to create a network and then ask it to run
scenarios:

```go
network.Run(
		runCtx,
		scenarios.Scenario{
			Name: "all",
			Steps: []*scenarios.Step{
				&scenarios.Step{Text: "InitSubscriptions", Args: []any{[]requests.SubMethod{requests.Methods.ETHNewHeads}}},
				&scenarios.Step{Text: "PingErigonRpc"},
				&scenarios.Step{Text: "CheckTxPoolContent", Args: []any{0, 0, 0}},
				&scenarios.Step{Text: "SendTxWithDynamicFee", Args: []any{recipientAddress, services.DevAddress, sendValue}},
				&scenarios.Step{Text: "AwaitBlocks", Args: []any{2 * time.Second}},
			},
		})
```
The steps here refer to step handlers which can be defined as follows:

```go
func init() {
	scenarios.MustRegisterStepHandlers(
		scenarios.StepHandler(GetBalance),
	)
}

func GetBalance(ctx context.Context, addr string, blockNum requests.BlockNumber, checkBal uint64) {
...
```
This commit is an initial implementation of the scenario running - which
is working, but will need to be enhanced to make it more usable &
developable.

The current version of the code is working and has been tested with the
dev network, and bor withoutheimdall. There is a multi miner bor
heimdall configuration but this is yet to be tested.

Note that by default the scenario runner picks nodes at random on the
network to send transactions to. this causes the dev network to run very
slowly as it seems to take a long time to include transactions where the
nonce is incremented across nodes. It seems to take a long time for the
nonce to catch up in the transaction pool processing. This is yet to be
investigated.
2023-06-14 12:35:22 +01:00
Andrew Ashikhmin
862d7b6bc6
Move ChainID from CommonTx to DynamicFeeTransaction (#7732)
For legacy transactions ChainID is optional (missing in
pre-[EIP155](https://eips.ethereum.org/EIPS/eip-155) transactions) and
is derived from `V` anyway.

Also, cherry pick https://github.com/ethereum/go-ethereum/pull/27452.
2023-06-14 11:44:05 +02:00
Alex Sharov
1b14785d79
ws unfreeze (#7724)
attempt to address next issue: 

> when I'm having a lot of websocket connections the node is freezing
and then it needs like 10 mins to sync. Then if I keep pushing requests
it falls out of sync all the time
2023-06-14 16:25:05 +07:00
racytech
8b312d54e8
eip-4844: desszing blobtx: initial commit (#7715) 2023-06-14 11:01:03 +02:00
Alex Sharov
5ff8246e2a
e4: release some tests (#7729) 2023-06-14 10:01:00 +07:00
Alex Sharov
ae1b193d6f
use 1 constant in tests to configure initialCycle variable (#7720) 2023-06-13 10:55:59 +07:00
Mark Holt
e99269ebce
added --log.dir.prefix flag (#7714)
This request adds an additional logging flag to change the name of the
logfiles produced by erigon to be prefixed by a name other than
'erigon'.

This allows multiple nodes to log to the same directory without
overwriting each others files. It is requires so that the devnet when
running can maintain all of its log files in a single consolidated
directory which survives devnet restarts.

---------

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-06-12 15:30:05 +01:00
Alex Sharov
31be6f6885
StagedSync: fix canRunCycleInOneTransaction logic (#7712) 2023-06-12 16:02:28 +07:00
Alex Sharov
afd8ea94d0
e3: move domains to tables.go (#7707) 2023-06-11 21:12:05 +07:00
Mark Holt
1e575ea172
rpc service and stageloop logger updates (#7696)
This is another update to logging to replace the root logger with a
contextual logger
2023-06-10 07:39:39 +01:00
Alex Sharov
afa4f53ae2
e3: use deserializeV3 for history (#7691) 2023-06-09 10:53:04 +07:00
Andrew Ashikhmin
fa0cfcbec7
Remove the Rinkeby testnet (#7677)
See https://github.com/ethereum/go-ethereum/pull/27406
2023-06-07 07:42:36 +02:00
Alex Sharov
bf9f5067f3
fix integration tests run (#7672) 2023-06-06 13:49:01 +07:00
Alex Sharov
e14664d53b
Store Canonical/NonCanonical/Bad blocks/senders/txs in same tables. Make re-org cheaper (#7648)
- allow store non-canonical blocks/senders
- optimize re-org: don't update/delete most of data
- allow mark chain as `Bad` - will be not visible by eth_getBlockByHash,
but can read if have hash+num
2023-06-05 12:36:24 +07:00
Alex Sharov
895e7dd96a
cleanup (#7655) 2023-06-04 10:37:20 +07:00
Alex Sharov
157b6de8e3
release small rename (#7647) 2023-06-03 17:26:20 +07:00
Alex Sharov
63c92010cd
remove txsV3 cli flag (#7644) 2023-06-03 15:54:27 +07:00
Alex Sharov
ad72b7178e
prune speedup. stage_senders: don't re-calc existing senders (#7643)
- stage_senders: don't re-calc existing senders
- stage_tx_lookup: prune less blocks per iteration - because
random-deletes are expensive. pruning must not slow-down sync.
- prune data even if --snap.stop is set
- "prune as-much-as-possible at startup" is not very good idea: at
initialCycle machine can be cold and prune will cause big downtime, no
reason to produce much freelist in 1 tx. People may also restart erigon
- because of some bug - and it will cause unexpected downtime (usually
Erigon startup very fast). So, I just remove all `initialSync`-related
logic in pruning.
- fix lost metrics about disk write byte/sec
2023-06-03 12:30:53 +07:00
Alex Sharov
d40317c905
preparation for --txs.v3, step 2 (#7636) 2023-06-03 07:20:22 +07:00
Andrew Ashikhmin
190bc9dfd8
EIP-4844: add data_gas_used (#7639)
Implement https://github.com/ethereum/EIPs/pull/7062 and
https://github.com/ethereum/EIPs/pull/7095. Pick up
https://github.com/ledgerwatch/erigon-lib/pull/1006.
2023-06-02 21:26:19 +01:00
Andrew Ashikhmin
b2a90e0ea9
Merge EngineGetPayloadWithBlobs into EngineGetPayload (#7637)
Pick up https://github.com/ledgerwatch/interfaces/pull/171 &
https://github.com/ledgerwatch/interfaces/pull/173
2023-06-02 13:02:13 +02:00
Alex Sharov
6439bdd835
txsv3: release small parts (#7632) 2023-06-02 11:38:20 +07:00
Alex Sharov
436656a40f
snapshots: DumpBodies - break dependency on body.BaseTxNum value in db (#7607)
it's step towards saving canonical and non-canonical bodies in same
table (and txs also in same own table). to reduce write amplification
(cheaper re-orgs)

PR change: reading BaseTxNum from existing snapshots instead of DB 
DB will store in field body.BaseTxNum - non-canonical TxnID  
Snapshots will store only canonical TxNum in field body.BaseTxNum
2023-06-02 11:01:57 +07:00
Alex Sharov
b163d3a877
e3: release some e4 parts (#7629) 2023-06-02 10:55:40 +07:00
Alex Sharov
c6b12ed7ca
stageLoop: unbound canRunCycleInOneTransaction logic from initialCycle variable (#7616) 2023-06-01 16:50:19 +07:00
Alex Sharov
6c0b531eca
add "erigon snapshots diff" sub-command to find difference between 2 snapshots (#7619) 2023-06-01 16:29:26 +07:00
Alex Sharov
5c7da48331
blockReader: fix blockNum check logic (#7615) 2023-06-01 15:04:21 +07:00
Alex Sharov
299c81e522
move "makeBodiesCanonical" method to blockWriter (#7605) 2023-05-31 15:44:01 +07:00
Alex Sharov
3cba3908dd
use BlockReader.CanonicalHash method instead of rawdb (#7604) 2023-05-31 13:41:10 +07:00
Alex Sharov
af83845f7e
cheaper isCanonicalHash func, less lookups of blockHash by blockNum in rpc (#7603) 2023-05-31 11:26:38 +07:00
Anshul Yadav
1fb053f260
Header downloader monitor for diagnostics system (#7590)
## What's this PR about?

- Added states to be sent to diagnostics system for header downloader
monitor
- Added the code for sending the states through the tunnel
- Code added for updating the states in the header_algos.go file

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-05-29 14:18:31 +07:00
Alex Sharov
279e1bec33
use blockReader as service-provider of RoSnapshots (#7584) 2023-05-26 17:12:33 +07:00
Alex Sharov
cfafa24481
tests for blocks snapshots creation data producer (#7579) 2023-05-25 14:58:31 +07:00
Alex Sharov
111db5f655
blockReader in tests - step8 (#7578) 2023-05-25 12:46:11 +07:00
Alex Sharov
a79033cef8
blockReader in tests, step6 (#7573) 2023-05-24 18:34:36 +07:00
Alex Sharov
31687be599
blockReaders in tests, step4 (#7570) 2023-05-24 15:52:51 +07:00
Alex Sharov
179b138000
add blockReader to mining, blockchain_test (#7569) 2023-05-24 13:53:22 +07:00
Alex Sharov
23bd14744b
blockReader: use in ethstat (#7565) 2023-05-23 16:30:47 +07:00
Alex Sharov
696ccb6edd
add constant to run tests on initialSync=true (#7563) 2023-05-23 14:57:15 +07:00
Alex Sharov
63afe65686
Use BlockReader in ForkValidator, CliqueAPI (#7562) 2023-05-23 14:49:17 +07:00
Alex Sharov
18990ff91a
e2: reduce StageSyncStep interface size (#7561) 2023-05-23 12:28:31 +07:00
Alex Sharov
4d0dee6fb0
Introduce BlockWriter object (txsV3 step 0) (#7559) 2023-05-22 15:49:21 +07:00
Alex Sharov
2865b85888
move e2 snapshots management closer to e3: step 1 (#7543)
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 10:09:46 +07:00
Anshul Yadav
2c194e10a1
Args usage msg bug fix (#7554)
## What's this PR is about?
Minor fix in args usage message of support flag. The current message
says that the flag should be 'metrics.url' but it reality it should be
'metrics.urls'
2023-05-20 21:57:23 +01:00
ledgerwatch
2a872b4d54
[devnet] separare logging - headers download (#7551)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 07:00:19 +01:00
ledgerwatch
b0117a7c30
[devnet] separate logging - p2p (#7547)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-19 18:41:53 +01:00
ledgerwatch
b382f96f6c
Introduce logger into etl (#7537)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 21:20:07 +01:00
ledgerwatch
05597cb195
[devnet tool] separare logging (#7531)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 17:36:15 +01:00
Alex Sharov
c0096ee8c8
StageLoopStep: if node synced - then run initialCycle in 1 tx also (for data consistency) (#7532) 2023-05-17 13:39:21 +01:00
ledgerwatch
e75ea786c0
[devnet tool] separate logging (#7526)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 07:36:06 +01:00
ledgerwatch
3f9ae3ec77
[devnet tool] separate logging (#7525)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-16 10:53:50 +01:00
Alex Sharov
b3aca15ff8
e2: ReadAhead of blocks, senders accounts, code (#7501)
It improves performance of initial sync (stage exec) by 5-20% when
blocks snapshots are mounted to high-latency drive and when chaindata is
on high-latency drive. And improving cold-start performance.

Current implementation using 2 goroutines for ReadAhead. It also
producing more garbage, can improve it later (here are dashboard with
impact).
```
mainnet2-1: with ReadAhead
mainnet2-3: no ReadAhead
```

<img width="949" alt="Screenshot 2023-05-12 at 09 24 31"
src="https://github.com/ledgerwatch/erigon/assets/46885206/b90b1fa8-9099-48ff-95b3-86e864a36d46">

<img width="845" alt="Screenshot 2023-05-12 at 09 24 13"
src="https://github.com/ledgerwatch/erigon/assets/46885206/39d90c0c-a9d5-4735-8c03-da1455b147aa">
2023-05-15 15:08:35 +07:00
ledgerwatch
bc169179b0
[devnet tool] Side-quest: logging step 7 (#7506)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-13 21:22:30 +01:00
ledgerwatch
5ae9cb7f39
[Devnet tool] side-quest logging step 6 (#7496)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-11 18:59:55 +01:00
ledgerwatch
6ef3fc3a3c
[devnet tool] Side-quest logging, step 5 (#7484)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 19:36:27 +01:00
ledgerwatch
20437faaf5
[Devnet tool] side-quest logging step 4 (#7481)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 17:12:23 +01:00
racytech
42e8db3958
eip-4844: modified DecodeTransaction version 1 (#7442)
Blob transactions are SSZ encoded, so it had to be added to decoding.
There are 2 encoding forms: `network` and `minimal` (usual). Network
encoded blob transactions include "wrapper data" which are `kzgs`,
`blobs` and `proofs`, and decoded by `DecodeWrappedTransaction`. For
previous types of transactions the network encoding is no different.
Execution-payloads / blocks use the minimal encoding of transactions. In
the transaction-pool and local transaction-journal the network encoding
is used.

Concerns: 
1. Possible performance reduction caused by these changes, not sure if
streams are better then slices. Go slices in this modifications are
read-only, so they should be referred to the same underlying array and
passed by a reference.
2. If `DecodeWrappedTransaction` and `DecodeTransaction` will create
confusion and should be merged into one function.
2023-05-09 18:44:53 +01:00
ledgerwatch
f38ec1e772
[devnet tool] side-quest: logging, step 3 (#7471)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-09 18:11:31 +01:00
Andrew Ashikhmin
f3144a6ed0
Proper Gnosis Chain rewards in trace_block (#7473) 2023-05-09 17:19:23 +02:00
ledgerwatch
3c1448afed
[devnet tool] Side-quest logging - replace quiet parameter (#7464)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-08 17:52:31 +01:00
Andrew Ashikhmin
cc11462860
Prioritize eth/68 by default (#7463)
Set `DefaultConfig.ProtocolVersion` to [68, 67, 66] instead of [67, 68].
See https://github.com/ethereum/hive/pull/776
2023-05-08 16:03:59 +02:00
Alex Sharov
1a5c019a2a
tests: less output (#7446) 2023-05-07 14:48:55 +07:00
ledgerwatch
fdd385cef1
[Devnet tool] Side-quest to improve logging - part 1 (#7445)
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-07 07:28:15 +01:00
alex.sharov
dcd376c949 tests: less output 2023-05-05 14:02:09 +07:00
alex.sharov
557c35339d tests: less output 2023-05-05 14:01:58 +07:00
alex.sharov
061d3ff1a4 fix cli metrics flag 2023-05-05 09:36:02 +07:00
alex.sharov
15eb9ea957 fix cli metrics flag 2023-05-05 09:33:47 +07:00
racytech
e751b9d219
eip-4844: minor additions (#7443) 2023-05-05 09:24:45 +07:00
a
30430d585a
begin refactor of beacon state (#7433)
this first major move separates the transient beacon state cache from
the underlying tree.

leaf updates are enforced in the setters, which should make programming
easier.

all exported methods of the raw.BeaconState should be safe to call
(without disrupting internal state)

changes many functions to consume *raw.BeaconState in perparation for
interface


beyond refactor it also:

adds a pool for the leaves of the validator ssz hash 

adds a pool for the snappy writers
  
removed the parallel hash experiment (high memory use)
2023-05-04 15:18:42 +02:00
Alex Sharov
268674cdde
e4: tests compat (#7436) 2023-05-04 11:30:21 +07:00
racytech
40947f6c98
eip-4844: adding data_gas to gaspool (#7428)
Adding `data_gas` to gas pool. EIP-4844 gas pool includes data_gas which
is used to fee blob transactions.
2023-05-03 09:02:30 +07:00
alex.sharov
b53c2b1775 comment out "erigon alpha_backup" cmd 2023-05-02 13:06:22 +07:00
Andrew Ashikhmin
1533674dad
Implement EIP-1153 transient storage (#7405)
Port https://github.com/ethereum/go-ethereum/pull/26003

---------

Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
2023-05-01 18:26:24 +02:00
naviechan
d475bab15f
Add diagnostics endpoint for flags (#7417)
To expose context flags through diagnostic endpoint
`/debug/metrics/flags`.

The current `/debug/metrics/cmdline` only provides the command run by
user. In the case when user runs Erigon using config file, `cmdline`’s
info does not truly reflect the ‘launch setting' and flags set that
Erigon is running on.

## Example

### Command
```
erigon --datadir /tmp/data --config test.yaml
```
### Pseudo config file (in yaml)
```
datadir : '/tmp/data'
chain : "sepolia"
http : true
metrics: true
private.api.addr : "localhost:9090"

http.api : ["eth","debug","net"]
```
### Output
```
SUCCESS
chain=sepolia
config=test.yaml
datadir=/tmp/data
http=true
http.api=eth,debug,net
metrics=true
private.api.addr=localhost:9090
```
2023-05-01 15:38:00 +01:00
alex.sharov
6cfd2cec61 erigon backup: docs 2023-05-01 09:27:35 +07:00
Alex Sharov
76580441b8
new command "erigon alpha_backup" - an alpha version of command to all Erigon's databases (#7415) 2023-05-01 08:27:34 +07:00
racytech
726ce264ef
eip-4844: RPCTransactions to support BlobTx (#7407)
Small additions to support eip-4844 transaction marshalling
2023-04-30 14:03:38 +07:00
Alex Sharov
ded8283df7
erigon backup: v0 of sub-command (#7396) 2023-04-27 10:42:12 +07:00
Jason Yellick
4e9b378a5d
Enable negative Merkle proofs for eth_getProof (#7393)
This addresses the last known deficiency of the eth_getProof
implementation. The previous code would return an error in the event
that the element was not found in the trie. EIP-1186 allows for
'negative' proofs where a proof demonstrates that an element cannot be
in the trie, so this commit updates the logic to support that case.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-27 10:38:45 +07:00
ledgerwatch
d002861f5f
[Diagnostics] fix erigon support connection to diagnostics system (#7391) 2023-04-26 15:57:39 +01:00
Jason Yellick
434ac76f79
Add additional trie proof testing (#7382)
This PR extends the function merged in #7337 to cover the proof
generation paths as well.

The first commit simply migrates the internal proof checking code from
the rpc `eth_getProof` test to be available externally exported under
`turbo/trie`. In the process, it translates the in place testing
assertions to return more normally as errors and adapts to use the
pre-existing trie node types which are defined slightly differently
(although this code is still intended only for testing purposes).

The second commit refactors the existing trie fuzzing tests from the
previous PR and adds new fuzzing tests for the proof generation. In
order to validate the proofs, these fuzzing tests fundamentally do two
things. Firstly, after bootstrapping the test (by seeding, and modifying
the db), for each key we compute the 'naive' proof utilizing the
existing code in `proof.go` and the in memory `trie.Trie` structure. The
`trie.Trie` code actually had a couple small bugs which are fixed in
this PR (not handling value nodes, and not honoring the `NewTestRLPTrie`
contract of pre-RLP encoded nodes in proof generation). Secondly, we
re-compute the same proof for the flatDB production variant, and verify
that it is exactly the same proof as computed by the naive
implementation.

This fuzzing has been run for ~72 hours locally with no errors. Although
this code hasn't identified any new bugs in the proof generation path,
it improves coverage and should help to prevent regressions. Additional
extensions will be provided in a subsequent PR.

---------

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-26 09:33:46 +07:00
Jason Yellick
23d5c7c47f
Fix bugs in trie hash computation (#7337)
There are currently a number of bugs in the flat DB trie hash
computation. These bugs are improbable in 'real' scenarios (hence the
ability to sync the assorted large chains), and, the repercussions of
encountering them are low (generally re-computing the hash at a later
block will succeed). Still, they can cause the process to crash, or
deadlock, and a clever adversary could feasibly construct a block or
storage update designed to trigger these bugs. (Or, an unlucky block may
trigger them inadvertently). Based on the tracing in the code, it seems
that some of these bugs may have been witnessed in the wild, but not
reliably reproduced (for instance when `maxlen >= len(curr)`).

1. There is an infinite loop that can occur in
_nextSiblingOfParentInMem. This occurs in the account path when the
c.k[1] entry is 'nil' and the code will continuously retry resolving
this entry indefinitely.

2. When the next trie node is deeper in the trie than the previous, but
is not a descendent of the previous trie node, then the old trie node is
inadvertently left in memory instead of nil-ed. This results in an
incorrect hash being computed.

3. When the last trie node being processed is comprised entirely of 'f'
nibbles, the 'FirstNotCoveredPrefix' returns an empty byte array,
because there is no next nibble sub-tree. This causes keys to
inappropriately be reprocessed triggering either an index out of bounds
panic or incorrect hash results.

4. When the _nextSiblingInDB path is triggered, if the next nibble
subtree contains no trie table entries, then any keys with a prefix in
that subtree will be skipped resulting in incorrect hash results.

The fuzzing seeds included cover all four of these cases (for both
accounts and storage where appropriate). As fuzzing is baked into the
native go 1.18 toolchain, running 'go test' (as the Makefile currently
does) is adequate to cover these failures. To run additional fuzzing the
`-fuzz` flag may be provided as documented in the go test doc.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-25 09:25:29 +07:00
ledgerwatch
d70c9f0979
[Diagnostics] initial support for body downloader visualisation (#7373)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-04-23 18:56:37 +01:00
racytech
6588bca40b
eip-4844: NewMessage now expectes maxFeePerDataGas & GetPayloadV3 impl (#7365)
types.NewMessage now expects maxFeePerDataGas param, which will be used
in transaction verification (preCheck). GetPayloadV3 method added to
EngineAPI. Some cosmetic changes applied.
2023-04-23 18:27:05 +01:00
Alex Sharov
e1f549ff01
cli: use SplitAndTrim instead of strgings.Split(",") (#7369) 2023-04-23 10:54:55 +07:00
Alex Sharov
b14ca7508f
e3: remove garbage files (#7357) 2023-04-21 09:51:03 +07:00
Jason Yellick
7f31b047f1
Fix eth_getProof element order (#7351)
According to EIP-1186 the `proof` parts of the response to eth_getProof
should be returned "starting with the stateRoot-Node, following the path
of the SHA3 (address) as key." Currently, the proof is returned in
traversal order, rather than from the root.

Although all of the proof elements are there and correct, this is
contrary to the EIP and will cause problems for some clients. The
existing rpc test uses a map to lookup proof elements by hash, rather
than by index, so this bug was not initially caught.

This commit fixes the behavior, updates the existing test, and adds
additional checks to the rpc test.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-20 10:06:19 +07:00
Giulio rebuffo
2ce5e761bb
added --internalcl flag to Erigon (#7349) 2023-04-19 14:37:35 +02:00
Alex Sharov
21d66d6c01
e3: state reader constructor in tests (#7338) 2023-04-19 03:10:33 +00:00
Andrew Ashikhmin
3f60a902d4
Fix nil withdrawals in eth_getBlockByHash (#7332) 2023-04-18 11:23:22 +00:00
Alex Sharov
558f5bdfdd
e3: less use testTx method, more sentryMock (to support e4) (#7329) 2023-04-18 04:15:01 +00:00
Alex Sharov
69a3396433
add flag --db.size.limit (#7325) 2023-04-17 12:48:57 +00:00
Alex Sharov
62d1e8a66c
e3: in-general merge must not see "overlaps/deleted" files, but merge of history need access to corresponding index files (even if they marked as deleted or already merged - before kill -9) (#7321) 2023-04-17 05:54:29 +00:00
racytech
7aa217f015
eip-4844: setting header's excess_data_gas, verifiying eip-4844 headers (#7308)
In this PR Header's ExcessDataGas is set to the actual value, but it's
still unused. It will be used to compute data fee for eip-4844 data
blobs, logic of which will be added in later PRs. Also eip-4844 header
verification logic added.
2023-04-16 08:12:40 +00:00
Andrew Ashikhmin
02f6cac7b7
Move hexutil.Bytes to erigon-lib (#7305) 2023-04-13 11:19:02 +00:00
ledgerwatch
a41ad2d94b
[Diagnostics] introduce versioning and improve usability of support c… (#7300)
…ommand

---------

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-04-12 21:23:49 +00:00
Andrew Ashikhmin
9387abf66b
[interfaces] lower_snake_case field names (#7293)
Pick up https://github.com/ledgerwatch/interfaces/pull/163
2023-04-12 10:35:54 +00:00
racytech
47fd86e4fb
eip-4844: assigned value to excessDataGas, unused packages removed (#7289)
`excessDataGas` has been partially made eip-4844 ready, so instead of
passing nils to functions, now it actually assigned to some value (it is
expected to be nil until cancun update).
2023-04-12 05:45:44 +00:00
Andrew Ashikhmin
7b576e9fda
Revert "return [] instead of null for withdrawals (#7279)" (#7292)
Pre-Shanghai blocks should have `nil` withdrawals, while post-Shanghai
blocks should have non-nil withdrawals (empty or non-empty slice, but
not `nil`). Judging from Issue #6976, that's not always a case. PR #7279
attempted to fix the issue, but unfortunately it only masks the root
cause.

This reverts commit c60a6a2962.
2023-04-11 13:39:41 +00:00
ledgerwatch
db52bbafb3
[Diagnostics] Re-enable log support, more reliable reconnection (#7286)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-04-10 07:35:24 +00:00
Giulio rebuffo
c60a6a2962
return [] instead of null for withdrawals (#7279) 2023-04-07 21:09:11 +00:00
ledgerwatch
9690228ede
[Diagnostics] Simplify logging settings, introduce correct log rotation with lumberjack (#7273)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-04-07 21:08:44 +00:00
ledgerwatch
3d904d509e
[Diagnostics] expose command line args via metrics (#7271)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-04-06 15:34:06 +00:00
ledgerwatch
56bc16b7ab
[Diagnostics] reconnect to diagnostics server when session is timed out (#7263)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-04-05 14:39:54 +00:00
Alex Sharov
64e7e788b5
e3: print background indexing logs in e2 style (summary of all files progress in 1 line) (#7256) 2023-04-05 04:49:02 +00:00
Jason Yellick
80530e10a9
Add storage proof support to eth_getProof (#7202)
This PR completes the implementation of `eth_getProof` by adding support
for storage proofs.

Because storage proofs are potentially overlapping, the existing
strategy of simply aggregating all proofs together into a single result
was challenging. Instead, this commit rewires things to introduce a
ProofRetainer, which aggregates proofs and their corresponding nibble
encoded paths in the trie. Once all of the proofs have been aggregated,
the caller requests the proof result, which then iterates over the
aggregated proofs, placing each into the relevant proof array into the
result.

Although there are tests for `eth_getProof` as an RPC and for the new
`ProofRetainer` code, the code coverage for the proof generation over
complex tries is lacking. But, since this is not a new problem I'll plan
to follow up this PR with an additional one adding more coverage into
`turbo/trie`.

---------

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-04-05 03:01:31 +00:00
Alex Sharov
a42d362cbd
move aura epoch data from chainDB to auraDB. remove epochReader parameter from consensus interface (#7250) 2023-04-04 03:30:07 +00:00
ledgerwatch
7258a2b872
Remove BSC support in Erigon (step 1) (#7246)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-04-03 14:46:21 +00:00
ledgerwatch
3a740ee0f2
Diagnostics system - step 3 (#7231)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-03-31 15:17:59 +00:00
Alex Sharov
a8e8bf4528
remove simd lib, because it doesn't work with ghcr.io/goreleaser/goreleaser-cross (which producing release binaries) (#7229)
@shyba hi, seems this lib doesn't work with
ghcr.io/goreleaser/goreleaser-cross (which producing release binaries)
removing it for now, feel free to add it in future - if can make it work
with goreleaser-cross
see: https://github.com/ledgerwatch/erigon/issues/7210
2023-03-31 05:07:43 +00:00
ledgerwatch
fac796d921
Diagnostics - step 2 (#7214)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-03-30 20:59:03 +00:00
racytech
04b57904b0
eip-4844: excessDataGas param added to block execution functions (#7217)
Small addition to block execution functions. Now they expect
excessDataGas param so the BlockContext can be created with it.
2023-03-30 20:42:43 +00:00
Alex Sharov
f541f616e6
e3: one more reconst deadlock fix (#7207) 2023-03-29 07:32:31 +00:00
Alex Sharov
417a437584
Break dependency of ethcfg package to core/consensus/etc... move genesis struct to 'types' package (#7206) 2023-03-29 07:27:06 +00:00
racytech
975e38a800
eip-4844: NewEVMBlockContext now expects excessDataGas (#7203)
Small change in core.NewEVMBlockContext and now it expects
excessDataGas. This will be used in state transition to compute data fee
for eip-4844 data blobs. The logic that computes it will be added in the
next PRs.
2023-03-29 06:39:36 +00:00
racytech
70bc7f7a25
eip-4844: small additions and modified gaspool (#7190)
This PR contains very small EIP-4844 additions. GasPool is modified and
now it is a struct with 2 fields "gas" and "dataGas" (blobs are priced
in dataGas). ExcessDataGas block header field added. ExcessDataGas
needed to compute the data gas price. EIP-4844 helper functions are
added as well.
2023-03-27 21:39:15 +00:00
Alex Sharov
201572c6f5
enable more linters #954 (#7179) 2023-03-25 05:13:27 +00:00
Alex Sharov
9001f668a7
go 1.19 atomics (#7164) 2023-03-23 05:11:28 +00:00
Jason Yellick
a09541c82c
Add eth_getProof support for historical blocks (#7115)
This PR starts with a few small commits of code cleanup. Reviewed
separately they should hopefully obviously be functionally no-ops. I'm
happy to strip these out and submit them separately if desired.

The final commit is to add support for older blocks as a parameter to
eth_getProof. In order to compute proofs, the function leverages the
staged sync unwinding code to bring the hashed state table back to its
historic state, as well as to build a list of trie nodes which need to
be invalidated/re-computed in the trie computation. Because these
operations could be expensive for very old blocks, it limits the
distance proofs are allowed from the head. It also adds some additional
checks for correctness, as well as tests which verify the
implementation.

This was discussed a bit on Discord in the db-format topic.

---------

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-03-23 03:22:33 +00:00
Temirlan
4e5dce1365
add flag txpool.commit_every (#7062)
Adds flag _--txpool.commit_every_ to control how often transactions
should be committed to the storage

[Related Issue](https://github.com/ledgerwatch/erigon/issues/7002)
2023-03-22 03:53:47 +00:00
Andrew Ashikhmin
56a3844656
Drop Default from GenesisBlock* functions (#7147)
Small cosmetic changes and clean-ups
2023-03-20 15:44:22 +00:00
Alex Sharov
157a380be7
e3: history no auto-increment (#7097) 2023-03-15 08:03:57 +00:00
hexoscott
4f91dff773
fix for reading yaml/toml config in cmd/integration (#7101) 2023-03-14 08:33:59 +00:00
Alex Sharov
bbe56620a3
move more parts to lru2 (#7098) 2023-03-14 07:37:23 +00:00
Victor Shyba
158fb2b606
Optimize memory buffer, simplify set32, use sha256-simd (#7060)
Hi,

I'm syncing Gnosis on a Celeron N5100 to get familiar with the codebase.
In the process I managed to optimize some things from profiling.
Since I'm not yet on the project Discord, I decided to open this PR as a
suggestion. This pass all tests here and gave me a nice boost for that
platform, although I didn't have time to benchmark it yet.

* reuse VM Memory objects with sync.Pool. It starts with 4k as `evmone`
[code
suggested](0897edb001/lib/evmone/execution_state.hpp (L49))
as a good value.
* set32 simplification: mostly cosmetic
* sha256-simd: Celeron has SHA instructions. We should probably do the
same for torrent later, but this already helped as it is very CPU bound
on such a low end processor. Maybe that helps ARM as well.
2023-03-14 07:17:04 +00:00
hexoscott
efd541028c
read metrics config from yaml file (#7089) 2023-03-14 00:07:05 +00:00
Jason Yellick
b21569ccc5
Backfill eth getproof tests (#7092)
This PR adds missing tests for eth_getProof and does some mild
refactoring with switching from strings to more strict types. It's
likely best/most easily reviewed commit by commit.

Note, the tests include quite a number of helper types and functions for
doing the proof validation. This is largely because unlike Geth,
Erigon's approach to trie computations only requires serializing the
trie nodes, not deserializing them. Consequently, it wasn't obvious how
to leverage the existing trie code for doing deserialization and proof
checks. I checked on Discord, but, there were no suggestions. Of course,
any feedback is welcome and I'd be happy to remove this code if it can
be avoided.

Additionally, I've opted to change the interface type for `GetProof` to
use a `common.Hash` for the storage keys instead of a `string`. I
_think_ this should be fairly safe, as until very recently it was
unimplemented. That being said, since it's an interface, it has the
potential to break other consumers, anyone who was generating mocks
against it etc. There's additionally a `GetStorageAt` that follows the
same parameter. I'd be happy to submit a PR modifying this one as well
if desired.

Also, as a small note, there is test code for checking storage proofs,
but, storage proofs aren't currently supported by the implementation. My
hope is to add storage proofs and historic proofs in a followup PR.

---------

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-03-14 00:04:52 +00:00
Jason Yellick
aec1dcdeef
Delete retain_list_builder.go (#7096)
I was walking the code to try to understand in a bit more detail how the
state root hash is constructed and stumbled across
`retain_list_builder.go` as a consumer of the retain list APIs. But, as
far as I can tell, this file doesn't seem to actually be used anywhere
(including tests), and, it's seen no development (other than import
fixes) since 2020 or so. All linting and tests still pass for me locally
without it, so, I believe it's safe to simply remove.

Co-authored-by: Jason Yellick <jason@enya.ai>
2023-03-14 00:02:07 +00:00
Alex Sharov
4ab63db6e2
nil ptr in delete ancient (#7081) 2023-03-11 05:46:32 +00:00
Michael Montour
d5fbc5011e
Partial EIP1186 eth_getProof implementation (#6560)
This is a partial implementation of eth_getProof (see issue #1349),
supporting only a request for the latest block and an empty list of
storage keys (i.e. Account proof only). I don't know if there's a better
way of implementing this, but this was what I could come up with.
Posting it here in case it's useful.

Example output:
```
> eth.getProof("0x67b1d87101671b127f5f8714789C7192f7ad340e", [], 'latest')
{
  accountProof: ["0xf90131a0252c9d4ed347b4cf3fdccaea3ccef0a507e6bd4dbe4dcd98609b7195347c4062a0ab8cdb808c8303bb61fb48e276217be9770fa83ecf3f90f2234d558885f5abf18080a01a697e814758281972fcd13bc9707dbcd2f195986b05463d7b78426508445a04a0b5d7a91be5ee273cce27e2ad9a160d2faadd5a6ba518d384019b68728a4f62f4a0c2c799b60a0cd6acd42c1015512872e86c186bcf196e85061e76842f3b7cf86080a0e73919d9f472eec11f6da95518503f5527a98b9428f7a02c4f55bf51854214e480a06301b39b2ea8a44df8b0356120db64b788e71f52e1d7a6309d0d2e5b86fee7cb8080a01b7779e149cadf24d4ffb77ca7e11314b8db7097e4d70b2a173493153ca2e5a0a066a7662811491b3d352e969506b420d269e8b51a224f574b3b38b3463f43f0098080", "0xf8518080808080a0a00135c9ec2655cb6a47ab7ad27d6fc150d9cba8b3d4a702e879179116a68a60808080808080a02fb46956347985b9870156b5747712899d213b1636ad4fe553c63e33521d567a80808080", "0xf873a02056274a27dd7524955417c11ecd917251cc7c4c8310f4c7e4bd3c304d3d9a79b850f84e808a021e19e0c9bab2400000a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470"],
  address: "0x67b1d87101671b127f5f8714789c7192f7ad340e",
  balance: "0x21e19e0c9bab2400000",
  codeHash: "0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470",
  nonce: "0x0",
  storageHash: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  storageProof: []
}
> eth.getBlock('latest').stateRoot
"0x6a0673c691edfa4c4528323986bb43c579316f436ff6f8b4ac70854bbd95340b"
```
2023-03-09 03:04:09 +00:00
Alex Sharov
92c444bc1a
--torrent.staticpeers (#7052) 2023-03-08 08:30:00 +00:00
alex.sharov
d748b686d3 save 2023-03-07 14:31:40 +07:00
Alex Sharov
d4e25daf22
etl: distinct empty values from nil (#7039)
Reverts ledgerwatch/erigon#7038
2023-03-07 04:19:00 +00:00
Alex Sharov
7206e482da
Revert "etl: distinct empty values from nil" (#7038)
Reverts ledgerwatch/erigon#6934
2023-03-07 03:22:23 +00:00
Alex Sharov
c5acfd0503
etl: distinct empty values from nil (#6934) 2023-03-07 02:33:21 +00:00
hexoscott
77065ee0f3
remove PeerUseless calls (#7032) 2023-03-06 18:59:14 +00:00
ledgerwatch
19cf4fedb6
Patch up erigon3 integration test (#7016)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-03-03 16:08:53 +00:00
ledgerwatch
7346c8fbb6
Extend mock_sentry to properly support forkValidator (#7008)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-03-03 07:00:54 +00:00
ledgerwatch
217ddc753d
Skeleton for unwind test, tidy up bodies cache (#7001)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-03-02 14:09:48 +00:00
ledgerwatch
d865b6405c
Fix nil pointer panic in tracing (#7000)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-03-02 10:34:29 +00:00
nanevardanyan
ab6239b30f
WIP: cmd, turbo, core, eth: TransactionsV3 flag and persist in new table (#6754) 2023-02-24 18:49:25 +00:00
ledgerwatch
fa1f5d4d9d
Improved test for intra_block_state (#6939)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-02-23 14:49:28 +00:00
ledgerwatch
9c9449a059
Mock of Erigon support - step 1 (#6930)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-02-22 12:38:22 +00:00
Stéphane Loeuillet
195a72bf74
Initial GraphQL interface implementation (#6821)
That's an initial PR mostly for code review, not ready for production
use

Got basic GraphQL working when querying a single block

---------

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-02-20 11:23:06 +00:00
Alex Sharov
c7a654cf05
E3: trace_filter to support of pos (#6909) 2023-02-20 09:42:40 +00:00
Anshal Shukla
7ce3f7b158
State sync transactions added to debug_traceBlockByNumber call (#6830)
State sync transactions can be enabled using `"borTraceEnabled": true`
parameter, this is to ensure that the current behaviour is unchanged.
2023-02-20 08:04:11 +00:00
Andrew Ashikhmin
e31678f355
Body downloader: fix empty withdrawals in empty bodies (#6893)
Potentially fixes Issue #6817.
2023-02-17 14:05:49 +00:00
Jochen Müller
4106a899d5
Chiado snapshots (#6886)
No change to `go.mod` since `erigon-snapshot` has already been updated
in `devel`.

I'll keep my Chiado node up to serve the snapshots for a while, until
there are more erigon nodes in the network.

Resolves #6307
2023-02-16 12:05:33 +00:00
Andrew Ashikhmin
58fcc04575
blockValue should use gasUsed rather than gasLimit (#6875)
Fees going to the gas fee recipient should be based on the actual gas
used (available in the receipt) rather than the gas limit in a
transaction. This fixes Hive test "GetPayloadV2 Block Value".

Also `engine_getPayloadBodiesByRangeV1` params should be encoded as hex
strings.
2023-02-14 15:05:27 +00:00
Alex Sharov
999899d66d
e3: read files list from db (#6833) 2023-02-13 05:17:01 +00:00
Alex Sharov
80b375b074
sentry: less sprintf in logs (#6851) 2023-02-12 11:23:58 +00:00
Alex Sharov
60a4b8bc7b
Snapshots Indexing: avoid loop with semaphore locking before logging (#6845) 2023-02-12 02:49:20 +00:00
ledgerwatch
ee83447fa1
CL lightclient to create memdb in the tmpdir which gets cleaned up on… (#6829)
… startup

---------

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-02-11 20:44:51 +00:00
Alex Sharov
983db55dc5
Downloader: "incomplete" markers are more important than "complete", so commit them with fsync (#6825) 2023-02-10 13:20:19 +07:00