Commit Graph

1336 Commits

Author SHA1 Message Date
Alex Sharov
436656a40f
snapshots: DumpBodies - break dependency on body.BaseTxNum value in db (#7607)
it's step towards saving canonical and non-canonical bodies in same
table (and txs also in same own table). to reduce write amplification
(cheaper re-orgs)

PR change: reading BaseTxNum from existing snapshots instead of DB 
DB will store in field body.BaseTxNum - non-canonical TxnID  
Snapshots will store only canonical TxNum in field body.BaseTxNum
2023-06-02 11:01:57 +07:00
Alex Sharov
b163d3a877
e3: release some e4 parts (#7629) 2023-06-02 10:55:40 +07:00
Alex Sharov
c6b12ed7ca
stageLoop: unbound canRunCycleInOneTransaction logic from initialCycle variable (#7616) 2023-06-01 16:50:19 +07:00
Alex Sharov
6c0b531eca
add "erigon snapshots diff" sub-command to find difference between 2 snapshots (#7619) 2023-06-01 16:29:26 +07:00
Alex Sharov
5c7da48331
blockReader: fix blockNum check logic (#7615) 2023-06-01 15:04:21 +07:00
Alex Sharov
299c81e522
move "makeBodiesCanonical" method to blockWriter (#7605) 2023-05-31 15:44:01 +07:00
Alex Sharov
3cba3908dd
use BlockReader.CanonicalHash method instead of rawdb (#7604) 2023-05-31 13:41:10 +07:00
Alex Sharov
af83845f7e
cheaper isCanonicalHash func, less lookups of blockHash by blockNum in rpc (#7603) 2023-05-31 11:26:38 +07:00
Anshul Yadav
1fb053f260
Header downloader monitor for diagnostics system (#7590)
## What's this PR about?

- Added states to be sent to diagnostics system for header downloader
monitor
- Added the code for sending the states through the tunnel
- Code added for updating the states in the header_algos.go file

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-05-29 14:18:31 +07:00
Alex Sharov
279e1bec33
use blockReader as service-provider of RoSnapshots (#7584) 2023-05-26 17:12:33 +07:00
Alex Sharov
cfafa24481
tests for blocks snapshots creation data producer (#7579) 2023-05-25 14:58:31 +07:00
Alex Sharov
111db5f655
blockReader in tests - step8 (#7578) 2023-05-25 12:46:11 +07:00
Alex Sharov
a79033cef8
blockReader in tests, step6 (#7573) 2023-05-24 18:34:36 +07:00
Alex Sharov
31687be599
blockReaders in tests, step4 (#7570) 2023-05-24 15:52:51 +07:00
Alex Sharov
179b138000
add blockReader to mining, blockchain_test (#7569) 2023-05-24 13:53:22 +07:00
Alex Sharov
23bd14744b
blockReader: use in ethstat (#7565) 2023-05-23 16:30:47 +07:00
Alex Sharov
696ccb6edd
add constant to run tests on initialSync=true (#7563) 2023-05-23 14:57:15 +07:00
Alex Sharov
63afe65686
Use BlockReader in ForkValidator, CliqueAPI (#7562) 2023-05-23 14:49:17 +07:00
Alex Sharov
18990ff91a
e2: reduce StageSyncStep interface size (#7561) 2023-05-23 12:28:31 +07:00
Alex Sharov
4d0dee6fb0
Introduce BlockWriter object (txsV3 step 0) (#7559) 2023-05-22 15:49:21 +07:00
Alex Sharov
2865b85888
move e2 snapshots management closer to e3: step 1 (#7543)
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 10:09:46 +07:00
Anshul Yadav
2c194e10a1
Args usage msg bug fix (#7554)
## What's this PR is about?
Minor fix in args usage message of support flag. The current message
says that the flag should be 'metrics.url' but it reality it should be
'metrics.urls'
2023-05-20 21:57:23 +01:00
ledgerwatch
2a872b4d54
[devnet] separare logging - headers download (#7551)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 07:00:19 +01:00
ledgerwatch
b0117a7c30
[devnet] separate logging - p2p (#7547)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-19 18:41:53 +01:00
ledgerwatch
b382f96f6c
Introduce logger into etl (#7537)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 21:20:07 +01:00
ledgerwatch
05597cb195
[devnet tool] separare logging (#7531)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 17:36:15 +01:00
Alex Sharov
c0096ee8c8
StageLoopStep: if node synced - then run initialCycle in 1 tx also (for data consistency) (#7532) 2023-05-17 13:39:21 +01:00
ledgerwatch
e75ea786c0
[devnet tool] separate logging (#7526)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 07:36:06 +01:00
ledgerwatch
3f9ae3ec77
[devnet tool] separate logging (#7525)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-16 10:53:50 +01:00
Alex Sharov
b3aca15ff8
e2: ReadAhead of blocks, senders accounts, code (#7501)
It improves performance of initial sync (stage exec) by 5-20% when
blocks snapshots are mounted to high-latency drive and when chaindata is
on high-latency drive. And improving cold-start performance.

Current implementation using 2 goroutines for ReadAhead. It also
producing more garbage, can improve it later (here are dashboard with
impact).
```
mainnet2-1: with ReadAhead
mainnet2-3: no ReadAhead
```

<img width="949" alt="Screenshot 2023-05-12 at 09 24 31"
src="https://github.com/ledgerwatch/erigon/assets/46885206/b90b1fa8-9099-48ff-95b3-86e864a36d46">

<img width="845" alt="Screenshot 2023-05-12 at 09 24 13"
src="https://github.com/ledgerwatch/erigon/assets/46885206/39d90c0c-a9d5-4735-8c03-da1455b147aa">
2023-05-15 15:08:35 +07:00
ledgerwatch
bc169179b0
[devnet tool] Side-quest: logging step 7 (#7506)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-13 21:22:30 +01:00
ledgerwatch
5ae9cb7f39
[Devnet tool] side-quest logging step 6 (#7496)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-11 18:59:55 +01:00
ledgerwatch
6ef3fc3a3c
[devnet tool] Side-quest logging, step 5 (#7484)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 19:36:27 +01:00
ledgerwatch
20437faaf5
[Devnet tool] side-quest logging step 4 (#7481)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 17:12:23 +01:00
racytech
42e8db3958
eip-4844: modified DecodeTransaction version 1 (#7442)
Blob transactions are SSZ encoded, so it had to be added to decoding.
There are 2 encoding forms: `network` and `minimal` (usual). Network
encoded blob transactions include "wrapper data" which are `kzgs`,
`blobs` and `proofs`, and decoded by `DecodeWrappedTransaction`. For
previous types of transactions the network encoding is no different.
Execution-payloads / blocks use the minimal encoding of transactions. In
the transaction-pool and local transaction-journal the network encoding
is used.

Concerns: 
1. Possible performance reduction caused by these changes, not sure if
streams are better then slices. Go slices in this modifications are
read-only, so they should be referred to the same underlying array and
passed by a reference.
2. If `DecodeWrappedTransaction` and `DecodeTransaction` will create
confusion and should be merged into one function.
2023-05-09 18:44:53 +01:00
ledgerwatch
f38ec1e772
[devnet tool] side-quest: logging, step 3 (#7471)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-09 18:11:31 +01:00
Andrew Ashikhmin
f3144a6ed0
Proper Gnosis Chain rewards in trace_block (#7473) 2023-05-09 17:19:23 +02:00
ledgerwatch
3c1448afed
[devnet tool] Side-quest logging - replace quiet parameter (#7464)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-08 17:52:31 +01:00
Andrew Ashikhmin
cc11462860
Prioritize eth/68 by default (#7463)
Set `DefaultConfig.ProtocolVersion` to [68, 67, 66] instead of [67, 68].
See https://github.com/ethereum/hive/pull/776
2023-05-08 16:03:59 +02:00
Alex Sharov
1a5c019a2a
tests: less output (#7446) 2023-05-07 14:48:55 +07:00
ledgerwatch
fdd385cef1
[Devnet tool] Side-quest to improve logging - part 1 (#7445)
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-07 07:28:15 +01:00
alex.sharov
dcd376c949 tests: less output 2023-05-05 14:02:09 +07:00
alex.sharov
557c35339d tests: less output 2023-05-05 14:01:58 +07:00
alex.sharov
061d3ff1a4 fix cli metrics flag 2023-05-05 09:36:02 +07:00
alex.sharov
15eb9ea957 fix cli metrics flag 2023-05-05 09:33:47 +07:00
racytech
e751b9d219
eip-4844: minor additions (#7443) 2023-05-05 09:24:45 +07:00
a
30430d585a
begin refactor of beacon state (#7433)
this first major move separates the transient beacon state cache from
the underlying tree.

leaf updates are enforced in the setters, which should make programming
easier.

all exported methods of the raw.BeaconState should be safe to call
(without disrupting internal state)

changes many functions to consume *raw.BeaconState in perparation for
interface


beyond refactor it also:

adds a pool for the leaves of the validator ssz hash 

adds a pool for the snappy writers
  
removed the parallel hash experiment (high memory use)
2023-05-04 15:18:42 +02:00
Alex Sharov
268674cdde
e4: tests compat (#7436) 2023-05-04 11:30:21 +07:00
racytech
40947f6c98
eip-4844: adding data_gas to gaspool (#7428)
Adding `data_gas` to gas pool. EIP-4844 gas pool includes data_gas which
is used to fee blob transactions.
2023-05-03 09:02:30 +07:00
alex.sharov
b53c2b1775 comment out "erigon alpha_backup" cmd 2023-05-02 13:06:22 +07:00