Commit Graph

1353 Commits

Author SHA1 Message Date
Alex Sharov
ae1b193d6f
use 1 constant in tests to configure initialCycle variable (#7720) 2023-06-13 10:55:59 +07:00
Mark Holt
e99269ebce
added --log.dir.prefix flag (#7714)
This request adds an additional logging flag to change the name of the
logfiles produced by erigon to be prefixed by a name other than
'erigon'.

This allows multiple nodes to log to the same directory without
overwriting each others files. It is requires so that the devnet when
running can maintain all of its log files in a single consolidated
directory which survives devnet restarts.

---------

Co-authored-by: Mark Holt <mark@distributed.vision>
2023-06-12 15:30:05 +01:00
Alex Sharov
31be6f6885
StagedSync: fix canRunCycleInOneTransaction logic (#7712) 2023-06-12 16:02:28 +07:00
Alex Sharov
afd8ea94d0
e3: move domains to tables.go (#7707) 2023-06-11 21:12:05 +07:00
Mark Holt
1e575ea172
rpc service and stageloop logger updates (#7696)
This is another update to logging to replace the root logger with a
contextual logger
2023-06-10 07:39:39 +01:00
Alex Sharov
afa4f53ae2
e3: use deserializeV3 for history (#7691) 2023-06-09 10:53:04 +07:00
Andrew Ashikhmin
fa0cfcbec7
Remove the Rinkeby testnet (#7677)
See https://github.com/ethereum/go-ethereum/pull/27406
2023-06-07 07:42:36 +02:00
Alex Sharov
bf9f5067f3
fix integration tests run (#7672) 2023-06-06 13:49:01 +07:00
Alex Sharov
e14664d53b
Store Canonical/NonCanonical/Bad blocks/senders/txs in same tables. Make re-org cheaper (#7648)
- allow store non-canonical blocks/senders
- optimize re-org: don't update/delete most of data
- allow mark chain as `Bad` - will be not visible by eth_getBlockByHash,
but can read if have hash+num
2023-06-05 12:36:24 +07:00
Alex Sharov
895e7dd96a
cleanup (#7655) 2023-06-04 10:37:20 +07:00
Alex Sharov
157b6de8e3
release small rename (#7647) 2023-06-03 17:26:20 +07:00
Alex Sharov
63c92010cd
remove txsV3 cli flag (#7644) 2023-06-03 15:54:27 +07:00
Alex Sharov
ad72b7178e
prune speedup. stage_senders: don't re-calc existing senders (#7643)
- stage_senders: don't re-calc existing senders
- stage_tx_lookup: prune less blocks per iteration - because
random-deletes are expensive. pruning must not slow-down sync.
- prune data even if --snap.stop is set
- "prune as-much-as-possible at startup" is not very good idea: at
initialCycle machine can be cold and prune will cause big downtime, no
reason to produce much freelist in 1 tx. People may also restart erigon
- because of some bug - and it will cause unexpected downtime (usually
Erigon startup very fast). So, I just remove all `initialSync`-related
logic in pruning.
- fix lost metrics about disk write byte/sec
2023-06-03 12:30:53 +07:00
Alex Sharov
d40317c905
preparation for --txs.v3, step 2 (#7636) 2023-06-03 07:20:22 +07:00
Andrew Ashikhmin
190bc9dfd8
EIP-4844: add data_gas_used (#7639)
Implement https://github.com/ethereum/EIPs/pull/7062 and
https://github.com/ethereum/EIPs/pull/7095. Pick up
https://github.com/ledgerwatch/erigon-lib/pull/1006.
2023-06-02 21:26:19 +01:00
Andrew Ashikhmin
b2a90e0ea9
Merge EngineGetPayloadWithBlobs into EngineGetPayload (#7637)
Pick up https://github.com/ledgerwatch/interfaces/pull/171 &
https://github.com/ledgerwatch/interfaces/pull/173
2023-06-02 13:02:13 +02:00
Alex Sharov
6439bdd835
txsv3: release small parts (#7632) 2023-06-02 11:38:20 +07:00
Alex Sharov
436656a40f
snapshots: DumpBodies - break dependency on body.BaseTxNum value in db (#7607)
it's step towards saving canonical and non-canonical bodies in same
table (and txs also in same own table). to reduce write amplification
(cheaper re-orgs)

PR change: reading BaseTxNum from existing snapshots instead of DB 
DB will store in field body.BaseTxNum - non-canonical TxnID  
Snapshots will store only canonical TxNum in field body.BaseTxNum
2023-06-02 11:01:57 +07:00
Alex Sharov
b163d3a877
e3: release some e4 parts (#7629) 2023-06-02 10:55:40 +07:00
Alex Sharov
c6b12ed7ca
stageLoop: unbound canRunCycleInOneTransaction logic from initialCycle variable (#7616) 2023-06-01 16:50:19 +07:00
Alex Sharov
6c0b531eca
add "erigon snapshots diff" sub-command to find difference between 2 snapshots (#7619) 2023-06-01 16:29:26 +07:00
Alex Sharov
5c7da48331
blockReader: fix blockNum check logic (#7615) 2023-06-01 15:04:21 +07:00
Alex Sharov
299c81e522
move "makeBodiesCanonical" method to blockWriter (#7605) 2023-05-31 15:44:01 +07:00
Alex Sharov
3cba3908dd
use BlockReader.CanonicalHash method instead of rawdb (#7604) 2023-05-31 13:41:10 +07:00
Alex Sharov
af83845f7e
cheaper isCanonicalHash func, less lookups of blockHash by blockNum in rpc (#7603) 2023-05-31 11:26:38 +07:00
Anshul Yadav
1fb053f260
Header downloader monitor for diagnostics system (#7590)
## What's this PR about?

- Added states to be sent to diagnostics system for header downloader
monitor
- Added the code for sending the states through the tunnel
- Code added for updating the states in the header_algos.go file

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-05-29 14:18:31 +07:00
Alex Sharov
279e1bec33
use blockReader as service-provider of RoSnapshots (#7584) 2023-05-26 17:12:33 +07:00
Alex Sharov
cfafa24481
tests for blocks snapshots creation data producer (#7579) 2023-05-25 14:58:31 +07:00
Alex Sharov
111db5f655
blockReader in tests - step8 (#7578) 2023-05-25 12:46:11 +07:00
Alex Sharov
a79033cef8
blockReader in tests, step6 (#7573) 2023-05-24 18:34:36 +07:00
Alex Sharov
31687be599
blockReaders in tests, step4 (#7570) 2023-05-24 15:52:51 +07:00
Alex Sharov
179b138000
add blockReader to mining, blockchain_test (#7569) 2023-05-24 13:53:22 +07:00
Alex Sharov
23bd14744b
blockReader: use in ethstat (#7565) 2023-05-23 16:30:47 +07:00
Alex Sharov
696ccb6edd
add constant to run tests on initialSync=true (#7563) 2023-05-23 14:57:15 +07:00
Alex Sharov
63afe65686
Use BlockReader in ForkValidator, CliqueAPI (#7562) 2023-05-23 14:49:17 +07:00
Alex Sharov
18990ff91a
e2: reduce StageSyncStep interface size (#7561) 2023-05-23 12:28:31 +07:00
Alex Sharov
4d0dee6fb0
Introduce BlockWriter object (txsV3 step 0) (#7559) 2023-05-22 15:49:21 +07:00
Alex Sharov
2865b85888
move e2 snapshots management closer to e3: step 1 (#7543)
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 10:09:46 +07:00
Anshul Yadav
2c194e10a1
Args usage msg bug fix (#7554)
## What's this PR is about?
Minor fix in args usage message of support flag. The current message
says that the flag should be 'metrics.url' but it reality it should be
'metrics.urls'
2023-05-20 21:57:23 +01:00
ledgerwatch
2a872b4d54
[devnet] separare logging - headers download (#7551)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 07:00:19 +01:00
ledgerwatch
b0117a7c30
[devnet] separate logging - p2p (#7547)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-19 18:41:53 +01:00
ledgerwatch
b382f96f6c
Introduce logger into etl (#7537)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 21:20:07 +01:00
ledgerwatch
05597cb195
[devnet tool] separare logging (#7531)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 17:36:15 +01:00
Alex Sharov
c0096ee8c8
StageLoopStep: if node synced - then run initialCycle in 1 tx also (for data consistency) (#7532) 2023-05-17 13:39:21 +01:00
ledgerwatch
e75ea786c0
[devnet tool] separate logging (#7526)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-17 07:36:06 +01:00
ledgerwatch
3f9ae3ec77
[devnet tool] separate logging (#7525)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-16 10:53:50 +01:00
Alex Sharov
b3aca15ff8
e2: ReadAhead of blocks, senders accounts, code (#7501)
It improves performance of initial sync (stage exec) by 5-20% when
blocks snapshots are mounted to high-latency drive and when chaindata is
on high-latency drive. And improving cold-start performance.

Current implementation using 2 goroutines for ReadAhead. It also
producing more garbage, can improve it later (here are dashboard with
impact).
```
mainnet2-1: with ReadAhead
mainnet2-3: no ReadAhead
```

<img width="949" alt="Screenshot 2023-05-12 at 09 24 31"
src="https://github.com/ledgerwatch/erigon/assets/46885206/b90b1fa8-9099-48ff-95b3-86e864a36d46">

<img width="845" alt="Screenshot 2023-05-12 at 09 24 13"
src="https://github.com/ledgerwatch/erigon/assets/46885206/39d90c0c-a9d5-4735-8c03-da1455b147aa">
2023-05-15 15:08:35 +07:00
ledgerwatch
bc169179b0
[devnet tool] Side-quest: logging step 7 (#7506)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-13 21:22:30 +01:00
ledgerwatch
5ae9cb7f39
[Devnet tool] side-quest logging step 6 (#7496)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-11 18:59:55 +01:00
ledgerwatch
6ef3fc3a3c
[devnet tool] Side-quest logging, step 5 (#7484)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-10 19:36:27 +01:00