Commit Graph

1219 Commits

Author SHA1 Message Date
Alex Sharov
afd8ea94d0
e3: move domains to tables.go (#7707) 2023-06-11 21:12:05 +07:00
Alex Sharov
2c39570077
e3: to produce stateChanges stream (#7692) 2023-06-09 11:13:06 +07:00
Mark Holt
37a8450d5f
Added non root logger to bor.ValidatorSet (#7668)
I've added a non root logger to bor.ValidatorSet validator set. This
creates a signature change on a number of calling functions to propagate
the logger. This is mostly constrained to the bor package but impacts a
number of tests and utilities which call the validators set.
2023-06-06 10:55:59 +01:00
Alex Sharov
d703c3cd8e
e3: Fix integration tests, step 2 (#7673) 2023-06-06 15:51:04 +07:00
Alex Sharov
bf9f5067f3
fix integration tests run (#7672) 2023-06-06 13:49:01 +07:00
Alex Sharov
e14664d53b
Store Canonical/NonCanonical/Bad blocks/senders/txs in same tables. Make re-org cheaper (#7648)
- allow store non-canonical blocks/senders
- optimize re-org: don't update/delete most of data
- allow mark chain as `Bad` - will be not visible by eth_getBlockByHash,
but can read if have hash+num
2023-06-05 12:36:24 +07:00
Alex Sharov
c8e717c957
rawdb methods to rw db schema version (#7653) 2023-06-04 09:20:22 +07:00
Alex Sharov
63c92010cd
remove txsV3 cli flag (#7644) 2023-06-03 15:54:27 +07:00
Alex Sharov
ad72b7178e
prune speedup. stage_senders: don't re-calc existing senders (#7643)
- stage_senders: don't re-calc existing senders
- stage_tx_lookup: prune less blocks per iteration - because
random-deletes are expensive. pruning must not slow-down sync.
- prune data even if --snap.stop is set
- "prune as-much-as-possible at startup" is not very good idea: at
initialCycle machine can be cold and prune will cause big downtime, no
reason to produce much freelist in 1 tx. People may also restart erigon
- because of some bug - and it will cause unexpected downtime (usually
Erigon startup very fast). So, I just remove all `initialSync`-related
logic in pruning.
- fix lost metrics about disk write byte/sec
2023-06-03 12:30:53 +07:00
Alex Sharov
d40317c905
preparation for --txs.v3, step 2 (#7636) 2023-06-03 07:20:22 +07:00
Andrew Ashikhmin
190bc9dfd8
EIP-4844: add data_gas_used (#7639)
Implement https://github.com/ethereum/EIPs/pull/7062 and
https://github.com/ethereum/EIPs/pull/7095. Pick up
https://github.com/ledgerwatch/erigon-lib/pull/1006.
2023-06-02 21:26:19 +01:00
Alex Sharov
3750ee6930
fix lint issue (#7633) 2023-06-02 13:26:56 +07:00
Alex Sharov
436656a40f
snapshots: DumpBodies - break dependency on body.BaseTxNum value in db (#7607)
it's step towards saving canonical and non-canonical bodies in same
table (and txs also in same own table). to reduce write amplification
(cheaper re-orgs)

PR change: reading BaseTxNum from existing snapshots instead of DB 
DB will store in field body.BaseTxNum - non-canonical TxnID  
Snapshots will store only canonical TxNum in field body.BaseTxNum
2023-06-02 11:01:57 +07:00
Alex Sharov
b163d3a877
e3: release some e4 parts (#7629) 2023-06-02 10:55:40 +07:00
Alex Sharov
c6b12ed7ca
stageLoop: unbound canRunCycleInOneTransaction logic from initialCycle variable (#7616) 2023-06-01 16:50:19 +07:00
Alex Sharov
5a60ebeca2
blocks retire: fix baseID logic (#7614) 2023-06-01 10:07:57 +07:00
Alex Sharov
0dc8e271a9
fix reset tx sequence value (#7606) 2023-05-31 16:06:09 +07:00
Alex Sharov
299c81e522
move "makeBodiesCanonical" method to blockWriter (#7605) 2023-05-31 15:44:01 +07:00
Alex Sharov
3cba3908dd
use BlockReader.CanonicalHash method instead of rawdb (#7604) 2023-05-31 13:41:10 +07:00
Alex Sharov
af83845f7e
cheaper isCanonicalHash func, less lookups of blockHash by blockNum in rpc (#7603) 2023-05-31 11:26:38 +07:00
Alex Sharov
3ec7d9b010
receipts: less allocs in DeriveFields, use blockReader, remove ReadReceiptsByHash, gasPriceOracle don't read block twice and use blockLru (#7592)
preparation for adding BlockID
2023-05-27 16:39:14 +07:00
Alex Sharov
94ceb6f6c5
nil ptr in test (#7586) 2023-05-26 17:46:50 +07:00
Alex Sharov
279e1bec33
use blockReader as service-provider of RoSnapshots (#7584) 2023-05-26 17:12:33 +07:00
Alex Sharov
111db5f655
blockReader in tests - step8 (#7578) 2023-05-25 12:46:11 +07:00
Alex Sharov
a79033cef8
blockReader in tests, step6 (#7573) 2023-05-24 18:34:36 +07:00
Alex Sharov
31687be599
blockReaders in tests, step4 (#7570) 2023-05-24 15:52:51 +07:00
alex.sharov
2181b6abfd Merge branch 'devel' of github.com:ledgerwatch/erigon into devel 2023-05-24 14:01:01 +07:00
alex.sharov
cb0434be87 block reader bodies test 2023-05-24 14:00:29 +07:00
Alex Sharov
179b138000
add blockReader to mining, blockchain_test (#7569) 2023-05-24 13:53:22 +07:00
Alex Sharov
4d0dee6fb0
Introduce BlockWriter object (txsV3 step 0) (#7559) 2023-05-22 15:49:21 +07:00
Alex Sharov
2865b85888
move e2 snapshots management closer to e3: step 1 (#7543)
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 10:09:46 +07:00
ledgerwatch
067f695fff
[devnet tool] Separate logging (#7553)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 14:48:16 +01:00
ledgerwatch
b382f96f6c
Introduce logger into etl (#7537)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 21:20:07 +01:00
ledgerwatch
3f9ae3ec77
[devnet tool] separate logging (#7525)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-16 10:53:50 +01:00
ledgerwatch
5d6b0ead8f
[devnet tool] separate logging (#7510)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-05-15 15:30:56 +01:00
Alex Sharov
b3aca15ff8
e2: ReadAhead of blocks, senders accounts, code (#7501)
It improves performance of initial sync (stage exec) by 5-20% when
blocks snapshots are mounted to high-latency drive and when chaindata is
on high-latency drive. And improving cold-start performance.

Current implementation using 2 goroutines for ReadAhead. It also
producing more garbage, can improve it later (here are dashboard with
impact).
```
mainnet2-1: with ReadAhead
mainnet2-3: no ReadAhead
```

<img width="949" alt="Screenshot 2023-05-12 at 09 24 31"
src="https://github.com/ledgerwatch/erigon/assets/46885206/b90b1fa8-9099-48ff-95b3-86e864a36d46">

<img width="845" alt="Screenshot 2023-05-12 at 09 24 13"
src="https://github.com/ledgerwatch/erigon/assets/46885206/39d90c0c-a9d5-4735-8c03-da1455b147aa">
2023-05-15 15:08:35 +07:00
Alex Sharov
9ab48c067b
e2: self-heal after accidental blocks delete (download blocks eventually, then work as usual) (#7516) 2023-05-15 15:07:32 +07:00
racytech
42e8db3958
eip-4844: modified DecodeTransaction version 1 (#7442)
Blob transactions are SSZ encoded, so it had to be added to decoding.
There are 2 encoding forms: `network` and `minimal` (usual). Network
encoded blob transactions include "wrapper data" which are `kzgs`,
`blobs` and `proofs`, and decoded by `DecodeWrappedTransaction`. For
previous types of transactions the network encoding is no different.
Execution-payloads / blocks use the minimal encoding of transactions. In
the transaction-pool and local transaction-journal the network encoding
is used.

Concerns: 
1. Possible performance reduction caused by these changes, not sure if
streams are better then slices. Go slices in this modifications are
read-only, so they should be referred to the same underlying array and
passed by a reference.
2. If `DecodeWrappedTransaction` and `DecodeTransaction` will create
confusion and should be merged into one function.
2023-05-09 18:44:53 +01:00
ledgerwatch
3c1448afed
[devnet tool] Side-quest logging - replace quiet parameter (#7464)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-08 17:52:31 +01:00
Alex Sharov
231d128d91
e3: parallel exec docs (#7459) 2023-05-08 14:32:12 +07:00
Alex Sharov
299535e7ea
LeakDetector: use it to find which resource was created but not closed (leaked) (#7457) 2023-05-08 12:22:26 +07:00
ledgerwatch
fdd385cef1
[Devnet tool] Side-quest to improve logging - part 1 (#7445)
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-07 07:28:15 +01:00
0xMaxMa
9440126ddf
Add clique API, support "clique" option in --http.api flag (#6985)
* Enable the `clique` option in the `--http.api` flag.

* List of Clique commands:
`clique_getSnapshot(block number)`
`clique_getSnapshotAtHash(block hash)`
`clique_getSnapshotAtHash(block hash)`
`clique_getSigners(block number)`
`clique_getSignersAtHash(block hash)`
`clique_proposals()`
`clique_propose(signer address, bool)`
`clique_discard(signer address)`
`clique_status()`
Example:
`curl --data
'{"method":"clique_getSigners","params":[],"id":1,"jsonrpc":"2.0"}' -H
"Content-Type: application/json" -X POST http://localhost:8545`

* Please be careful while using the Clique API. Do not make the HTTP API
public on the Clique's signer node, as anyone can directly call a Clique
command. Instead, it should only be allowed in the localhost by using
the flag `--http.addr "127.0.0.1"`.
2023-05-05 09:20:40 +07:00
Andrew Ashikhmin
1533674dad
Implement EIP-1153 transient storage (#7405)
Port https://github.com/ethereum/go-ethereum/pull/26003

---------

Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
2023-05-01 18:26:24 +02:00
Alex Sharov
4344136973
e3: rethink meaning of kv_temporal.BeginRw() (#7413) 2023-04-30 10:08:41 +07:00
Alex Sharov
7b7b2fca0a
e3: drain when no new tasks causing deadlock (#7395) 2023-04-27 09:10:24 +07:00
Alex Sharov
367f3e9afa
e3: log slow (unclosed) contexts to detect leaks (#7375) 2023-04-24 12:52:15 +07:00
ledgerwatch
d70c9f0979
[Diagnostics] initial support for body downloader visualisation (#7373)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-04-23 18:56:37 +01:00
Alex Sharov
b14ca7508f
e3: remove garbage files (#7357) 2023-04-21 09:51:03 +07:00
Alex Sharov
b23a0267d3
e3: deadlock fix part2 (#7344) 2023-04-19 08:12:13 +00:00