Commit Graph

1199 Commits

Author SHA1 Message Date
Alex Sharov
3ec7d9b010
receipts: less allocs in DeriveFields, use blockReader, remove ReadReceiptsByHash, gasPriceOracle don't read block twice and use blockLru (#7592)
preparation for adding BlockID
2023-05-27 16:39:14 +07:00
Alex Sharov
94ceb6f6c5
nil ptr in test (#7586) 2023-05-26 17:46:50 +07:00
Alex Sharov
279e1bec33
use blockReader as service-provider of RoSnapshots (#7584) 2023-05-26 17:12:33 +07:00
Alex Sharov
111db5f655
blockReader in tests - step8 (#7578) 2023-05-25 12:46:11 +07:00
Alex Sharov
a79033cef8
blockReader in tests, step6 (#7573) 2023-05-24 18:34:36 +07:00
Alex Sharov
31687be599
blockReaders in tests, step4 (#7570) 2023-05-24 15:52:51 +07:00
alex.sharov
2181b6abfd Merge branch 'devel' of github.com:ledgerwatch/erigon into devel 2023-05-24 14:01:01 +07:00
alex.sharov
cb0434be87 block reader bodies test 2023-05-24 14:00:29 +07:00
Alex Sharov
179b138000
add blockReader to mining, blockchain_test (#7569) 2023-05-24 13:53:22 +07:00
Alex Sharov
4d0dee6fb0
Introduce BlockWriter object (txsV3 step 0) (#7559) 2023-05-22 15:49:21 +07:00
Alex Sharov
2865b85888
move e2 snapshots management closer to e3: step 1 (#7543)
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-22 10:09:46 +07:00
ledgerwatch
067f695fff
[devnet tool] Separate logging (#7553)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-20 14:48:16 +01:00
ledgerwatch
b382f96f6c
Introduce logger into etl (#7537)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-18 21:20:07 +01:00
ledgerwatch
3f9ae3ec77
[devnet tool] separate logging (#7525)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-16 10:53:50 +01:00
ledgerwatch
5d6b0ead8f
[devnet tool] separate logging (#7510)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2023-05-15 15:30:56 +01:00
Alex Sharov
b3aca15ff8
e2: ReadAhead of blocks, senders accounts, code (#7501)
It improves performance of initial sync (stage exec) by 5-20% when
blocks snapshots are mounted to high-latency drive and when chaindata is
on high-latency drive. And improving cold-start performance.

Current implementation using 2 goroutines for ReadAhead. It also
producing more garbage, can improve it later (here are dashboard with
impact).
```
mainnet2-1: with ReadAhead
mainnet2-3: no ReadAhead
```

<img width="949" alt="Screenshot 2023-05-12 at 09 24 31"
src="https://github.com/ledgerwatch/erigon/assets/46885206/b90b1fa8-9099-48ff-95b3-86e864a36d46">

<img width="845" alt="Screenshot 2023-05-12 at 09 24 13"
src="https://github.com/ledgerwatch/erigon/assets/46885206/39d90c0c-a9d5-4735-8c03-da1455b147aa">
2023-05-15 15:08:35 +07:00
Alex Sharov
9ab48c067b
e2: self-heal after accidental blocks delete (download blocks eventually, then work as usual) (#7516) 2023-05-15 15:07:32 +07:00
racytech
42e8db3958
eip-4844: modified DecodeTransaction version 1 (#7442)
Blob transactions are SSZ encoded, so it had to be added to decoding.
There are 2 encoding forms: `network` and `minimal` (usual). Network
encoded blob transactions include "wrapper data" which are `kzgs`,
`blobs` and `proofs`, and decoded by `DecodeWrappedTransaction`. For
previous types of transactions the network encoding is no different.
Execution-payloads / blocks use the minimal encoding of transactions. In
the transaction-pool and local transaction-journal the network encoding
is used.

Concerns: 
1. Possible performance reduction caused by these changes, not sure if
streams are better then slices. Go slices in this modifications are
read-only, so they should be referred to the same underlying array and
passed by a reference.
2. If `DecodeWrappedTransaction` and `DecodeTransaction` will create
confusion and should be merged into one function.
2023-05-09 18:44:53 +01:00
ledgerwatch
3c1448afed
[devnet tool] Side-quest logging - replace quiet parameter (#7464)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-08 17:52:31 +01:00
Alex Sharov
231d128d91
e3: parallel exec docs (#7459) 2023-05-08 14:32:12 +07:00
Alex Sharov
299535e7ea
LeakDetector: use it to find which resource was created but not closed (leaked) (#7457) 2023-05-08 12:22:26 +07:00
ledgerwatch
fdd385cef1
[Devnet tool] Side-quest to improve logging - part 1 (#7445)
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established

---------

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-05-07 07:28:15 +01:00
0xMaxMa
9440126ddf
Add clique API, support "clique" option in --http.api flag (#6985)
* Enable the `clique` option in the `--http.api` flag.

* List of Clique commands:
`clique_getSnapshot(block number)`
`clique_getSnapshotAtHash(block hash)`
`clique_getSnapshotAtHash(block hash)`
`clique_getSigners(block number)`
`clique_getSignersAtHash(block hash)`
`clique_proposals()`
`clique_propose(signer address, bool)`
`clique_discard(signer address)`
`clique_status()`
Example:
`curl --data
'{"method":"clique_getSigners","params":[],"id":1,"jsonrpc":"2.0"}' -H
"Content-Type: application/json" -X POST http://localhost:8545`

* Please be careful while using the Clique API. Do not make the HTTP API
public on the Clique's signer node, as anyone can directly call a Clique
command. Instead, it should only be allowed in the localhost by using
the flag `--http.addr "127.0.0.1"`.
2023-05-05 09:20:40 +07:00
Andrew Ashikhmin
1533674dad
Implement EIP-1153 transient storage (#7405)
Port https://github.com/ethereum/go-ethereum/pull/26003

---------

Co-authored-by: Mark Tyneway <mark.tyneway@gmail.com>
2023-05-01 18:26:24 +02:00
Alex Sharov
4344136973
e3: rethink meaning of kv_temporal.BeginRw() (#7413) 2023-04-30 10:08:41 +07:00
Alex Sharov
7b7b2fca0a
e3: drain when no new tasks causing deadlock (#7395) 2023-04-27 09:10:24 +07:00
Alex Sharov
367f3e9afa
e3: log slow (unclosed) contexts to detect leaks (#7375) 2023-04-24 12:52:15 +07:00
ledgerwatch
d70c9f0979
[Diagnostics] initial support for body downloader visualisation (#7373)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-04-23 18:56:37 +01:00
Alex Sharov
b14ca7508f
e3: remove garbage files (#7357) 2023-04-21 09:51:03 +07:00
Alex Sharov
b23a0267d3
e3: deadlock fix part2 (#7344) 2023-04-19 08:12:13 +00:00
Alex Sharov
695b9f887b
e3: close input chan in the end of rwloop (to fix deadlock) (#7342) 2023-04-19 07:26:22 +00:00
Alex Sharov
a5b9f2d774
e3: close input chan when no work left (#7340) 2023-04-19 04:47:53 +00:00
alex.sharov
c733596f59 cumulative index stage: ctrl+C support 2023-04-19 10:56:02 +07:00
alex.sharov
ef73159657 save 2023-04-18 08:30:27 +07:00
Alex Sharov
62d1e8a66c
e3: in-general merge must not see "overlaps/deleted" files, but merge of history need access to corresponding index files (even if they marked as deleted or already merged - before kill -9) (#7321) 2023-04-17 05:54:29 +00:00
racytech
7aa217f015
eip-4844: setting header's excess_data_gas, verifiying eip-4844 headers (#7308)
In this PR Header's ExcessDataGas is set to the actual value, but it's
still unused. It will be used to compute data fee for eip-4844 data
blobs, logic of which will be added in later PRs. Also eip-4844 header
verification logic added.
2023-04-16 08:12:40 +00:00
Andrew Ashikhmin
96bb5d544b
Remove Parlia (#7306)
Pre-requisite: https://github.com/ledgerwatch/erigon-lib/pull/969
2023-04-14 06:24:10 +00:00
racytech
47fd86e4fb
eip-4844: assigned value to excessDataGas, unused packages removed (#7289)
`excessDataGas` has been partially made eip-4844 ready, so instead of
passing nils to functions, now it actually assigned to some value (it is
expected to be nil until cancun update).
2023-04-12 05:45:44 +00:00
alex.sharov
6e57f3ef92 e3: less conlicts 2023-04-10 09:38:08 +07:00
Alex Sharov
895e61d1ab
e3: merge out chan+heap to one class (#7287) 2023-04-10 02:33:20 +00:00
Reinhard Schu
cbe92dc6a4
Update README.md (#7284)
Update description of stages
2023-04-10 01:46:10 +00:00
Alex Sharov
e26c607922
e3: combine input tasks channel+heap to class PriorityQueueWithRetry and add docs (#7275) 2023-04-07 06:43:05 +00:00
Alex Sharov
9b5d7357dd
e3: remove sync.Cond and make ownership of first queue clearer (producer will close it). (#7268)
Separate: 
- new tasks coming by channel with limited capacity (touching the limit
will block - like sync.Cond.Wait() does, but composable). as a result -
don't need track queue size (it will not increase conflict-rate).
- re-exec tasks are going to priority-queue (which is also higher
priority than 1-st `chan`).
2023-04-06 09:03:54 +00:00
alex.sharov
962ab2a767 e3: deadlock fix - handle close chanel well 2023-04-05 11:50:58 +07:00
Alex Sharov
64e7e788b5
e3: print background indexing logs in e2 style (summary of all files progress in 1 line) (#7256) 2023-04-05 04:49:02 +00:00
Alex Sharov
a42d362cbd
move aura epoch data from chainDB to auraDB. remove epochReader parameter from consensus interface (#7250) 2023-04-04 03:30:07 +00:00
racytech
d67087f7fd
eip-4844: ApplyTransaction now expects excessDataGas (#7233)
Tiny addition to ApplyTransaction. Now it expects excessDataGas param so
the BlockContext can be created with it
2023-04-03 14:30:28 +00:00
Alex Sharov
65fcec8d85
e3: replace sync.Cond to fix deadlock (#7243) 2023-04-03 11:33:43 +00:00
Alex Sharov
06da707d58
e3: add metric for repeat count (#7239) 2023-04-03 04:34:45 +00:00
racytech
04b57904b0
eip-4844: excessDataGas param added to block execution functions (#7217)
Small addition to block execution functions. Now they expect
excessDataGas param so the BlockContext can be created with it.
2023-03-30 20:42:43 +00:00