Commit Graph

11893 Commits

Author SHA1 Message Date
Igor Mandrigin
fd77eaf86a
Staged Sync: Execution phase should use "plain state" (#548)
* introduce PlainStateReader with fallbacks

* no 10.000 changes in tests

* even less iterations

* remove even more iterations

* add `go run ./cmd/geth --syncmode staged --plainstate` flag

* fix serialization calls

* make a more sensible file default

doesn’t affect anything, because this flag is always overriden when parsing CLI. but still.
2020-05-15 08:52:45 +01:00
ledgerwatch
729303eb6b
Start transforming SubTrieLoader into iterator/stream (#547)
* Splitting up into interations

* Some fixes

* Fixes

* Fix linter

* Fix linter

* Improvements after review

* Add more diagnostics

* Fix and better printing
2020-05-14 18:12:33 +01:00
ledgerwatch
9f3f768ab4
Rename Resolver => SubTrieLoader, ResolveSet => RetainList (#544)
* Rename Resolver => SubTrieLoader, ResolveSet => RetainList

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 15:24:43 +01:00
ledgerwatch
82a67f4d48
eth_getProof step2 - Decouple Resolver from Trie, remove ResolveRequest (#543)
* Removing PrepareResolveParams and NeedResolution

* Remove ResolveRequest

* Decouple resolver from trie

* Remove hooks from resolver

* Fix util compilation

* More test fixes

* Compiled tests in trie

* Fix trie tests

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 08:22:45 +01:00
ledgerwatch
41b5fce078
resolver: merge HashBuilder-s and ResolveSet-s, decouple from ResolveRequest-s (#540)
* merge hbStorage and hb in resolver

* Fix prepare

* Remove tracing by default

* cleanup

* Fix linter

* Not to include incarnation into the ResolveRequest

* Fix linter

* Fix linter

* Fix test

* Fix linter

* Revert

* Fix for infinite loop

* Fix linter

* Collapse rss into rs

* remove reliance of requests in resolver

* Remove currentReq

* Remove fields from ResolveRequest

* Fix linter
2020-05-11 05:46:07 +01:00
ledgerwatch
cc9163a41c
Remove empty file (#541)
* Remove empty file

* Remove another file
2020-05-10 21:43:49 +01:00
Giulio rebuffo
89a4f8ddff
Fix --rinkeby and --goerli (#539)
* fixed goerli and rinkeby testnets

* merge
2020-05-09 20:29:00 +01:00
ledgerwatch
8ded73fc4f
Refactoring of resolver_stateful (#537) 2020-05-09 12:37:09 +01:00
ledgerwatch
5b2a11137b
[WIP] Fix block tests that were failing due to CalcTrieRoots (#536)
* Fix block tests, add support for code nodes in the stream

* Fix typo

* Fix test

* Comment cleanup
2020-05-09 05:44:56 +01:00
ledgerwatch
ba27c39615
Optimise resolver (#535)
* Avoid repetitive skipping in resolver

* Fix linter

* Stop overshooting
2020-05-08 14:41:37 +01:00
ledgerwatch
b4d2e57428
Introduce incarnationMap in both DbStateReader and DbStateWriter (#534)
* Add incarnationMap into db_state_reader

* Fix test

* Add incarnationMap back to the DbStateWriter
2020-05-08 05:52:55 +01:00
ledgerwatch
d60e35e80e
Fix for the sync failure on block 2792476 (#533)
* Commit after every block

* Commit after every block

* Read account

* Read account

* Small rewrite of resolver logic

* Revert "Commit after every block"

This reverts commit 6a67392f99860ca544fb6d3f4333ea36b9399b7a.

* Revert "Commit after every block"

This reverts commit 6da4c2296fa61d952de5f51e29ed68905d33c3ee.

* Fix linter

* Fix

* Fix

* Fix

* Fix

* Optimisation

* Fixes

* Skip irrelevant storage always

* Fix

* testResolve

* Another fix

* Fix

* Fix fix

* Cleanup
2020-05-08 05:52:43 +01:00
b00ris
2a6f12520c
remove pruning option. Inverse enabling pruning logic. (#532) 2020-05-07 14:31:14 +01:00
ledgerwatch
2fd8f1704c
Restore EVMC integration for staged sync (accidentally broken), resetState (#529)
* Add resetState

* resetState not to disturb header chain

* Fix linter

* Fix linter

* Restore evmc integration
2020-05-07 06:59:00 +01:00
ledgerwatch
c391731dc4
Potential fix for sync failure at block 2792476 (#520)
* Fixes

* Fix

* Fix to stream
2020-05-07 06:58:37 +01:00
ledgerwatch
8cde4d5c4c
Skip flaky test (#526) 2020-05-06 18:42:21 +01:00
ledgerwatch
f47d661c34
Proper unwinding in the staged sync (#521)
* introduce fork

* Fix unwind test

* Unwind now really works in the test

* Fix linter

* Fix linter

* Fix linter

* Not to handle block and block hash announced in staged sync
2020-05-06 18:42:10 +01:00
b00ris
abe54cd7a2
Remove EIP2027 (#512)
* save state

* fmt

* save state

* save state

* fix resolve test

* fix lint

* fix test
2020-05-06 09:59:50 +01:00
Alex Sharov
56022d8d85
- parent node type - doesn't matter for IH - just check that node itself is fullNode or duoNode is enough (#518)
- in unloading - 64 nibbles - it's hex of a.storage, but not a - then 64 nibbles hex must be stored in IH with incarnation
- EmptyRoot can't come to `WillUnloadBranchNode` method - because if account has no storage - then accountNode will have no branches in a.storage, a.storage will equal to `hashNode(EmptyRoot)` - and we don't unload hashNodes (only fullNode/duoNode).
2020-05-06 09:58:37 +01:00
Andrew Ashikhmin
39a37f2dd9
Don't check root in accountsEqual, that's covered by storageChanged anyway. (Fixing checkChangeSets for contract 0xe2942418b2d87a1d88cce66d0b11c7ef16a72971 in block 1467584.) (#519) 2020-05-06 09:58:12 +01:00
ledgerwatch
395faa635e
Switch to the latest bolt, disable prefix compression (#517)
* Disable prefix compression

* Switch to the latest bolt
2020-05-05 21:08:28 +01:00
ledgerwatch
d894150948
Correctly compress default incarnation in the storage changesets (#516)
* Correctly compress default incarnation in the storage changesets

* Add test
2020-05-05 21:08:14 +01:00
Alex Sharov
2686bf2f3b
Remove topLevels param of resolver (#515)
* remove topLevels param of resolver

* revert to master version of ih
2020-05-05 21:08:00 +01:00
Alex Sharov
73d5a8e905
Prometheus dashboard to support multiple-turbo-geth-insances (#514)
add goroutines and threads chart
2020-05-04 23:30:51 +01:00
Igor Mandrigin
cea453be74
Support EVMC (EVMone) (#509)
* add stuff

* vendor evmc and patch import paths

* some fixups

* compiles!

* fixups for evm initialization

* less chatty logs

* fix wrong import in test

* better (time based) logging

* remove unnecessary printf

* evmc as a module

* glue evmc code

* use a proper fork for evmc

* post-rebase fixups

* fix linterz
2020-05-04 17:13:51 +01:00
b00ris
d0af8a2139
Blocks compression (#510)
* block compression

* rerun lint

* fix lint
2020-05-04 06:55:51 +01:00
b00ris
f608a5592d
Storage encoding docs (#511)
* add docs

* clean up

* fix lint

* fix lint

* remove trace.go

* rm tmp file
2020-05-04 06:55:37 +01:00
ledgerwatch
5d6fb16a97
[WIP] Rewinding for the staged sync, test for the staged sync (#508)
* Initial rework

* Fix lint

* boilerplate for the test

* Fix linter

* More to the test

* fix linter

* Save progress

* Test works

* Fix linter

* More on test

* Fix test, linter

* Fix lint

* Fix test
2020-05-03 13:39:50 +01:00
Alex Sharov
e55c34ccb5
reuse accAddrHashWithInc buffer, remove non-essential copying (#507) 2020-05-03 07:27:21 +01:00
Alex Sharov
af4a99b977
Merge account and storage resolvers (#504)
* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* Debug

This reverts commit 9c5eb69465f25607d546b03359b2cbcb1bd46689.

* Fix linters

* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* remove inc

* remove inc from rss

* tr.succStorage.Reset()

* remove inc from rss

* Remove hard-coding

* succ.Reset

* Enable CalcTrieRoots

* Proper dumping of the trie

* Debug

* Fix for CalcTrieRoot

* Fix another inteference bug

* Temp

* Fix test

* Cleanup

* remove STORE_ACCOUNT_ROOT=true flag

* Fix linter

* Fix linter

* Disable getnodedata by default

* Fix test

* Fix test

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-05-02 19:00:57 +01:00
Andrew Ashikhmin
7b40cbb6fa
Incarnation should be read by StateReader, not StateWriter (#506)
* GetCurrentAccountIncarnation

* Incarnation should be read by StateReader, not StateWriter

* Use GetHistoricalAccountIncarnation in DbState

* RemoteReader ReadAccountIncarnation

* Handle the case where a contract has self-destructed, then Eth sent to it, then it got recreated again
2020-05-02 19:00:42 +01:00
ledgerwatch
d795bd5a90
Shortcut for 2^e (#505) 2020-05-02 15:42:37 +01:00
ledgerwatch
4991ce6858
Small fixes for download and execute sync stages (#503)
* Small fixes

* Disable tests that require double validation
2020-04-30 17:19:24 +01:00
Igor Mandrigin
30d41380ee
Parallel signature verification (#499)
* create context

* parallel recovery

* recovery should use the parallel contexts

* linters

* linters

* remove a leftover

* fix refactoring leftover
2020-04-29 14:17:28 +03:00
Igor Mandrigin
7d9bcc1e01
Fix initial state of the ExportFileBlockProvider (#500)
* Fix initial state of the `ExportFileBlockProvider`

* fixup
2020-04-29 14:17:06 +03:00
Alex Sharov
8ae055d04c
up syntax of solidity pragma (#498) 2020-04-29 11:51:07 +01:00
ledgerwatch
051323edbb
Tinkering with storage changesets (#497)
* Adjustments

* fix tests

* Fix tests

* Widen most numbers to 32 bits
2020-04-28 16:36:00 +01:00
Igor Mandrigin
f130d00b8c
Parallel sender recovery in staged sync (#496)
* fix stuff

* recover senders in a separate goroutine

* better way of saving state

* dont fail hash checks because of block 0

* debug stuff

* Revert "debug stuff"

This reverts commit 6d4be2255b29944f6e52516b37656a8eafdb415e.

* fix linter
2020-04-28 13:29:09 +01:00
Alexey Akhunov
f762df30dd Revert "Adjustments"
This reverts commit 216fbc8468.
2020-04-28 12:24:59 +01:00
Alexey Akhunov
216fbc8468 Adjustments 2020-04-28 12:24:32 +01:00
b00ris
cc137af1ce
[WIP] New storage changeset encoding (#493)
* new storage changeset encoding

* fix lint
2020-04-28 08:41:55 +01:00
ledgerwatch
7211b0f261
Add senders recovery stage to staged sync (#494)
* Maintenance commit

* Fixes

* Fixes

* Fix linters

* Fix linter

* Fix linter
2020-04-28 06:36:51 +01:00
ledgerwatch
a09547e822
Truncate history index after reorgs (#492)
* Refactor RewindData

* Add cpu profile

* Index truncation

* Fix compile

* Fix linter

* Fix linter

* Fix tests

* Create test

* Remove Remove

* Clarifying comment
2020-04-28 06:36:33 +01:00
Igor Mandrigin
a1facf39be
Fix broken --block in the stateless prototype. 2020-04-27 17:40:34 +03:00
Igor Mandrigin
c516110121
Staged Sync: check trie root hash stage (#490) 2020-04-27 16:27:37 +03:00
ledgerwatch
ccc0c4eeb8
Fix receipts (#488) 2020-04-27 07:42:21 +01:00
ledgerwatch
e1cda82eb8
Fix canonical hashes in staged bodies sync (#487)
* Fix canonical hashes

Remove changes in hack.go

* incarnations and db_state_reader

* Fix compilation

* Fix linter

* Test fixes
2020-04-26 22:58:26 +01:00
Igor Mandrigin
468bd65109
Add "execute blocks" stage to the staged sync. (#486)
* extract functions to different files

* post-rebase fixups

* stage v0

* add a log

* better log entries

* fixes in log messages

* fix some stuff

* review fixes

* fix linters

* pw as a variable

* fix a test

* add Byzantium check

* batch save progress too
2020-04-26 17:02:38 +01:00
ledgerwatch
971024aab1
Stage reorgs (#485)
* Reorg for staged sync

* Fix test

* Fix linter
2020-04-26 12:22:11 +01:00
ledgerwatch
0c6a44b5e5
Index bits to mark creation of accounts or storage items (#483)
* Index bits

* Fix linter

* Fix linter

* Not to call getModifiedAccountsByNumber for go-ethereum

* Remove the 0x000 kludge
2020-04-25 15:50:32 +01:00