Commit Graph

16 Commits

Author SHA1 Message Date
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes (#597)
* Introduce hashCollector

* Add HashCollector to SubTrieLoader

* Fix linter

* Reset hashed state

* Not to regenerate the hashed state

* Not to delete state

* Fix linter

* Print expected hash in the beginning

* Simplify

* Remove initialTrie

* Use etl to buffer intermediate hashes

* Copy values, not insert empty key

* Compress instead of decompress

* Enhance file buffer logging, fix linter

* Fix compile errors

* Fix log

* Fix logging

* Exclude zero key again

* Add rewind

* Restrict timestamps

* Fix

* Fix formatting

* Incorporate separation

* Extract identityLoadFunction

* Fix formatting
2020-05-31 13:23:34 +01:00
ledgerwatch
e322961dd2
Non scalable but correct implementation of eth_getProof (#562)
* Prototype works

* Fix linter, start actual implementation

* More on getProof implementation

* Fixes

* Fix storage proofs

* Fix linter, start adding to rpctest

* Fixes in eth_getProof and rpctest

* Fix linter

* Reenable check for debug_traceTransation

* Cleanup
2020-05-20 15:32:50 +01:00
Igor Mandrigin
0dae3ade0d post-rebase fixups 2020-05-20 15:26:22 +03:00
ledgerwatch
9f3f768ab4
Rename Resolver => SubTrieLoader, ResolveSet => RetainList (#544)
* Rename Resolver => SubTrieLoader, ResolveSet => RetainList

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 15:24:43 +01:00
ledgerwatch
82a67f4d48
eth_getProof step2 - Decouple Resolver from Trie, remove ResolveRequest (#543)
* Removing PrepareResolveParams and NeedResolution

* Remove ResolveRequest

* Decouple resolver from trie

* Remove hooks from resolver

* Fix util compilation

* More test fixes

* Compiled tests in trie

* Fix trie tests

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 08:22:45 +01:00
ledgerwatch
41b5fce078
resolver: merge HashBuilder-s and ResolveSet-s, decouple from ResolveRequest-s (#540)
* merge hbStorage and hb in resolver

* Fix prepare

* Remove tracing by default

* cleanup

* Fix linter

* Not to include incarnation into the ResolveRequest

* Fix linter

* Fix linter

* Fix test

* Fix linter

* Revert

* Fix for infinite loop

* Fix linter

* Collapse rss into rs

* remove reliance of requests in resolver

* Remove currentReq

* Remove fields from ResolveRequest

* Fix linter
2020-05-11 05:46:07 +01:00
Alex Sharov
2686bf2f3b
Remove topLevels param of resolver (#515)
* remove topLevels param of resolver

* revert to master version of ih
2020-05-05 21:08:00 +01:00
Alex Sharov
af4a99b977
Merge account and storage resolvers (#504)
* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* Debug

This reverts commit 9c5eb69465f25607d546b03359b2cbcb1bd46689.

* Fix linters

* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* remove inc

* remove inc from rss

* tr.succStorage.Reset()

* remove inc from rss

* Remove hard-coding

* succ.Reset

* Enable CalcTrieRoots

* Proper dumping of the trie

* Debug

* Fix for CalcTrieRoot

* Fix another inteference bug

* Temp

* Fix test

* Cleanup

* remove STORE_ACCOUNT_ROOT=true flag

* Fix linter

* Fix linter

* Disable getnodedata by default

* Fix test

* Fix test

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-05-02 19:00:57 +01:00
Giulio rebuffo
6e2fc0d936
[WIP] 459 merge storage account (#466)
* bt commit

* merged buckets

* adjusted cmd/

* fixed sync
2020-04-19 20:51:32 +01:00
Alex Sharov
59f997a663
Move account root to ih (#465)
* move_account_root_to_ih

* temptorary comment test
2020-04-18 21:09:44 +01:00
ledgerwatch
9bb7f3056d
Fix hanging resolver (#424)
* Debugging the cached resolver

* make logic of comparison more explicit

* fix stuck on block 696817

* Remove tracing

* Fix CI

* Search for root hash

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-04-01 17:04:41 +01:00
ledgerwatch
767a374c2e
Tx pool use triedbstate, fixes to GetNodeData and move it to experimental (#364)
* Use TrieDbState for tx pool

* Not initialise tx pool until state is loaded

* Add preimage

* Fix account

* Print codehash

* Print correct code hash

* Print incarnatin

* Print incarnatin

* Use proper incarnation

* Print dbValue

* Actually fix

* Actually fix

* Fix verifySnapshot

* readAccount to get code hash

* Next incarnation

* Print addrHashes with 0 incarnations

* Print storage history

* Print storage history

* Print storage history

* Print storage history

* Print all storage history

* print change set keys

* print change set keys

* print change set keys

* print change set keys

* Not print codebucket info

* Fixes

* Fix for incarnation

* Fix for storage history bucket

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Fix embedded nodes

* Hasher

* Fix

* Test fixes

* Add experimental debug flag

* Fix tx_pool_test

* Disable GetNodeData test unless in experiment

* Fix more tests

* Fix lint and revert some changes

* Fix lint

* Fix lint
2020-02-10 17:05:32 +00:00
b00ris
df8798b94b
Thin history for storage (#324)
* save state

* save state

* fmt

* add check to test

* move incarnation length to common

* remove line

* fix lint
2020-01-15 17:55:43 +03:00
ledgerwatch
b2ca635d86
Debug tool for recursively comparing state (in DB or in a file) with geth archive node (#191)
* Fetching results of eth_getProof

* Dump 5 levels of the trie in a file for repeated runs

* Drill down to 6 levels of the trie

* Fix lint

* Fix lint

* Fix lint

* verifySnapshot to check accounts with emptyRoot

* Descend into short nodes

* Latest tool fixes

* Fix lint

* Fix state properly working
2019-11-25 13:36:21 +00:00