Commit Graph

34 Commits

Author SHA1 Message Date
Alex Sharov
325785aba4
add net namespace (#974) 2020-08-25 20:04:01 +07:00
Alex Sharov
1dcc2b141a
Rpcdaemon as lib (#940)
* share config object

* create default config and logger

* move db connection to common func

* move server start to cli package

* clear

* clear

* rename cli to rpc

* use unified SetupLogger func

* make all root flag persistent

* use common flags in different packages

* use common flags in different packages

* move TraceTx method to eth package

* use native slice flags

* create package "turbo"

* disable geth api

* disable geth api

* move more data types to turbo/adapter package

* add support for customApiList

* run more

* run more

* run more

* dog-food

* move DoCall

* move DoCall

* fix tests

* fix test
2020-08-19 12:46:20 +01:00
b00ris
eb46dd68df
Account range testing (#938)
* save state

* save state

* save new timeouts

* compare passed

* save state

* cleanup
2020-08-19 07:33:49 +01:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
b00ris
12aa55bcbb
Remove incarnation inversion (#755)
* remove incarnation inversion

* fix lint

* remove incarnation inversion

* remove comment

* remove inversion from new code

* revert for unwind
2020-07-17 15:18:31 +01:00
Evgeny Danilenko
5ead20e288
close tx_cache and peer.handle goruotines properly (#748)
* close goruotines

* fix using defer

* close pm managers a bit later

* fmt

* finalizer for in-mem db

* after cr

* revert finalizer

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-07-16 14:27:24 +01:00
ledgerwatch
72dcf478d3
Fix debug_storageRangeAt in rpc daemon (#732)
* Fix storageRangeAt

* Fix formatting

* Fix lint

* Fix lint

* Prevent racing by closing db after the blockchain
2020-07-10 06:44:01 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692)
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
2020-06-28 07:10:27 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes (#597)
* Introduce hashCollector

* Add HashCollector to SubTrieLoader

* Fix linter

* Reset hashed state

* Not to regenerate the hashed state

* Not to delete state

* Fix linter

* Print expected hash in the beginning

* Simplify

* Remove initialTrie

* Use etl to buffer intermediate hashes

* Copy values, not insert empty key

* Compress instead of decompress

* Enhance file buffer logging, fix linter

* Fix compile errors

* Fix log

* Fix logging

* Exclude zero key again

* Add rewind

* Restrict timestamps

* Fix

* Fix formatting

* Incorporate separation

* Extract identityLoadFunction

* Fix formatting
2020-05-31 13:23:34 +01:00
ledgerwatch
e322961dd2
Non scalable but correct implementation of eth_getProof (#562)
* Prototype works

* Fix linter, start actual implementation

* More on getProof implementation

* Fixes

* Fix storage proofs

* Fix linter, start adding to rpctest

* Fixes in eth_getProof and rpctest

* Fix linter

* Reenable check for debug_traceTransation

* Cleanup
2020-05-20 15:32:50 +01:00
Igor Mandrigin
0dae3ade0d post-rebase fixups 2020-05-20 15:26:22 +03:00
ledgerwatch
9f3f768ab4
Rename Resolver => SubTrieLoader, ResolveSet => RetainList (#544)
* Rename Resolver => SubTrieLoader, ResolveSet => RetainList

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 15:24:43 +01:00
ledgerwatch
82a67f4d48
eth_getProof step2 - Decouple Resolver from Trie, remove ResolveRequest (#543)
* Removing PrepareResolveParams and NeedResolution

* Remove ResolveRequest

* Decouple resolver from trie

* Remove hooks from resolver

* Fix util compilation

* More test fixes

* Compiled tests in trie

* Fix trie tests

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 08:22:45 +01:00
ledgerwatch
41b5fce078
resolver: merge HashBuilder-s and ResolveSet-s, decouple from ResolveRequest-s (#540)
* merge hbStorage and hb in resolver

* Fix prepare

* Remove tracing by default

* cleanup

* Fix linter

* Not to include incarnation into the ResolveRequest

* Fix linter

* Fix linter

* Fix test

* Fix linter

* Revert

* Fix for infinite loop

* Fix linter

* Collapse rss into rs

* remove reliance of requests in resolver

* Remove currentReq

* Remove fields from ResolveRequest

* Fix linter
2020-05-11 05:46:07 +01:00
Alex Sharov
2686bf2f3b
Remove topLevels param of resolver (#515)
* remove topLevels param of resolver

* revert to master version of ih
2020-05-05 21:08:00 +01:00
Alex Sharov
af4a99b977
Merge account and storage resolvers (#504)
* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* Debug

This reverts commit 9c5eb69465f25607d546b03359b2cbcb1bd46689.

* Fix linters

* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* remove inc

* remove inc from rss

* tr.succStorage.Reset()

* remove inc from rss

* Remove hard-coding

* succ.Reset

* Enable CalcTrieRoots

* Proper dumping of the trie

* Debug

* Fix for CalcTrieRoot

* Fix another inteference bug

* Temp

* Fix test

* Cleanup

* remove STORE_ACCOUNT_ROOT=true flag

* Fix linter

* Fix linter

* Disable getnodedata by default

* Fix test

* Fix test

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-05-02 19:00:57 +01:00
ledgerwatch
0c6a44b5e5
Index bits to mark creation of accounts or storage items (#483)
* Index bits

* Fix linter

* Fix linter

* Not to call getModifiedAccountsByNumber for go-ethereum

* Remove the 0x000 kludge
2020-04-25 15:50:32 +01:00
ledgerwatch
091811c3d1
Account range crash fix (#481)
* Pass error from accountRange

* Fix linter

* Fixes for accountRange

* Fix tests

* Fix tests

* Eliminate deadlocks

* Fixes

* Fix linter

* Fix linter

* Code hash fixes

* Fix linter

* Fix linter

* Fix linter

* Fix linter

* Fix in rpctest

* Ignore storage roots

* Fix linter

* Fix dump test
2020-04-23 10:35:43 +01:00
Igor Mandrigin
7ad429ae5f
add debug_accountRange to rpctest (#476) 2020-04-22 15:36:32 +03:00
ledgerwatch
92800eb66e
Remove duplicate and spurious items from WalkAsOf (not optimised) (#480) 2020-04-22 13:14:40 +03:00
Giulio rebuffo
6e2fc0d936
[WIP] 459 merge storage account (#466)
* bt commit

* merged buckets

* adjusted cmd/

* fixed sync
2020-04-19 20:51:32 +01:00
Alex Sharov
59f997a663
Move account root to ih (#465)
* move_account_root_to_ih

* temptorary comment test
2020-04-18 21:09:44 +01:00
ledgerwatch
076a1005aa
Fix eth_getLogs and restore it in rpctest (#462)
* Modifications to rpctest

* Park getLogs

* Fix in bloombits

* Add tolerance to rpc to geth

* Fix linter

* Fix linter

* Fix

* More fixes to rpctest

* Fixes in rpctest
2020-04-17 15:48:30 +01:00
ledgerwatch
9bb7f3056d
Fix hanging resolver (#424)
* Debugging the cached resolver

* make logic of comparison more explicit

* fix stuck on block 696817

* Remove tracing

* Fix CI

* Search for root hash

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-04-01 17:04:41 +01:00
ledgerwatch
767a374c2e
Tx pool use triedbstate, fixes to GetNodeData and move it to experimental (#364)
* Use TrieDbState for tx pool

* Not initialise tx pool until state is loaded

* Add preimage

* Fix account

* Print codehash

* Print correct code hash

* Print incarnatin

* Print incarnatin

* Use proper incarnation

* Print dbValue

* Actually fix

* Actually fix

* Fix verifySnapshot

* readAccount to get code hash

* Next incarnation

* Print addrHashes with 0 incarnations

* Print storage history

* Print storage history

* Print storage history

* Print storage history

* Print all storage history

* print change set keys

* print change set keys

* print change set keys

* print change set keys

* Not print codebucket info

* Fixes

* Fix for incarnation

* Fix for storage history bucket

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Try to fix the leak

* Fix embedded nodes

* Hasher

* Fix

* Test fixes

* Add experimental debug flag

* Fix tx_pool_test

* Disable GetNodeData test unless in experiment

* Fix more tests

* Fix lint and revert some changes

* Fix lint

* Fix lint
2020-02-10 17:05:32 +00:00
b00ris
df8798b94b
Thin history for storage (#324)
* save state

* save state

* fmt

* add check to test

* move incarnation length to common

* remove line

* fix lint
2020-01-15 17:55:43 +03:00
Alex Sharov
438476d1ba
Serialization bench (#288)
* bench shows that need to pass pointer to []byte when use encoder.Encode()
2020-01-08 13:09:07 +07:00
Alex Sharov
805f9aa501
benchmark rpc daemon
* tracing, write bench1() result into files

* add context to computeTxEnv, remote_db cursor lazy cache allocation,

* more results

* added

* added

* added

* more info

* make linter happy

* exit from tracing gorutines

* remove unused param

* re-run CI

* re-run CI

* remove concept txHandle

* batch cursor in reports

* remove debug output

* fix shadow variables and unhandled errors
2019-12-22 21:10:46 +06:00
alex.sharov
73b50e080d debug_StorageRangeAt v0, got wrong response on request: {"jsonrpc":"2.0","method":"debug_storageRangeAt","params":["0x2bf07c790737be3bc4c57cbf3dedb231806f6bfef434657d59dcc9ddbe4665ab", 1,"0x8b3b3b624c3c0397d3da8fd861512393d51dcbac","0xfade75560a6cfb895f5dc7c4ab3fa10089ac2372c98aa78280d029ab36285ad6",1024],"id":1377} 2019-12-13 09:56:30 +06:00
ledgerwatch
b2ca635d86
Debug tool for recursively comparing state (in DB or in a file) with geth archive node (#191)
* Fetching results of eth_getProof

* Dump 5 levels of the trie in a file for repeated runs

* Drill down to 6 levels of the trie

* Fix lint

* Fix lint

* Fix lint

* verifySnapshot to check accounts with emptyRoot

* Descend into short nodes

* Latest tool fixes

* Fix lint

* Fix state properly working
2019-11-25 13:36:21 +00:00
Giulio
138241bd2c linting 2019-11-05 22:51:11 +01:00
Alexey Akhunov
fe01bccbb8 Apply Turbo-Geth modifications to go-ethereum codebase 2019-11-01 21:52:03 +01:00