Commit Graph

2032 Commits

Author SHA1 Message Date
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. (#592) 2020-05-30 10:00:35 +03:00
Andrew Ashikhmin
12d6851be4
[Issue 552] Disable debug protocol by default (#588) 2020-05-28 18:58:08 +01:00
Alex Sharov
d74732a662
bolt db stats export to prometheus (#585) 2020-05-28 12:34:37 +01:00
Alex Sharov
daa359c363
Mgr schedule iterator (#566)
* db based version of PrefixByCumulativeWitnessSize

* db based version of PrefixByCumulativeWitnessSize

* retain all in Trie by default

* fix WitnessLen logic in calcTrie roots

* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket

* handle corner cases in WL

* Use correct incarnation for IH bucket

* use name WitnessSize

* save progress towards db-only witness estimation

* results from trie and from db are still different

* less recursion

* correct incarnation in CumulativeSearch

* reuse results from previous Tick, separate concepts of parent and startKey

* experiment: if not including trie structure to WitnessSize will reduce cumulative error

* tool to generate all IH and tool to calculate assessment of cumulative error

* tool to generate all IH

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* better stats

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* calculate ticks size distribution

* estimate cumulative error

* fix linter

* resetIH from scratch if needed

* cleanup

* fix test

* fix test
2020-05-28 12:33:05 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
Andrew Ashikhmin
5bbdb6a4b9
Use uint256 for account balance (#580)
* Use uint256 for account balance

* linter
2020-05-26 17:53:50 +01:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices (#569)
* save state

* add current index feature

* fix test

* remove logs

* Only execute 1000 blocks

* Reset history index

* Correct action

* Increase batch size

* Increase chunk size, print memory stats

* Fix linter

* Remove unused from

* Split into 2 staged

* Use storage history gen

* remove log

* Not to run tx_cacher in staged mode

* Not to recover during stage 2

* Not to recover during stage 2

* Remove counter

Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
ledgerwatch
6d883b5350
Fix staged sync issue introduced by LRU caches (#568)
* readPlainAccount

* readPlainAccount

* Try to disable accountCache

* Try to not separate changeBatch

* The actual fix
?

* Move to fastcache, include incarnation

* Fix linter

* Fix linter

* Disable caches

* Fix linter

* Reenable all except storage cache

* Cosmetics

* Disable account cache

* Disable code size cache

* Fix code caching

* Apply the same fix to db_state_writer
2020-05-22 16:11:00 +01:00
ledgerwatch
8b97c5c620
Remove memory leak caused by accountCache, count batch size in bytes (#565)
* Lower cache sizes

* Add memory profiling over http

* No code cache

* Calculate db batch size in bytes

* Fixes

* Fixes

* Increase batch

* Fix linter

* Restore account caching, with copying

* Reintroduce code cache

* Add fixed overhead per key

* Print batch size

* Fix batch size

* Reduce batch size

* 50 Mb

* Fix linter
2020-05-21 21:55:39 +01:00
ledgerwatch
5904332035
hack --action resetState to create both hashed and plain genesis, clean all buckets (#563)
* Reset plain genesis

* Change to optimised freelist implementation

* Fix linter

* Revert "Change to optimised freelist implementation"

This reverts commit 65e28f1737801178e95585381a789785504ec318.

* Noerrcheck
2020-05-21 06:18:25 +01:00
ledgerwatch
e322961dd2
Non scalable but correct implementation of eth_getProof (#562)
* Prototype works

* Fix linter, start actual implementation

* More on getProof implementation

* Fixes

* Fix storage proofs

* Fix linter, start adding to rpctest

* Fixes in eth_getProof and rpctest

* Fix linter

* Reenable check for debug_traceTransation

* Cleanup
2020-05-20 15:32:50 +01:00
Igor Mandrigin
0dae3ade0d post-rebase fixups 2020-05-20 15:26:22 +03:00
Martin Holst Swende
f66c118cad cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	consensus/clique/snapshot_test.go
#	core/bench_test.go
#	core/block_validator_test.go
#	core/blockchain.go
#	core/blockchain_test.go
#	core/chain_makers_test.go
#	core/dao_test.go
#	core/rawdb/accessors_indexes.go
#	core/rawdb/schema.go
#	eth/config.go
#	eth/helper_test.go
#	eth/sync.go
#	light/odr_test.go
#	light/trie_test.go
#	light/txpool_test.go
#	miner/worker_test.go
#	tests/block_test_util.go
2020-05-20 15:26:22 +03:00
Marius van der Wijden
322f4f31c5 params, cmd/utils: remove outdated discv5 bootnodes, deprecate flags (#20949)
* params: remove outdated discv5 bootnodes

* cmd/utils: deprecated bootnodesv4/v5 flags
# Conflicts:
#	mobile/params.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
28d3cbb176 cmd/geth: handle memfixes on 32bit arch with large RAM 2020-05-20 15:26:22 +03:00
rene
2b0cf4d1ee cmd/utils: renames flags related to http-rpc server (#20935)
* rpc flags related to starting http server renamed to http

* old rpc flags aliased and still functional

* pprof flags fixed

* renames gpo related flags

* linted

* renamed rpc flags for consistency and clarity

* added warn logs

* added more warn logs for all deprecated flags for consistency

* moves legacy flags to separate file, hides older flags under show-deprecated-flags command

* legacy prefix and moved some more legacy flags to legacy file

* fixed circular import

* added docs

* fixed imports lint error

* added notes about when flags were deprecated

* cmd/utils: group flags by deprecation date + reorder by date,

* modified deprecated comments for consistency, added warn log for --rpc

* making sure deprecated flags are still functional

* show-deprecated-flags command cleaned up

* fixed lint errors

* corrected merge conflict

* IsSet --> GlobalIsSet

* uncategorized flags, if not deprecated, displayed under misc

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	internal/debug/flags.go
2020-05-20 15:26:22 +03:00
Steven E. Harris
8856454979 node: shut down all node-related HTTP servers gracefully (#20956)
Rather than just closing the underlying network listener to stop our
HTTP servers, use the graceful shutdown procedure, waiting for any
in-process requests to finish.
2020-05-20 15:26:22 +03:00
Marius van der Wijden
a7e120dc4c accounts/keystore: fix double import race (#20915)
* accounts/keystore: fix race in Import/ImportECDSA

* accounts/keystore: added import/export tests

* cmd/geth: improved TestAccountImport test

* accounts/keystore: added import/export tests

* accounts/keystore: fixed naming

* accounts/keystore: fixed typo

* accounts/keystore: use mutex instead of rwmutex

* accounts: use errors instead of fmt
# Conflicts:
#	accounts/keystore/keystore_test.go
#	cmd/geth/accountcmd_test.go
2020-05-20 15:26:22 +03:00
gary rong
ad9e4241b1 all: seperate consensus error and evm internal error (#20830)
* all: seperate consensus error and evm internal error

There are actually two types of error will be returned when
a tranaction/message call is executed: (a) consensus error
(b) evm internal error. The former should be converted to
a consensus issue, e.g. The sender doesn't enough asset to
purchase the gas it specifies. The latter is allowed since
evm itself is a blackbox and internal error is allowed to happen.

This PR emphasizes the difference by introducing a executionResult
structure. The evm error is embedded inside. So if any error
returned, it indicates consensus issue happens.

And also this PR improve the `EstimateGas` API to return the concrete
revert reason if the transaction always fails

* all: polish

* accounts/abi/bind/backends: add tests

* accounts/abi/bind/backends, internal: cleanup error message

* all: address comments

* core: fix lint

* accounts, core, eth, internal: address comments

* accounts, internal: resolve revert reason if possible

* accounts, internal: address comments
# Conflicts:
#	accounts/abi/abi.go
#	accounts/abi/bind/backends/simulated.go
#	cmd/geth/retesteth.go
#	core/state/snapshot/difflayer_test.go
#	core/state/snapshot/disklayer_test.go
#	core/state/snapshot/iterator_test.go
#	core/state_processor.go
#	core/state_transition.go
#	core/vm/evm.go
#	core/vm/instructions.go
#	core/vm/jump_table.go
#	eth/api_tracer.go
#	internal/ethapi/api.go
#	les/odr_test.go
#	light/odr_test.go
#	tests/state_test_util.go
2020-05-20 15:26:22 +03:00
rene
aa2f2b742a cmd, core: remove override muir glacier and override istanbul (#20942)
# Conflicts:
#	core/genesis.go
#	eth/backend.go
#	les/client.go
2020-05-20 15:26:22 +03:00
Andrew Ashikhmin
5bc45f2426
Use uint256 in EVM implementation (kudos to Pawel Bylica) (#551)
* core/vm: improve jumpdest lookup

* Use uint256 in EVM implementation (kudos to Pawel Bylica)

* Safety precaution in opMulmod

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-05-18 08:10:59 +01:00
ledgerwatch
1fb11f4323
Decompose SubTrieLoader into DB loading and trie building part (#555)
* First change

* Fix

* Fix

* Completely decompose finaliseRoot and finaliseStorageRoot

* Further decomposition

* Fix linter

* Created receiver function

* Separate DefaultReceiver

* Introduce middle receiver

* Prototype of getProof, not working correctly yet

* Fix linter

* Fix linter

* Fix lint
2020-05-18 08:02:58 +01:00
Alex Sharov
02dd405e93
MGR Schedule v0 (#550)
* MGR Make Schedule v0
2020-05-15 15:58:36 +07:00
Igor Mandrigin
fd77eaf86a
Staged Sync: Execution phase should use "plain state" (#548)
* introduce PlainStateReader with fallbacks

* no 10.000 changes in tests

* even less iterations

* remove even more iterations

* add `go run ./cmd/geth --syncmode staged --plainstate` flag

* fix serialization calls

* make a more sensible file default

doesn’t affect anything, because this flag is always overriden when parsing CLI. but still.
2020-05-15 08:52:45 +01:00
ledgerwatch
729303eb6b
Start transforming SubTrieLoader into iterator/stream (#547)
* Splitting up into interations

* Some fixes

* Fixes

* Fix linter

* Fix linter

* Improvements after review

* Add more diagnostics

* Fix and better printing
2020-05-14 18:12:33 +01:00
ledgerwatch
9f3f768ab4
Rename Resolver => SubTrieLoader, ResolveSet => RetainList (#544)
* Rename Resolver => SubTrieLoader, ResolveSet => RetainList

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 15:24:43 +01:00
ledgerwatch
82a67f4d48
eth_getProof step2 - Decouple Resolver from Trie, remove ResolveRequest (#543)
* Removing PrepareResolveParams and NeedResolution

* Remove ResolveRequest

* Decouple resolver from trie

* Remove hooks from resolver

* Fix util compilation

* More test fixes

* Compiled tests in trie

* Fix trie tests

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 08:22:45 +01:00
ledgerwatch
41b5fce078
resolver: merge HashBuilder-s and ResolveSet-s, decouple from ResolveRequest-s (#540)
* merge hbStorage and hb in resolver

* Fix prepare

* Remove tracing by default

* cleanup

* Fix linter

* Not to include incarnation into the ResolveRequest

* Fix linter

* Fix linter

* Fix test

* Fix linter

* Revert

* Fix for infinite loop

* Fix linter

* Collapse rss into rs

* remove reliance of requests in resolver

* Remove currentReq

* Remove fields from ResolveRequest

* Fix linter
2020-05-11 05:46:07 +01:00
ledgerwatch
ba27c39615
Optimise resolver (#535)
* Avoid repetitive skipping in resolver

* Fix linter

* Stop overshooting
2020-05-08 14:41:37 +01:00
ledgerwatch
b4d2e57428
Introduce incarnationMap in both DbStateReader and DbStateWriter (#534)
* Add incarnationMap into db_state_reader

* Fix test

* Add incarnationMap back to the DbStateWriter
2020-05-08 05:52:55 +01:00
ledgerwatch
d60e35e80e
Fix for the sync failure on block 2792476 (#533)
* Commit after every block

* Commit after every block

* Read account

* Read account

* Small rewrite of resolver logic

* Revert "Commit after every block"

This reverts commit 6a67392f99860ca544fb6d3f4333ea36b9399b7a.

* Revert "Commit after every block"

This reverts commit 6da4c2296fa61d952de5f51e29ed68905d33c3ee.

* Fix linter

* Fix

* Fix

* Fix

* Fix

* Optimisation

* Fixes

* Skip irrelevant storage always

* Fix

* testResolve

* Another fix

* Fix

* Fix fix

* Cleanup
2020-05-08 05:52:43 +01:00
b00ris
2a6f12520c
remove pruning option. Inverse enabling pruning logic. (#532) 2020-05-07 14:31:14 +01:00
ledgerwatch
2fd8f1704c
Restore EVMC integration for staged sync (accidentally broken), resetState (#529)
* Add resetState

* resetState not to disturb header chain

* Fix linter

* Fix linter

* Restore evmc integration
2020-05-07 06:59:00 +01:00
ledgerwatch
8cde4d5c4c
Skip flaky test (#526) 2020-05-06 18:42:21 +01:00
b00ris
abe54cd7a2
Remove EIP2027 (#512)
* save state

* fmt

* save state

* save state

* fix resolve test

* fix lint

* fix test
2020-05-06 09:59:50 +01:00
Alex Sharov
2686bf2f3b
Remove topLevels param of resolver (#515)
* remove topLevels param of resolver

* revert to master version of ih
2020-05-05 21:08:00 +01:00
Alex Sharov
73d5a8e905
Prometheus dashboard to support multiple-turbo-geth-insances (#514)
add goroutines and threads chart
2020-05-04 23:30:51 +01:00
Alex Sharov
af4a99b977
Merge account and storage resolvers (#504)
* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* Debug

This reverts commit 9c5eb69465f25607d546b03359b2cbcb1bd46689.

* Fix linters

* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* remove inc

* remove inc from rss

* tr.succStorage.Reset()

* remove inc from rss

* Remove hard-coding

* succ.Reset

* Enable CalcTrieRoots

* Proper dumping of the trie

* Debug

* Fix for CalcTrieRoot

* Fix another inteference bug

* Temp

* Fix test

* Cleanup

* remove STORE_ACCOUNT_ROOT=true flag

* Fix linter

* Fix linter

* Disable getnodedata by default

* Fix test

* Fix test

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-05-02 19:00:57 +01:00
Andrew Ashikhmin
7b40cbb6fa
Incarnation should be read by StateReader, not StateWriter (#506)
* GetCurrentAccountIncarnation

* Incarnation should be read by StateReader, not StateWriter

* Use GetHistoricalAccountIncarnation in DbState

* RemoteReader ReadAccountIncarnation

* Handle the case where a contract has self-destructed, then Eth sent to it, then it got recreated again
2020-05-02 19:00:42 +01:00
Igor Mandrigin
7d9bcc1e01
Fix initial state of the ExportFileBlockProvider (#500)
* Fix initial state of the `ExportFileBlockProvider`

* fixup
2020-04-29 14:17:06 +03:00
Alex Sharov
8ae055d04c
up syntax of solidity pragma (#498) 2020-04-29 11:51:07 +01:00
b00ris
cc137af1ce
[WIP] New storage changeset encoding (#493)
* new storage changeset encoding

* fix lint
2020-04-28 08:41:55 +01:00
Igor Mandrigin
a1facf39be
Fix broken --block in the stateless prototype. 2020-04-27 17:40:34 +03:00
ledgerwatch
0c6a44b5e5
Index bits to mark creation of accounts or storage items (#483)
* Index bits

* Fix linter

* Fix linter

* Not to call getModifiedAccountsByNumber for go-ethereum

* Remove the 0x000 kludge
2020-04-25 15:50:32 +01:00
ledgerwatch
091811c3d1
Account range crash fix (#481)
* Pass error from accountRange

* Fix linter

* Fixes for accountRange

* Fix tests

* Fix tests

* Eliminate deadlocks

* Fixes

* Fix linter

* Fix linter

* Code hash fixes

* Fix linter

* Fix linter

* Fix linter

* Fix linter

* Fix in rpctest

* Ignore storage roots

* Fix linter

* Fix dump test
2020-04-23 10:35:43 +01:00
Igor Mandrigin
7ad429ae5f
add debug_accountRange to rpctest (#476) 2020-04-22 15:36:32 +03:00
ledgerwatch
92800eb66e
Remove duplicate and spurious items from WalkAsOf (not optimised) (#480) 2020-04-22 13:14:40 +03:00
Igor Mandrigin
b661a52bd3
Implement debug_accountRange with geth 1.9.13 changes. (#475) 2020-04-22 13:14:33 +03:00
ledgerwatch
623545a47c
Fix index chunks (#478)
* First stab at fixing index chunks

* Fix compilation

* Fix tests

* Fix walkAsOf for storage

* Fix

* Fix for walkAsOf accounts

* Fix linter
2020-04-21 09:15:40 +01:00
b00ris
a9f71298aa
Index chunking (#473)
* index generation

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* fix

* save state

* fix
2020-04-20 11:35:33 +01:00