Commit Graph

12059 Commits

Author SHA1 Message Date
b00ris
266f9a5208
Plain state index (#595)
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
2020-05-31 07:57:47 +01:00
Evgeny Danilenko
32ec443572
Restore graceful shutdown (#591)
* initial

* gracefull shutdown for staged

* more wg fixes

* fmt

* linters

* remove generalization

* linters

* linters

* fix

* fix

* fmt

* quit into etl

* after CR

* after CR
2020-05-30 16:44:54 +03:00
Igor Mandrigin
a49fa497df
add go mod tidy check (#593) 2020-05-30 09:12:32 +01:00
Alex Sharov
916a1f8b83
[wip] Lmdb: AbstractKV and DB interfaces (#589)
* resetIH from scratch if needed

* lmdb

* add AbstractKV to loader, added new Object accessor around AbstractKV

* add lmdb cli flag

* add requirement of k!=nil on error in docs

* add Size method for compatibility

* read after put tests

* fix multiput nils

* simplify loops

* increase mmap size

* better error messages

* better error messages

* fix tests

* better error messages

* cleanup

* avoid bolt usage in test

* move hardcoded bucket name to dbutils

* register more buckets

* register more buckets

* fix test
2020-05-30 09:12:21 +01:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. (#592) 2020-05-30 10:00:35 +03:00
Andrew Ashikhmin
12d6851be4
[Issue 552] Disable debug protocol by default (#588) 2020-05-28 18:58:08 +01:00
ledgerwatch
dd4db2b36e
Use AbstractKV instead of BoltKV in the SubTrieLoader (#587) 2020-05-28 15:05:35 +01:00
ledgerwatch
f752ac4810
Increase max table size (#586) 2020-05-28 12:52:09 +01:00
Alex Sharov
d74732a662
bolt db stats export to prometheus (#585) 2020-05-28 12:34:37 +01:00
Alex Sharov
daa359c363
Mgr schedule iterator (#566)
* db based version of PrefixByCumulativeWitnessSize

* db based version of PrefixByCumulativeWitnessSize

* retain all in Trie by default

* fix WitnessLen logic in calcTrie roots

* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket

* handle corner cases in WL

* Use correct incarnation for IH bucket

* use name WitnessSize

* save progress towards db-only witness estimation

* results from trie and from db are still different

* less recursion

* correct incarnation in CumulativeSearch

* reuse results from previous Tick, separate concepts of parent and startKey

* experiment: if not including trie structure to WitnessSize will reduce cumulative error

* tool to generate all IH and tool to calculate assessment of cumulative error

* tool to generate all IH

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* better stats

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* calculate ticks size distribution

* estimate cumulative error

* fix linter

* resetIH from scratch if needed

* cleanup

* fix test

* fix test
2020-05-28 12:33:05 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
ledgerwatch
e348dcb846 Move promotion before the shortcut (#582)
* Move promotion before the shortcut

* Correct the logs

* Trying without the encoder

* Revert "Trying without the encoder"

This reverts commit 24c526bc2791fb6806b1c0d4f52a43b87c3959b9.

* Commit to db, not to the mutation

* Fix log more

* No need to log every deleted file
2020-05-27 09:26:56 +01:00
ledgerwatch
5456b2371a
Switch to the latest bolt for hashmap-based freelist (#581) 2020-05-26 18:45:28 +01:00
Andrew Ashikhmin
5bbdb6a4b9
Use uint256 for account balance (#580)
* Use uint256 for account balance

* linter
2020-05-26 17:53:50 +01:00
Igor Mandrigin
4f339a1c81
reset the size of sortableBuffer when flushing to disk (#579) 2020-05-26 17:02:19 +03:00
Igor Mandrigin
c60aff8058
Promote hashed state initially (#577) 2020-05-26 16:37:25 +03:00
ledgerwatch
810cf13bf8
Explicit incarnation map in the database for deleted contracts (#578)
* Explicit map of incarnations

* Fix linter
2020-05-26 13:27:21 +01:00
Andrew Ashikhmin
be6e1cc3b8
Merge pull request #576 from ledgerwatch/gc-7
Post-merge fix
2020-05-25 13:37:16 +02:00
andrew
8d6ff0b769 Post-merge fix 2020-05-25 13:36:19 +02:00
Andrew Ashikhmin
b16e560529
Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore (#575)
* Produce less garbage in GetState

* Still playing with mem allocation in GetCommittedState

* Pass key by pointer in GetState as well

* linter

* Avoid a memory allocation in opSload

* Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore

* linter

* linters

* small clean up
2020-05-25 12:12:25 +01:00
Igor Mandrigin
db746bba7a
"Unwind" for the execution phase when plain state is selected (#571)
* "Unwind" for the execution phase when plain state is selected

* test stub (fails)

* tests (one with incarnations fails)

* test fixups

* fix tests: cleanup contract code bucket
2020-05-25 12:11:56 +01:00
ledgerwatch
154b9a9ca2
Stop blockchain when terminating downloader (#574)
* Print

* Stop blockchain on Terminate

* Remove prints
2020-05-25 08:19:08 +01:00
Andrew Ashikhmin
df28575420
Use pointers to hashes in Get(Commited)State to reduce memory allocation (#573)
* Produce less garbage in GetState

* Still playing with mem allocation in GetCommittedState

* Pass key by pointer in GetState as well

* linter

* Avoid a memory allocation in opSload
2020-05-24 18:43:54 +02:00
ledgerwatch
f6be0b601e
Fix history generation and optimise via pre-sorting in the files (#572)
* Rewrite

* Fix linter, make counter

* Skip hash stage

* Fix linter

* Add common

* Fix wrong bucket

* Use temp files to generate index

* Fix hack, defer

* Add logging when creating files

* Properly pass datadir

* Bigger buffers, less logging

* Log current key, enforce batching

* Limit to 4m, print more

* Use ReadFull

* Optimised version of storage generation

* Don't print entire key

* Commit at the end

* Fix linter

* Remove a bit of copy-paste

* More copy-paste reduction

* Fixes

* Fix key length

* Not delete files too early

* Fix linter

* Fix logging

* Fix starting block

* Skip test

* Remove limit on the stage 4

* Fix test while disabling the hash check

* Fix tests

* Fix unreachable
2020-05-24 15:48:15 +01:00
Andrew Ashikhmin
9f89cf810d
mutation.puts.Size was confused for its number of entries (#570) 2020-05-23 11:27:05 +01:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices (#569)
* save state

* add current index feature

* fix test

* remove logs

* Only execute 1000 blocks

* Reset history index

* Correct action

* Increase batch size

* Increase chunk size, print memory stats

* Fix linter

* Remove unused from

* Split into 2 staged

* Use storage history gen

* remove log

* Not to run tx_cacher in staged mode

* Not to recover during stage 2

* Not to recover during stage 2

* Remove counter

Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
ledgerwatch
6d883b5350
Fix staged sync issue introduced by LRU caches (#568)
* readPlainAccount

* readPlainAccount

* Try to disable accountCache

* Try to not separate changeBatch

* The actual fix
?

* Move to fastcache, include incarnation

* Fix linter

* Fix linter

* Disable caches

* Fix linter

* Reenable all except storage cache

* Cosmetics

* Disable account cache

* Disable code size cache

* Fix code caching

* Apply the same fix to db_state_writer
2020-05-22 16:11:00 +01:00
ledgerwatch
8b97c5c620
Remove memory leak caused by accountCache, count batch size in bytes (#565)
* Lower cache sizes

* Add memory profiling over http

* No code cache

* Calculate db batch size in bytes

* Fixes

* Fixes

* Increase batch

* Fix linter

* Restore account caching, with copying

* Reintroduce code cache

* Add fixed overhead per key

* Print batch size

* Fix batch size

* Reduce batch size

* 50 Mb

* Fix linter
2020-05-21 21:55:39 +01:00
ledgerwatch
e2848ac666
Staged exec: RLU cache between StateReader and StateWriter, separate batches for state and changesets (#542)
* Add cpu profiling

* Splt state and change batches

* Log commits

* Remove

* Log memory

* Add lru caches for staged execution

* Fix

* Fix for nil account

* Fix linter

* Fix linter

* Increase state batch size

* Add memory profiling over http

* Reduce cache sizes

* Fix storage cache

* Fix

* Fix

* Fix

* Fix

* Cleanup

* Fix linter

* Reduce logging

* Reduce logging
2020-05-21 13:27:52 +01:00
Andrew Ashikhmin
d92d0fc706
Don't copy jump table for every new EVMInterpreter since it's quite heavy (#564)
* Avoid copying operation in EVMInterpreter.Run

* Don't copy jump table for every new EVMInterpreter since it's quite heavy
2020-05-21 10:12:09 +01:00
ledgerwatch
5904332035
hack --action resetState to create both hashed and plain genesis, clean all buckets (#563)
* Reset plain genesis

* Change to optimised freelist implementation

* Fix linter

* Revert "Change to optimised freelist implementation"

This reverts commit 65e28f1737801178e95585381a789785504ec318.

* Noerrcheck
2020-05-21 06:18:25 +01:00
Igor Mandrigin
12a6b2f9f6
Use upstream EVMC (#561)
* Use upstream EVMC

* update to 7.3.0
2020-05-20 19:35:32 +01:00
ledgerwatch
e322961dd2
Non scalable but correct implementation of eth_getProof (#562)
* Prototype works

* Fix linter, start actual implementation

* More on getProof implementation

* Fixes

* Fix storage proofs

* Fix linter, start adding to rpctest

* Fixes in eth_getProof and rpctest

* Fix linter

* Reenable check for debug_traceTransation

* Cleanup
2020-05-20 15:32:50 +01:00
Andrew Ashikhmin
9c47df0a2c
Use math.bits in common math Safe* for performance (#556)
* Use math.bits in comoon math Safe* for performance (the compiler optimizes away  math.bits)

* Prettier code in opExp
2020-05-20 14:31:23 +02:00
Andrew Ashikhmin
ff23652d50
GC: reducing copying in stateObject.updateTrie (#557)
* It's unnecessary to copy key in stateObject.updateTrie

* Move value copying from stateObject.updateTrie down to TrieStateWriter
2020-05-20 14:31:03 +02:00
Igor Mandrigin
0dae3ade0d post-rebase fixups 2020-05-20 15:26:22 +03:00
Péter Szilágyi
7cdf2aa95a params: release go-ethereum v1.9.14 2020-05-20 15:26:22 +03:00
Hao Duan
6fe80da4c8 ethstats: stop report ticker in each loop cycle #21070 (#21071)
Co-authored-by: Hao Duan <duan.hao@hyperchain.cn>
2020-05-20 15:26:22 +03:00
gary rong
1aa6ead45f core: fix missing receipt on Clique crashes (#21045)
* core: fix missing receipt

* core: address comment
# Conflicts:
#	core/blockchain.go
2020-05-20 15:26:22 +03:00
Marius van der Wijden
c9fb0f9a3f accounts/abi: allow overloaded argument names (#21060)
* accounts/abi: allow overloaded argument names

In solidity it is possible to create the following contract:
```
contract Overloader {
    struct F { uint _f; uint __f; uint f; }
    function f(F memory f) public {}
}
```
This however resulted in a panic in the abi package.

* accounts/abi fixed error handling
2020-05-20 15:26:22 +03:00
Marius van der Wijden
fc91ce52eb accounts/abi: accounts/abi/bind: Move topics to abi package (#21057)
* accounts/abi/bind: added test cases for waitDeployed

* accounts/abi/bind: added test case for boundContract

* accounts/abi/bind: removed unnecessary resolve methods

* accounts/abi: moved topics from /bind to /abi

* accounts/abi/bind: cleaned up format... functions

* accounts/abi: improved log message

* accounts/abi: added type tests

* accounts/abi/bind: remove superfluous template methods
# Conflicts:
#	accounts/abi/bind/bind.go
#	accounts/abi/topics.go
#	accounts/abi/topics_test.go
2020-05-20 15:26:22 +03:00
gary rong
f86fe20ddc miner: support disabling empty blockprecommits form the Go API (#20736)
* cmd, miner: add noempty-precommit flag

* cmd, miner: get rid of external flag

* miner: change bool to atomic int

* miner: fix tiny typo

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	miner/worker.go
2020-05-20 15:26:22 +03:00
ucwong
0d2edd5653 core/rawdb: remove unused math (#21065) 2020-05-20 15:26:22 +03:00
ucwong
1ed0a13d69 core/rawdb : log format fix for Unindexing transaction (#21064)
* core/rawdb : log format fix for Unindexing transaction

* core/rawdb: tiny fixup

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-20 15:26:22 +03:00
gary rong
2b9820a167 les: drop the message if the entire p2p connection is stuck (#21033)
* les: drop the message if the entire p2p connection is stuck

* les: fix lint
# Conflicts:
#	les/peer.go
#	les/server_handler.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
42a23703f2 core: fixup blockchain tests (#21062)
core: fixup blockchain tests
# Conflicts:
#	core/blockchain_test.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
f66c118cad cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	consensus/clique/snapshot_test.go
#	core/bench_test.go
#	core/block_validator_test.go
#	core/blockchain.go
#	core/blockchain_test.go
#	core/chain_makers_test.go
#	core/dao_test.go
#	core/rawdb/accessors_indexes.go
#	core/rawdb/schema.go
#	eth/config.go
#	eth/helper_test.go
#	eth/sync.go
#	light/odr_test.go
#	light/trie_test.go
#	light/txpool_test.go
#	miner/worker_test.go
#	tests/block_test_util.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
c7cfd94cfc params: bump CHTs for the v1.9.14 release 2020-05-20 15:26:22 +03:00
Felix Lange
698d5eb28e p2p: add 0 port check in dialer (#21008)
* p2p: add low port check in dialer

We already have a check like this for UDP ports, add a similar one in
the dialer. This prevents dials to port zero and it's also an extra
layer of protection against spamming HTTP servers.

* p2p/discover: use errLowPort in v4 code

* p2p: change port check

* p2p: add comment

* p2p/simulations/adapters: ensure assigned port is in all node records
2020-05-20 15:26:22 +03:00
AusIV
2060287625 core/rawdb: stop freezer process as part of freezer.Close() (#21010)
* core/rawdb: Stop freezer process as part of freezer.Close()

When you call db.Close(), it was closing the leveldb database first,
then closing the freezer, but never stopping the freezer process.
This could cause the freezer to attempt to write to leveldb after
leveldb had been closed, leading to a crash with a non-zero exit code.

This change adds a quit channel to the freezer, and freezer.Close()
will not return until the freezer process has stopped.

Additionally, when you call freezerdb.Close(), it will close the
AncientStore before closing leveldb, to ensure that the freezer goroutine
will be stopped before leveldb is closed.

* core/rawdb: Fix formatting for golint

* core/rawdb: Use backoff flag to avoid repeating select

* core/rawdb: Include accidentally omitted backoff
# Conflicts:
#	core/rawdb/database.go
#	core/rawdb/freezer.go
2020-05-20 15:26:22 +03:00