Commit Graph

11986 Commits

Author SHA1 Message Date
ledgerwatch
f6be0b601e
Fix history generation and optimise via pre-sorting in the files (#572)
* Rewrite

* Fix linter, make counter

* Skip hash stage

* Fix linter

* Add common

* Fix wrong bucket

* Use temp files to generate index

* Fix hack, defer

* Add logging when creating files

* Properly pass datadir

* Bigger buffers, less logging

* Log current key, enforce batching

* Limit to 4m, print more

* Use ReadFull

* Optimised version of storage generation

* Don't print entire key

* Commit at the end

* Fix linter

* Remove a bit of copy-paste

* More copy-paste reduction

* Fixes

* Fix key length

* Not delete files too early

* Fix linter

* Fix logging

* Fix starting block

* Skip test

* Remove limit on the stage 4

* Fix test while disabling the hash check

* Fix tests

* Fix unreachable
2020-05-24 15:48:15 +01:00
Andrew Ashikhmin
9f89cf810d
mutation.puts.Size was confused for its number of entries (#570) 2020-05-23 11:27:05 +01:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices (#569)
* save state

* add current index feature

* fix test

* remove logs

* Only execute 1000 blocks

* Reset history index

* Correct action

* Increase batch size

* Increase chunk size, print memory stats

* Fix linter

* Remove unused from

* Split into 2 staged

* Use storage history gen

* remove log

* Not to run tx_cacher in staged mode

* Not to recover during stage 2

* Not to recover during stage 2

* Remove counter

Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
ledgerwatch
6d883b5350
Fix staged sync issue introduced by LRU caches (#568)
* readPlainAccount

* readPlainAccount

* Try to disable accountCache

* Try to not separate changeBatch

* The actual fix
?

* Move to fastcache, include incarnation

* Fix linter

* Fix linter

* Disable caches

* Fix linter

* Reenable all except storage cache

* Cosmetics

* Disable account cache

* Disable code size cache

* Fix code caching

* Apply the same fix to db_state_writer
2020-05-22 16:11:00 +01:00
ledgerwatch
8b97c5c620
Remove memory leak caused by accountCache, count batch size in bytes (#565)
* Lower cache sizes

* Add memory profiling over http

* No code cache

* Calculate db batch size in bytes

* Fixes

* Fixes

* Increase batch

* Fix linter

* Restore account caching, with copying

* Reintroduce code cache

* Add fixed overhead per key

* Print batch size

* Fix batch size

* Reduce batch size

* 50 Mb

* Fix linter
2020-05-21 21:55:39 +01:00
ledgerwatch
e2848ac666
Staged exec: RLU cache between StateReader and StateWriter, separate batches for state and changesets (#542)
* Add cpu profiling

* Splt state and change batches

* Log commits

* Remove

* Log memory

* Add lru caches for staged execution

* Fix

* Fix for nil account

* Fix linter

* Fix linter

* Increase state batch size

* Add memory profiling over http

* Reduce cache sizes

* Fix storage cache

* Fix

* Fix

* Fix

* Fix

* Cleanup

* Fix linter

* Reduce logging

* Reduce logging
2020-05-21 13:27:52 +01:00
Andrew Ashikhmin
d92d0fc706
Don't copy jump table for every new EVMInterpreter since it's quite heavy (#564)
* Avoid copying operation in EVMInterpreter.Run

* Don't copy jump table for every new EVMInterpreter since it's quite heavy
2020-05-21 10:12:09 +01:00
ledgerwatch
5904332035
hack --action resetState to create both hashed and plain genesis, clean all buckets (#563)
* Reset plain genesis

* Change to optimised freelist implementation

* Fix linter

* Revert "Change to optimised freelist implementation"

This reverts commit 65e28f1737801178e95585381a789785504ec318.

* Noerrcheck
2020-05-21 06:18:25 +01:00
Igor Mandrigin
12a6b2f9f6
Use upstream EVMC (#561)
* Use upstream EVMC

* update to 7.3.0
2020-05-20 19:35:32 +01:00
ledgerwatch
e322961dd2
Non scalable but correct implementation of eth_getProof (#562)
* Prototype works

* Fix linter, start actual implementation

* More on getProof implementation

* Fixes

* Fix storage proofs

* Fix linter, start adding to rpctest

* Fixes in eth_getProof and rpctest

* Fix linter

* Reenable check for debug_traceTransation

* Cleanup
2020-05-20 15:32:50 +01:00
Andrew Ashikhmin
9c47df0a2c
Use math.bits in common math Safe* for performance (#556)
* Use math.bits in comoon math Safe* for performance (the compiler optimizes away  math.bits)

* Prettier code in opExp
2020-05-20 14:31:23 +02:00
Andrew Ashikhmin
ff23652d50
GC: reducing copying in stateObject.updateTrie (#557)
* It's unnecessary to copy key in stateObject.updateTrie

* Move value copying from stateObject.updateTrie down to TrieStateWriter
2020-05-20 14:31:03 +02:00
Igor Mandrigin
0dae3ade0d post-rebase fixups 2020-05-20 15:26:22 +03:00
Péter Szilágyi
7cdf2aa95a params: release go-ethereum v1.9.14 2020-05-20 15:26:22 +03:00
Hao Duan
6fe80da4c8 ethstats: stop report ticker in each loop cycle #21070 (#21071)
Co-authored-by: Hao Duan <duan.hao@hyperchain.cn>
2020-05-20 15:26:22 +03:00
gary rong
1aa6ead45f core: fix missing receipt on Clique crashes (#21045)
* core: fix missing receipt

* core: address comment
# Conflicts:
#	core/blockchain.go
2020-05-20 15:26:22 +03:00
Marius van der Wijden
c9fb0f9a3f accounts/abi: allow overloaded argument names (#21060)
* accounts/abi: allow overloaded argument names

In solidity it is possible to create the following contract:
```
contract Overloader {
    struct F { uint _f; uint __f; uint f; }
    function f(F memory f) public {}
}
```
This however resulted in a panic in the abi package.

* accounts/abi fixed error handling
2020-05-20 15:26:22 +03:00
Marius van der Wijden
fc91ce52eb accounts/abi: accounts/abi/bind: Move topics to abi package (#21057)
* accounts/abi/bind: added test cases for waitDeployed

* accounts/abi/bind: added test case for boundContract

* accounts/abi/bind: removed unnecessary resolve methods

* accounts/abi: moved topics from /bind to /abi

* accounts/abi/bind: cleaned up format... functions

* accounts/abi: improved log message

* accounts/abi: added type tests

* accounts/abi/bind: remove superfluous template methods
# Conflicts:
#	accounts/abi/bind/bind.go
#	accounts/abi/topics.go
#	accounts/abi/topics_test.go
2020-05-20 15:26:22 +03:00
gary rong
f86fe20ddc miner: support disabling empty blockprecommits form the Go API (#20736)
* cmd, miner: add noempty-precommit flag

* cmd, miner: get rid of external flag

* miner: change bool to atomic int

* miner: fix tiny typo

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	miner/worker.go
2020-05-20 15:26:22 +03:00
ucwong
0d2edd5653 core/rawdb: remove unused math (#21065) 2020-05-20 15:26:22 +03:00
ucwong
1ed0a13d69 core/rawdb : log format fix for Unindexing transaction (#21064)
* core/rawdb : log format fix for Unindexing transaction

* core/rawdb: tiny fixup

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-20 15:26:22 +03:00
gary rong
2b9820a167 les: drop the message if the entire p2p connection is stuck (#21033)
* les: drop the message if the entire p2p connection is stuck

* les: fix lint
# Conflicts:
#	les/peer.go
#	les/server_handler.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
42a23703f2 core: fixup blockchain tests (#21062)
core: fixup blockchain tests
# Conflicts:
#	core/blockchain_test.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
f66c118cad cmd, core, eth: background transaction indexing (#20302)
* cmd, core, eth: init tx lookup in background

* core/rawdb: tiny log fixes to make it clearer what's happening

* core, eth: fix rebase errors

* core/rawdb: make reindexing less generic, but more optimal

* rlp: implement rlp list iterator

* core/rawdb: new implementation of tx indexing/unindex using generic tx iterator and hashing rlp-data

* core/rawdb, cmd/utils: fix review concerns

* cmd/utils: fix merge issue

* core/rawdb: add some log formatting polishes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	consensus/clique/snapshot_test.go
#	core/bench_test.go
#	core/block_validator_test.go
#	core/blockchain.go
#	core/blockchain_test.go
#	core/chain_makers_test.go
#	core/dao_test.go
#	core/rawdb/accessors_indexes.go
#	core/rawdb/schema.go
#	eth/config.go
#	eth/helper_test.go
#	eth/sync.go
#	light/odr_test.go
#	light/trie_test.go
#	light/txpool_test.go
#	miner/worker_test.go
#	tests/block_test_util.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
c7cfd94cfc params: bump CHTs for the v1.9.14 release 2020-05-20 15:26:22 +03:00
Felix Lange
698d5eb28e p2p: add 0 port check in dialer (#21008)
* p2p: add low port check in dialer

We already have a check like this for UDP ports, add a similar one in
the dialer. This prevents dials to port zero and it's also an extra
layer of protection against spamming HTTP servers.

* p2p/discover: use errLowPort in v4 code

* p2p: change port check

* p2p: add comment

* p2p/simulations/adapters: ensure assigned port is in all node records
2020-05-20 15:26:22 +03:00
AusIV
2060287625 core/rawdb: stop freezer process as part of freezer.Close() (#21010)
* core/rawdb: Stop freezer process as part of freezer.Close()

When you call db.Close(), it was closing the leveldb database first,
then closing the freezer, but never stopping the freezer process.
This could cause the freezer to attempt to write to leveldb after
leveldb had been closed, leading to a crash with a non-zero exit code.

This change adds a quit channel to the freezer, and freezer.Close()
will not return until the freezer process has stopped.

Additionally, when you call freezerdb.Close(), it will close the
AncientStore before closing leveldb, to ensure that the freezer goroutine
will be stopped before leveldb is closed.

* core/rawdb: Fix formatting for golint

* core/rawdb: Use backoff flag to avoid repeating select

* core/rawdb: Include accidentally omitted backoff
# Conflicts:
#	core/rawdb/database.go
#	core/rawdb/freezer.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
470c003d1e console: fix some crashes/errors in the bridge (#21050)
Fixes #21046
2020-05-20 15:26:22 +03:00
Marius van der Wijden
322f4f31c5 params, cmd/utils: remove outdated discv5 bootnodes, deprecate flags (#20949)
* params: remove outdated discv5 bootnodes

* cmd/utils: deprecated bootnodesv4/v5 flags
# Conflicts:
#	mobile/params.go
2020-05-20 15:26:22 +03:00
gary rong
5f7d454115 accounts/abi/bind/backend, internal/ethapi: recap gas limit with balance (#21043)
* accounts/abi/bind/backend, internal/ethapi: recap gas limit with balance

* accounts, internal: address comment and fix lint

* accounts, internal: extend log message

* tiny nits to format hexutil.Big and nil properly

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
27063780f4 core/state: make GetCodeSize mirror GetCode implementation wise
# Conflicts:
#	core/state/state_object.go
#	core/state/statedb.go
2020-05-20 15:26:22 +03:00
ligi
67589d1658 build: fix CLI params for windows LNK files (#21055)
* build: Fix CLI params for windows LNK files

closes #21054

* Remove parameters
2020-05-20 15:26:22 +03:00
Martin Holst Swende
4b33806719 core/state: avoid statedb.dbErr due to emptyCode (#21051)
* core/state: more verbose stateb errors

* core/state: fix flaw

* core/state: fixed lint

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
# Conflicts:
#	core/state/statedb.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
dc5eee4860 core/state: abort commit if read errors have occurred (#21039)
This finally adds the error check that the documentation of StateDB.dbErr
promises to do. dbErr was added in 9e5f03b6c (June 2017), and the check was
already missing in that commit. We somehow survived without it for three years.
# Conflicts:
#	core/state/statedb.go
#	core/state/statedb_test.go
2020-05-20 15:26:22 +03:00
Martin Holst Swende
ab83455837 core/state: include zero-address in state dump if present (#21038)
* Include 0x0000 address into the dump if it is present

* core/state: go fmt

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
# Conflicts:
#	core/state/dump.go
2020-05-20 15:26:22 +03:00
Marius van der Wijden
a222f5f6cb accounts/abi/bind: add void if no return args specified (#21002)
* accounts/abi/bind: add void if no return args specified

Currently the java generator generates invalid input on pure/view functions
that have no return type. e.g. `function f(uint u) view public {}`
This is not a problem in practice as people rarely ever write functions like this.

* accounts/abi/bind: use elseif instead of nested if
2020-05-20 15:26:22 +03:00
gary rong
cd6eacfd2c core/state/snapshot: fix typo (#21037)
# Conflicts:
#	core/state/snapshot/difflayer.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
f30a6fd32a core/state/snapshot: don't create storage list for non-existing accounts
# Conflicts:
#	core/state/snapshot/difflayer.go
2020-05-20 15:26:22 +03:00
Boqin Qin
b42bb4db08 core: avoid double-lock in tx_pool_test (#20984) 2020-05-20 15:26:22 +03:00
Guillaume Ballet
61d8a49ece eth/downloader: minor typo fixes in comments (#21035)
# Conflicts:
#	trie/sync.go
2020-05-20 15:26:22 +03:00
gary rong
f0ffa8a8b4 trie: fix TestBadRangeProof unit test (#21034)
# Conflicts:
#	trie/proof_test.go
2020-05-20 15:26:22 +03:00
ploui
1d0688c7cc eth: don't inadvertently enable snapshots in archive nodes (#21025)
* eth: don't reassign more cache than is being made available

* eth: don't inadvertently enable snapshot in a case where --snapshot wasn't given
# Conflicts:
#	eth/backend.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
60086a33e3 eth: skip transaction announcer goroutine on eth<65 2020-05-20 15:26:22 +03:00
Marius van der Wijden
ffc0f9e85b account/abi: remove superfluous type checking (#21022)
* accounts/abi: added getType func to Type struct

* accounts/abi: fixed tuple unpack

* accounts/abi: removed type.Type

* accounts/abi: added comment

* accounts/abi: removed unused types

* accounts/abi: removed superfluous declarations

* accounts/abi: typo
# Conflicts:
#	accounts/abi/numbers.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
3b2cbc4572 build: raise test timeout back to 10 mins (#21027) 2020-05-20 15:26:22 +03:00
Péter Szilágyi
28d3cbb176 cmd/geth: handle memfixes on 32bit arch with large RAM 2020-05-20 15:26:22 +03:00
Péter Szilágyi
99b990b0af tests: skip consensus test using 1GB RAM 2020-05-20 15:26:22 +03:00
rene
2b0cf4d1ee cmd/utils: renames flags related to http-rpc server (#20935)
* rpc flags related to starting http server renamed to http

* old rpc flags aliased and still functional

* pprof flags fixed

* renames gpo related flags

* linted

* renamed rpc flags for consistency and clarity

* added warn logs

* added more warn logs for all deprecated flags for consistency

* moves legacy flags to separate file, hides older flags under show-deprecated-flags command

* legacy prefix and moved some more legacy flags to legacy file

* fixed circular import

* added docs

* fixed imports lint error

* added notes about when flags were deprecated

* cmd/utils: group flags by deprecation date + reorder by date,

* modified deprecated comments for consistency, added warn log for --rpc

* making sure deprecated flags are still functional

* show-deprecated-flags command cleaned up

* fixed lint errors

* corrected merge conflict

* IsSet --> GlobalIsSet

* uncategorized flags, if not deprecated, displayed under misc

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	internal/debug/flags.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
3bea6f911c core/state/snapshot: release iterator after verification 2020-05-20 15:26:22 +03:00
Igor Mandrigin
46f9df7821 accounts/abi: move U256Bytes to common/math (#21020)
# Conflicts:
#	accounts/abi/abi_test.go
#	accounts/abi/numbers.go
#	signer/core/signed_data.go
2020-05-20 15:26:22 +03:00