Commit Graph

1389 Commits

Author SHA1 Message Date
Alex Sharov
90fa69a9e5
Swap IH and HashState stages (#922)
* move ih stage v0

* save progress

* save working version

* baby step

* baby step

* baby step

* non-baby step

* cleanup
2020-08-15 14:49:36 +01:00
Igor Mandrigin
f4102e115d
Update README.md 2020-08-13 10:42:32 +02:00
Evgeny Danilenko
5d39fe8873
RPCDeamon coinbase, net_version (#911)
* use eth.backend

* net_version

* add coinbase to the console test
2020-08-12 14:47:59 +01:00
Evgeny Danilenko
3980fa7d45
Grps eth_sendTransaction (#882)
* implementation

* tidy gomod

* linters

* fix cmd test

* fix

* fix lint
2020-08-11 22:09:30 +01:00
Alex Sharov
3899f2defe
recover_from_hashstate_stage_interruption (#903) 2020-08-11 17:41:06 +07:00
Alex Sharov
6df97dec22
save progress (#886) 2020-08-11 17:36:29 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Giulio rebuffo
057894994d
add blockhashes stage (#862)
* add blockhashes stage

* unwindOrder specified

* fixed typo

* added common.copybytes

* fixed little typo

* better progress

* better progress

* refactoring with etl

* added blockhashes to disabled stages in cmd/integration

* added migraations

* fixed go.mod

* better migrations

* rename unwind

* simplified migrations

* added onloadcommit
2020-08-10 18:46:06 +01:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Martin Holst Swende
e3e00089f4 eth/downloader: refactor downloader + queue (#21263)
* eth/downloader: refactor downloader + queue

downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache

downloader: more accurate deliverytime calculation, less mem overhead in state requests

downloader/queue: increase underlying buffer of results, new throttle mechanism

eth/downloader: updates to tests

eth/downloader: fix up some review concerns

eth/downloader/queue: minor fixes

eth/downloader: minor fixes after review call

eth/downloader: testcases for queue.go

eth/downloader: minor change, don't set progress unless progress...

eth/downloader: fix flaw which prevented useless peers from being dropped

eth/downloader: try to fix tests

eth/downloader: verify non-deliveries against advertised remote head

eth/downloader: fix flaw with checking closed-status causing hang

eth/downloader: hashing avoidance

eth/downloader: review concerns + simplify resultcache and queue

eth/downloader: add back some locks, address review concerns

downloader/queue: fix remaining lock flaw

* eth/downloader: nitpick fixes

* eth/downloader: remove the *2*3/4 throttling threshold dance

* eth/downloader: print correct throttle threshold in stats

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	eth/downloader/downloader.go
#	eth/downloader/downloader_test.go
#	eth/downloader/queue.go
#	eth/downloader/statesync.go
#	eth/fetcher/block_fetcher.go
2020-08-07 14:23:49 +02:00
Martin Holst Swende
a4b70f9ee9 core/vm: less allocations for various call variants (#21222)
* core/vm/runtime/tests: add more benchmarks

* core/vm: initial work on improving alloc count for calls to precompiles

name                                  old time/op    new time/op    delta
SimpleLoop/identity-precompile-10M-6     117ms ±75%      43ms ± 1%  -63.09%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   79.6ms ± 4%    70.5ms ± 1%  -11.42%  (p=0.008 n=5+5)

name                                  old alloc/op   new alloc/op   delta
SimpleLoop/identity-precompile-10M-6    24.4MB ± 0%     4.9MB ± 0%  -79.94%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   13.2kB ± 0%    13.2kB ± 0%     ~     (p=0.357 n=5+5)

name                                  old allocs/op  new allocs/op  delta
SimpleLoop/identity-precompile-10M-6      382k ± 0%      153k ± 0%  -59.99%  (p=0.000 n=5+4)
SimpleLoop/loop-10M-6                     40.0 ± 0%      40.0 ± 0%     ~     (all equal)

* core/vm: don't allocate big.int for touch

name                                  old time/op    new time/op    delta
SimpleLoop/identity-precompile-10M-6    43.3ms ± 1%    42.4ms ± 7%     ~     (p=0.151 n=5+5)
SimpleLoop/loop-10M-6                   70.5ms ± 1%    76.7ms ± 1%   +8.67%  (p=0.008 n=5+5)

name                                  old alloc/op   new alloc/op   delta
SimpleLoop/identity-precompile-10M-6    4.90MB ± 0%    2.46MB ± 0%  -49.83%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   13.2kB ± 0%    13.2kB ± 1%     ~     (p=0.571 n=5+5)

name                                  old allocs/op  new allocs/op  delta
SimpleLoop/identity-precompile-10M-6      153k ± 0%       76k ± 0%  -49.98%  (p=0.029 n=4+4)
SimpleLoop/loop-10M-6                     40.0 ± 0%      40.0 ± 0%     ~     (all equal)

* core/vm: reduce allocs in staticcall

name                                  old time/op    new time/op    delta
SimpleLoop/identity-precompile-10M-6    42.4ms ± 7%    37.5ms ± 6%  -11.68%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   76.7ms ± 1%    69.1ms ± 1%   -9.82%  (p=0.008 n=5+5)

name                                  old alloc/op   new alloc/op   delta
SimpleLoop/identity-precompile-10M-6    2.46MB ± 0%    0.02MB ± 0%  -99.35%  (p=0.008 n=5+5)
SimpleLoop/loop-10M-6                   13.2kB ± 1%    13.2kB ± 0%     ~     (p=0.143 n=5+5)

name                                  old allocs/op  new allocs/op  delta
SimpleLoop/identity-precompile-10M-6     76.4k ± 0%      0.1k ± 0%     ~     (p=0.079 n=4+5)
SimpleLoop/loop-10M-6                     40.0 ± 0%      40.0 ± 0%     ~     (all equal)

* trie: better use of hasher keccakState

* core/state/statedb: reduce allocations in getDeletedStateObject

* core/vm: reduce allocations in all call derivates

* core/vm: reduce allocations in call variants

- Make returnstack `uint32`
- Use a `sync.Pool` of `stack`s

* core/vm: fix tests

* core/vm: goimports

* core/vm: tracer fix + staticcall gas fix

* core/vm: add back snapshot to staticcall

* core/vm: review concerns + make returnstack pooled + enable returndata in traces

* core/vm: fix some test tracer method signatures

* core/vm: run gencodec, minor comment polish

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/state/statedb.go
#	core/vm/contracts_test.go
#	core/vm/evm.go
#	core/vm/instructions.go
#	core/vm/interpreter.go
#	core/vm/logger.go
#	core/vm/logger_json.go
#	core/vm/logger_test.go
#	core/vm/runtime/runtime_test.go
#	core/vm/stack/stack.go
#	eth/tracers/tracer.go
#	eth/tracers/tracer_test.go
#	trie/secure_trie.go
2020-08-07 13:46:26 +02:00
gary rong
8738c90219 les: historical data garbage collection (#19570)
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
# Conflicts:
#	cmd/geth/main.go
#	cmd/utils/flags.go
#	core/blockchain.go
#	core/chain_indexer.go
#	core/chain_indexer_test.go
#	core/chain_makers.go
#	core/dao_test.go
#	core/genesis.go
#	core/rawdb/accessors_chain_test.go
#	core/rawdb/accessors_indexes_test.go
#	core/rawdb/freezer.go
#	core/state/statedb_test.go
#	eth/downloader/downloader.go
#	eth/downloader/downloader_test.go
#	eth/downloader/testchain_test.go
#	eth/gen_config.go
#	internal/ethapi/api.go
#	internal/ethapi/backend.go
#	les/api_backend.go
#	les/client.go
#	les/odr_requests.go
#	les/odr_test.go
#	les/request_test.go
#	les/server.go
#	les/sync_test.go
#	les/test_helper.go
#	les/ulc_test.go
#	light/lightchain.go
#	light/odr.go
#	light/odr_util.go
#	light/postprocess.go
#	trie/database.go
#	trie/iterator_test.go
#	trie/trie_test.go
2020-08-07 13:16:46 +02:00
Martin Holst Swende
0d23ff42aa eth/downloader: fix peer idleness tracking when restarting state sync (#21260)
This fixes two issues with state sync restarts:

When sync restarts with a new root, some peers can have in-flight requests.
Since all peers with active requests were marked idle when exiting sync,
the new sync would schedule more requests for those peers. When the
response for the earlier request arrived, the new sync would reject it and
mark the peer idle again, rendering the peer useless until it disconnected.

The other issue was that peers would not be marked idle when they had
delivered a response, but the response hadn't been processed before
restarting the state sync. This also made the peer useless because it
would be permanently marked busy.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	eth/downloader/downloader.go
#	eth/downloader/statesync.go
2020-08-07 12:52:02 +02:00
Martin Holst Swende
45140488f0 eth: increase timeout in TestBroadcastBlock (#21299) 2020-08-07 12:49:19 +02:00
gary rong
1a99c54719 eth/gasprice: lighter gas price oracle for light client (#20409)
This PR reduces the bandwidth used by the light client to compute the
recommended gas price. The current mechanism for suggesting the price is:

- retrieve recent 20 blocks
- get the lowest gas price of these blocks
- sort the price array and return the middle(60%) one

This works for full nodes, which have all blocks available locally.
However, this is very expensive for the light client because the light
client needs to retrieve block bodies from the network.

The PR changes the default options for light client. With the new config,
the light client only retrieves the two latest blocks, but in order to
collect more sample transactions, the 3 lowest prices are collected from
each block.

This PR also changes the behavior for empty blocks. If the block is empty,
the lastest price is reused for sampling.

# Conflicts:
#	eth/gasprice/gasprice.go
2020-08-07 12:29:19 +02:00
Martin Holst Swende
5d0b4ca3ff rpc, internal/ethapi: default rpc gascap at 25M + better error message (#21229)
* rpc, internal/ethapi: default rpc gascap at 50M + better error message

* eth,internal: make globalgascap uint64

* core/tests: fix compilation failure

* eth/config: gascap at 25M + minor review concerns
# Conflicts:
#	core/tx_pool_test.go
#	internal/ethapi/api.go
#	les/api_backend.go
2020-08-07 12:27:17 +02:00
Marius van der Wijden
12de9c3f65 eth/downloader: fixes data race between synchronize and other methods (#21201)
* eth/downloaded: fixed datarace between synchronize and Progress

There was a race condition between `downloader.synchronize()` and `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders`
This PR changes the behavior of the downloader a bit.
Previously the functions `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders` read the syncMode anew within their loops. Now they read the syncMode at the start of their function and don't change it during their runtime.

* eth/downloaded: comment

* eth/downloader: added comment
# Conflicts:
#	eth/downloader/downloader.go
2020-08-07 11:51:38 +02:00
Marius van der Wijden
9de5600229 core: types: less allocations when hashing and tx handling (#21265)
* core, crypto: various allocation savings regarding tx handling

* core: reduce allocs for gas price comparison

This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees

* core: added tx.GasPriceIntCmp for comparison without allocation

adds a method to remove unneeded allocation in comparison to tx.gasPrice

* core/types: pool legacykeccak256 objects in rlpHash

rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.

* reverted some changes

* reverted some changes

* trie: use crypto.KeccakState instead of replicating code

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	core/tx_list.go
#	core/tx_pool.go
#	core/types/block.go
#	core/types/transaction.go
#	trie/committer.go
#	trie/hasher.go
2020-08-07 11:46:33 +02:00
Marius van der Wijden
0cd5872378 eth: returned revert reason in traceTx (#21195)
* eth: returned revert reason in traceTx

* eth: return result data
2020-08-07 11:23:14 +02:00
gary rong
20de87c1bf eth: don't block if transaction broadcast loop fails (#21255)
* eth: don't block if transaction broadcast loop is returned

* eth: kick out peer if we failed to send message

* eth: address comment
2020-08-07 11:22:09 +02:00
gary rong
2740fc90d9 cmd, eth, internal, les: add txfee cap (#21212)
* cmd, eth, internal, les: add gasprice cap

* cmd/utils, eth: add default value for gasprice cap

* all: use txfee cap

* cmd, eth: add fix

* cmd, internal: address comments
# Conflicts:
#	les/api_backend.go
2020-08-07 11:18:39 +02:00
sixdays
287abd928b eth/downloader: don't use defer for unlock before return (#21227)
Co-authored-by: linjing <linjingjing@baidu.com>
2020-08-07 11:17:56 +02:00
Martin Holst Swende
bb5f890c0c eth/downloader: fix spuriously failing tests (#21149)
* eth/downloader tests: fix spurious failing test due to race between receipts/headers

* miner tests: fix travis failure on arm64

* eth/downloader: tests - store td in ancients too
# Conflicts:
#	eth/downloader/downloader_test.go
#	miner/worker_test.go
2020-08-07 11:15:45 +02:00
Alexey Akhunov
b72fae75c9 Fix some snags in stage 6 2020-08-05 11:31:41 +01:00
Alexey Akhunov
11637c4c89 Fix some snags in stage 6 2020-08-05 11:25:57 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
Igor Mandrigin
8bc293b3f7
Remove unnecessary memory profiling (#872)
* remove separate memory profiling from staged sync

* don’t run pprof globally always
2020-08-05 10:54:19 +01:00
Igor Mandrigin
8b5903f6f2 docs: add a piechart with stages 2020-08-05 10:29:15 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
pk
91baa688bd
Update stagedsync.go (#866)
fixed a typo from `compiting` to `computing `
2020-08-04 09:03:47 +01:00
Alex Sharov
9c0f6cd166
stage3_block_compression_support (#859) 2020-08-03 14:42:10 +03:00
Igor Mandrigin
ad15e52391 fixup 2020-08-02 14:47:30 +02:00
Igor Mandrigin
3aedcbcd7d Use datadir for temp files when generating indexes 2020-08-02 12:32:41 +02:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
Igor Mandrigin
79ed493678
Fixup for TestSendTransactions65 (#838) 2020-07-31 10:44:42 +02:00
Alex Sharov
8c8433302e
remove eth/63, add tests for eth/65 (#823) 2020-07-30 11:16:55 +01:00
ledgerwatch
ddfec0828d
Delete receipts in unwind stage4 (#818) 2020-07-29 18:04:49 +01:00
ledgerwatch
ba6c6aaef2
Move BadHashes to stage1 (#812) 2020-07-29 15:33:01 +01:00
ledgerwatch
7bc45fab80
Remove preimage storage mode, write receipts by default (#809) 2020-07-29 14:00:39 +01:00
Alex Sharov
c5b5eacc7a
fix stage5 interruption (#800)
* fix stage5 interruption

* clean

* fix test

* Simplification

* Fix to the txpool logic and renaming

* Fix array size in stage3

* remove 'interrupted stage' concept, simplify integration state_stages

* remove blockchain interface

* revive staged sync tests

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-29 05:30:44 +01:00
ledgerwatch
32fa0ed3ec
Txpool test6 (#797)
* Fix tx propagation and announcement tests

* Fix checkpoint challenge test

* Save current state of code

* Fix compile errors

* Fixes

* Fix formatting

* Printouts

* Printouts

* Printouts

* Too much printing, not close db when stopping

* Better handing of interruptions

* No need to reset after incremental interruption

* Fix stages

* Fixes

* Giving up

* Fix formatting

* Cleanup
2020-07-27 23:00:08 +01:00
Alex Sharov
f500148b86
remove BoltDatabase, BadgerDatabase, RemoteDatabase classes (replaced by objectDb) (#794) 2020-07-27 22:59:54 +01:00
Alex Sharov
f037aed6e5
wait for verify_headers method (#793) 2020-07-27 19:26:24 +07:00
Igor Mandrigin
6ca3e36024
Use CalVer for turbo-geth versioning. (#792) 2020-07-27 14:18:26 +02:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb (#788) 2020-07-27 19:15:48 +07:00
b00ris
9fc51a3efd
Account range (#781)
* save state

* cleanup

* add walkasof test for storage with chunks

* fix lint

* fix lint

* remove useless comments

* fix dump test

* move accountRange test to plain state

* get last block fixes
2020-07-27 07:09:50 +01:00
Igor Mandrigin
d05cfa22f6
Make plain state + staged sync the default (#782)
* make plain state + staged sync the default

* remove sync mode flag completely

* one thing

* fix the console

* simplify code

* fix and skip tests

* fixup for console tests
2020-07-25 18:18:18 +01:00
ledgerwatch
cf5b0c8836
Integrate txPool into staged sync, fix tx_pool_test (#778)
* cmd/tester to initiate reorg

* Second commit

* Turn on profile

* More

* Unwind function

* More fixes

* Before fixing tests

* Fix compile error

* Fix lint

* Fix formatting

* Fix lint

* Fix nil in tests

* Fix lint

* Decouple txpool from rawdb

* Fix some tx tests

* One more test

* Fix the rest of tests in tx_pool_test.go

* Fix formatting
2020-07-23 21:52:41 +01:00
Igor Mandrigin
9b599b8ce4
Workaround panic in staged sync. (#777)
Fixes #776
2020-07-23 21:52:25 +01:00