Commit Graph

440 Commits

Author SHA1 Message Date
Igor Mandrigin
7bc892e886
Prefetch block bodies (#1013)
* prefetched blocks 1

* fixups

* fix logs

* fixups

* linters

* remove logs

* fix an NPE in tests

* setup the correct origin

* run blocks on prefetched bodies

* fix log

* smaller cache size
2020-08-30 18:34:40 +01:00
Igor Mandrigin
5365b8e402
stagedsync: create a StagedSync object, initialize in ProtocolManager (#1012)
* stagedsync: create a StagedSync object, initialize in ProtocolManager

* fix compile error
2020-08-30 13:24:58 +02:00
Igor Mandrigin
15e86b4b34 fix compilation issue 2020-08-29 15:30:06 +02:00
Igor Mandrigin
637493e8f1 Merge branch 'master' into geth-1.9.20 2020-08-29 15:28:03 +02:00
Igor Mandrigin
bd61245ece test fixups 2020-08-29 14:35:39 +02:00
Igor Mandrigin
4b4326020c geth compiles 2020-08-29 14:03:41 +02:00
Igor Mandrigin
948cd29be7 fixes 2020-08-29 13:43:44 +02:00
Igor Mandrigin
78e466d06a fix imports 2020-08-29 13:31:46 +02:00
Martin Holst Swende
fa3ee4bf6b eth: utilize sync bloom for getNodeData (#21445)
* eth/downloader, eth/handler: utilize sync bloom for getNodeData

* trie: handle if bloom is nil

* trie, downloader: check bloom nilness externally
# Conflicts:
#	eth/handler.go
2020-08-29 13:26:15 +02:00
gary rong
550bd11f9d core, eth, les, trie: add a prefix to contract code (#21080)
# Conflicts:
#	cmd/evm/internal/t8ntool/execution.go
#	consensus/clique/clique.go
#	consensus/ethash/consensus.go
#	core/block_validator.go
#	core/blockchain.go
#	core/blockchain_test.go
#	core/genesis.go
#	core/rawdb/accessors_indexes_test.go
#	core/rawdb/accessors_metadata.go
#	core/rawdb/chain_iterator_test.go
#	core/rawdb/database.go
#	core/rawdb/schema.go
#	core/state/database.go
#	core/state/iterator_test.go
#	core/state/statedb.go
#	core/state/sync.go
#	core/state/sync_test.go
#	core/tx_pool_test.go
#	core/types/block_test.go
#	core/types/derive_sha.go
#	eth/downloader/downloader.go
#	eth/downloader/queue.go
#	eth/downloader/statesync.go
#	eth/fetcher/block_fetcher.go
#	eth/fetcher/block_fetcher_test.go
#	eth/handler.go
#	les/odr_requests.go
#	les/server_handler.go
#	light/odr.go
#	light/odr_test.go
#	light/trie.go
#	miner/worker.go
#	trie/database.go
#	trie/secure_trie.go
#	trie/sync.go
#	trie/sync_bloom.go
#	trie/sync_test.go
2020-08-29 13:25:20 +02:00
Martin Holst Swende
b670cc44a5 eth/downloader: fix rollback issue on short chains 2020-08-29 12:30:17 +02:00
Péter Szilágyi
bfac425c87 core: define and test chain rewind corner cases (#21409)
* core: define and test chain reparation cornercases

* core: write up a variety of set-head tests

* core, eth: unify chain rollbacks, handle all the cases

* core: make linter smile

* core: remove commented out legacy code

* core, eth/downloader: fix review comments

* core: revert a removed recovery mechanism
# Conflicts:
#	core/blockchain.go
#	core/blockchain_test.go
#	core/headerchain.go
#	core/rawdb/database.go
#	core/rawdb/freezer.go
#	core/rawdb/schema.go
#	eth/downloader/downloader.go
#	eth/sync.go
#	trie/sync.go
2020-08-29 12:29:00 +02:00
Alex Sharov
46a5d9c275
clean (#1001) 2020-08-29 17:07:12 +07:00
Alex Sharov
ca0bc897dc
clear logs (#1000) 2020-08-29 17:06:53 +07:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
Alex Sharov
de16812a94
move tx limit from 16 to 32 blocks (#988) 2020-08-28 07:28:08 +01:00
Alex Sharov
df05a581dd
Add tx pool stage to transaction (#981)
* add tx_pool stage to tx

* clean

* don't call commit in read tx

* don't call commit in read tx

* clean

* more logs

* fix tx pool race

* more logs

* clean
2020-08-28 07:27:48 +01:00
Alex Sharov
714e5041e2
Better logs3 (#976) 2020-08-26 19:06:51 +07:00
Alex Sharov
522287ac18
Transactional cycle (#966)
* v0

* v1

* v3

* v4

* clean

* temporary fix of txpool

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* add condition

* tx pool to not hold own db

* try to enable TxPool in integration

* exclude tx pool from tx

* exclude tx pool from integration

* reduce limit

* fix integration

* clean

* clean

* clean

* clean

* clean

* exclude tx pool unwind

* exclude tx pool unwind in integration

* fix integration tx pool

* fix commit

* fix current stage after unwind

* fix current stage after unwind

* fix linter

* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.

* move body and headers unwind out of tx

* fix unwind order after reboot

* add support external tx to exec stage

* clean

* clean

* clean

* clean

* clean

* add logs

* better id check

* better id check
2020-08-26 07:02:10 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Igor Mandrigin
db6a0c43ee linter 2020-08-15 22:17:38 +02:00
Igor Mandrigin
9cc5690483 fix downloader tests 2020-08-15 20:32:50 +02:00
rene
b03de86a5c downloader: add eth65 tests (#21383)
* eth65 tests

linted

* remove non-latest eth light tests
# Conflicts:
#	eth/downloader/downloader_test.go
2020-08-14 16:39:16 +02:00
rene
018cfd0b6d eth/downloader: remove eth62 (#21378)
* init

notes

removed some mentions of eth62, bumped protocol err too old to >=63

* remove sanity checks and bump supported protocol version up to 63

* remove 62 tests, still need to add 65

* remove 65 tests
# Conflicts:
#	eth/downloader/downloader_test.go
#	eth/downloader/peer.go
2020-08-12 12:42:36 +02:00
Giulio rebuffo
057894994d
add blockhashes stage (#862)
* add blockhashes stage

* unwindOrder specified

* fixed typo

* added common.copybytes

* fixed little typo

* better progress

* better progress

* refactoring with etl

* added blockhashes to disabled stages in cmd/integration

* added migraations

* fixed go.mod

* better migrations

* rename unwind

* simplified migrations

* added onloadcommit
2020-08-10 18:46:06 +01:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Martin Holst Swende
e3e00089f4 eth/downloader: refactor downloader + queue (#21263)
* eth/downloader: refactor downloader + queue

downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache

downloader: more accurate deliverytime calculation, less mem overhead in state requests

downloader/queue: increase underlying buffer of results, new throttle mechanism

eth/downloader: updates to tests

eth/downloader: fix up some review concerns

eth/downloader/queue: minor fixes

eth/downloader: minor fixes after review call

eth/downloader: testcases for queue.go

eth/downloader: minor change, don't set progress unless progress...

eth/downloader: fix flaw which prevented useless peers from being dropped

eth/downloader: try to fix tests

eth/downloader: verify non-deliveries against advertised remote head

eth/downloader: fix flaw with checking closed-status causing hang

eth/downloader: hashing avoidance

eth/downloader: review concerns + simplify resultcache and queue

eth/downloader: add back some locks, address review concerns

downloader/queue: fix remaining lock flaw

* eth/downloader: nitpick fixes

* eth/downloader: remove the *2*3/4 throttling threshold dance

* eth/downloader: print correct throttle threshold in stats

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	eth/downloader/downloader.go
#	eth/downloader/downloader_test.go
#	eth/downloader/queue.go
#	eth/downloader/statesync.go
#	eth/fetcher/block_fetcher.go
2020-08-07 14:23:49 +02:00
gary rong
8738c90219 les: historical data garbage collection (#19570)
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
# Conflicts:
#	cmd/geth/main.go
#	cmd/utils/flags.go
#	core/blockchain.go
#	core/chain_indexer.go
#	core/chain_indexer_test.go
#	core/chain_makers.go
#	core/dao_test.go
#	core/genesis.go
#	core/rawdb/accessors_chain_test.go
#	core/rawdb/accessors_indexes_test.go
#	core/rawdb/freezer.go
#	core/state/statedb_test.go
#	eth/downloader/downloader.go
#	eth/downloader/downloader_test.go
#	eth/downloader/testchain_test.go
#	eth/gen_config.go
#	internal/ethapi/api.go
#	internal/ethapi/backend.go
#	les/api_backend.go
#	les/client.go
#	les/odr_requests.go
#	les/odr_test.go
#	les/request_test.go
#	les/server.go
#	les/sync_test.go
#	les/test_helper.go
#	les/ulc_test.go
#	light/lightchain.go
#	light/odr.go
#	light/odr_util.go
#	light/postprocess.go
#	trie/database.go
#	trie/iterator_test.go
#	trie/trie_test.go
2020-08-07 13:16:46 +02:00
Martin Holst Swende
0d23ff42aa eth/downloader: fix peer idleness tracking when restarting state sync (#21260)
This fixes two issues with state sync restarts:

When sync restarts with a new root, some peers can have in-flight requests.
Since all peers with active requests were marked idle when exiting sync,
the new sync would schedule more requests for those peers. When the
response for the earlier request arrived, the new sync would reject it and
mark the peer idle again, rendering the peer useless until it disconnected.

The other issue was that peers would not be marked idle when they had
delivered a response, but the response hadn't been processed before
restarting the state sync. This also made the peer useless because it
would be permanently marked busy.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	eth/downloader/downloader.go
#	eth/downloader/statesync.go
2020-08-07 12:52:02 +02:00
Marius van der Wijden
12de9c3f65 eth/downloader: fixes data race between synchronize and other methods (#21201)
* eth/downloaded: fixed datarace between synchronize and Progress

There was a race condition between `downloader.synchronize()` and `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders`
This PR changes the behavior of the downloader a bit.
Previously the functions `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders` read the syncMode anew within their loops. Now they read the syncMode at the start of their function and don't change it during their runtime.

* eth/downloaded: comment

* eth/downloader: added comment
# Conflicts:
#	eth/downloader/downloader.go
2020-08-07 11:51:38 +02:00
sixdays
287abd928b eth/downloader: don't use defer for unlock before return (#21227)
Co-authored-by: linjing <linjingjing@baidu.com>
2020-08-07 11:17:56 +02:00
Martin Holst Swende
bb5f890c0c eth/downloader: fix spuriously failing tests (#21149)
* eth/downloader tests: fix spurious failing test due to race between receipts/headers

* miner tests: fix travis failure on arm64

* eth/downloader: tests - store td in ancients too
# Conflicts:
#	eth/downloader/downloader_test.go
#	miner/worker_test.go
2020-08-07 11:15:45 +02:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
Alex Sharov
8c8433302e
remove eth/63, add tests for eth/65 (#823) 2020-07-30 11:16:55 +01:00
Alex Sharov
c5b5eacc7a
fix stage5 interruption (#800)
* fix stage5 interruption

* clean

* fix test

* Simplification

* Fix to the txpool logic and renaming

* Fix array size in stage3

* remove 'interrupted stage' concept, simplify integration state_stages

* remove blockchain interface

* revive staged sync tests

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-29 05:30:44 +01:00
ledgerwatch
32fa0ed3ec
Txpool test6 (#797)
* Fix tx propagation and announcement tests

* Fix checkpoint challenge test

* Save current state of code

* Fix compile errors

* Fixes

* Fix formatting

* Printouts

* Printouts

* Printouts

* Too much printing, not close db when stopping

* Better handing of interruptions

* No need to reset after incremental interruption

* Fix stages

* Fixes

* Giving up

* Fix formatting

* Cleanup
2020-07-27 23:00:08 +01:00
Igor Mandrigin
d05cfa22f6
Make plain state + staged sync the default (#782)
* make plain state + staged sync the default

* remove sync mode flag completely

* one thing

* fix the console

* simplify code

* fix and skip tests

* fixup for console tests
2020-07-25 18:18:18 +01:00
ledgerwatch
cf5b0c8836
Integrate txPool into staged sync, fix tx_pool_test (#778)
* cmd/tester to initiate reorg

* Second commit

* Turn on profile

* More

* Unwind function

* More fixes

* Before fixing tests

* Fix compile error

* Fix lint

* Fix formatting

* Fix lint

* Fix nil in tests

* Fix lint

* Decouple txpool from rawdb

* Fix some tx tests

* One more test

* Fix the rest of tests in tx_pool_test.go

* Fix formatting
2020-07-23 21:52:41 +01:00
Alex Sharov
621e3a2074
Ci lmdb - reduce memory usage (#762) 2020-07-21 08:58:00 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
Alex Sharov
8b8094d72c
integration: fix problem with interruption (#743)
* store stageData correctly

* integration: use State of sync, use stage data, don't skip persistent steps even if they complete before interruption

* reset_state: clear unwind stages info

* stage4: stop where previous stage stopped, print to logs digits which represent what exactly is in DB, rename variable 'nextBlockNumber' to 'stateProgress'

* rebase to master

* simplify chainconfig

* try increase sleep time on ci

* try increase sleep time on ci

* try increase sleep time on ci

* try increase sleep time on ci
2020-07-15 12:53:12 +01:00
ledgerwatch
3359ba7a04
Make cmd/tester initiate staged sync, remove some pools and caches (#746)
* no pools

* Remove excessive logging

* Revert

* Initial commit

* Cleanup

* Fix some compile errors

* Fix core tests

* Fix more tests

* Fix eth/downloader

* go mod tidy

* Fix core/vm

* Fix formatting

* Fix formatting

* Fix lint

* Fix lint

* Switch tests to StagedSync

* Cleanup

* Reuse cache in stage4

* Fix

* Fix formatting

* Try to fix test
2020-07-15 07:15:48 +01:00
ledgerwatch
665b1d88b9
Speed up GenerateChain by using intermediate hashes (#736)
* GenerateChain to support intermediate hashes

* Fix formatting

* Changeset stats

* Fix formatting
2020-07-10 22:37:34 +01:00
Evgeny Danilenko
5f77570556
Make transaction pool startable and stoppable. Staged sync. (#730)
* after concurrency testing

* fixes for linters

* add an extra parameter

* change init order for protocolManager

* green tests
2020-07-09 13:53:35 +01:00
ledgerwatch
887fc4f99c
GenerateChain without using TrieDbState (#726)
* Different chain generation

* More fixes

* Fix TestSideLogRebirth

* Fix the rest of core tests

* Fix accounts/abi/bind

* Fix compile errors for cmd/pics

* Fix consensus/clique tests

* More fixes

* Fix core/state tests

* Fix eth tests

* Fix eth/downloader tests

* Fix more tests

* Fix compile errors

* Some more fixes

* Save

* Exists

* Fix last tests

* Clean up

* Fix formatting

* Fix linter

* Fix formatting

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint
2020-07-09 07:15:28 +01:00
Giulio rebuffo
011a15350b
fix stage 2 (#705)
* removed unused actions in stage 2

* fixed stage 2
2020-07-02 19:44:08 +01:00
ledgerwatch
5812a649ce
Combine stage5 and stage6 (#679)
* Handle bucket error

* Replace with 0

* SetMaxDBs

* Set MaxDb before opening'

* Merge stage5 and stage6

* Fix lint

* Make downloader tests not parallel
2020-06-20 10:07:22 +01:00
Alex Sharov
8d04689041
close db in tests (#669) 2020-06-16 12:05:40 +07:00
ledgerwatch
dec30b698d
Fix execution for block 10094566 (#665)
* First sketch of the fix

* Re-enable stage4

* Fix formatting

* Park the changes

* Save updates

* Save update

* Cleanup

* Cleanup

* Cleanup

* Fix lint

* Cleanup

* Update

* Tx tracing

* Update

* Cleanup

* Switch tests back to bolt

* Test cleanup

* Update

* Polymorthic CREATE2 test

* Add assertions

* Fix formatting

* Fix test

* Cleanup

* Cleanup

* Cleanup
2020-06-15 22:24:08 +01:00
Igor Mandrigin
93f6c4c01a post-rebase fixups 2020-06-15 19:38:54 +03:00