Commit Graph

84 Commits

Author SHA1 Message Date
Alexey Akhunov
b72fae75c9 Fix some snags in stage 6 2020-08-05 11:31:41 +01:00
Alexey Akhunov
11637c4c89 Fix some snags in stage 6 2020-08-05 11:25:57 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
Igor Mandrigin
8bc293b3f7
Remove unnecessary memory profiling (#872)
* remove separate memory profiling from staged sync

* don’t run pprof globally always
2020-08-05 10:54:19 +01:00
Igor Mandrigin
8b5903f6f2 docs: add a piechart with stages 2020-08-05 10:29:15 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
pk
91baa688bd
Update stagedsync.go (#866)
fixed a typo from `compiting` to `computing `
2020-08-04 09:03:47 +01:00
Alex Sharov
9c0f6cd166
stage3_block_compression_support (#859) 2020-08-03 14:42:10 +03:00
Igor Mandrigin
ad15e52391 fixup 2020-08-02 14:47:30 +02:00
Igor Mandrigin
3aedcbcd7d Use datadir for temp files when generating indexes 2020-08-02 12:32:41 +02:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
ledgerwatch
ddfec0828d
Delete receipts in unwind stage4 (#818) 2020-07-29 18:04:49 +01:00
ledgerwatch
ba6c6aaef2
Move BadHashes to stage1 (#812) 2020-07-29 15:33:01 +01:00
Alex Sharov
c5b5eacc7a
fix stage5 interruption (#800)
* fix stage5 interruption

* clean

* fix test

* Simplification

* Fix to the txpool logic and renaming

* Fix array size in stage3

* remove 'interrupted stage' concept, simplify integration state_stages

* remove blockchain interface

* revive staged sync tests

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-29 05:30:44 +01:00
ledgerwatch
32fa0ed3ec
Txpool test6 (#797)
* Fix tx propagation and announcement tests

* Fix checkpoint challenge test

* Save current state of code

* Fix compile errors

* Fixes

* Fix formatting

* Printouts

* Printouts

* Printouts

* Too much printing, not close db when stopping

* Better handing of interruptions

* No need to reset after incremental interruption

* Fix stages

* Fixes

* Giving up

* Fix formatting

* Cleanup
2020-07-27 23:00:08 +01:00
Alex Sharov
f037aed6e5
wait for verify_headers method (#793) 2020-07-27 19:26:24 +07:00
Igor Mandrigin
d05cfa22f6
Make plain state + staged sync the default (#782)
* make plain state + staged sync the default

* remove sync mode flag completely

* one thing

* fix the console

* simplify code

* fix and skip tests

* fixup for console tests
2020-07-25 18:18:18 +01:00
ledgerwatch
cf5b0c8836
Integrate txPool into staged sync, fix tx_pool_test (#778)
* cmd/tester to initiate reorg

* Second commit

* Turn on profile

* More

* Unwind function

* More fixes

* Before fixing tests

* Fix compile error

* Fix lint

* Fix formatting

* Fix lint

* Fix nil in tests

* Fix lint

* Decouple txpool from rawdb

* Fix some tx tests

* One more test

* Fix the rest of tests in tx_pool_test.go

* Fix formatting
2020-07-23 21:52:41 +01:00
Alex Sharov
e65b743878
Prometheus dashboard revisit (#767) 2020-07-21 15:33:03 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
Alex Sharov
7f60d1b535
Remove load start key from etl (#757)
* limit incremental step size of stages 5 and 6

* remove limit, make incremental

* remove_load_start_key_from_etl

* switch to "struct of arrays" from "array of structs"

* switch to "struct of arrays" from "array of structs"

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-18 10:17:04 +01:00
Evgeny Danilenko
f79b156a0b
delete keys (#756) 2020-07-17 08:34:52 +01:00
Alex Sharov
c8a594dec6
Limit incremental step size of stages 5 and 6. (#754) 2020-07-17 12:52:09 +07:00
Alex Sharov
4d54f0ec0c
fix error "could not apply NextKey for the empty key" (#753) 2020-07-17 06:59:22 +07:00
Alex Sharov
086435117f
remove atomic from stage4 (#751) 2020-07-17 06:59:03 +07:00
ledgerwatch
18e67cd07d
cmd/tester to initiate correct staged sync (but not reorg yet) (#747)
* Disable caches

* Fix cmd/integration?

* state root hash matches

* Cleanup

* Skip TestForkIDSplit

* Try to fix the test
2020-07-15 13:26:59 +01:00
Alex Sharov
8b8094d72c
integration: fix problem with interruption (#743)
* store stageData correctly

* integration: use State of sync, use stage data, don't skip persistent steps even if they complete before interruption

* reset_state: clear unwind stages info

* stage4: stop where previous stage stopped, print to logs digits which represent what exactly is in DB, rename variable 'nextBlockNumber' to 'stateProgress'

* rebase to master

* simplify chainconfig

* try increase sleep time on ci

* try increase sleep time on ci

* try increase sleep time on ci

* try increase sleep time on ci
2020-07-15 12:53:12 +01:00
ledgerwatch
3359ba7a04
Make cmd/tester initiate staged sync, remove some pools and caches (#746)
* no pools

* Remove excessive logging

* Revert

* Initial commit

* Cleanup

* Fix some compile errors

* Fix core tests

* Fix more tests

* Fix eth/downloader

* go mod tidy

* Fix core/vm

* Fix formatting

* Fix formatting

* Fix lint

* Fix lint

* Switch tests to StagedSync

* Cleanup

* Reuse cache in stage4

* Fix

* Fix formatting

* Try to fix test
2020-07-15 07:15:48 +01:00
Evgeny Danilenko
2ce24c3274
dont stop txPool stage on error (#745) 2020-07-15 06:10:17 +01:00
ledgerwatch
16695eb100
Log Info and duplication fix (#735) 2020-07-10 11:40:13 +01:00
Alex Sharov
8335700331
Integration: compare to reference_db after all cycles, better cancelation, add to makefile (#733)
* cmp to reference db if --reference_chaindata provided

* graceful shutdown

* add ./cmd/integration to makefile

* add interruption support for different stages

* don't compare errors directly
2020-07-10 07:03:18 +01:00
Alex Sharov
d98382fdf8
Integration add checkChangeSet into state_stages command (#728)
Integration add checkChangeSet into state_stages command (#728)
2020-07-10 10:25:40 +07:00
Andrew Ashikhmin
bae2e7094a
Fix limit = 0 (unlimited) (#731) 2020-07-09 16:08:26 +01:00
Evgeny Danilenko
5f77570556
Make transaction pool startable and stoppable. Staged sync. (#730)
* after concurrency testing

* fixes for linters

* add an extra parameter

* change init order for protocolManager

* green tests
2020-07-09 13:53:35 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
2689b35da9
TxLookup unwind: Remove entries for blocks between unwindPoint+1 and stage.BlockNumber (#723) 2020-07-08 05:46:04 +01:00
Alex Sharov
495f95f688
Add commands "compare_states", add command "stage9", fix unwind of stage9 (it walked over whole db) (#722)
* ./cmd/integration compare_states, ./cmd/integration stage9

* TxLookup fix unwind - it walked over all db
2020-07-07 17:07:14 +07:00
Alex Sharov
305f8aff02
Integration tests2 (#714)
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```
2020-07-07 11:00:25 +07:00
Alex Sharov
8363cb0535
pass deleted code to collector (#718) 2020-07-07 09:33:26 +07:00
Alex Sharov
d3ede58ff0
stage3, fix: if header stage is ehead of body stage (#717)
* fix case: if header stage is ehead of body stage

* fix case: if header stage is ehead of body stage

* fix case: if header stage is ehead of body stage

* allow stage3 to sync to given block

* allow stage3 to sync to given block
2020-07-06 11:11:01 +01:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes (#716)
* Fixing history generation

* Try fixes

* Reset from 0

* Cleanup
2020-07-06 07:34:24 +01:00
Alex Sharov
98f8ccc561
Command for long and heavy integration tests (#712)
* cmd for integration tests

* cmd for integration tests
2020-07-05 07:18:21 +01:00
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature (#711) 2020-07-05 07:17:27 +01:00
ledgerwatch
cccc3ab460
Stage3 optimise (#708)
* Cherry-picked

* Reset3

* Fix formatting

* Print before commit

* Use ETL

* Fix linter

* Use different keys for ETL

* Fix formatting

* Update progress correctly

* Reset stage5

* Fix stage3 hack
2020-07-03 18:55:43 +01:00
ledgerwatch
49181a298b
Special bucket for transaction senders (#707)
* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test
2020-07-03 08:23:49 +01:00
ledgerwatch
b498419dcb
Fix loader crash (kludge) (#706)
* Kludge to remove duplicates

* Fix

* Fix formatting

* Fix in Len function
2020-07-03 08:08:35 +01:00
Alex Sharov
ec70cd2a9c
Revert "Only useful IH cursor (#694)" (#703)
This reverts commit 63c998c91f.

# Conflicts:
#	cmd/hack/hack.go
#	eth/stagedsync/stage_interhashes.go
2020-07-02 17:55:35 +07:00
ledgerwatch
aaf3ab9161
Fix (#702) 2020-07-02 11:23:21 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00