Commit Graph

390 Commits

Author SHA1 Message Date
Alex Sharov
a6be18b915
ticker-based logs (#954)
* timer-based logs

* timer-based logs

* delegate progress calculation to user

* delegate progress calculation to user

* delegate progress calculation to user

* clear

* add logs to senders recovery

* use default dir in integration

* more logs

* more logs
2020-08-22 12:12:33 +02:00
Alex Sharov
f827f86c11
Rpcdaemon as lib 2 (#943) 2020-08-20 10:52:27 +07:00
Alex Sharov
1dcc2b141a
Rpcdaemon as lib (#940)
* share config object

* create default config and logger

* move db connection to common func

* move server start to cli package

* clear

* clear

* rename cli to rpc

* use unified SetupLogger func

* make all root flag persistent

* use common flags in different packages

* use common flags in different packages

* move TraceTx method to eth package

* use native slice flags

* create package "turbo"

* disable geth api

* disable geth api

* move more data types to turbo/adapter package

* add support for customApiList

* run more

* run more

* run more

* dog-food

* move DoCall

* move DoCall

* fix tests

* fix test
2020-08-19 12:46:20 +01:00
Alex Sharov
90fa69a9e5
Swap IH and HashState stages (#922)
* move ih stage v0

* save progress

* save working version

* baby step

* baby step

* baby step

* non-baby step

* cleanup
2020-08-15 14:49:36 +01:00
Alex Sharov
3aed624521
[do not merge] Trie Loader: use only "retain=false" IH (#891) 2020-08-15 07:52:36 +07:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
091819a51c
adopt --metrics.addr flag in integration (#889) 2020-08-11 06:38:34 +07:00
Alex Sharov
6c0ff33366
integration: clear unwind stack (#890) 2020-08-11 06:38:19 +07:00
Giulio rebuffo
057894994d
add blockhashes stage (#862)
* add blockhashes stage

* unwindOrder specified

* fixed typo

* added common.copybytes

* fixed little typo

* better progress

* better progress

* refactoring with etl

* added blockhashes to disabled stages in cmd/integration

* added migraations

* fixed go.mod

* better migrations

* rename unwind

* simplified migrations

* added onloadcommit
2020-08-10 18:46:06 +01:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
Igor Mandrigin
6c815b3f6a fix cmd/integration 2020-08-05 10:21:08 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
Alex Sharov
590b59dc30
integration_use_storage_mode_from_db (#857) 2020-08-02 12:53:42 +01:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) (#854)
* return error from rawdb

* squash

* v14

* improve performance of put

* clean

* clean

* hide feature behind ENV variable

* cleanup

* cleanup

* disable ipc and make Readme less confusing (people thought points are depend on each-other)

* fix test

* cleanup

* cleanup
2020-08-02 12:53:08 +01:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
ledgerwatch
ddfec0828d
Delete receipts in unwind stage4 (#818) 2020-07-29 18:04:49 +01:00
Alex Sharov
c5b5eacc7a
fix stage5 interruption (#800)
* fix stage5 interruption

* clean

* fix test

* Simplification

* Fix to the txpool logic and renaming

* Fix array size in stage3

* remove 'interrupted stage' concept, simplify integration state_stages

* remove blockchain interface

* revive staged sync tests

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-29 05:30:44 +01:00
ledgerwatch
c6f0f8897f
Small fixes in cmd/integration (#799)
* Add datadir option
2020-07-28 20:07:36 +07:00
ledgerwatch
32fa0ed3ec
Txpool test6 (#797)
* Fix tx propagation and announcement tests

* Fix checkpoint challenge test

* Save current state of code

* Fix compile errors

* Fixes

* Fix formatting

* Printouts

* Printouts

* Printouts

* Too much printing, not close db when stopping

* Better handing of interruptions

* No need to reset after incremental interruption

* Fix stages

* Fixes

* Giving up

* Fix formatting

* Cleanup
2020-07-27 23:00:08 +01:00
Alex Sharov
b854a642aa
fix status check (#796) 2020-07-27 20:51:59 +07:00
Alex Sharov
5f171a550f
print compaction progress (#784) 2020-07-25 18:18:45 +01:00
Alex Sharov
b2bcbaa895
Compact flag (#783)
* compact flag

* compact flag

* compact flag

* compact flag

* compact flag
2020-07-25 08:35:08 +01:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
Alex Sharov
c8a594dec6
Limit incremental step size of stages 5 and 6. (#754) 2020-07-17 12:52:09 +07:00
Evgeny Danilenko
5ead20e288
close tx_cache and peer.handle goruotines properly (#748)
* close goruotines

* fix using defer

* close pm managers a bit later

* fmt

* finalizer for in-mem db

* after cr

* revert finalizer

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-07-16 14:27:24 +01:00
Alex Sharov
8b8094d72c
integration: fix problem with interruption (#743)
* store stageData correctly

* integration: use State of sync, use stage data, don't skip persistent steps even if they complete before interruption

* reset_state: clear unwind stages info

* stage4: stop where previous stage stopped, print to logs digits which represent what exactly is in DB, rename variable 'nextBlockNumber' to 'stateProgress'

* rebase to master

* simplify chainconfig

* try increase sleep time on ci

* try increase sleep time on ci

* try increase sleep time on ci

* try increase sleep time on ci
2020-07-15 12:53:12 +01:00
ledgerwatch
3359ba7a04
Make cmd/tester initiate staged sync, remove some pools and caches (#746)
* no pools

* Remove excessive logging

* Revert

* Initial commit

* Cleanup

* Fix some compile errors

* Fix core tests

* Fix more tests

* Fix eth/downloader

* go mod tidy

* Fix core/vm

* Fix formatting

* Fix formatting

* Fix lint

* Fix lint

* Switch tests to StagedSync

* Cleanup

* Reuse cache in stage4

* Fix

* Fix formatting

* Try to fix test
2020-07-15 07:15:48 +01:00
Alex Sharov
9da86a2737
stop exactly on "--block" (#740) 2020-07-14 08:57:41 +07:00
b00ris
6e8c027d68
fix go run for integration (#741) 2020-07-14 08:56:29 +07:00
Alex Sharov
8335700331
Integration: compare to reference_db after all cycles, better cancelation, add to makefile (#733)
* cmp to reference db if --reference_chaindata provided

* graceful shutdown

* add ./cmd/integration to makefile

* add interruption support for different stages

* don't compare errors directly
2020-07-10 07:03:18 +01:00
Alex Sharov
d98382fdf8
Integration add checkChangeSet into state_stages command (#728)
Integration add checkChangeSet into state_stages command (#728)
2020-07-10 10:25:40 +07:00
Alex Sharov
686f186545
integration - to use same blockchain config as production (#727) 2020-07-09 13:54:10 +01:00
Alex Sharov
2689b35da9
TxLookup unwind: Remove entries for blocks between unwindPoint+1 and stage.BlockNumber (#723) 2020-07-08 05:46:04 +01:00
Alex Sharov
495f95f688
Add commands "compare_states", add command "stage9", fix unwind of stage9 (it walked over whole db) (#722)
* ./cmd/integration compare_states, ./cmd/integration stage9

* TxLookup fix unwind - it walked over all db
2020-07-07 17:07:14 +07:00
Alex Sharov
f7cc86452c
Move stages from hack.go to ./cmd/integration (#719)
* move stages from hack.go to ./cmd/integration

* move stages from hack.go to ./cmd/integration
2020-07-07 08:11:10 +01:00
Alex Sharov
305f8aff02
Integration tests2 (#714)
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```
2020-07-07 11:00:25 +07:00
Alex Sharov
98f8ccc561
Command for long and heavy integration tests (#712)
* cmd for integration tests

* cmd for integration tests
2020-07-05 07:18:21 +01:00