Alex Sharov
305f8aff02
Integration tests2 ( #714 )
...
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out // to file
--pprof --pprof.port=6060 // launch pprof server
--metrics // sends to prometheus
```
2020-07-07 11:00:25 +07:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes ( #716 )
...
* Fixing history generation
* Try fixes
* Reset from 0
* Cleanup
2020-07-06 07:34:24 +01:00
Alex Sharov
98f8ccc561
Command for long and heavy integration tests ( #712 )
...
* cmd for integration tests
* cmd for integration tests
2020-07-05 07:18:21 +01:00
ledgerwatch
b498419dcb
Fix loader crash (kludge) ( #706 )
...
* Kludge to remove duplicates
* Fix
* Fix formatting
* Fix in Len function
2020-07-03 08:08:35 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets ( #698 )
...
* Debugging changesets
* Kinda works
* Fix compile error
* Fix formatting
* Fix lint
* Duplicate entries kludge
* Fix compile error
* Cleanup
2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon ( #692 )
...
* Fixing history index
* Remove chunk generation, fix formatting
* Fix compile error, clean up hack.go
* Fix output tests
* Fix index generator test
* Fixed checkChangeSets
* Fix linter
2020-06-28 07:10:27 +01:00
Alex Sharov
b26b00f72b
increase IO buffers size ( #686 )
...
* increase IO buffers size
* found SSD threshold
2020-06-25 08:34:40 +01:00
b00ris
7ac009a7da
Fix last chunk order ( #683 )
...
* fix commit order
* fix lint
2020-06-21 15:13:17 +01:00
ledgerwatch
fd98914c28
Stage6 - intermediate hashes ( #677 )
...
* First cut of the stage6
* Fix formatter
* Introduce state6
* Fix target number for stage6
* Fix linter
* Fix linter
* Reset in regenerate
* Correct block number
* Fix linter
* Fix linter
* Fix
* Upper case
* Skeleton to debug'
* Fix formatting
* Added codehash correction
* Fix TestUnwind
* fix test
* Fix linter
* Introduce unwind6
* Report adjustment error
* Code hashes included into stage5 incremental promotion and unwind
* Fix formatting
* Cleanup
* fix TestUnwind
* Cleanup
* Fix compile error
* Fix formatting
* unwind4
* Disable verifyRoot in stage5, fix hacks
* Remove verifyRoot function
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-06-18 22:27:11 +01:00
b00ris
7df7880a5a
[etl] Split collect&commit to separate threads ( #676 )
...
* fast search
* fix
* remove copy bytes
* speedup collector
* move logs to commit gr
* fix lint
* fix test
2020-06-18 19:14:10 +01:00
Igor Mandrigin
96750b3711
Support ETL interruptions while promoting plain state ( #662 )
...
* lil etl changez
* fix test compile
* fixups to the unwinds
* add new buffer mode
* support unfinished transitions
* fixups
* fix tests
* linters
* linters
2020-06-13 16:03:38 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage ( #646 )
...
* save state
* txlookup full results
* save state
* save state
* remove experiments
* some fix&lint
* add end key to txLookup and index generation
* change log message
* change log
* fix lint
* lint
* fix test
2020-06-10 23:07:14 +03:00
ledgerwatch
a8b438d21d
Incremental promotion for stage 5 ( #628 )
...
* Incremental promotion
* Fix compile errors
* Fix linter
* Transform state keys before loading
* Re-enable and fix the tests
* Fix formatting
2020-06-06 06:36:29 +01:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger ( #624 )
...
* lmdb tests
* trigger ci
* fix tests
* disable parallelism
* disable parallelism
* cleanup resources
* cleanup resources
* reduce concurency
* try run tests on bolt
* try run tests on bolt
* fix downloader test
* run bolt tests
* rely on interface instead of exact instance
* Rename AbstractKV to KV
* don't use separator for badger
* don't initialize badger cursor - because it not used here
* fix linter
* try reduce badger compactors
* compat with master
* try lmdb
* try lmdb
* try lmdb
* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option
* allow to close closed db
* release
* release
* ideal batch size for badger
* ideal batch size for badger
2020-06-05 10:25:33 +01:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments ( #608 )
...
* First commit
* Fix the one-off error
* Fix formatting
* Ability to execute stage5 separately
* Clean up intermediate hashes and stage5 progress
* Fix linter
* Print original keys when extracting
* channel
* More logging
* More logging
* Remove excess logging
* fix stage2
* Revert
* Fix stage2
* Add provider exhausted
* Sort sortable buffer
* Fix test
* Another cleanup
* Remove exhaust log
2020-06-03 13:03:40 +01:00
Igor Mandrigin
c16d3da1b4
etl: startkey for Load + OnLoadCommit callback ( #601 )
2020-06-01 17:14:40 +03:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes ( #597 )
...
* Introduce hashCollector
* Add HashCollector to SubTrieLoader
* Fix linter
* Reset hashed state
* Not to regenerate the hashed state
* Not to delete state
* Fix linter
* Print expected hash in the beginning
* Simplify
* Remove initialTrie
* Use etl to buffer intermediate hashes
* Copy values, not insert empty key
* Compress instead of decompress
* Enhance file buffer logging, fix linter
* Fix compile errors
* Fix log
* Fix logging
* Exclude zero key again
* Add rewind
* Restrict timestamps
* Fix
* Fix formatting
* Incorporate separation
* Extract identityLoadFunction
* Fix formatting
2020-05-31 13:23:34 +01:00
Igor Mandrigin
786561e65e
don't write files if all data fits into one buffer. ( #594 )
2020-05-31 08:32:33 +01:00
b00ris
266f9a5208
Plain state index ( #595 )
...
* plain state indexes
* generalize index generation&tests
* add plain state tests
* fix regenerate index
* сoncurrent changeset chunks processing
* remove concurrency
* fix lint
* remove comments
* add test to truncate
* fix conflicts
* fmt
* remove shadowing
2020-05-31 07:57:47 +01:00
Evgeny Danilenko
32ec443572
Restore graceful shutdown ( #591 )
...
* initial
* gracefull shutdown for staged
* more wg fixes
* fmt
* linters
* remove generalization
* linters
* linters
* fix
* fix
* fmt
* quit into etl
* after CR
* after CR
2020-05-30 16:44:54 +03:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. ( #592 )
2020-05-30 10:00:35 +03:00
