Igor Mandrigin
1495979983
etl: only print file sizes for actual files, not RAM ( #877 )
2020-08-06 09:33:09 +02:00
Igor Mandrigin
fa89cdefd5
log ETL temp files datasize ( #873 )
2020-08-05 16:33:58 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key ( #868 )
2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key ( #858 )" ( #867 )
...
This reverts commit 7290bf519e
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key ( #858 )
2020-08-04 16:25:28 +07:00
Alex Sharov
7f60d1b535
Remove load start key from etl ( #757 )
...
* limit incremental step size of stages 5 and 6
* remove limit, make incremental
* remove_load_start_key_from_etl
* switch to "struct of arrays" from "array of structs"
* switch to "struct of arrays" from "array of structs"
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-18 10:17:04 +01:00
Alex Sharov
305f8aff02
Integration tests2 ( #714 )
...
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out // to file
--pprof --pprof.port=6060 // launch pprof server
--metrics // sends to prometheus
```
2020-07-07 11:00:25 +07:00
Alex Sharov
98f8ccc561
Command for long and heavy integration tests ( #712 )
...
* cmd for integration tests
* cmd for integration tests
2020-07-05 07:18:21 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets ( #698 )
...
* Debugging changesets
* Kinda works
* Fix compile error
* Fix formatting
* Fix lint
* Duplicate entries kludge
* Fix compile error
* Cleanup
2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon ( #692 )
...
* Fixing history index
* Remove chunk generation, fix formatting
* Fix compile error, clean up hack.go
* Fix output tests
* Fix index generator test
* Fixed checkChangeSets
* Fix linter
2020-06-28 07:10:27 +01:00
Igor Mandrigin
96750b3711
Support ETL interruptions while promoting plain state ( #662 )
...
* lil etl changez
* fix test compile
* fixups to the unwinds
* add new buffer mode
* support unfinished transitions
* fixups
* fix tests
* linters
* linters
2020-06-13 16:03:38 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage ( #646 )
...
* save state
* txlookup full results
* save state
* save state
* remove experiments
* some fix&lint
* add end key to txLookup and index generation
* change log message
* change log
* fix lint
* lint
* fix test
2020-06-10 23:07:14 +03:00
ledgerwatch
a8b438d21d
Incremental promotion for stage 5 ( #628 )
...
* Incremental promotion
* Fix compile errors
* Fix linter
* Transform state keys before loading
* Re-enable and fix the tests
* Fix formatting
2020-06-06 06:36:29 +01:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments ( #608 )
...
* First commit
* Fix the one-off error
* Fix formatting
* Ability to execute stage5 separately
* Clean up intermediate hashes and stage5 progress
* Fix linter
* Print original keys when extracting
* channel
* More logging
* More logging
* Remove excess logging
* fix stage2
* Revert
* Fix stage2
* Add provider exhausted
* Sort sortable buffer
* Fix test
* Another cleanup
* Remove exhaust log
2020-06-03 13:03:40 +01:00
Igor Mandrigin
c16d3da1b4
etl: startkey for Load + OnLoadCommit callback ( #601 )
2020-06-01 17:14:40 +03:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes ( #597 )
...
* Introduce hashCollector
* Add HashCollector to SubTrieLoader
* Fix linter
* Reset hashed state
* Not to regenerate the hashed state
* Not to delete state
* Fix linter
* Print expected hash in the beginning
* Simplify
* Remove initialTrie
* Use etl to buffer intermediate hashes
* Copy values, not insert empty key
* Compress instead of decompress
* Enhance file buffer logging, fix linter
* Fix compile errors
* Fix log
* Fix logging
* Exclude zero key again
* Add rewind
* Restrict timestamps
* Fix
* Fix formatting
* Incorporate separation
* Extract identityLoadFunction
* Fix formatting
2020-05-31 13:23:34 +01:00
Igor Mandrigin
786561e65e
don't write files if all data fits into one buffer. ( #594 )
2020-05-31 08:32:33 +01:00
b00ris
266f9a5208
Plain state index ( #595 )
...
* plain state indexes
* generalize index generation&tests
* add plain state tests
* fix regenerate index
* сoncurrent changeset chunks processing
* remove concurrency
* fix lint
* remove comments
* add test to truncate
* fix conflicts
* fmt
* remove shadowing
2020-05-31 07:57:47 +01:00
Evgeny Danilenko
32ec443572
Restore graceful shutdown ( #591 )
...
* initial
* gracefull shutdown for staged
* more wg fixes
* fmt
* linters
* remove generalization
* linters
* linters
* fix
* fix
* fmt
* quit into etl
* after CR
* after CR
2020-05-30 16:44:54 +03:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. ( #592 )
2020-05-30 10:00:35 +03:00
