ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets ( #698 )
...
* Debugging changesets
* Kinda works
* Fix compile error
* Fix formatting
* Fix lint
* Duplicate entries kludge
* Fix compile error
* Cleanup
2020-07-01 15:56:56 +01:00
b00ris
7ac009a7da
Fix last chunk order ( #683 )
...
* fix commit order
* fix lint
2020-06-21 15:13:17 +01:00
b00ris
7df7880a5a
[etl] Split collect&commit to separate threads ( #676 )
...
* fast search
* fix
* remove copy bytes
* speedup collector
* move logs to commit gr
* fix lint
* fix test
2020-06-18 19:14:10 +01:00
Igor Mandrigin
96750b3711
Support ETL interruptions while promoting plain state ( #662 )
...
* lil etl changez
* fix test compile
* fixups to the unwinds
* add new buffer mode
* support unfinished transitions
* fixups
* fix tests
* linters
* linters
2020-06-13 16:03:38 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage ( #646 )
...
* save state
* txlookup full results
* save state
* save state
* remove experiments
* some fix&lint
* add end key to txLookup and index generation
* change log message
* change log
* fix lint
* lint
* fix test
2020-06-10 23:07:14 +03:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger ( #624 )
...
* lmdb tests
* trigger ci
* fix tests
* disable parallelism
* disable parallelism
* cleanup resources
* cleanup resources
* reduce concurency
* try run tests on bolt
* try run tests on bolt
* fix downloader test
* run bolt tests
* rely on interface instead of exact instance
* Rename AbstractKV to KV
* don't use separator for badger
* don't initialize badger cursor - because it not used here
* fix linter
* try reduce badger compactors
* compat with master
* try lmdb
* try lmdb
* try lmdb
* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option
* allow to close closed db
* release
* release
* ideal batch size for badger
* ideal batch size for badger
2020-06-05 10:25:33 +01:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments ( #608 )
...
* First commit
* Fix the one-off error
* Fix formatting
* Ability to execute stage5 separately
* Clean up intermediate hashes and stage5 progress
* Fix linter
* Print original keys when extracting
* channel
* More logging
* More logging
* Remove excess logging
* fix stage2
* Revert
* Fix stage2
* Add provider exhausted
* Sort sortable buffer
* Fix test
* Another cleanup
* Remove exhaust log
2020-06-03 13:03:40 +01:00
Igor Mandrigin
c16d3da1b4
etl: startkey for Load + OnLoadCommit callback ( #601 )
2020-06-01 17:14:40 +03:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes ( #597 )
...
* Introduce hashCollector
* Add HashCollector to SubTrieLoader
* Fix linter
* Reset hashed state
* Not to regenerate the hashed state
* Not to delete state
* Fix linter
* Print expected hash in the beginning
* Simplify
* Remove initialTrie
* Use etl to buffer intermediate hashes
* Copy values, not insert empty key
* Compress instead of decompress
* Enhance file buffer logging, fix linter
* Fix compile errors
* Fix log
* Fix logging
* Exclude zero key again
* Add rewind
* Restrict timestamps
* Fix
* Fix formatting
* Incorporate separation
* Extract identityLoadFunction
* Fix formatting
2020-05-31 13:23:34 +01:00
Igor Mandrigin
786561e65e
don't write files if all data fits into one buffer. ( #594 )
2020-05-31 08:32:33 +01:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. ( #592 )
2020-05-30 10:00:35 +03:00
