Main Target: reduce RAM usage of huffman tables. If possible - improve
decompression speed. Compression speed not so important.
Experiments on 74Gb uncompressed file (bsc
012500-013000-transactions.seg)
Ram - needed just to open compressed file (Huff tables, etc...)
dec_speed - loop with `word, _ = g.Next(word[:0])`
skip_speed - loop with `g.Skip()`
```
| DictSize | Ram | file_size | dec_speed | skip_speed |
| -------- | ---- | --------- | --------- | ---------- |
| 1M | 70Mb | 35871Mb | 4m06s | 1m58s |
| 512K | 42Mb | 36496Mb | 3m49s | 1m51s |
| 256K | 21Mb | 37100Mb | 3m44s | 1m48s |
| 128K | 11Mb | 37782Mb | 3m25s | 1m44s |
| 64K | 7Mb | 38597Mb | 3m16s | 1m34s |
| 32K | 5Mb | 39626Mb | 3m0s | 1m29s |
```
Also about small sampling: skip superstrings if superstringNumber % 4 !=
0 does reduce compression ratio by 1% - checked on big BSC file and
small (1gb) goerli file.
so, I feel it's not so bad idea to use:
maxDictPatterns=64k
samplingFactor=4
Tradeoffs: sacrify 5% compression ratio to 4x compression speedup (i
think even more), 30% decompression speedup, 10x RAM reduction
Release: I will not change existing snapshots - now will focus on
releasing new block snapshots and releasing new history snapshots
(Erigon3). If have time will re-compress existing snapshots later.
* added commitment to aggregator
* added commitment evaluation by updates, fixed mainnet roothash mismatch
* added ability to change starting state of hph
* replayable erigon23 with commitment
* possible fix for eliasfano index read after close
* fixed db pruning and restart
* Initial fixes
* Debug
* clear downHashedLen for branch nodes
* Fix key length, cleanup
* Cleanup
* Cleanup
* picked aggregator updates
* fixed empty cell hash for ProcessUpdate evaluation
* hashBuffer moved from Cell to HexPatriciaHashed
* fixed codeHash incorrect renewal
* lint
* removed valuemergefn from history
* fixed lint
* fixed test
* rewritten fuzz test on hph
* fix for Win tests - do not remove tmp dir after test
* win
* fixup after merge
* close aggregator after test
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
* Split History from Domain
* Add History.prune
* More on history
* Fix HistoryHistory test
* Merge history files
* Scan file test for history
* Add aggregator for erigon 2.2
* Change to generics, introduce contexts
* Delete to belong to Aggregator
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Use pointers to InvertedIndex again
* Remove prints
* Close embedded InvertedIndex
* Fix closing files
* Print
* Update ci.yml
* More printing
* Fix
* Make InvertedIndex pointer inside History
* Fix
* Update ci.yml
* Remove print
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Add temporary table for Plain state reconstitution
* Add 2 more temp tables
* FinishTx with delay
* Fix search in history
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Reduce allocations in domain and aggregator
* Make keys always uncompressed, values compressed only for code
* Functions to remake index
* Fix index recreation
* Test for reindex, fix
* Use uncompress vals in history
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Better control of compress/uncompressed
* Add new function
* more careful pruning
* Printf
* Printf
* Fix DupSort
* Remove copying in prune
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Merge fixes, add historical access
* Change API from AfterTxNum to BeforeTxNum
* Change API functions
* Change API functions
* Print
* Fix for non-existent items
* Remove prints
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Add scan files tests, create new aggregator type
* Fix lint
* windows test fix
* Add delelte test
* AggCollation
* More functions to Aggregator
* More aggregator functions
* Update
* More functions
* More functions
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
* [erigon2] Introduce inverted index type
* More inverted index code
* More tests for inverted index
* Think about public and non-public APIs
* Minimise DB access when accessing history
* Work on iterator
* Implementation of inverted iterator
* Test for inverted index
* Assert end of iterators
* Merge of inverted index files and test
* Fix lint
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
* Iteration over files - initial
* Fix interator for multistep
* Add function
* More functions for merge
* Merge files
* More work on the merge
* Fix buildIndex
* Fix history test for test of not completely pruned db
* Prepare for merge test
* Merge file test
* Close files
* Move functions into separate file
* Print
* fix for closing index
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp.lan>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Domain
* First functions
* change year
* More on domain
* More to test
* More on test
* More on domains
* buildFiles
* More on domains
* Collation test
* Fix collate
* Add test for decompressors
* Restructure history tables
* Split history into 2 tables
* Fix lint
* Check index files in the test
* Close files
* Add file scanning
* Fix lint
* Fix lint
* Add readFromFiles
* Add ef history idx file
* Start cleanup
* More to cleanup, test for ef history
* More test
* Add prune to test
* Test for prune and fix
* Start history access
* History test
* Test for LastDup
* Fix one lint
* Workaround
* History tests
* Debug
* Fix
* Fix in history
* Fix lint
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-macbook-pro.home>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp.lan>
