mirror/erigon-pulse
1
0
Fork 0
mirror of https://gitlab.com/pulsechaincom/erigon-pulse.git synced 2024-12-28 14:47:16 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
968 Commits 5 Branches 14 Tags 294 MiB
dd32747db2
Commit Graph

32 Commits

Author SHA1 Message Date
ledgerwatch
fadc9b21d1
[erigon2.2] Split 2.2 and 2.3 prototype (#548) 
* Introduce access functions to history

* Add missing functions

* Add missing functions

* Add missing functions

* Changeover in the aggregator

* Intermediate

* Fix domain tests

* Fix lint

* Fix lint

* Fix lint

* Close files

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2022-07-28 08:47:13 +01:00
Alex Sharov
f23061eed9
compressor: generic sort (#524) 2022-07-18 17:12:39 +07:00
Alex Sharov
ceafdded8f
Compress: reduce etl buffers to save RAM (#502) 2022-06-25 19:39:36 +06:00
Alex Sharov
fdf7c6598b
compress.Count() method (#478) 2022-06-03 12:14:58 +07:00
Alex Sharov
04337fd090
Compress: reduce maxlen to 512 (#416) 2022-04-17 07:59:29 +07:00
ledgerwatch
f18e05186d
Compact huffman representation in files (#414) 
* More compact huffman represenation

* Intermediate

* Intermediate

* fix

* Fix lint

* Fix lint

* Fix lint

* Change min file size

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2022-04-13 12:55:15 +01:00
Alex Sharov
75b64f01a3
compressor: log lvl #408 2022-04-01 10:44:25 +07:00
Alex Sharov
83951a1d62
Enable more linters (#381) 2022-03-19 11:38:37 +07:00
Alex Sharov
c1f1365f92
cancel compress (#362) 2022-03-12 16:34:58 +07:00
Alex Sharov
6512e3c941
add emptyWordsCount field to .seg file header (breaking .seg format) (#355) 
* up torrent

* save

* save

* save

* save

* save

* save

* save
 2022-03-10 07:48:37 +00:00
ledgerwatch
75b52ac25e
[compress] Allow uncompressed words (#350) 
* Intermediate work

* Allow uncompressed words

* Fix

* Fix tests

* Add NextUncompressed, remove g.word buffer

* Code simplifications, no goroutines when workers == 1

* Fix lint|

* Add test for MatchPrefix

* Work on patricia

* Beginning of new matcher

* Fuzz test for new longest match

* No skip

* Fixes

* Fixes

* More tracing

* Fixes

* Fixes

* Change back to old FindLongestMatches

* Switch to old match finder

* Print mismatches

* Fix

* After fix

* After fix

* After fix

* Print pointers

* Fixes and tests

* Print

* Print

* Print

* More tests

* Intermediate

* Fix

* Fix

* Prints

* Fix

* Fix

* Initialise matchStack

* Compute only once

* Compute only once

* Switch back

* Switch to old Find

* Introduce sais

* Switch patricia to sais

* Use sais in compressor

* Use sais in compressor

* Remove unused code

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2022-03-09 17:25:22 +00:00
Alex Sharov
3205770ee0
snapshots: fix test (#346) 2022-02-24 08:35:13 +07:00
Alex Sharov
dfdf7c8a66
[wip] parallel compress: less read of dat file (#284) 
* save

* save

* save
 2022-01-27 17:13:26 +07:00
Alex Sharov
ec11eb3d91
parallel compressor: don't save dict (#283) 
* save

* save
 2022-01-27 12:54:38 +07:00
ledgerwatch
7ec016b160
Fixes in compress (#260) 
* Fixes in compress

* Reuse outputFile also as uncompressed file

* Close file before renaming

* Trace

* Untrace

* Use 8 threads

* Print aggregations

* Print merge and timing

* Print merge and timing

* readonly mode for patricia

* Fix to infinite loop

* Fix file names

* Cleanup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2022-01-24 22:13:48 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alex Sharov
7c2104e2e1
fix to no prealloc (because max size unknown) 2022-01-17 17:05:37 +07:00
Alex Sharov
51220cfe43
ParallelCompressor class, DecompressedFile class (#234) 
* save

* save

* save

* remove major jump check

* remove major jump check

* log

* log

* save

* format docs

* format docs

* issue-260

* issue-260

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
 2022-01-17 08:50:42 +00:00
Alex Sharov
f5733d438f
Parallel compression (#223) 2022-01-06 14:13:03 +07:00
alex.sharov
dd363aea75 Revert "save" 
This reverts commit ef20b06b01.
 2021-12-27 15:11:24 +07:00
alex.sharov
ef20b06b01 save 2021-12-27 15:11:00 +07:00
Alex Sharov
583c3f9b5b
count in .seg (#212) 2021-12-21 10:45:20 +07:00
Alex Sharov
a4ee647902
Recsplit: move files read/write helpers to erigon-lib (#176) 2021-11-19 12:03:36 +07:00
ledgerwatch
ea56078a4c
state aggregator - tests for contract storage (#165) 
* First test for storage

* Debugging

* Fixes to delete account test

* Fix the lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-11-16 00:11:40 +00:00
Alex Sharov
5b7f67deae
Snapshot naming (#163) 
* save

* save

* save

* save

* save

* save
 2021-11-15 14:19:56 +00:00
Alex Sharov
7faa84fe9f
Compress: add maxPatternLen=64 (#155) 2021-11-10 22:09:54 +07:00
Alex Sharov
3a17502353
Compress: fix compress bytes share (#154) 
* fix compress bytes share

* fix compress bytes share
 2021-11-10 19:50:36 +07:00
Alex Sharov
dad84cccc9
move DictAggregator to erigon-lib (#153) 2021-11-09 10:12:20 +07:00
Alex Sharov
be8c96bbef
Use bufio limit in compressor (#151) 2021-11-07 14:32:01 +07:00
Alex Sharov
ba51a5966a
etl.collector - move logPrefix to constructor (#128) 2021-10-25 09:12:00 +07:00
ledgerwatch
967937151d
Fixes for compress, decompressor, and tests (#110) 
* Fixes for compress, and first test

* Add decompressor and memory mapping

* Add decompressor and memory mapping

* Fix for windows

* Fix lint

* Fix compile for windows

* More on decompressor

* Fix lint

* Decompress

* Fix lint

* Use decompressor in tests, fixes

* Introduce Index for RecSplit

* Fix compilation on Windows

* close index file on failure

* Fixes to the tests

* Add single Elias Fano, fix recsplit fuzz test

* Fix elias fano

* Add two layer index

* Add two level index to the tests

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-10-16 10:43:41 +01:00
ledgerwatch
083ee83906
Generalise patricia tree, initial compress (#103) 
* Generalise patricia tree, initial compress

* Include tranform

* Generalise Insert

* More on compression

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-10-11 18:31:49 +01:00