mirror/erigon-pulse
1
0
Fork 0
mirror of https://gitlab.com/pulsechaincom/erigon-pulse.git synced 2025-01-01 00:31:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
777 Commits 5 Branches 14 Tags 294 MiB
2fc62cd2f5
Commit Graph

48 Commits

Author SHA1 Message Date
racytech
7763945374
Reverted 3 last commits (#348) 
* Revert "unnecessary includes removed"

This reverts commit 76406bb78b.

* Revert "local dev setup"

This reverts commit ac06fd9400.

* Revert "compress/cgo-addition"

This reverts commit fae7683d46, reversing
changes made to e3e108c6c4.
 2022-02-24 14:39:42 +00:00
Kairat Abylkasymov
76406bb78b unnecessary includes removed 2022-02-24 06:21:25 -05:00
Kairat Abylkasymov
ac06fd9400 local dev setup 2022-02-24 06:15:14 -05:00
Alex Sharov
3205770ee0
snapshots: fix test (#346) 2022-02-24 08:35:13 +07:00
ledgerwatch
c71ac02a0f
[erigon2] Optimisations in etl collector and compressor (#339) 
* Optimisations in etl collector and compressor

* Not copy k and v in the collector

* Fix lint

* Optimisations

* Change Load1 back to Load

* Reduce allocations for tests

* preallocate inv

* counting hits and misses

* Try to fix

* Try to fix

* Relaxation 1

* Relaxation 2

* Add arch tables

* Fix

* Update arch tables and use them

* Not to override larger value

* Increase arch table size

* Increase arch table size

* Fixes to arch

* Print

* Off by one

* Print

* Fix

* Remove print

* Perform update of arch in the background

* Build up huffman tree

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2022-02-20 22:14:06 +00:00
Alex Sharov
1f5a1ab9cd
fuzz cases (#328) 2022-02-14 11:53:20 +07:00
Alex Sharov
6f85066c7e
path -> filepath (path package is for urls) (#321) 2022-02-12 20:11:30 +07:00
Alex Sharov
e649f7ea91
Less alloc etl recsplit (#307) 
* less allocs recsplit

* save

* save
 2022-02-09 13:22:45 +07:00
Alex Sharov
567d9ddfed
ParallelCompressor: Remove intermediate ETL collectors (#302) 2022-02-04 16:48:02 +07:00
ledgerwatch
55080d5c01
Proper reset of decompressor getter (#299) 
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2022-02-03 17:58:56 +00:00
Alex Sharov
0feb7fd591
Decompressor.WithReadAhead (#290) 2022-02-01 11:19:11 +07:00
ledgerwatch
4e8840256e
[erigon2] Use shorter references instead of full plain keys in the commitment files (#289) 
* Rearrange aggregations

* More rearranging before introducing 3 threads

* Background aggregation|

* Concurrency fixes

* Remove files under lock

* Better logging

* Remove files without lock

* Fix lint

* Fix locking

* Try

* Fix background Merge

* Log merging

* Log merging

* Less logging

* Millisecond

* Add Stats function

* Log merge only after 1m

* Wrong counting

* plain key extract and replace functions

* Insert valTransform function

* Not parse first byte

* Not parse first byte

* Fix lint

* Switch to thin state references

* Fix lint

* Fix lint

* Debug print|

* Fix decoding

* Turn off valTransform

* Not to reuse transformer

* Print

* Print

* Print

* Derive hashed keys later

* Fix

* Fix log

* Fix

* Debug

* Another fix

* Fix

* Fix

* Print

* Print

* Data race

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2022-01-31 22:32:00 +00:00
ledgerwatch
586ab3e6b3
Separate state btree files (#287) 
* Separate state file btrees, fix Match in the decompressor

* fix match

* Fix to match

* Switch back from Match

* Try to use match, close indices

* Fixing Match

* Use Skip

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2022-01-29 11:12:38 +00:00
Alex Sharov
dfdf7c8a66
[wip] parallel compress: less read of dat file (#284) 
* save

* save

* save
 2022-01-27 17:13:26 +07:00
Alex Sharov
ec11eb3d91
parallel compressor: don't save dict (#283) 
* save

* save
 2022-01-27 12:54:38 +07:00
ledgerwatch
7ec016b160
Fixes in compress (#260) 
* Fixes in compress

* Reuse outputFile also as uncompressed file

* Close file before renaming

* Trace

* Untrace

* Use 8 threads

* Print aggregations

* Print merge and timing

* Print merge and timing

* readonly mode for patricia

* Fix to infinite loop

* Fix file names

* Cleanup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2022-01-24 22:13:48 +00:00
primal_concrete_sledge
d8a33270e8
issue/issue-249-add_index_reader (#273) 
* issue/issue-249-add_index_reader

* Add licence
 2022-01-24 20:39:04 +00:00
primal_concrete_sledge
e69a5da702
Issue 248 refinements for decompressor api (#271) 
* issue/ISSUE-248-refinements_for_decompressor_api

* Fix match tedst expectations

* Remove unneeded comment
 2022-01-24 09:18:08 +00:00
ledgerwatch
340195df93
Less verbose parallel compressor (#247) 
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2022-01-18 14:20:05 +00:00
Alex Sharov
11ab5bdbb8
Parallel compressor - allow empty words (#245) 
* save

* save

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2022-01-18 13:57:35 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alex Sharov
7c2104e2e1
fix to no prealloc (because max size unknown) 2022-01-17 17:05:37 +07:00
Alex Sharov
51220cfe43
ParallelCompressor class, DecompressedFile class (#234) 
* save

* save

* save

* remove major jump check

* remove major jump check

* log

* log

* save

* format docs

* format docs

* issue-260

* issue-260

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
 2022-01-17 08:50:42 +00:00
Alex Sharov
01a6417505
snapshots: same workers amount #233 2022-01-15 11:23:19 +07:00
Alex Sharov
1647faec37
Fix bigChunk helper (#229) 2022-01-12 10:46:26 +07:00
Alex Sharov
f92c12855d
Decompressor: fast .Count method(#226) 2022-01-09 17:32:56 +07:00
alex.sharov
a8c2481967 create huffman_codes.txt in tmpdir 2022-01-09 14:52:52 +07:00
alex.sharov
8bc0f26a49 create .seg in tmpdir 2022-01-09 14:49:56 +07:00
Alex Sharov
0d5d8975d9
Snapshots: create .dat in tmpdir (#225) 2022-01-09 14:43:55 +07:00
Alex Sharov
f5733d438f
Parallel compression (#223) 2022-01-06 14:13:03 +07:00
Alex Sharov
2fe3f06702
Decompressor internal file path getter (#220) 2021-12-31 18:42:43 +07:00
alex.sharov
dd363aea75 Revert "save" 
This reverts commit ef20b06b01.
 2021-12-27 15:11:24 +07:00
alex.sharov
ef20b06b01 save 2021-12-27 15:11:00 +07:00
Alex Sharov
583c3f9b5b
count in .seg (#212) 2021-12-21 10:45:20 +07:00
Alex Sharov
bb1d712834
Hack: dump bodies and headers (#177) 
* save

* save

* save

* save
 2021-11-19 22:00:55 +07:00
Alex Sharov
a4ee647902
Recsplit: move files read/write helpers to erigon-lib (#176) 2021-11-19 12:03:36 +07:00
ledgerwatch
ea56078a4c
state aggregator - tests for contract storage (#165) 
* First test for storage

* Debugging

* Fixes to delete account test

* Fix the lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-11-16 00:11:40 +00:00
Alex Sharov
5b7f67deae
Snapshot naming (#163) 
* save

* save

* save

* save

* save

* save
 2021-11-15 14:19:56 +00:00
Alex Sharov
7faa84fe9f
Compress: add maxPatternLen=64 (#155) 2021-11-10 22:09:54 +07:00
Alex Sharov
3a17502353
Compress: fix compress bytes share (#154) 
* fix compress bytes share

* fix compress bytes share
 2021-11-10 19:50:36 +07:00
Alex Sharov
dad84cccc9
move DictAggregator to erigon-lib (#153) 2021-11-09 10:12:20 +07:00
Alex Sharov
d79f87a0e9
Recsplit: single offset bucket (#152) 2021-11-08 14:27:21 +07:00
alex.sharov
4398d3f8da save 2021-11-08 10:35:30 +07:00
alex.sharov
377bc94675 save 2021-11-08 07:59:18 +07:00
Alex Sharov
be8c96bbef
Use bufio limit in compressor (#151) 2021-11-07 14:32:01 +07:00
Alex Sharov
ba51a5966a
etl.collector - move logPrefix to constructor (#128) 2021-10-25 09:12:00 +07:00
ledgerwatch
967937151d
Fixes for compress, decompressor, and tests (#110) 
* Fixes for compress, and first test

* Add decompressor and memory mapping

* Add decompressor and memory mapping

* Fix for windows

* Fix lint

* Fix compile for windows

* More on decompressor

* Fix lint

* Decompress

* Fix lint

* Use decompressor in tests, fixes

* Introduce Index for RecSplit

* Fix compilation on Windows

* close index file on failure

* Fixes to the tests

* Add single Elias Fano, fix recsplit fuzz test

* Fix elias fano

* Add two layer index

* Add two level index to the tests

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-10-16 10:43:41 +01:00
ledgerwatch
083ee83906
Generalise patricia tree, initial compress (#103) 
* Generalise patricia tree, initial compress

* Include tranform

* Generalise Insert

* More on compression

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-10-11 18:31:49 +01:00