Alex Sharov
b683ed435c
Compress params change ( #651 )
...
Main Target: reduce RAM usage of huffman tables. If possible - improve
decompression speed. Compression speed not so important.
Experiments on 74Gb uncompressed file (bsc
012500-013000-transactions.seg)
Ram - needed just to open compressed file (Huff tables, etc...)
dec_speed - loop with `word, _ = g.Next(word[:0])`
skip_speed - loop with `g.Skip()`
```
| DictSize | Ram | file_size | dec_speed | skip_speed |
| -------- | ---- | --------- | --------- | ---------- |
| 1M | 70Mb | 35871Mb | 4m06s | 1m58s |
| 512K | 42Mb | 36496Mb | 3m49s | 1m51s |
| 256K | 21Mb | 37100Mb | 3m44s | 1m48s |
| 128K | 11Mb | 37782Mb | 3m25s | 1m44s |
| 64K | 7Mb | 38597Mb | 3m16s | 1m34s |
| 32K | 5Mb | 39626Mb | 3m0s | 1m29s |
```
Also about small sampling: skip superstrings if superstringNumber % 4 !=
0 does reduce compression ratio by 1% - checked on big BSC file and
small (1gb) goerli file.
so, I feel it's not so bad idea to use:
maxDictPatterns=64k
samplingFactor=4
Tradeoffs: sacrify 5% compression ratio to 4x compression speedup (i
think even more), 30% decompression speedup, 10x RAM reduction
Release: I will not change existing snapshots - now will focus on
releasing new block snapshots and releasing new history snapshots
(Erigon3). If have time will re-compress existing snapshots later.
2022-10-05 17:54:48 +07:00
Alex Sharov
9860f845fe
mdbx mergeThreshold option #665
2022-10-03 16:53:23 +07:00
Artem Tsebrovskiy
4f5232504f
E3 agg commitment ( #647 )
...
* added commitment to aggregator
* added commitment evaluation by updates, fixed mainnet roothash mismatch
* added ability to change starting state of hph
* replayable erigon23 with commitment
* possible fix for eliasfano index read after close
* fixed db pruning and restart
* Initial fixes
* Debug
* clear downHashedLen for branch nodes
* Fix key length, cleanup
* Cleanup
* Cleanup
* picked aggregator updates
* fixed empty cell hash for ProcessUpdate evaluation
* hashBuffer moved from Cell to HexPatriciaHashed
* fixed codeHash incorrect renewal
* lint
* removed valuemergefn from history
* fixed lint
* fixed test
* rewritten fuzz test on hph
* fix for Win tests - do not remove tmp dir after test
* win
* fixup after merge
* close aggregator after test
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-09-26 15:59:24 +01:00
Enrique Jose Avila Asapche
183d2718ca
allow mem-mutation to create buckets ( #650 )
2022-09-25 16:35:30 +07:00
giuliorebuffo
f38bce356d
added verkle tree buckets
2022-09-22 18:05:02 +02:00
Alex Sharov
cd20a6862f
tests: one-liner to create tx ( #644 )
2022-09-20 13:18:30 +07:00
Alex Sharov
f05cd214bd
aggregator22: read dir without idx ( #638 )
2022-09-18 17:38:43 +07:00
hexoscott
547ac03eeb
ensure kv semaphore has at least 1 count ( #641 )
2022-09-15 22:14:10 +02:00
Andrew Ashikhmin
ef693175fe
Bump map size of in-memory MDBX ( #640 )
2022-09-15 13:18:08 +02:00
hexoscott
6f6b03d7f0
semaphore for remote kv and reduce semaphore max count ( #639 )
2022-09-15 08:49:12 +07:00
Giulio rebuffo
2017964398
added temporary database ( #633 )
...
Co-authored-by: giuliorebuffo <giuliorebuffo@system76-pc.localdomain>
2022-09-10 23:56:59 +02:00
Andrea Lanfranchi
09680a841f
MDBX does support empty keys ( #631 )
...
* MDBX does support empty keys
* Lint errors
* Amend test
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2022-09-09 16:25:00 +02:00
Enrique Jose Avila Asapche
7505532113
got rid of bor related things ( #618 )
2022-09-02 10:50:59 +07:00
Håvard Anda Estensen
a9459aea68
Enable thelper linter ( #610 )
2022-08-31 10:09:44 +07:00
Alex Sharov
1508952df9
mdbx more metrics ( #614 )
2022-08-31 10:09:23 +07:00
Alex Sharov
22beae4579
mdbx more metrics ( #613 )
2022-08-31 10:04:53 +07:00
Håvard Anda Estensen
3b0c5f75f8
Enable prealloc linter ( #607 )
2022-08-29 11:07:53 +07:00
alex.sharov
5110fa3a13
save
2022-08-24 16:07:32 +07:00
Andrew Ashikhmin
23c7f503e0
WithTablessCfg -> WithTableCfg ( #601 )
2022-08-24 11:02:47 +02:00
Alex Sharov
8db5419002
wait_block_reduce ( #594 )
2022-08-18 12:50:41 +07:00
ledgerwatch
e160c1ad9c
Optimise state erigon2.2 reconstitution ( #570 )
...
* Start iterator1
* No parallel buildFiles and mergeFiles
* Optimise GetNoState
* Fixes
* Fix 2
* Another fix
* Fix
* More changes iter
* Provide keys in ScanIterator
* Tables for bitmaps
* Add X tables
* Change signature of GeNoState
* More on changes iterator
* Test for changed keys iterator
* ReconDb tables
* Changed key iterator
* Fix lint
* Fix lint
* uncovert
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp.lan>
2022-08-14 14:56:47 +01:00
Alex Sharov
dfb87cc33b
expose_growth_step ( #582 )
2022-08-12 22:59:28 +07:00
Leonard Chinonso
c96a22dcdc
WIP: Fix null pointer reference error in rpcdaemon ( #578 )
...
* saving changes for draft PR
* typo
* Revert "typo"
This reverts commit a7d96e384e87eeceff1a6c51a2bed659eb014ad0.
* checking if reclected value is nil
* Removed print statements
* Removed unused import
* forcing lint ignore
Co-authored-by: Enrique Avila <eavilaasapche@gmail.com>
2022-08-12 12:12:20 +03:00
Andrew Ashikhmin
55fa5d7006
Various fixes and improvements of MemoryMutation ( #573 )
...
* Remove duplicate function
* Close cursor in ForAmount
* Correct ForEach & ForPrefix
* Switch GetOne implementation to cursor
* Switch Has implementation to cursor
* Small fixes
* statelessCursors should be cleared in UpdateTxn
* cursorentry -> cursorEntry
* Fix memoryMutationCursor.NextNoDup
* Don't swallow errors
* Fix memoryMutationCursor.First for DupSort tables
* TestFirstAfterClearBucket
* Add TestAutoDupSort
* WithTablessCfg -> WithTablesCfg
* WithTablesCfg -> WithTableCfg
* Add TestAutoDupSort
* Remove memoryMutationCursor.currentPair
* Merge duplicated cursors
* goForward -> resolveCursorPriority
* Extend TestAutoDupSort
* Amend skipIntersection
* Restore currentPair
* Revert "Merge duplicated cursors"
This reverts commit 13ba28f2fd4c63d6ef17254aa93ac05172e37429.
* Revert WithTablessCfg -> WithTableCfg
* Small fix to AppendDup
* Revert "Revert "Merge duplicated cursors""
This reverts commit b94b7612617750d26eebe8b149e5a8d7f219a4db.
* Pay more attention to isTableCleared
* Remove convertAutoDupsort
* Fix DeleteCurrentDuplicates
* Small simplification of memoryMutationCursor.SeekBothRange
* Revert BaseCase -> baseCase
* Revert parameter renaming
* Restore an assertion
* Add TestAutoConversionSeekBothRange
* CursorDupSort doesn't do AutoDupSortKeysConversion
2022-08-11 09:55:28 +02:00
Alex Sharov
1e029ac6d8
go1.19 gofmt ( #576 )
...
* save
* save
2022-08-10 19:00:19 +07:00
Ryan Schneider
8fbf01ca91
mdbx: fix possible semaphore exhaustion on cancelled context ( #567 )
2022-08-04 11:43:56 +07:00
Andrew Ashikhmin
1d6c297797
Make Delete() remove all values for DupSort tables ( #565 )
...
* panic in MemoryMutation.Reset
* Make Delete remove all values for DupSort tables
2022-08-03 15:36:49 +02:00
Håvard Anda Estensen
ad2344a6cc
Replace ioutil with io and os ( #560 )
2022-08-01 11:03:48 +07:00
Alex Sharov
dd32747db2
parity_listStorageKeys: *mdbx.MdbxCursor is not kv.CursorDupSort ( #559 )
2022-07-29 09:13:40 +07:00
Giulio rebuffo
5d7fb4ddb0
Expose Reset ( #558 )
...
* exposed Reset()
* exposed Reset()
* unimplemented commit
* complete reset
* complete reset
* complete reset
* better reset
* finished Reset
* fixed memory mutation
Co-authored-by: giuliorebuffo <giuliorebuffo@system76-pc.localdomain>
2022-07-28 22:42:23 +02:00
Alex Sharov
471d790348
kv.Del() remove second parameter ( #554 )
...
* save
* save
* save
* save
* save
* save
* save
* save
2022-07-26 12:47:08 +07:00
Alex Sharov
7a1c2f700b
use PrioritizedSend method ( #553 )
2022-07-26 10:37:29 +07:00
Alex Sharov
e995e5a7a1
kv.Snapshots() implementation #551
2022-07-26 10:13:37 +07:00
TKartist
0436e97a72
memory_mutation_test added ( #546 )
...
* tests for GetOne and Put added
* test configured
* multiple tests added
* conflict resolve
* optimized and tests updated
* conflict fix
* tests added and tested with iteration
* conflict resolve
* lint error resolve
* resolve unnecessary code
* memory_mutation_test implemented
* test error fixed
* test error fixed
2022-07-25 09:53:58 +01:00
TKartist
684c3d3697
Tests with iteration have been added ( #538 )
2022-07-22 16:50:12 +07:00
Alex Sharov
b9efe56597
mdbx: use OS pagesize by default (but > 4Kb, and < 64Kb) #530
2022-07-19 10:39:48 +07:00
TKartist
8b2c02d1fb
Test for kv_mdbx optimized ( #528 )
...
* tests for GetOne and Put added
* test configured
* multiple tests added
* conflict resolve
* optimized and tests updated
* conflict fix
2022-07-18 21:36:58 +02:00
TKartist
5821ae7bbc
kv_mdbx_test updated ( #522 )
...
* tests for GetOne and Put added
* test configured
* multiple tests added
* conflict resolve
2022-07-18 14:27:03 +07:00
Giulio rebuffo
21aa65c1c3
Silence errors for mdbx_get no data ( #521 )
...
* removed empty data cursors operations
* ops
* ops
2022-07-14 18:05:25 +02:00
TKartist
33df7d785c
tests for GetOne and Put added ( #519 )
...
* tests for GetOne and Put added
* test configured
2022-07-14 17:57:52 +02:00
Max Revitt
d629e31df7
fix( #4543 ): BeginRo use semaphore ( #520 )
2022-07-13 14:37:45 +02:00
Igor Mandrigin
95e361fa1e
Fixup for a possible deadlock if db.env.BeginTxn fails ( #516 )
...
* attempt to fix the deadlock
* slightly more correct check
Co-authored-by: Igor Mandrigin <i@mandrigin.ru>
2022-07-07 08:52:30 +02:00
Alex Sharov
a86baf3e81
TxPool: btree with generics ( #514 )
2022-07-07 11:07:43 +06:00
ledgerwatch
e77f25bce6
[erigon2.2] Temporary tables for state reconstitution ( #512 )
...
* Add temporary table for Plain state reconstitution
* Add 2 more temp tables
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-07-04 08:11:49 +01:00
Alex Sharov
8873903605
Snapshots table ( #510 )
...
* save
* save
2022-07-04 10:56:35 +06:00
Alex Sharov
7aa540e6fa
save ( #508 )
2022-07-01 10:10:23 +06:00
Giulio rebuffo
b2c211d9bf
Refactor sequence handling in mutation
2022-06-25 01:05:27 +02:00
Giulio rebuffo
541dbe8c47
SeekBothRange instead of FirstDup
2022-06-25 01:04:51 +02:00
Giulio rebuffo
8e0df524d8
Added more functionality to the mutation.
2022-06-24 22:10:45 +02:00
ledgerwatch
46bebb3317
[erigon2.2] Add ReadIndices aggregator to collect data ( #500 )
...
* [erigon2.2] Add ReadIndices aggregator to collect data
* Try
* Fix for history access
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-20 08:39:29 +01:00
