mirror of
https://gitlab.com/pulsechaincom/erigon-pulse.git
synced 2025-01-01 00:31:21 +00:00
b683ed435c
Main Target: reduce RAM usage of huffman tables. If possible - improve decompression speed. Compression speed not so important. Experiments on 74Gb uncompressed file (bsc 012500-013000-transactions.seg) Ram - needed just to open compressed file (Huff tables, etc...) dec_speed - loop with `word, _ = g.Next(word[:0])` skip_speed - loop with `g.Skip()` ``` | DictSize | Ram | file_size | dec_speed | skip_speed | | -------- | ---- | --------- | --------- | ---------- | | 1M | 70Mb | 35871Mb | 4m06s | 1m58s | | 512K | 42Mb | 36496Mb | 3m49s | 1m51s | | 256K | 21Mb | 37100Mb | 3m44s | 1m48s | | 128K | 11Mb | 37782Mb | 3m25s | 1m44s | | 64K | 7Mb | 38597Mb | 3m16s | 1m34s | | 32K | 5Mb | 39626Mb | 3m0s | 1m29s | ``` Also about small sampling: skip superstrings if superstringNumber % 4 != 0 does reduce compression ratio by 1% - checked on big BSC file and small (1gb) goerli file. so, I feel it's not so bad idea to use: maxDictPatterns=64k samplingFactor=4 Tradeoffs: sacrify 5% compression ratio to 4x compression speedup (i think even more), 30% decompression speedup, 10x RAM reduction Release: I will not change existing snapshots - now will focus on releasing new block snapshots and releasing new history snapshots (Erigon3). If have time will re-compress existing snapshots later.
46 lines
1.7 KiB
Modula-2
46 lines
1.7 KiB
Modula-2
module github.com/ledgerwatch/erigon-lib
|
|
|
|
go 1.18
|
|
|
|
require (
|
|
github.com/RoaringBitmap/roaring v1.2.1
|
|
github.com/VictoriaMetrics/metrics v1.22.2
|
|
github.com/c2h5oh/datasize v0.0.0-20220606134207-859f65c6625b
|
|
github.com/go-stack/stack v1.8.1
|
|
github.com/google/btree v1.1.2
|
|
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
|
|
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d
|
|
github.com/holiman/uint256 v1.2.1
|
|
github.com/ledgerwatch/interfaces v0.0.0-20220914022748-0bc2888c95ce
|
|
github.com/ledgerwatch/log/v3 v3.4.1
|
|
github.com/ledgerwatch/secp256k1 v1.0.0
|
|
github.com/quasilyte/go-ruleguard/dsl v0.3.21
|
|
github.com/spaolacci/murmur3 v1.1.0
|
|
github.com/stretchr/testify v1.8.0
|
|
github.com/torquem-ch/mdbx-go v0.26.0
|
|
go.uber.org/atomic v1.10.0
|
|
golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4
|
|
golang.org/x/exp v0.0.0-20220921164117-439092de6870
|
|
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
|
|
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f
|
|
google.golang.org/grpc v1.48.0
|
|
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.2.0
|
|
google.golang.org/protobuf v1.28.1
|
|
)
|
|
|
|
require (
|
|
github.com/bits-and-blooms/bitset v1.2.0 // indirect
|
|
github.com/davecgh/go-spew v1.1.1 // indirect
|
|
github.com/golang/protobuf v1.5.2 // indirect
|
|
github.com/mattn/go-colorable v0.1.11 // indirect
|
|
github.com/mattn/go-isatty v0.0.14 // indirect
|
|
github.com/mschoch/smat v0.2.0 // indirect
|
|
github.com/pmezard/go-difflib v1.0.0 // indirect
|
|
github.com/valyala/fastrand v1.1.0 // indirect
|
|
github.com/valyala/histogram v1.2.0 // indirect
|
|
golang.org/x/net v0.0.0-20220607020251-c690dde0001d // indirect
|
|
golang.org/x/text v0.3.7 // indirect
|
|
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 // indirect
|
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
|
)
|