Commit Graph

735 Commits

Author SHA1 Message Date
Alex Sharov
e649f7ea91
Less alloc etl recsplit (#307)
* less allocs recsplit

* save

* save
2022-02-09 13:22:45 +07:00
ledgerwatch
f5deff3ac9
[erigon2] accumulating chain history and bitmap indices (#306)
* Aggregator refactoring

* Restore the test

* Simplify commitment

* Lint

* onBefore

* Wip smaller commitments

* Fixes

* Fix lint

* Check for zero len

* Print

* Remove restriction

* Correct restrictions

* Correct restrictions

* More merging

* delBitmap

* Switch off merging

* Reintroduce background merge

* Try to fix

* Slight optimisation

* Produce initial changeset files

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-08 21:19:29 +00:00
ledgerwatch
92fc7215c8
[erigon2] fix commitment error (#304)
* Try to fix

* Fix

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-04 18:48:50 +00:00
Alex Sharov
567d9ddfed
ParallelCompressor: Remove intermediate ETL collectors (#302) 2022-02-04 16:48:02 +07:00
alex.sharov
d4b9053aed save 2022-02-04 11:20:02 +07:00
ledgerwatch
55080d5c01
Proper reset of decompressor getter (#299)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-03 17:58:56 +00:00
ledgerwatch
11615db276
Fix memory leak in txpool (#298)
* Fix memory leak in txpool

* Cannot delete during iteration

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-03 17:52:20 +00:00
alex.sharov
d814adb13e save 2022-02-03 16:28:43 +07:00
alex.sharov
02c407a14e save 2022-02-03 16:28:32 +07:00
ledgerwatch
cbfd733672
[erigon2] optimisations (#297)
* Reuse getter and indexReader

* Properly reset Getter

* Use separate getter and indexReader for merge thread

* Remove DB

* Fixes

* Print

* Fixes

* Cleanup signatures

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-03 08:49:47 +00:00
Alex Sharov
ec354d1615
Fuzz fixes (#295)
* fuzz tests fixes

* fuzz tests fixes
2022-02-02 09:18:04 +00:00
Alex Sharov
a53f232322
pool: fix race in async broadcast (#294)
* pool: better async broadcast

* save

* save
2022-02-02 14:49:26 +07:00
Alex Sharov
74d49a9de1
non blocking broadcast of new txs (#292) 2022-02-01 12:16:43 +00:00
ledgerwatch
1cc07746a4
[erigon2] Try to fix deadlock (#291)
* Try to fix deadlock

* Another lock

* Try to optimise

* comments and cleanup

* Prevent deadlock on panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-01 10:28:05 +00:00
Alex Sharov
0feb7fd591
Decompressor.WithReadAhead (#290) 2022-02-01 11:19:11 +07:00
ledgerwatch
4e8840256e
[erigon2] Use shorter references instead of full plain keys in the commitment files (#289)
* Rearrange aggregations

* More rearranging before introducing 3 threads

* Background aggregation|

* Concurrency fixes

* Remove files under lock

* Better logging

* Remove files without lock

* Fix lint

* Fix locking

* Try

* Fix background Merge

* Log merging

* Log merging

* Less logging

* Millisecond

* Add Stats function

* Log merge only after 1m

* Wrong counting

* plain key extract and replace functions

* Insert valTransform function

* Not parse first byte

* Not parse first byte

* Fix lint

* Switch to thin state references

* Fix lint

* Fix lint

* Debug print|

* Fix decoding

* Turn off valTransform

* Not to reuse transformer

* Print

* Print

* Print

* Derive hashed keys later

* Fix

* Fix log

* Fix

* Debug

* Another fix

* Fix

* Fix

* Print

* Print

* Data race

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-31 22:32:00 +00:00
ledgerwatch
586ab3e6b3
Separate state btree files (#287)
* Separate state file btrees, fix Match in the decompressor

* fix match

* Fix to match

* Switch back from Match

* Try to use match, close indices

* Fixing Match

* Use Skip

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-29 11:12:38 +00:00
Enrique Jose Avila Asapche
a5c2d83108
Added bor tables (#286)
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
2022-01-28 21:34:03 +00:00
primal_concrete_sledge
24098d94d8
WIP: issue/issue-250-use_compressed_keys (#285)
* issue/issue-250-use_compressed_keys

* Step into the twilight zone

* fix zero byte

* Add zerolen check

* Remove branhc data check

* Check lengths
2022-01-28 19:30:32 +00:00
Alex Sharov
c8397a0433
Mdbx: more 4tb fixes (#282) 2022-01-27 23:10:30 +07:00
Alex Sharov
dfdf7c8a66
[wip] parallel compress: less read of dat file (#284)
* save

* save

* save
2022-01-27 17:13:26 +07:00
Alex Sharov
ec11eb3d91
parallel compressor: don't save dict (#283)
* save

* save
2022-01-27 12:54:38 +07:00
ledgerwatch
1d4022960a
Fix RLP inconsistency in txpool (#280)
* Fix RLP inconsistency in txpool

* Add debug

* hasEnvelope argument when parsing transaction

* Handle incorrect packets without panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-26 16:09:02 +00:00
ledgerwatch
df95313a31
Add blockGasLimit to the state stream (#261)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit eff9f8f

git-subtree-dir: interfaces
git-subtree-split: eff9f8f7273dda7c78575f2474e3103da61825db

* Add blockGasLimit to the state stream

* Pass blockGasLimit into txpool

* Add NotTooMuchGas bit, refactored promote

* Replace effectiveTip with minTip in the queues and slices

* Fix lint

* Refactor

* Improve terminology of ordering function

* Optimise baseFee change

* cleanup

* Safe operations

* Fix UnsafeAdd

* also print pendingBaseFee

* trace txpool_content

* trace txpool_content

* user sender ID inverted index to speed up txpool_content

* copy rlp from Db

* remove debug printing

* change from string to []byte

* safe storage of sender in the map

* Fix compile

* safe storage of sender in the map

* Print errors

* Prints

* Prints

* Prints

* Change worst function

* Change worst function

* Change worst function

* Change worst function

* Fix minTip

* Fix minTip

* Warn on db error, but not stop

* Fix interface

* More prints

* Cleanup

* Cleanup

* Differentiate baseFee pool

* Bump version of remove KV interface

* Not print every block

* Not print every block

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-25 10:20:11 +00:00
Alex Sharov
5932009110
MDBX: configurable corrupt error recommendations (#275)
* save

* save
2022-01-25 15:25:09 +07:00
ledgerwatch
a7ec201c0a
Update aggregator.go (#274) 2022-01-24 22:43:41 +00:00
ledgerwatch
7ec016b160
Fixes in compress (#260)
* Fixes in compress

* Reuse outputFile also as uncompressed file

* Close file before renaming

* Trace

* Untrace

* Use 8 threads

* Print aggregations

* Print merge and timing

* Print merge and timing

* readonly mode for patricia

* Fix to infinite loop

* Fix file names

* Cleanup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-24 22:13:48 +00:00
primal_concrete_sledge
d8a33270e8
issue/issue-249-add_index_reader (#273)
* issue/issue-249-add_index_reader

* Add licence
2022-01-24 20:39:04 +00:00
primal_concrete_sledge
e69a5da702
Issue 248 refinements for decompressor api (#271)
* issue/ISSUE-248-refinements_for_decompressor_api

* Fix match tedst expectations

* Remove unneeded comment
2022-01-24 09:18:08 +00:00
Alex Sharov
3a2d474940
MDBX: Add recommendations to corruption error message #3337 2022-01-24 10:35:38 +07:00
Alex Sharov
5ba56ae360
Mdbx 0.11.4 (#266) 2022-01-23 09:07:46 +07:00
Alex Sharov
1c92f7bc00
mdbx: consensus label #265 2022-01-22 12:49:25 +07:00
alex.sharov
ec2a482087 allow configure pagesize 2022-01-22 12:46:22 +07:00
Alex Sharov
8791c5eb7a
Etl less allocs (#263)
* etl less allocs

* etl less allocs

* allow configure pagesize
2022-01-22 12:21:04 +07:00
Alex Sharov
bcef5f091c
allow configure pagesize (#264) 2022-01-22 12:13:35 +07:00
Alex Sharov
79eb27d3f1
Helper to prohibit cli flag changes (#262) 2022-01-22 10:48:22 +07:00
canepat
e58d0fa537
Add B+ tree prototype implementation (#258)
* Add B+ tree prototype implementation

* Add Apache license header to all files
Fix linter errors
2022-01-21 07:06:09 +00:00
Alex Sharov
f16aa09cb1
mdbx: MDBX_FORCE_ASSERTIONS=0 (#256) 2022-01-21 11:45:32 +07:00
e-danko
43105470fd
added starknet gointerfaces (#259) 2022-01-20 14:11:08 +00:00
e-danko
3dea173d42
Added starknet proto to gointerfaces (#254) 2022-01-20 09:31:34 +00:00
Alex Sharov
ac8c29cacf
kv helpers (#255) 2022-01-20 15:31:01 +07:00
Giulio rebuffo
ac26beb190
added cumulative index tables (#253) 2022-01-20 08:25:03 +00:00
Alex Sharov
302946c5e5
Move cli root context to erigon-lib (#251)
* save

* save
2022-01-19 10:48:58 +07:00
ledgerwatch
340195df93
Less verbose parallel compressor (#247)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 14:20:05 +00:00
Alex Sharov
11ab5bdbb8
Parallel compressor - allow empty words (#245)
* save

* save

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 13:57:35 +00:00
Zhang Zhuo
10b163bbf1
doc: fix a broken link in etl/README.md (#246) 2022-01-18 13:43:30 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alexandr Borodulin
ac4119191d
Encoding StarknetTx salt parameter (#243)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2022-01-17 13:47:16 +01:00
Alex Sharov
7c2104e2e1
fix to no prealloc (because max size unknown) 2022-01-17 17:05:37 +07:00
Alexandr Borodulin
fa6c709fcd
Encoding StarknetTx salt parameter (#240)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2022-01-17 10:54:29 +01:00