Commit Graph

923 Commits

Author SHA1 Message Date
Alex Sharov
74d49a9de1
non blocking broadcast of new txs (#292) 2022-02-01 12:16:43 +00:00
ledgerwatch
1cc07746a4
[erigon2] Try to fix deadlock (#291)
* Try to fix deadlock

* Another lock

* Try to optimise

* comments and cleanup

* Prevent deadlock on panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-01 10:28:05 +00:00
Alex Sharov
0feb7fd591
Decompressor.WithReadAhead (#290) 2022-02-01 11:19:11 +07:00
ledgerwatch
4e8840256e
[erigon2] Use shorter references instead of full plain keys in the commitment files (#289)
* Rearrange aggregations

* More rearranging before introducing 3 threads

* Background aggregation|

* Concurrency fixes

* Remove files under lock

* Better logging

* Remove files without lock

* Fix lint

* Fix locking

* Try

* Fix background Merge

* Log merging

* Log merging

* Less logging

* Millisecond

* Add Stats function

* Log merge only after 1m

* Wrong counting

* plain key extract and replace functions

* Insert valTransform function

* Not parse first byte

* Not parse first byte

* Fix lint

* Switch to thin state references

* Fix lint

* Fix lint

* Debug print|

* Fix decoding

* Turn off valTransform

* Not to reuse transformer

* Print

* Print

* Print

* Derive hashed keys later

* Fix

* Fix log

* Fix

* Debug

* Another fix

* Fix

* Fix

* Print

* Print

* Data race

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-31 22:32:00 +00:00
ledgerwatch
586ab3e6b3
Separate state btree files (#287)
* Separate state file btrees, fix Match in the decompressor

* fix match

* Fix to match

* Switch back from Match

* Try to use match, close indices

* Fixing Match

* Use Skip

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-29 11:12:38 +00:00
Enrique Jose Avila Asapche
a5c2d83108
Added bor tables (#286)
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
2022-01-28 21:34:03 +00:00
primal_concrete_sledge
24098d94d8
WIP: issue/issue-250-use_compressed_keys (#285)
* issue/issue-250-use_compressed_keys

* Step into the twilight zone

* fix zero byte

* Add zerolen check

* Remove branhc data check

* Check lengths
2022-01-28 19:30:32 +00:00
Alex Sharov
c8397a0433
Mdbx: more 4tb fixes (#282) 2022-01-27 23:10:30 +07:00
Alex Sharov
dfdf7c8a66
[wip] parallel compress: less read of dat file (#284)
* save

* save

* save
2022-01-27 17:13:26 +07:00
Alex Sharov
ec11eb3d91
parallel compressor: don't save dict (#283)
* save

* save
2022-01-27 12:54:38 +07:00
ledgerwatch
1d4022960a
Fix RLP inconsistency in txpool (#280)
* Fix RLP inconsistency in txpool

* Add debug

* hasEnvelope argument when parsing transaction

* Handle incorrect packets without panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-26 16:09:02 +00:00
ledgerwatch
df95313a31
Add blockGasLimit to the state stream (#261)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit eff9f8f

git-subtree-dir: interfaces
git-subtree-split: eff9f8f7273dda7c78575f2474e3103da61825db

* Add blockGasLimit to the state stream

* Pass blockGasLimit into txpool

* Add NotTooMuchGas bit, refactored promote

* Replace effectiveTip with minTip in the queues and slices

* Fix lint

* Refactor

* Improve terminology of ordering function

* Optimise baseFee change

* cleanup

* Safe operations

* Fix UnsafeAdd

* also print pendingBaseFee

* trace txpool_content

* trace txpool_content

* user sender ID inverted index to speed up txpool_content

* copy rlp from Db

* remove debug printing

* change from string to []byte

* safe storage of sender in the map

* Fix compile

* safe storage of sender in the map

* Print errors

* Prints

* Prints

* Prints

* Change worst function

* Change worst function

* Change worst function

* Change worst function

* Fix minTip

* Fix minTip

* Warn on db error, but not stop

* Fix interface

* More prints

* Cleanup

* Cleanup

* Differentiate baseFee pool

* Bump version of remove KV interface

* Not print every block

* Not print every block

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-25 10:20:11 +00:00
Alex Sharov
5932009110
MDBX: configurable corrupt error recommendations (#275)
* save

* save
2022-01-25 15:25:09 +07:00
ledgerwatch
a7ec201c0a
Update aggregator.go (#274) 2022-01-24 22:43:41 +00:00
ledgerwatch
7ec016b160
Fixes in compress (#260)
* Fixes in compress

* Reuse outputFile also as uncompressed file

* Close file before renaming

* Trace

* Untrace

* Use 8 threads

* Print aggregations

* Print merge and timing

* Print merge and timing

* readonly mode for patricia

* Fix to infinite loop

* Fix file names

* Cleanup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-24 22:13:48 +00:00
primal_concrete_sledge
d8a33270e8
issue/issue-249-add_index_reader (#273)
* issue/issue-249-add_index_reader

* Add licence
2022-01-24 20:39:04 +00:00
primal_concrete_sledge
e69a5da702
Issue 248 refinements for decompressor api (#271)
* issue/ISSUE-248-refinements_for_decompressor_api

* Fix match tedst expectations

* Remove unneeded comment
2022-01-24 09:18:08 +00:00
Alex Sharov
3a2d474940
MDBX: Add recommendations to corruption error message #3337 2022-01-24 10:35:38 +07:00
Alex Sharov
5ba56ae360
Mdbx 0.11.4 (#266) 2022-01-23 09:07:46 +07:00
Alex Sharov
1c92f7bc00
mdbx: consensus label #265 2022-01-22 12:49:25 +07:00
alex.sharov
ec2a482087 allow configure pagesize 2022-01-22 12:46:22 +07:00
Alex Sharov
8791c5eb7a
Etl less allocs (#263)
* etl less allocs

* etl less allocs

* allow configure pagesize
2022-01-22 12:21:04 +07:00
Alex Sharov
bcef5f091c
allow configure pagesize (#264) 2022-01-22 12:13:35 +07:00
Alex Sharov
79eb27d3f1
Helper to prohibit cli flag changes (#262) 2022-01-22 10:48:22 +07:00
canepat
e58d0fa537
Add B+ tree prototype implementation (#258)
* Add B+ tree prototype implementation

* Add Apache license header to all files
Fix linter errors
2022-01-21 07:06:09 +00:00
Alex Sharov
f16aa09cb1
mdbx: MDBX_FORCE_ASSERTIONS=0 (#256) 2022-01-21 11:45:32 +07:00
e-danko
43105470fd
added starknet gointerfaces (#259) 2022-01-20 14:11:08 +00:00
e-danko
3dea173d42
Added starknet proto to gointerfaces (#254) 2022-01-20 09:31:34 +00:00
Alex Sharov
ac8c29cacf
kv helpers (#255) 2022-01-20 15:31:01 +07:00
Giulio rebuffo
ac26beb190
added cumulative index tables (#253) 2022-01-20 08:25:03 +00:00
Alex Sharov
302946c5e5
Move cli root context to erigon-lib (#251)
* save

* save
2022-01-19 10:48:58 +07:00
ledgerwatch
340195df93
Less verbose parallel compressor (#247)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 14:20:05 +00:00
Alex Sharov
11ab5bdbb8
Parallel compressor - allow empty words (#245)
* save

* save

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 13:57:35 +00:00
Zhang Zhuo
10b163bbf1
doc: fix a broken link in etl/README.md (#246) 2022-01-18 13:43:30 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alexandr Borodulin
ac4119191d
Encoding StarknetTx salt parameter (#243)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2022-01-17 13:47:16 +01:00
Alex Sharov
7c2104e2e1
fix to no prealloc (because max size unknown) 2022-01-17 17:05:37 +07:00
Alexandr Borodulin
fa6c709fcd
Encoding StarknetTx salt parameter (#240)
Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2022-01-17 10:54:29 +01:00
Alex Sharov
5042e16218
enforce exactly once semantic in BigChunks helper 2022-01-17 16:49:02 +07:00
Alex Sharov
51220cfe43
ParallelCompressor class, DecompressedFile class (#234)
* save

* save

* save

* remove major jump check

* remove major jump check

* log

* log

* save

* format docs

* format docs

* issue-260

* issue-260

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2022-01-17 08:50:42 +00:00
Neo
26dcd28433
Update PriceBump, change tx replacement logic, update test (#238) 2022-01-17 07:45:57 +00:00
ledgerwatch
0eaa0333e1
[erigon2] Optional writing of before changes, commitment decoupled from blocks (#237)
* Separate commitment from blocks

* Optional beforeOn

* Optional beforeOn

* Optional beforeOn

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-16 20:27:44 +00:00
ledgerwatch
e5c07ec901
[erigon2] Resumable prototype (#236)
* Not warn about files that don't match at all

* Not warn about files that don't match at all

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-15 22:09:06 +00:00
ledgerwatch
b95cff2ce3
No empty RLPs in TxPacket messages, consistent parsing of TxPacket and PooledTxPacket (#232)
* Not include empty RLPs into TransactionPacket messages

* Consistent parsing of TransactionPacket and PooledTransactionPacket

* The other way around

* Cleaner form

* Add test for encoding and parsing TransctionsPacket

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-15 09:09:33 +00:00
Alex Sharov
01a6417505
snapshots: same workers amount #233 2022-01-15 11:23:19 +07:00
ledgerwatch
c08ed12256
[erigon2] Measure change set sizes with perfect hash table (#230)
* Remove warning message

* Calculate sizes of change sets for transactions

* Calculate sizes of change sets for transactions

* Produce changeset files

* Not to produce changesets for commitments

* Remove rewindTx

* Close decompressor

* Not generate changesets by default

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-14 14:10:10 +00:00
alex.sharov
3e856aba62 format docs 2022-01-13 11:08:05 +07:00
ledgerwatch
ae6f7202d2
[erigon2] Introduce commitment files and table (#201)
* Introduce commitment files and table

* Introduce commitment files and table

* Introduce words buffer

* Move changes out of aggregator into the writer

* Start calc commitment

* Add commitment package

* More on hex patricia

* More

* account decorator

* More

* Add test

* More

* More to the test

* Use hex in tests

* More

* More

* More

* More

* Simplified cells

* Add delBitmap

* More advanced MockState

* More compact unfolding

* Separation of hashed Keys

* Sepatate downHashedKey and upHashedKey

* Carry extension node through accounts

* optimised row allocations

* Fix encoding/decoding, add trace flag

* Added account and storage into the cells

* Make accountKeyLen global setting

* Remove BranchNodeUpdaqte intermediate

* Started on computeHash

* Initial hash calculations

* Fix lint

* Change account encoding

* Fix commitment issues

* Fix lint

* Fix lint

* Fix lint

* Trace

* Small fix and tracing

* Print branch hashes, fix empty

* Fix

* Fix

* Fix

* Fix

* Print accountFn

* Trigger accountFn

* Trigger accountFn

* Another fix for extension nodes

* Return root hash and set trace

* Print depth for computeCellHash

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Ineffectual deletes

* Trace needUnfolding

* Fix

* Fix

* Fix

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* not overwrite downHashedKey in updateAccount

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* propagate deletes when unfolding

* remove deletes when folding

* remove deletes when folding

* remove upHashedKey when delete cell

* remove upHashedKey when delete cell

* aggregate storage changes later

* remove del bit when updating account

* remove del bit when updating account

* remove del bit when updating account

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit 0234ea6c3e1a0b2e7599df7c470d4b8b8d56736a.

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit cfa7b75327e4d5d1236f905c3c04eb640b21838d.

* Introduce FinishTx

* Separate commitment separation from computation

* Prevent spurious delete

* Spurious deletes

* Trace param

* Preserve upHashedKey when unfolding

* Fix

* Fix

* Correctly compute firstInsert

* Try to fix delete + fold

* Remove commented out

* updateStorage to remove deleted flag

* rename upHashedKey to extension, fix clear-up

* Carry extension up if account plain key is present

* Carry extension up if account plain key is present

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Copy keys before putting them into commTree

* Preserve changes to AccountData

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-12 11:21:52 +00:00
Alex Sharov
1647faec37
Fix bigChunk helper (#229) 2022-01-12 10:46:26 +07:00
Alex Sharov
e98482670a
mdbx: fix gc "retry" issue (slowness of gc during commit) (#227)
* mdbx: fix gc "retry" issue (slowness of gc during commit)

* mdbx: fix gc "retry" issue (slowness of gc during commit)
2022-01-11 08:12:20 +00:00