Commit Graph

22 Commits

Author SHA1 Message Date
Alex Sharov
763370a3ce
commitment: generic btree #523 2022-07-18 11:46:38 +07:00
Artem Tsebrovskiy
21c6baf287
fixes for bin commitment trie, test updates (#517)
* fixes for bin commitment trie, test updates

* lint fix

* removed graphics deps
2022-07-10 12:08:25 +01:00
Alex Sharov
e824fdff60
remove fuzzbeta build tag, because now go1.18 is minimum requirement (#428) 2022-07-03 14:38:53 +06:00
Artem Tsebrovskiy
f8bdadf3e0
HPH with direct reading from state by plainKey (#472)
* dirty trie with direct reading of account/storage data from state

run with fixes

implemented trie with direct reading from state

* cleaner version without updates
2022-06-09 13:46:11 +01:00
Artem Tsebrovskiy
56a6afc8c9
commitment: fixed hash evaluation for hex patricia trie (#471) 2022-05-28 07:52:24 +01:00
Artem Vorotnikov
15d2a37c0b
Typify commitment (#454) 2022-05-20 15:01:44 +07:00
Artem Tsebrovskiy
59d810650c
[erigon2] cleanup obsolete code for bin-commitment-trie (#451)
* cleanup obsolete code for bin-commitment-trie

* renamed back test functions for hex, renamed test file for bin trie for consistency
2022-05-19 21:18:03 +01:00
Alex Sharov
b1dc1bfbbf
run go fix ./... (#453) 2022-05-17 14:48:16 +07:00
Alex Sharov
91f7d84e60
Generic sort of slices (no allocs, inlinable) (#449)
* save

* save
2022-05-16 08:23:43 +01:00
ledgerwatch
900c03b1ba
Fixes for bin_patricia_hashed (#441)
* Debug bin commitment

* Fix print

* Fix print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Fix binString

* Fix binString

* Fix binString

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* No print

* No print

* Clean up

* Fix lint

* Fixing test

* Skip some tests

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-05-14 20:45:37 +01:00
Artem Tsebrovskiy
abd93fe9c9
implement bin_patricia_hashed trie (#430)
* commitment: implemented semi-working bin patricia trie

* commitment: added initialize function to select commitment implementation

* deleted reference implementation of binary trie

* added branch merge function selection in accordance with current commitment type

* smarter branch prefix convolution to reduce disk usage

* implemented DELETE update

* commitment/bin-trie: fixed merge processing and storage encoding

* added changed hex to bin patricia trie

* fixed trie variant select

* allocate if bufPos larger than buf size

* added tracing code

* Fix lint

* Skip test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-05 13:08:58 +01:00
Artem Tsebrovskiy
2a7ebd8e87
[#389] fuzz tests on hex patricia hashed trie (#420)
* commitment: added UniqueRepresentation and fuzz test on HexPatriciaHashed trie

* commitment: added another fuzz test with different inputs

* commitment: fix of roothash of size 33

* adds new updates for unique representation  test to broke it

* Fix lint

* Fix test

* Remove prints

* updated fuzz test and removed useless code

* Remove print

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-04-22 11:48:59 +01:00
ledgerwatch
4e8d577d1d
[erigon2] Switch back to MDBX (#388)
* Switch back to MDBX

* Fix test

* No restore

* Fix overwrite

* Fix change file opening

* Verify state vs change files

* Add WriteMap

* Print

* Not to go to the end of the file when reading

* Fix rebuild

* prefixLen

* Print

* Print

* Print

* Fix for rebuild

* Remove prints

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-03-23 14:35:13 +00:00
ledgerwatch
2b14d6da29
[erigon2] debug delays in the main thread (#379)
* Split handover time

* Clean up

* Correctly compute handover time

* not use compression when aggregating

* Fix for etl collector

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-03-19 12:02:27 +00:00
Alex Sharov
83951a1d62
Enable more linters (#381) 2022-03-19 11:38:37 +07:00
Artem Tsebrovskiy
6687e6b4d6
during computeCommitment exits with RootHash when there are no updates to apply (#366) 2022-03-15 17:01:51 +00:00
ledgerwatch
1c6e82c2b6
[erigon2] Thin commitment (2nd attempt) (#329)
* Another fix for history files

* Half way through

* Another fix

* Correct closing sequence

* Remove first byte insert marker

* More on think commitments

* Fixes

* Fixes

* Print

* Skip touchMap

* Merge branchData from trees and from files

* Fill branch commitment

* Fill branch commitment

* Print

* Fix?

* Merge branchData when updating in the tree

* Better panic

* Prints

* Prints

* Prints

* Create complete branch data if it did not exist before

* Cleanup printing

* Fix merge

* Fix merge use

* Fix transform

* Better startBlock panic

* Preserve touchMap

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Include fieldbits during transform

* Fix history reads

* Print

* Print

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Print largestMerge, lock files in branchFn

* Add storage lock

* Prints

* prefixLen fix

* prefixLen fix

* Fixes

* Remove print

* Remove print

* Set changesets and commitments flags upfront

* Logging instead of printing

* Fix history merge, recsplit panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-16 16:44:00 +00:00
ledgerwatch
f5deff3ac9
[erigon2] accumulating chain history and bitmap indices (#306)
* Aggregator refactoring

* Restore the test

* Simplify commitment

* Lint

* onBefore

* Wip smaller commitments

* Fixes

* Fix lint

* Check for zero len

* Print

* Remove restriction

* Correct restrictions

* Correct restrictions

* More merging

* delBitmap

* Switch off merging

* Reintroduce background merge

* Try to fix

* Slight optimisation

* Produce initial changeset files

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-08 21:19:29 +00:00
ledgerwatch
1cc07746a4
[erigon2] Try to fix deadlock (#291)
* Try to fix deadlock

* Another lock

* Try to optimise

* comments and cleanup

* Prevent deadlock on panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-01 10:28:05 +00:00
ledgerwatch
4e8840256e
[erigon2] Use shorter references instead of full plain keys in the commitment files (#289)
* Rearrange aggregations

* More rearranging before introducing 3 threads

* Background aggregation|

* Concurrency fixes

* Remove files under lock

* Better logging

* Remove files without lock

* Fix lint

* Fix locking

* Try

* Fix background Merge

* Log merging

* Log merging

* Less logging

* Millisecond

* Add Stats function

* Log merge only after 1m

* Wrong counting

* plain key extract and replace functions

* Insert valTransform function

* Not parse first byte

* Not parse first byte

* Fix lint

* Switch to thin state references

* Fix lint

* Fix lint

* Debug print|

* Fix decoding

* Turn off valTransform

* Not to reuse transformer

* Print

* Print

* Print

* Derive hashed keys later

* Fix

* Fix log

* Fix

* Debug

* Another fix

* Fix

* Fix

* Print

* Print

* Data race

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-31 22:32:00 +00:00
primal_concrete_sledge
24098d94d8
WIP: issue/issue-250-use_compressed_keys (#285)
* issue/issue-250-use_compressed_keys

* Step into the twilight zone

* fix zero byte

* Add zerolen check

* Remove branhc data check

* Check lengths
2022-01-28 19:30:32 +00:00
ledgerwatch
ae6f7202d2
[erigon2] Introduce commitment files and table (#201)
* Introduce commitment files and table

* Introduce commitment files and table

* Introduce words buffer

* Move changes out of aggregator into the writer

* Start calc commitment

* Add commitment package

* More on hex patricia

* More

* account decorator

* More

* Add test

* More

* More to the test

* Use hex in tests

* More

* More

* More

* More

* Simplified cells

* Add delBitmap

* More advanced MockState

* More compact unfolding

* Separation of hashed Keys

* Sepatate downHashedKey and upHashedKey

* Carry extension node through accounts

* optimised row allocations

* Fix encoding/decoding, add trace flag

* Added account and storage into the cells

* Make accountKeyLen global setting

* Remove BranchNodeUpdaqte intermediate

* Started on computeHash

* Initial hash calculations

* Fix lint

* Change account encoding

* Fix commitment issues

* Fix lint

* Fix lint

* Fix lint

* Trace

* Small fix and tracing

* Print branch hashes, fix empty

* Fix

* Fix

* Fix

* Fix

* Print accountFn

* Trigger accountFn

* Trigger accountFn

* Another fix for extension nodes

* Return root hash and set trace

* Print depth for computeCellHash

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Ineffectual deletes

* Trace needUnfolding

* Fix

* Fix

* Fix

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* not overwrite downHashedKey in updateAccount

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* propagate deletes when unfolding

* remove deletes when folding

* remove deletes when folding

* remove upHashedKey when delete cell

* remove upHashedKey when delete cell

* aggregate storage changes later

* remove del bit when updating account

* remove del bit when updating account

* remove del bit when updating account

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit 0234ea6c3e1a0b2e7599df7c470d4b8b8d56736a.

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit cfa7b75327e4d5d1236f905c3c04eb640b21838d.

* Introduce FinishTx

* Separate commitment separation from computation

* Prevent spurious delete

* Spurious deletes

* Trace param

* Preserve upHashedKey when unfolding

* Fix

* Fix

* Correctly compute firstInsert

* Try to fix delete + fold

* Remove commented out

* updateStorage to remove deleted flag

* rename upHashedKey to extension, fix clear-up

* Carry extension up if account plain key is present

* Carry extension up if account plain key is present

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Copy keys before putting them into commTree

* Preserve changes to AccountData

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-12 11:21:52 +00:00