Commit Graph

31 Commits

Author SHA1 Message Date
ledgerwatch
43b38539b3
[erigon2] Remove empty commitments and add comments (#331)
* Remove empty commitments and add comments

* Remove logging

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-16 22:38:37 +00:00
ledgerwatch
1c6e82c2b6
[erigon2] Thin commitment (2nd attempt) (#329)
* Another fix for history files

* Half way through

* Another fix

* Correct closing sequence

* Remove first byte insert marker

* More on think commitments

* Fixes

* Fixes

* Print

* Skip touchMap

* Merge branchData from trees and from files

* Fill branch commitment

* Fill branch commitment

* Print

* Fix?

* Merge branchData when updating in the tree

* Better panic

* Prints

* Prints

* Prints

* Create complete branch data if it did not exist before

* Cleanup printing

* Fix merge

* Fix merge use

* Fix transform

* Better startBlock panic

* Preserve touchMap

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Include fieldbits during transform

* Fix history reads

* Print

* Print

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Print largestMerge, lock files in branchFn

* Add storage lock

* Prints

* prefixLen fix

* prefixLen fix

* Fixes

* Remove print

* Remove print

* Set changesets and commitments flags upfront

* Logging instead of printing

* Fix history merge, recsplit panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-16 16:44:00 +00:00
Alex Sharov
9789f53041
aggregator: close idx after build (#327) 2022-02-14 18:38:39 +00:00
ledgerwatch
1363859958
[erigon2] Fix history bug, optimise history search (#326)
* Another fix for DeleteAccount

* Optimise bitmap search

* Optimise bitmap search

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-13 18:56:36 +00:00
ledgerwatch
d9367b582a
[erigon2] fix change files for commitments (#318)
* fix change files for commitments

* Not create commitment files unless enabled

* Print loaded files

* More fixes to commitment changes

* More fixes to commitment changes

* Fix for DeleteAccount storage values

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-12 14:51:23 +00:00
Alex Sharov
6f85066c7e
path -> filepath (path package is for urls) (#321) 2022-02-12 20:11:30 +07:00
alex.sharov
b041b959ca direct: txpool, mining 2022-02-12 19:33:47 +07:00
ledgerwatch
5f7283d0f1
[erigon2] History reader (#312)
* Initial code for history reader

* Print

* Print

* Fix npe

* Open files of all types

* don't close file in the constructor

* Print foundTxNum

* Print foundTxNum

* Print adding to bitmap

* Print when reducing

* Print when reducing

* Print when reducing

* Enum false for index

* Remove prints

* Remove prints

* Print loaded files

* Tracing

* Tracing

* Tracing

* Tracing

* Tracing

* Optinal commitments

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-11 23:44:30 +00:00
ledgerwatch
441a4c3cde
[erigon2] Chain history and bitmap indices - part 2 (#308)
* corretly shutdown history goroutine

* Different final merge for history files

* Skip value

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Handle collision

* Handle collision

* Debug

* Debug

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-10 15:30:55 +00:00
ledgerwatch
f5deff3ac9
[erigon2] accumulating chain history and bitmap indices (#306)
* Aggregator refactoring

* Restore the test

* Simplify commitment

* Lint

* onBefore

* Wip smaller commitments

* Fixes

* Fix lint

* Check for zero len

* Print

* Remove restriction

* Correct restrictions

* Correct restrictions

* More merging

* delBitmap

* Switch off merging

* Reintroduce background merge

* Try to fix

* Slight optimisation

* Produce initial changeset files

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-08 21:19:29 +00:00
ledgerwatch
92fc7215c8
[erigon2] fix commitment error (#304)
* Try to fix

* Fix

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-04 18:48:50 +00:00
ledgerwatch
cbfd733672
[erigon2] optimisations (#297)
* Reuse getter and indexReader

* Properly reset Getter

* Use separate getter and indexReader for merge thread

* Remove DB

* Fixes

* Print

* Fixes

* Cleanup signatures

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-03 08:49:47 +00:00
ledgerwatch
1cc07746a4
[erigon2] Try to fix deadlock (#291)
* Try to fix deadlock

* Another lock

* Try to optimise

* comments and cleanup

* Prevent deadlock on panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-01 10:28:05 +00:00
ledgerwatch
4e8840256e
[erigon2] Use shorter references instead of full plain keys in the commitment files (#289)
* Rearrange aggregations

* More rearranging before introducing 3 threads

* Background aggregation|

* Concurrency fixes

* Remove files under lock

* Better logging

* Remove files without lock

* Fix lint

* Fix locking

* Try

* Fix background Merge

* Log merging

* Log merging

* Less logging

* Millisecond

* Add Stats function

* Log merge only after 1m

* Wrong counting

* plain key extract and replace functions

* Insert valTransform function

* Not parse first byte

* Not parse first byte

* Fix lint

* Switch to thin state references

* Fix lint

* Fix lint

* Debug print|

* Fix decoding

* Turn off valTransform

* Not to reuse transformer

* Print

* Print

* Print

* Derive hashed keys later

* Fix

* Fix log

* Fix

* Debug

* Another fix

* Fix

* Fix

* Print

* Print

* Data race

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-31 22:32:00 +00:00
ledgerwatch
586ab3e6b3
Separate state btree files (#287)
* Separate state file btrees, fix Match in the decompressor

* fix match

* Fix to match

* Switch back from Match

* Try to use match, close indices

* Fixing Match

* Use Skip

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-29 11:12:38 +00:00
primal_concrete_sledge
24098d94d8
WIP: issue/issue-250-use_compressed_keys (#285)
* issue/issue-250-use_compressed_keys

* Step into the twilight zone

* fix zero byte

* Add zerolen check

* Remove branhc data check

* Check lengths
2022-01-28 19:30:32 +00:00
ledgerwatch
a7ec201c0a
Update aggregator.go (#274) 2022-01-24 22:43:41 +00:00
ledgerwatch
7ec016b160
Fixes in compress (#260)
* Fixes in compress

* Reuse outputFile also as uncompressed file

* Close file before renaming

* Trace

* Untrace

* Use 8 threads

* Print aggregations

* Print merge and timing

* Print merge and timing

* readonly mode for patricia

* Fix to infinite loop

* Fix file names

* Cleanup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-24 22:13:48 +00:00
primal_concrete_sledge
d8a33270e8
issue/issue-249-add_index_reader (#273)
* issue/issue-249-add_index_reader

* Add licence
2022-01-24 20:39:04 +00:00
Alex Sharov
11ab5bdbb8
Parallel compressor - allow empty words (#245)
* save

* save

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 13:57:35 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
ledgerwatch
0eaa0333e1
[erigon2] Optional writing of before changes, commitment decoupled from blocks (#237)
* Separate commitment from blocks

* Optional beforeOn

* Optional beforeOn

* Optional beforeOn

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-16 20:27:44 +00:00
ledgerwatch
e5c07ec901
[erigon2] Resumable prototype (#236)
* Not warn about files that don't match at all

* Not warn about files that don't match at all

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Add trace

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

* Cleanup

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-15 22:09:06 +00:00
ledgerwatch
c08ed12256
[erigon2] Measure change set sizes with perfect hash table (#230)
* Remove warning message

* Calculate sizes of change sets for transactions

* Calculate sizes of change sets for transactions

* Produce changeset files

* Not to produce changesets for commitments

* Remove rewindTx

* Close decompressor

* Not generate changesets by default

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-14 14:10:10 +00:00
ledgerwatch
ae6f7202d2
[erigon2] Introduce commitment files and table (#201)
* Introduce commitment files and table

* Introduce commitment files and table

* Introduce words buffer

* Move changes out of aggregator into the writer

* Start calc commitment

* Add commitment package

* More on hex patricia

* More

* account decorator

* More

* Add test

* More

* More to the test

* Use hex in tests

* More

* More

* More

* More

* Simplified cells

* Add delBitmap

* More advanced MockState

* More compact unfolding

* Separation of hashed Keys

* Sepatate downHashedKey and upHashedKey

* Carry extension node through accounts

* optimised row allocations

* Fix encoding/decoding, add trace flag

* Added account and storage into the cells

* Make accountKeyLen global setting

* Remove BranchNodeUpdaqte intermediate

* Started on computeHash

* Initial hash calculations

* Fix lint

* Change account encoding

* Fix commitment issues

* Fix lint

* Fix lint

* Fix lint

* Trace

* Small fix and tracing

* Print branch hashes, fix empty

* Fix

* Fix

* Fix

* Fix

* Print accountFn

* Trigger accountFn

* Trigger accountFn

* Another fix for extension nodes

* Return root hash and set trace

* Print depth for computeCellHash

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Fix for storage leaves

* Ineffectual deletes

* Trace needUnfolding

* Fix

* Fix

* Fix

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* trace deletes

* not overwrite downHashedKey in updateAccount

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* simplify needUnfolding

* propagate deletes when unfolding

* remove deletes when folding

* remove deletes when folding

* remove upHashedKey when delete cell

* remove upHashedKey when delete cell

* aggregate storage changes later

* remove del bit when updating account

* remove del bit when updating account

* remove del bit when updating account

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* perform delete after code update

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit 0234ea6c3e1a0b2e7599df7c470d4b8b8d56736a.

* latest update has precedence when aggregating

* Revert "latest update has precedence when aggregating"

This reverts commit cfa7b75327e4d5d1236f905c3c04eb640b21838d.

* Introduce FinishTx

* Separate commitment separation from computation

* Prevent spurious delete

* Spurious deletes

* Trace param

* Preserve upHashedKey when unfolding

* Fix

* Fix

* Correctly compute firstInsert

* Try to fix delete + fold

* Remove commented out

* updateStorage to remove deleted flag

* rename upHashedKey to extension, fix clear-up

* Carry extension up if account plain key is present

* Carry extension up if account plain key is present

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Not to fail for deleteAccount

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Print

* Copy keys before putting them into commTree

* Preserve changes to AccountData

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-12 11:21:52 +00:00
ledgerwatch
e83c9f276a
Tracing in the aggregator (#194)
* Tracing in the aggregator

* Trace merge of keys

* print value when merging

* More tracing

* Fix trace keys

* Fix tracing

* Fix skipping

* Fix tracing for deleteAccount

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-11-27 08:34:56 +00:00
ledgerwatch
d5f6afc92d
Delete code with account, add code test (#171)
* Delete code with account, add code test

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-11-17 15:49:48 +00:00
ledgerwatch
14b6423388
[state aggregator] contract recreation storage test (#169)
* Remove incarnation, add recreation

* Fix recreation test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-11-16 21:59:26 +00:00
ledgerwatch
ea56078a4c
state aggregator - tests for contract storage (#165)
* First test for storage

* Debugging

* Fixes to delete account test

* Fix the lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-11-16 00:11:40 +00:00
Alex Sharov
5b7f67deae
Snapshot naming (#163)
* save

* save

* save

* save

* save

* save
2021-11-15 14:19:56 +00:00
ledgerwatch
fd19ad8148
State aggregator (#114)
* State aggregator

* Compile fix

* More

* Add

* More

* More on aggregator

* Writes (still incorrect)

* Move table names

* More

* Start of aggregation

* Change files instead of db

* More on change files

* More

* More

* Dealing with state and change files

* More

* More

* More boilerplate

* More

* More

* Iteration over storage

* More boilerplate

* More fixes

* Insert flag

* More

* Unit test

* Add more to the test

* Expand the test a bit

* More testing

* Keep fixing the test

* More fixes to the test

* Clean up DB tables upon aggregation

* More fixes

* Remove update/insert indicator from returned values

* Add assertions

* close files before deleting

* close files before deleting

* close files before deleting

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-11-13 12:12:29 +00:00