Commit Graph

50 Commits

Author SHA1 Message Date
Alex Sharov
e87d4932f7
Changeset writer move to tx (#1871) 2021-05-05 14:31:40 +07:00
Artem Vorotnikov
2148885637
Simplify ChangeSet implementation and tests (#1878)
* Move storage changeset code into one file

* Simplify storage changeset tests

* simplify more

* more

* more

* more

* move account changeset into one file

* simplify account changeset test

* NewStorageChangesetPlain -> NewStorageChangeSet

* EncodeStoragePlain -> EncodeStorage

* StorageChangeSetPlain -> StorageChangeSet

* AccountChangeSetPlain -> AccountChangeSet

* more

* more

* more

* Simplify EncodeStorage

* more

* Simplify FromDBFormat

* more rename

* rename changeset buckets (constant names only)
2021-05-04 13:34:08 +01:00
Alex Sharov
c40c3b7a7a
exec unwind etl (#1806) 2021-04-26 10:41:31 +01:00
Alex Sharov
0af3085461
Revert "GetAsOf: do not return error on nil (#1769)" (#1782)
This reverts commit 60196abf50.
2021-04-22 13:45:04 +01:00
Alex Sharov
e7a78a2d9f
Mdbx: speedup tests (#1776) 2021-04-21 20:03:08 +07:00
Artem Vorotnikov
60196abf50
GetAsOf: do not return error on nil (#1769) 2021-04-21 10:47:46 +07:00
Artem Vorotnikov
d66555268d
Harmonize DB APIs (#1669) 2021-04-03 09:26:00 +03:00
Alex Sharov
9a9c5b298a
bitmapdb to use rwtx (#1661) 2021-04-03 08:52:45 +07:00
Alex Sharov
fbd9690251
Remove tds in few tests (#1657) 2021-04-02 12:14:43 +07:00
Alex Sharov
abe5bd879c
Remove old loader, step 3 (remove some tds and blockchain objects in tests) (#1647) 2021-04-01 12:36:22 +07:00
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface (#1627) 2021-03-30 12:53:54 +03:00
Alex Sharov
91e4126ca7
db commit - remove unused return param (#1584) 2021-03-22 19:41:52 +07:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions (#1577) 2021-03-21 20:15:25 +07:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Alex Sharov
d7d9ce0a8e
fix go vet (#1425) 2020-12-26 10:18:16 +07:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
ledgerwatch
1a0e78a2e2
Remove changeset-based decorators from WalkAsOf (#1356)
* Remove decorators

* Current changes

* Fix and skip some tests for now

* Fix lint

* Specialise WalkAsOf

* Fix compile errors

* Fix lint

* Scan index files for key sizes

* More cleanup

* Fix to remote server

* Fix for walkasofstorage

* Fix/skip tests

* Fix tests

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-18 14:58:52 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
ledgerwatch
b747ab5324
[WIP] CallTraces index (#1157)
* Initial commit for CallTraces index

* Fix compilation

* fix lint, add comment

* Fix integration

* Add Close function to ethdb.Cursor, fix some compile errors

* Try to stop cursor leak in Get

* Fix compile errors in RPC daemon

* Fix compile errors

* fixing another way

* Some fixes

* More fixes

* More fixes

* More fixes

* Fixes to core/state

* Fix lint

* Fix lint

* Fixes

* Stage caching for call trace stage

* Add mem stats

* Try to stop the leak

* Turn off debug

* Chunks for 10k blocks

* Print

* Revert "Print"

This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.

* Revert "Chunks for 10k blocks"

This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.

* Trying to fix the leak

* Don't compute receipts in re-tracing

* Not compose block

* Print speed, fix receipts, bigger caches

* Fix lint

* Utilise changeset info

* Counters

* Use NoReceipts and ReadOnly

* ReadOnly is incompatible with caching

* Skip test leaking transactions

* Fix block test

* Change disable message for call-traces stage

* Use block option for call traces integration

* Fix retracing due to incarnation
2020-10-12 09:39:04 +01:00
Igor Mandrigin
adf52465e3
move ./trie to ./turbo/trie (#1114)
Useful for minimizing merge conflicts when rebasing new geth
2020-09-14 11:33:39 +01:00
Igor Mandrigin
2a6478d799
turbo-api: create a second executable file with a custom stage example (#1055) 2020-09-05 18:07:27 +02:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
b00ris
9fc51a3efd
Account range (#781)
* save state

* cleanup

* add walkasof test for storage with chunks

* fix lint

* fix lint

* remove useless comments

* fix dump test

* move accountRange test to plain state

* get last block fixes
2020-07-27 07:09:50 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
8d04689041
close db in tests (#669) 2020-06-16 12:05:40 +07:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index (#648)
* Debug infos

* bucketStats lmdb by default

* Use bools instead of strings

* Fix linters

* Fix linter

* Finally fix

* Fix compile error

* Fix test

* Fix formatting

* Fix shadowing

* Cleanup

* cleanup
2020-06-11 07:19:15 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger (#624)
* lmdb tests

* trigger ci

* fix tests

* disable parallelism

* disable parallelism

* cleanup resources

* cleanup resources

* reduce concurency

* try run tests on bolt

* try run tests on bolt

* fix downloader test

* run bolt tests

* rely on interface instead of exact instance

* Rename AbstractKV to KV

* don't use separator for badger

* don't initialize badger cursor - because it not used here

* fix linter

* try reduce badger compactors

* compat with master

* try lmdb

* try lmdb

* try lmdb

* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option

* allow to close closed db

* release

* release

* ideal batch size for badger

* ideal batch size for badger
2020-06-05 10:25:33 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
Andrew Ashikhmin
5bbdb6a4b9
Use uint256 for account balance (#580)
* Use uint256 for account balance

* linter
2020-05-26 17:53:50 +01:00
Andrew Ashikhmin
b16e560529
Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore (#575)
* Produce less garbage in GetState

* Still playing with mem allocation in GetCommittedState

* Pass key by pointer in GetState as well

* linter

* Avoid a memory allocation in opSload

* Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore

* linter

* linters

* small clean up
2020-05-25 12:12:25 +01:00
Andrew Ashikhmin
7b40cbb6fa
Incarnation should be read by StateReader, not StateWriter (#506)
* GetCurrentAccountIncarnation

* Incarnation should be read by StateReader, not StateWriter

* Use GetHistoricalAccountIncarnation in DbState

* RemoteReader ReadAccountIncarnation

* Handle the case where a contract has self-destructed, then Eth sent to it, then it got recreated again
2020-05-02 19:00:42 +01:00
ledgerwatch
051323edbb
Tinkering with storage changesets (#497)
* Adjustments

* fix tests

* Fix tests

* Widen most numbers to 32 bits
2020-04-28 16:36:00 +01:00
ledgerwatch
a09547e822
Truncate history index after reorgs (#492)
* Refactor RewindData

* Add cpu profile

* Index truncation

* Fix compile

* Fix linter

* Fix linter

* Fix tests

* Create test

* Remove Remove

* Clarifying comment
2020-04-28 06:36:33 +01:00
Igor Mandrigin
468bd65109
Add "execute blocks" stage to the staged sync. (#486)
* extract functions to different files

* post-rebase fixups

* stage v0

* add a log

* better log entries

* fixes in log messages

* fix some stuff

* review fixes

* fix linters

* pw as a variable

* fix a test

* add Byzantium check

* batch save progress too
2020-04-26 17:02:38 +01:00
ledgerwatch
0c6a44b5e5
Index bits to mark creation of accounts or storage items (#483)
* Index bits

* Fix linter

* Fix linter

* Not to call getModifiedAccountsByNumber for go-ethereum

* Remove the 0x000 kludge
2020-04-25 15:50:32 +01:00
ledgerwatch
091811c3d1
Account range crash fix (#481)
* Pass error from accountRange

* Fix linter

* Fixes for accountRange

* Fix tests

* Fix tests

* Eliminate deadlocks

* Fixes

* Fix linter

* Fix linter

* Code hash fixes

* Fix linter

* Fix linter

* Fix linter

* Fix linter

* Fix in rpctest

* Ignore storage roots

* Fix linter

* Fix dump test
2020-04-23 10:35:43 +01:00
ledgerwatch
623545a47c
Fix index chunks (#478)
* First stab at fixing index chunks

* Fix compilation

* Fix tests

* Fix walkAsOf for storage

* Fix

* Fix for walkAsOf accounts

* Fix linter
2020-04-21 09:15:40 +01:00
b00ris
a9f71298aa
Index chunking (#473)
* index generation

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* fix

* save state

* fix
2020-04-20 11:35:33 +01:00
Giulio rebuffo
6e2fc0d936
[WIP] 459 merge storage account (#466)
* bt commit

* merged buckets

* adjusted cmd/

* fixed sync
2020-04-19 20:51:32 +01:00
Alex Sharov
59f997a663
Move account root to ih (#465)
* move_account_root_to_ih

* temptorary comment test
2020-04-18 21:09:44 +01:00
Igor Mandrigin
acbf8bcbdf
Enable thin history by default (#456)
* Enable thin history by default

* remove thin history flags

* remove from tests

* fix test compile error

* test fixups

* fix more stuff
2020-04-15 12:33:22 +03:00
ledgerwatch
fbe7a2cb06
THIN_HISTORY - Reinstate incarnation encoding in ChangeSets, but remove from index keys (#443)
* Cutting incarnation out of history and changeset keys

* Rollback the remove of incarnation only in changesets

* Fix linter

* Remove debug printing

* Reinstate encoding of incarnations in the ChangeSets

* Fix linter

* Fix linter

* Fix DefaultIncarnation

* Fixes

* Fix lint

* Fix to history writing

* Fix linter
2020-04-12 19:36:27 +01:00
Alex Sharov
535d73be5f
Run CI on bolt with prefix_compression_without_allocation (fixes) (#380) 2020-04-11 20:28:15 +07:00
ledgerwatch
ad64ae2085
[THIN_HISTORY] Decouple changesets and history from ethdb.Database (mutation/bolt_db/badger_db) (#432)
* Prepare for refactoring

* Decoupling

* Further cleanup of mutation

* Further cleanup of mutation

* Remove some tests (temporarily)

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Recover mutation_test cases

* Fix lint

* Fix WalkAsOf test

* Fix lint

* Fix TODO, linter

* Fix lint

* Fix lint

* Fix tests for THIN_HISTORY

* Fix lint

* Fix lint

* Reduce visibility of DecodeStorage

* Revert "Reduce visibility of DecodeStorage"

This reverts commit 2e11e16ea095b75a62cdddb77b9477eabc259e67.
2020-04-09 18:23:29 +01:00