Commit Graph

33 Commits

Author SHA1 Message Date
Alex Sharov
b7ada2a595
create new Change object if incarnation increased (#2539)
* create new Change object if incarnation increased

* create new Change object if incarnation increased
2021-08-18 09:41:07 +07:00
Alex Sharov
6bd44eb26c
move kv to erigon-lib (#2467) 2021-07-29 18:53:13 +07:00
Alex Sharov
21cb7befa4
finish remove bucket suffix (#2458) 2021-07-28 10:43:51 +07:00
Alex Sharov
838e5f9ef2
Move bucket constants into kv package, move kv interface to kv package (#2455) 2021-07-28 09:47:38 +07:00
Alex Sharov
6801208497
move some tests to kv interface (#2438) 2021-07-24 14:14:11 +07:00
Alex Sharov
a36a613e61
move migrations to kv interfaces (#2430) 2021-07-24 11:28:05 +07:00
Alex Sharov
9683b26483
remove ctx from state writer interface (#2302) 2021-07-05 15:42:44 +07:00
Alex Sharov
ccac3c6566
Add tx.Append method for ergonomic (#2088)
* close cursor

* stateless append

* stateless append

* faster pruning del and close cursor

* faster pruning del and close cursor

* less wrappers

* less wrappers

* less wrappers

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 14:32:48 +01:00
Artem Vorotnikov
3a86048af6
Remove block number from PlainStateWriter (#2091) 2021-06-04 11:13:19 +03:00
ledgerwatch
835b954f5c
State stream to RPC daemon (#1986)
* Remove interfaces

* Squashed 'interfaces/' content from commit e06631eb4

git-subtree-dir: interfaces
git-subtree-split: e06631eb4d0926c2d6a4f552497b920b4ed8d1bd

* Update KV interface

* Squashed 'interfaces/' changes from e06631eb4..014677ffe

014677ffe Merge remote-tracking branch 'origin/master' into stream
08c32a09e add version method to txPool and ethbackend
5b6bf70b9 Update README.md
7712cb267 Update README.md
f895ece4c save (#37)

git-subtree-dir: interfaces
git-subtree-split: 014677ffe5bff0dee1a333f06c92e6110d791468

* Fix old interfaces

* Squashed 'interfaces/' changes from 014677ffe..df31e1146

df31e1146 remove action from storage change

git-subtree-dir: interfaces
git-subtree-split: df31e1146c368eda2e2b15ab252b78fba7a0a6f3

* add Accumulator

* add location

* Squashed 'interfaces/' changes from df31e1146..472584639

472584639 Merge remote-tracking branch 'origin/master' into stream
dd6a42724 Refactor of consensus interface (#28)

git-subtree-dir: interfaces
git-subtree-split: 472584639f637189dfb906ef1ed03665f98d55d2

* Fix compilation in cons

* Pass accumulator to Execution stage

* Fix test

* Pass accumulator to unwind and plain writer

* Add accumulator use to plain writer

* Squashed 'interfaces/' changes from 472584639f..5c36f038b8

5c36f038b8 State change stream to KV (#38)
REVERT: 472584639f Merge remote-tracking branch 'origin/master' into stream
REVERT: df31e1146c remove action from storage change
REVERT: 014677ffe5 Merge remote-tracking branch 'origin/master' into stream
REVERT: e06631eb4d Fix
REVERT: 9c10d79d2d Fix
REVERT: 61ae9307de Fix
REVERT: 4fcf34ecc5 State change stream to KV

git-subtree-dir: interfaces
git-subtree-split: 5c36f038b87096ffb6b07e90c6762c21b864cd3b

* Add state.stream flag

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-23 15:53:18 +01:00
Artem Vorotnikov
5655409616
Do not write old block changesets in pruning storage mode (#1964)
* Remove incremental staged sync

* Do not write old block changesets in pruning storage mode

* Remove pruning flag, put behaviour behind !StorageMode.History
2021-05-22 11:57:10 +01:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
e87d4932f7
Changeset writer move to tx (#1871) 2021-05-05 14:31:40 +07:00
Artem Vorotnikov
2148885637
Simplify ChangeSet implementation and tests (#1878)
* Move storage changeset code into one file

* Simplify storage changeset tests

* simplify more

* more

* more

* more

* move account changeset into one file

* simplify account changeset test

* NewStorageChangesetPlain -> NewStorageChangeSet

* EncodeStoragePlain -> EncodeStorage

* StorageChangeSetPlain -> StorageChangeSet

* AccountChangeSetPlain -> AccountChangeSet

* more

* more

* more

* Simplify EncodeStorage

* more

* Simplify FromDBFormat

* more rename

* rename changeset buckets (constant names only)
2021-05-04 13:34:08 +01:00
Artem Vorotnikov
d66555268d
Harmonize DB APIs (#1669) 2021-04-03 09:26:00 +03:00
Alex Sharov
9a9c5b298a
bitmapdb to use rwtx (#1661) 2021-04-03 08:52:45 +07:00
ledgerwatch
3287b44c2e
Bad unwind of recreation of CREATE2 contract - unit test and potentia… (#1412)
* Bad unwind of recreation of CREATE2 contract - unit test and potential fix

* Fix unit test

* Fix lint

* Fix lint

* Fixup hack

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-16 17:16:37 +00:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
342c5fee89
[to discuss] More compact append implementation (#1372)
* more compact append implementation

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* fix tests

* fix tests
2020-11-28 14:24:47 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
ledgerwatch
c06dbdadf9
Use Append for changesets and receipts (#1018)
* Use Append for changesets and receipts

* Fix lint

* Delete all receipts on unwind

* Fix lint

* Fix unwind keys for changeset

* Fix test

* Use writeReceipts flag

* Revert to Append

* Delete receipts too when resetting
2020-09-01 07:48:25 +01:00
Alex Sharov
8c9a55bb21
[To test on HDD] Write changesets to tx instead of mutation (#978)
* use mutation over tx

* clear

* add .CommitAndBegin() method

* multip put 2

* right sorting

* write non-random things to tx

* write non-random things to tx

* disable multi put 2

* clean

* clean

* remove multiput2

* clean

* clean

* clean

* clean

* clean

* add receipts
2020-08-29 08:32:51 +01:00
Alex Sharov
d98382fdf8
Integration add checkChangeSet into state_stages command (#728)
Integration add checkChangeSet into state_stages command (#728)
2020-07-10 10:25:40 +07:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
b00ris
266f9a5208
Plain state index (#595)
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
2020-05-31 07:57:47 +01:00
ledgerwatch
810cf13bf8
Explicit incarnation map in the database for deleted contracts (#578)
* Explicit map of incarnations

* Fix linter
2020-05-26 13:27:21 +01:00
Andrew Ashikhmin
b16e560529
Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore (#575)
* Produce less garbage in GetState

* Still playing with mem allocation in GetCommittedState

* Pass key by pointer in GetState as well

* linter

* Avoid a memory allocation in opSload

* Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore

* linter

* linters

* small clean up
2020-05-25 12:12:25 +01:00
ledgerwatch
6d883b5350
Fix staged sync issue introduced by LRU caches (#568)
* readPlainAccount

* readPlainAccount

* Try to disable accountCache

* Try to not separate changeBatch

* The actual fix
?

* Move to fastcache, include incarnation

* Fix linter

* Fix linter

* Disable caches

* Fix linter

* Reenable all except storage cache

* Cosmetics

* Disable account cache

* Disable code size cache

* Fix code caching

* Apply the same fix to db_state_writer
2020-05-22 16:11:00 +01:00
ledgerwatch
e2848ac666
Staged exec: RLU cache between StateReader and StateWriter, separate batches for state and changesets (#542)
* Add cpu profiling

* Splt state and change batches

* Log commits

* Remove

* Log memory

* Add lru caches for staged execution

* Fix

* Fix for nil account

* Fix linter

* Fix linter

* Increase state batch size

* Add memory profiling over http

* Reduce cache sizes

* Fix storage cache

* Fix

* Fix

* Fix

* Fix

* Cleanup

* Fix linter

* Reduce logging

* Reduce logging
2020-05-21 13:27:52 +01:00
Igor Mandrigin
fd77eaf86a
Staged Sync: Execution phase should use "plain state" (#548)
* introduce PlainStateReader with fallbacks

* no 10.000 changes in tests

* even less iterations

* remove even more iterations

* add `go run ./cmd/geth --syncmode staged --plainstate` flag

* fix serialization calls

* make a more sensible file default

doesn’t affect anything, because this flag is always overriden when parsing CLI. but still.
2020-05-15 08:52:45 +01:00