Commit Graph

32 Commits

Author SHA1 Message Date
Alex Sharov
6bd44eb26c
move kv to erigon-lib (#2467) 2021-07-29 18:53:13 +07:00
Alex Sharov
838e5f9ef2
Move bucket constants into kv package, move kv interface to kv package (#2455) 2021-07-28 09:47:38 +07:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Thomas Jay Rush
dcdcc1a30f
Fixes issue #1232 (#1233)
* Fixes issue #1232

* Adding/updating test cases

* Updating as per comments

* Use tx instead of db in GetModifiedAccounts and clean up a bit

* option to specify LMDB map size for cmd/integration commands (#1238)

* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Use mapSize for first opening

* Fix for cobra

* Extend lmdb.mapSize for all commands

* Fix lint

* Fixes issue #1232

* Adding/updating test cases

* Updating as per comments

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-10-14 12:01:44 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124)
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt (#1085)
* kill bolt

* kill bolt

* fix test

* clean
2020-09-09 11:09:55 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
b00ris
e8b617bbf2
Add sort.Search to account changeset (#671)
* fast search

* fix

* remove copy bytes
2020-06-16 15:03:35 +01:00
Alex Sharov
916a1f8b83
[wip] Lmdb: AbstractKV and DB interfaces (#589)
* resetIH from scratch if needed

* lmdb

* add AbstractKV to loader, added new Object accessor around AbstractKV

* add lmdb cli flag

* add requirement of k!=nil on error in docs

* add Size method for compatibility

* read after put tests

* fix multiput nils

* simplify loops

* increase mmap size

* better error messages

* better error messages

* fix tests

* better error messages

* cleanup

* avoid bolt usage in test

* move hardcoded bucket name to dbutils

* register more buckets

* register more buckets

* fix test
2020-05-30 09:12:21 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
ledgerwatch
810cf13bf8
Explicit incarnation map in the database for deleted contracts (#578)
* Explicit map of incarnations

* Fix linter
2020-05-26 13:27:21 +01:00
Igor Mandrigin
db746bba7a
"Unwind" for the execution phase when plain state is selected (#571)
* "Unwind" for the execution phase when plain state is selected

* test stub (fails)

* tests (one with incarnations fails)

* test fixups

* fix tests: cleanup contract code bucket
2020-05-25 12:11:56 +01:00
Igor Mandrigin
fd77eaf86a
Staged Sync: Execution phase should use "plain state" (#548)
* introduce PlainStateReader with fallbacks

* no 10.000 changes in tests

* even less iterations

* remove even more iterations

* add `go run ./cmd/geth --syncmode staged --plainstate` flag

* fix serialization calls

* make a more sensible file default

doesn’t affect anything, because this flag is always overriden when parsing CLI. but still.
2020-05-15 08:52:45 +01:00
ledgerwatch
41b5fce078
resolver: merge HashBuilder-s and ResolveSet-s, decouple from ResolveRequest-s (#540)
* merge hbStorage and hb in resolver

* Fix prepare

* Remove tracing by default

* cleanup

* Fix linter

* Not to include incarnation into the ResolveRequest

* Fix linter

* Fix linter

* Fix test

* Fix linter

* Revert

* Fix for infinite loop

* Fix linter

* Collapse rss into rs

* remove reliance of requests in resolver

* Remove currentReq

* Remove fields from ResolveRequest

* Fix linter
2020-05-11 05:46:07 +01:00
Andrew Ashikhmin
7b40cbb6fa
Incarnation should be read by StateReader, not StateWriter (#506)
* GetCurrentAccountIncarnation

* Incarnation should be read by StateReader, not StateWriter

* Use GetHistoricalAccountIncarnation in DbState

* RemoteReader ReadAccountIncarnation

* Handle the case where a contract has self-destructed, then Eth sent to it, then it got recreated again
2020-05-02 19:00:42 +01:00
ledgerwatch
a09547e822
Truncate history index after reorgs (#492)
* Refactor RewindData

* Add cpu profile

* Index truncation

* Fix compile

* Fix linter

* Fix linter

* Fix tests

* Create test

* Remove Remove

* Clarifying comment
2020-04-28 06:36:33 +01:00
Igor Mandrigin
acbf8bcbdf
Enable thin history by default (#456)
* Enable thin history by default

* remove thin history flags

* remove from tests

* fix test compile error

* test fixups

* fix more stuff
2020-04-15 12:33:22 +03:00
ledgerwatch
14bcd3f142
Fixes for debug_storageRangeAt RPC method (#454)
* Fixes for debug_getStorageRangeAt

* Fix linter
2020-04-14 15:22:05 +01:00
Andrew Ashikhmin
82b80c2ebe
Thin history: Fix an error in mutation.getNoLock (#409)
* CheckChangeSets and thin history

* small code de-duplication

* small code clean-up

* Fix an error in mutation.getNoLock

* CheckChangeSets: truly make historyfile = chaindata by default
2020-03-26 21:52:22 +00:00
b00ris
2b74b43678 History index encoding
save state

save state

remove old history index

fix lint

compress index
2020-02-24 23:29:46 +03:00
b00ris
958ace4088 split storage and account encoding for changeset 2020-02-24 22:36:16 +03:00
b00ris
e69f53db8d Merge branch 'master' of github.com:ledgerwatch/turbo-geth into split_cs_buckets_squash 2020-02-24 21:47:26 +03:00
b00ris
aedecfc820 split changeset bucket
fmt

badger changes

badget puts

fix shadowing

fix mutation test. restore DecodeChangeSet

fix pruner test
2020-02-24 20:56:32 +03:00
ledgerwatch
0cd4f65ffe
Copy values from the ChangeSet bucket during Rewind (#348)
* Fix rewinding

* Fix rewinding
2020-02-01 16:24:38 +00:00
Giulio rebuffo
32375df7af made changeSet an helper type (#314)
* made changeSet an helper type

* added comment

* removed dbutils.Add

* remove common.CopyBytes from walk

* fixes
2020-01-15 20:51:10 +00:00
b00ris
bcee4845dc
Thin history (#272)
* save state

* remove repair

* save state

* remove emptydb check

* save state

* add walkAsOf test

* add WalkAsOf and MultiWalkAsOf tests

* deployed contracts counter

* reference counter for contract code

* drop storage root&contract hash for changesets

* start incarnation is 1(save state)

* fix ReorgOverSelfDestruct test

* hack fix TestReorgOverSelfDestruct

* test benchmark

* cleanup

* remove useless debug

* remove print trie

* return remove subtrie call  to updateTrieRoot

* save state

* add mutation test

* remove useless test

* fix

* added mutation commit test

* rename experiment to thin history

* thin history mutation commit test

* fix ethdb tests

* getAsOf test

* add test&fix history index

* fix test

* make test for index search

* compute trie root incarnation fix

* tests fixes

* done job in case of panic

* fix lint

* fix&test for bad incarnation

* fix initial incarnation for genesis

* fix lint

* fix changeset test

* fix storage ranges test

* fix lint

* move set incarnation to create contract

* add comment

Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Evgeny Danilenko <6655321@bk.ru>
2019-12-20 15:25:40 +03:00
alex.sharov
73b50e080d debug_StorageRangeAt v0, got wrong response on request: {"jsonrpc":"2.0","method":"debug_storageRangeAt","params":["0x2bf07c790737be3bc4c57cbf3dedb231806f6bfef434657d59dcc9ddbe4665ab", 1,"0x8b3b3b624c3c0397d3da8fd861512393d51dcbac","0xfade75560a6cfb895f5dc7c4ab3fa10089ac2372c98aa78280d029ab36285ad6",1024],"id":1377} 2019-12-13 09:56:30 +06:00
Andrew Ashikhmin
8e5e5d881f
Issue 123 Make encoding of ChangeSet ordered by keys and binary-searchable (#207) 2019-12-03 12:13:12 +01:00
Evgeny Danilenko
992e34745c
Replace red black trees (#184)
* use map instead of rb tree

* GetModifiedAccounts

* introduce tuples

* linters

* linters

* init puts

* init maps

* remove GetSortedKeys

* fix string bucket case

* use append in tuples

* fix tuples

* fix tests

* all tests are green

* fmt

* fmt

* rename tuple to tuples
2019-11-21 18:38:00 +00:00
b00ris
9eb7d8b1c2 Suffix reorg (#113) 2019-11-04 14:15:26 +01:00
Alexey Akhunov
fe01bccbb8 Apply Turbo-Geth modifications to go-ethereum codebase 2019-11-01 21:52:03 +01:00