Alex Sharov
7c60cc0fee
step towards kv interface ( #2313 )
2021-07-08 09:56:09 +07:00
ledgerwatch
9023f2a29c
Investigate slow receipt repair ( #2306 )
...
* Collect data about broken receipts
* Actually collect data
* Fix CBOR problem more efficiently
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-07-05 19:14:58 +01:00
ledgerwatch
954304e43a
Turn off receipt repair migration for investigation ( #2303 )
...
* receipt repair migration - check for r storage mode
* Turn off migration
* Fix lint
* Better way to disable migration
* Better way to disable migration
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-07-05 11:25:18 +01:00
ledgerwatch
63ccabc72f
Receipt repair migration ( #2291 )
...
* Fixes in rpctest and rpcdaemon for debug_traceTransaction
* Fix for opcode tracer
* Tool to fix receipts
* Better mechanism for detecting broken receipts
* Fixes
* Introduce receipt_repair migration
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-07-04 14:48:13 +01:00
Alex Sharov
ba902f7ef3
separate_config_from_node_and_eth ( #2289 )
2021-07-04 08:50:32 +01:00
ledgerwatch
5ad449ef22
Migration to fix receipts (broken due to change of CBOR codec) ( #2221 )
...
* Scan receipts
* Not check tx type before Berlin
* Reverse
* scan receipt code in the migration
* Turn migration on
* Apply migration
* Fix
* Print last
* Remove last
* Not remove last
* Not updata in migration
* no cursors in migration
* Handle empty blocks
* Handle empty blocks
* Use the same code in hack
* Print
* Print
* Save
* Fix
* Fix lint
* Fix lint
* Fix for empty DB
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-06-23 15:05:16 +01:00
Artem Vorotnikov
72e9a34365
Remove unused tables, soft rename, take 3 ( #2207 )
...
* Remove unused tables, soft rename, take 3
* Bump db schema version
* nil check in migration
2021-06-20 20:46:57 +01:00
Artem Vorotnikov
af836a6200
Nuke LMDB ( #2167 )
2021-06-16 13:57:58 +03:00
Alex Sharov
eee871aedf
Add db label (later will add txpool db) ( #2081 )
...
* add db label - later will add txpool db
* move commit marker to kv_mdbx behind Chain label
* Update migrations.go
* clean
Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 15:56:49 +01:00
ledgerwatch
43915a73a8
Migration to fix trace_filter ( #2095 )
...
* Migration to fix trace_filter
* Fix to db/tx
* Fix to db/tx
* Error fixes
* Bump KV version
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-06-04 12:45:06 +01:00
ledgerwatch
729bfab988
Split DBSchema version depending on the database type ( #1990 )
...
* Split DBSchema version depending on the database type
* Fix test
* Fix test
* Fix lint
* Update migrations.go
* Update bucket.go
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 10:20:43 +01:00
Alex Sharov
11d1e9119e
remove pre-berlin migrations ( #1987 )
2021-05-22 10:29:00 +07:00
Alex Sharov
545fe32f68
Mdbx - make it default db. Lazy buckets renaming. ( #1982 )
2021-05-22 10:03:02 +07:00
Alex Sharov
0be3044b7e
rename ( #1978 )
...
* rename
* rename "make grpc"
* rename "abi bindings templates"
* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
01ae962459
Clean tmp on start ( #1808 )
...
* fininish to remove chain context
* clean
* move migrations folder from
<datadir>/etl-tmp/migrations
to
<datadir>/migrations
2021-04-26 13:51:01 +01:00
ledgerwatch
e3f3dd3c9b
Integration tests 1 ( #1793 )
...
* Initial commit
* Add sentry gRPC interface
* p2psentry directory
* Update README.md
* Update README.md
* Update README.md
* Add go package
* Correct syntax
* add external downloader interface (#2 )
* Add txpool (#3 )
* Add private API (#4 )
* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5 )
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Separate upload messages into its own stream (#6 )
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Only send changed accounts to listeners (#7 )
* Txpool interface doc (#9 )
* Add architecture diagram source and picture (#10 )
* Typed hashes (#11 )
* Typed hashes
* Fix PeerId
* 64-bit tx nonce
* Add proper golang packages, max_block into p2p sentry Status (#12 )
* Add proper golang packages, max_block into p2p sentry Status
* Change EtherReply to address
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Add Rust infrastructure (#13 )
* DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665
* more p2p methods (#15 )
* add mining methods (#16 )
* First draft of Consensus gRPC interface (#14 )
* Update Rust build
* Fix interfaces in architecture diagram (#17 )
* Fix KV interface provider
* Fix Consensus interface provider
* drop java attributes (#18 )
* tx pool remove unused import (#19 )
* ethbackend: add protocol version and client version (#20 )
* Add missing ethbackend I/F (#21 )
* Add interface versioning mechanism (#23 )
Add versioning in KV interface
Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
* spec of tx pool method (#24 )
* spec of tx pool method (#25 )
* Update version.proto
* Refactor interface versioning
* Refactor interface versioning
* Testing interface
* Remove tree
* Fix
* Build testing protos
* Fix
* Fix
* Update to the newer interfaces
* Add ProtocolVersion and ClientVersion stubs
* Hook up ProtocolVersion and ClientVersion
* Remove service
* Add compatibility checks for RPC daemon
* Fix typos
* Properly update DB schema version
* Fix test
* Add test for KV compatibility|
* Info messages about compability for RPC daemon
* DB schema version to be one key
* Update release intructions
Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: canepat <tullio.canepa@gmail.com>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 16:46:29 +01:00
Alex Sharov
bbe9af3e25
Stages as const ( #1774 )
2021-04-21 15:02:21 +07:00
Evgeny Danilenko
17c07c50a5
Move clique buckets to separate DB ( #1703 )
...
* debug
* debug
* it works
* rename clique bucket
* remove genesis special case
* copy snapshot
* remove debug
* migration
* debug
* regenerate snapshots
* simplify
* regeneration
* after merge
* tests
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-19 22:58:05 +01:00
Artem Vorotnikov
b71c6323a0
KVGetter, replace old geth interfaces with it ( #1676 )
2021-04-05 20:04:58 +07:00
b00ris
c03da8fb8e
Split header prefix bucket to headers, td and canonical ( #1556 )
...
* split headers prefix
* migration
* fix downloader bug
* test for migration
* fix lint
* uncomment t.Prallel
* fix postprocessing test
2021-03-19 12:54:47 +00:00
Alex Sharov
5ef9a3feb9
Trie: use APPEND when re-generating trie. Also including db-migration for PR#1535 ( #1549 )
2021-03-12 16:26:39 +07:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record ( #1500 )
...
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
* SE
2021-02-21 18:41:59 +00:00
Alex Sharov
98b36126b2
History bitmap 64 ( #1374 )
...
* squash
* squash
* squash
* squash
* squash
* squash
* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Alex Sharov
068463dff4
Store transactions individually ( #1358 )
...
* Store transactions individually
* Store transactions individually
* save progress
* checkIndex
* merge
2020-11-22 21:25:26 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort ( #1342 )
...
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* squash
* squash
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* history_early_stop
* history_early_stop
* vmConfig with ReadOnly false
* auto_increment
* auto_increment
* rebase master
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values ( #1297 )
...
* aloow_delete_dupsort_values
* check canonical receipt
* exclusive lock for mdbx
* readme_load_dump_tables
* readme_load_dump_tables
* readme_load_dump_tables
* Remove IdealBatchSize change
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Alex Sharov
a139f31dcd
Exclusive lock manual ( #1302 )
...
* exclusive lock for chaindata folder
* exclusive lock for chaindata folder
* exclusive lock for chaindata folder
* mdb tools
* resolve abs path
* rpcdaemon to check exclusive lock
* rpcdaemon to check exclusive lock
* exclusive lock for mdbx
* exclusive lock for mdbx
2020-10-28 09:52:15 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx ( #1271 )
...
* squash
* add --database flag to integration
* clean
* split to 2 buckets
* split to 2 buckets
* split to 2 buckets
* split to 2 buckets
* split to 2 buckets
* save progress
* save progress
* improve test
* improve test
* save progress
* change app logic
* change app logic
* return err from rawdb package
* don't clean automatically
* don't clean automatically
* clean
* clean
* clean
* don't rely on `make clean`
* improve cbor code
* clean
* clean
* clean
* fix tests
* rebase master
* stop on error: headers stage
* make TxDb walk and multiwalk safe
* Fix panics
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
6584b43b12
Allow start read transactions from ethdb.Database interface ( #1287 )
...
* allow_read_transactions
* fix lint
* fix lint
* more cases
2020-10-24 07:55:43 +01:00
Alex Sharov
76f1b05cb2
Move tmpdir definition to app-start, move migrations folder inside tmpdir ( #1282 )
...
* extract tmpdir to app-level-code
* extract tmpdir to app-level-code
* save progresss
2020-10-23 12:18:45 +01:00
Alex Sharov
1e237069e9
migrations: make tmp dir name predictable and unique ( #1276 )
2020-10-22 10:30:04 +02:00
Alex Sharov
44bbe97a1f
Migration idempotency: handle corner-cases of CriticalCollector ( #1257 )
...
* save progress of db migration
* save progress of db migration
* delete
* delete
* delete
* delete
* delete
* print full key if it has 8 leading zeroes
* print full key if it has 8 leading zeroes
* print full key if it has 8 leading zeroes
2020-10-19 20:20:18 +01:00
Alex Sharov
d7bcb29802
do commit by don't save partial progress ( #1256 )
2020-10-19 07:43:30 +01:00
Alex Sharov
809d79b15f
Switch to cbor ( #1172 )
...
* switch receipts to cbor
* switch receipts to cbor
* rpcdaemon to cache chainconfig
* rpcdaemon to cache chainconfig
* rpcdaemon to cache chainconfig
* rpcdaemon to cache chainconfig
2020-10-02 13:51:20 +01:00
Alex Sharov
dae72a8589
(related to lmdb assert) Avoid deleteCurrent on cursor which used for iterations ( #1151 )
...
* save progress
* save progress
* avoid delete current
* undupsort_ih
2020-09-29 10:28:49 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs ( #1124 )
...
* save progress
* try now
* don't create bloom inside rlpDecode
* don't create bloom inside ApplyTransaction
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* rename method
* print timings
* print timings
* print timings
* sort before flush
* fix err lint
* clean
* move tests to transactions
* compressed version
* up bound
* up bound
* more tests
* more tests
* more tests
* more tests
* better removal
* clean
* better performance of get/put methods
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* optimize rpcdaemon
* fix test
* fix rpcdaemon
* fix test
* simplify
* simplify
* fix nil pointer
* clean
* revert some changes
* add some logs
* clean
* try without optimize
* clean
* clean
* clean
* clean
* try
* move log_index to own stage
* move log_index to own stage
* integration add log_index stage
* integration add log_index stage
* clean
* clean
* print timing
* remove duplicates at unwind
* extract truncateBitmaps func
* try detect
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* add blackList of topics
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* sharding 1
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 3
* sharded 3
* sharded 3
* speedup things by putCurrent and putReserve
* clean
* optimize trim
* clean
* remove blacklist
* add more info to err
* ?
* clean
* clean
* clean
* clean
* clean
* working version
* switch to cgo version of roaring bitmaps
* clean
* clean
* clean
* clean
* more docs
* clean
* clean
* fix logs bloom field
* Fix debug_getModifiedAccountsByNumber
* Try to fix crash
* fix problem with "absent block"
* fix problem with "absent block"
* remove optimize method call
* remove roaring iterator
* fix problem with rebuild indicess
* remove debug prints
* tests for eth_getLogs involving topics
* add tests for new stage, speparate topics into 2 buckets
* version up
* remove debug logs
* remove debug logs
* remove bloom filter implementation
* Optimisation
* Optimisatin not required, make rpctest lenient to geth errors
* Lenient to geth failures
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Alex Sharov
62fe81e4be
IH stage speedup and lmdb custom comparators support ( #1080 )
...
* etl.Loader - allow use of custom comparator
* log timing
* try now
* try now
* more performance
* etl.Loader - allow use of custom comparator
* working version
* simplify IH cursor
* clean
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* clean
* add only unwind support
* squash
* squash
* clean
* fix test
* clean
* clean
* clean
2020-09-10 13:35:58 +01:00
Alex Sharov
da31494cc3
transactional migrations ( #1079 )
2020-09-08 20:39:43 +01:00
Igor Mandrigin
2a6478d799
turbo-api: create a second executable file with a custom stage example ( #1055 )
2020-09-05 18:07:27 +02:00
Alex Sharov
bf596c26f3
[merge after release] dupsort of plain state ( #913 )
...
* dupsort of plain state
* rebase master
2020-08-15 08:11:40 +01:00
Alex Sharov
760fe0f0b6
DupSort of hash state ( #896 )
2020-08-12 09:57:55 +07:00
Alex Sharov
088bb58023
Migrations validation: duplicate names and commit call ( #902 )
2020-08-11 18:23:41 +07:00
Giulio rebuffo
057894994d
add blockhashes stage ( #862 )
...
* add blockhashes stage
* unwindOrder specified
* fixed typo
* added common.copybytes
* fixed little typo
* better progress
* better progress
* refactoring with etl
* added blockhashes to disabled stages in cmd/integration
* added migraations
* fixed go.mod
* better migrations
* rename unwind
* simplified migrations
* added onloadcommit
2020-08-10 18:46:06 +01:00
Alex Sharov
5c75003456
fix migration idempotency ( #898 )
2020-08-10 16:16:14 +07:00
Alex Sharov
115cc7d736
use transactional putter ( #893 )
2020-08-10 14:28:38 +07:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key ( #868 )
2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key ( #858 )" ( #867 )
...
This reverts commit 7290bf519e
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key ( #858 )
2020-08-04 16:25:28 +07:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices ( #569 )
...
* save state
* add current index feature
* fix test
* remove logs
* Only execute 1000 blocks
* Reset history index
* Correct action
* Increase batch size
* Increase chunk size, print memory stats
* Fix linter
* Remove unused from
* Split into 2 staged
* Use storage history gen
* remove log
* Not to run tx_cacher in staged mode
* Not to recover during stage 2
* Not to recover during stage 2
* Remove counter
Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
b00ris
cc137af1ce
[WIP] New storage changeset encoding ( #493 )
...
* new storage changeset encoding
* fix lint
2020-04-28 08:41:55 +01:00
