Alex Sharov
52d35930e8
simplify receipts reading ( #2094 )
...
* simplify receipts reading
* test
2021-06-04 13:28:18 +01:00
Alex Sharov
29fa1aa35a
sentry join streams ( #2059 )
2021-06-01 17:41:10 +07:00
Alex Sharov
3016b96e88
1 Sentry per protocol ( #2046 )
2021-05-30 09:53:30 +07:00
Alex Sharov
14c15cba43
Check version of remote services ( #1989 )
...
* save
* save
* Squashed 'interfaces/' content from commit 08c32a09e
git-subtree-dir: interfaces
git-subtree-split: 08c32a09e40b1e6fcb5922e723191c9477545356
* Revert "Squashed 'interfaces/' content from commit 08c32a09e"
This reverts commit 8393d9fd
* save
* seve
* Squashed 'interfaces/' content from commit dd6a42724
git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d
* ensure versions compatibility of all remote services
* Revert "Squashed 'interfaces/' content from commit dd6a42724"
This reverts commit 2a764bf9
* Squashed 'interfaces/' content from commit dd6a42724
git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d
* Revert "Squashed 'interfaces/' content from commit dd6a42724"
This reverts commit 52621846
* Squashed 'interfaces/' content from commit dd6a42724
git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d
* a
* a
* a
* a
* a
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 11:00:13 +01:00
Alex Sharov
0be3044b7e
rename ( #1978 )
...
* rename
* rename "make grpc"
* rename "abi bindings templates"
* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
50934fa5d3
Move chain maiker to kv ( #1972 )
2021-05-20 18:49:33 +07:00
Alex Sharov
0163329a5a
KV and RemoteKV: to reuse stateless cursors during transaction ( #1909 )
2021-05-11 15:23:17 +07:00
Alex Sharov
f3d2af1e33
Rpcdaemon: move tx pool to own grpc service, subscribe to pending txs ( #1863 )
2021-05-04 08:37:17 +07:00
alex.sharov
8a74b65ba2
remove kv_remove prefix and prefetch
2021-05-02 10:53:19 +07:00
Alex Sharov
9cc4a0a97c
rebort db metrics - not guilty ( #1828 )
2021-04-27 15:32:41 +07:00
ledgerwatch
117838e213
Move compatibility check ( #1814 )
...
* Move compatibility check
* Implement Alex's suggestion
* Use root context
* Fix lint
* Fix lint
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-26 13:39:34 +01:00
ledgerwatch
e3f3dd3c9b
Integration tests 1 ( #1793 )
...
* Initial commit
* Add sentry gRPC interface
* p2psentry directory
* Update README.md
* Update README.md
* Update README.md
* Add go package
* Correct syntax
* add external downloader interface (#2 )
* Add txpool (#3 )
* Add private API (#4 )
* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5 )
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Separate upload messages into its own stream (#6 )
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Only send changed accounts to listeners (#7 )
* Txpool interface doc (#9 )
* Add architecture diagram source and picture (#10 )
* Typed hashes (#11 )
* Typed hashes
* Fix PeerId
* 64-bit tx nonce
* Add proper golang packages, max_block into p2p sentry Status (#12 )
* Add proper golang packages, max_block into p2p sentry Status
* Change EtherReply to address
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Add Rust infrastructure (#13 )
* DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665
* more p2p methods (#15 )
* add mining methods (#16 )
* First draft of Consensus gRPC interface (#14 )
* Update Rust build
* Fix interfaces in architecture diagram (#17 )
* Fix KV interface provider
* Fix Consensus interface provider
* drop java attributes (#18 )
* tx pool remove unused import (#19 )
* ethbackend: add protocol version and client version (#20 )
* Add missing ethbackend I/F (#21 )
* Add interface versioning mechanism (#23 )
Add versioning in KV interface
Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
* spec of tx pool method (#24 )
* spec of tx pool method (#25 )
* Update version.proto
* Refactor interface versioning
* Refactor interface versioning
* Testing interface
* Remove tree
* Fix
* Build testing protos
* Fix
* Fix
* Update to the newer interfaces
* Add ProtocolVersion and ClientVersion stubs
* Hook up ProtocolVersion and ClientVersion
* Remove service
* Add compatibility checks for RPC daemon
* Fix typos
* Properly update DB schema version
* Fix test
* Add test for KV compatibility|
* Info messages about compability for RPC daemon
* DB schema version to be one key
* Update release intructions
Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: canepat <tullio.canepa@gmail.com>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 16:46:29 +01:00
Artem Vorotnikov
19924040bc
KV: Unified Has ( #1671 )
2021-04-03 19:04:58 +07:00
Artem Vorotnikov
d66555268d
Harmonize DB APIs ( #1669 )
2021-04-03 09:26:00 +03:00
Alex Sharov
ad342b27ab
Mdb no db size log ( #1665 )
2021-04-03 10:30:28 +07:00
Alex Sharov
6179933872
more ( #1658 )
2021-04-02 16:15:41 +07:00
Alex Sharov
ef77eb3053
tx.RwCursor() to return err (first step of removing lazy cursors) ( #1656 )
2021-04-02 13:36:49 +07:00
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface ( #1627 )
2021-03-30 12:53:54 +03:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions ( #1577 )
2021-03-21 20:15:25 +07:00
Artem Vorotnikov
b5f9cdd7b2
KV: split Sequence into ReadSequence and MakeSequence ( #1569 )
2021-03-20 21:12:54 +07:00
Artem Vorotnikov
01f6542729
KV: Remove reserve, putcurrent, cmp, dcmp ( #1552 )
2021-03-20 16:35:02 +07:00
Alex Sharov
ba37706681
move remote eth backend from ethdb to core package ( #1567 )
2021-03-20 10:52:00 +07:00
ledgerwatch
81ea5bab78
More updates to downloader, new p2psentry protocol ( #1559 )
...
* Initial commit
* Add sentry gRPC interface
* p2psentry directory
* Update README.md
* Update README.md
* Update README.md
* Add go package
* Correct syntax
* add external downloader interface (#2 )
* Add txpool (#3 )
* Add private API (#4 )
* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5 )
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Separate upload messages into its own stream (#6 )
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Only send changed accounts to listeners (#7 )
* Txpool interface doc (#9 )
* More additions
* More additions
* Fix locking
* Intermediate
* Fix separation of phases
* Intermediate
* Fix test
* More transformations
* New simplified way of downloading headers
* Fix hard-coded header sync
* Fixed syncing near the tip of the chain
* Add architecture diagram source and picture (#10 )
* More fixes
* rename tip to link
* Use preverified hashes instead of preverified headers
* Fix preverified hashes generation
* more parametrisation
* Continue parametrisation
* Fix grpc data limit, interruption of headers stage
* Add ropsten preverified hashes
* Typed hashes (#11 )
* Typed hashes
* Fix PeerId
* 64-bit tx nonce
* Disable penalties
* Add goerli settings, bootstrap nodes
* Try to fix goerly sync
* Remove interfaces
* Add proper golang packages, max_block into p2p sentry Status
* Prepare for proto overhaul
* Squashed 'interfaces/' content from commit ce36053c2
git-subtree-dir: interfaces
git-subtree-split: ce36053c24db2f56e48ac752808de60afa1dfb4b
* Change EtherReply to address
* Adaptations to new types
* Switch to new types
* Fixes
* Fix formatting
* Fix lint
* Lint fixes, reverse order in types
* Fix lint
* Fix lint
* Fix lint
* Fix test
* Not supporting eth/66 yet
* Fix shutdown
* Fix lint
* Fix lint
* Fix lint
* return stopped check
Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
2021-03-19 21:24:49 +00:00
Artem Vorotnikov
0b2e4df0ff
Align Cursor API with LMDB/MDBX ( #1558 )
2021-03-19 14:45:01 +07:00
Alex Sharov
4fe019a099
add periodical db statistic metrics ( #1545 )
2021-03-09 13:34:13 +07:00
ledgerwatch
6a1f000cf7
Bring kv interfaces ( #1493 )
...
* Remove interfaces
* Squashed 'interfaces/' content from commit 0941b0992
git-subtree-dir: interfaces
git-subtree-split: 0941b09926db64934ba7dd161fb9ca2a20ba25b8
* Change generation accordingly
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-13 07:41:47 +00:00
Artem Vorotnikov
5c34713e71
Remove nested transactions ( #1483 )
2021-02-10 17:04:22 +00:00
Artem Vorotnikov
739c0d30a7
Remove DupFixed ( #1482 )
2021-02-10 16:56:44 +07:00
Artem Vorotnikov
f7c5ac42d2
Remove unused fields from remoteCursor ( #1437 )
2021-01-09 12:29:04 +07:00
Andrew Ashikhmin
8a877c227a
Integrate Silkworm execution ( #1344 )
...
* Expose C Handle of lmdbTx
* LoadExecutionFunctionPointer
* silkworm ExecuteBlocks
* linter
* CLI flag for SilkwormExecutionFunc
* linter
* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage
* Commit transaction after Silkworm execution
* Fix batch initialization
* Make batch_size uint64_t instead of size_t
* max_block in silkworm_execute_blocks
* More accurate logProgress
* Use CHandle exposed by lmdb-go
* Enable Silkworm on Linux only
* Update lmdb-go
* Error when attempting to use Silkworm not on Linux
* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go
* Integrate Silkworm into cmd/integration
* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution
* Refactor SpawnExecuteBlocksStage
* linters
* Small clean-ups
* Move ChangeSetHook inside executeBlockWithGo
* No need to tamper with batch size in executeBlocksWithSilkworm
* fix
* Fix
* Fixup
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-28 15:08:02 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads ( #1359 )
...
* fix `make grpc` on new checkouts
* update proto files
* add some stub
* prototype with fake events
* notifying about events
* pass events
* events are being sent
* transfer headers to filters
* create the “filters” struct
* implement new heads
* PoC of New Heads subscription
* fix keep alive
* fixups for the client
* add “type” to the event
* support header event type on client
* better stage refactor
* fixup for the eth backend
* fixups
* fix tests
* fix tests
* fix linters
* address comments
* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort ( #1342 )
...
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* squash
* squash
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* history_early_stop
* history_early_stop
* vmConfig with ReadOnly false
* auto_increment
* auto_increment
* rebase master
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Alex Sharov
1dd283efe2
Sequence for each table ( #1354 )
...
* history_early_stop
* auto_increment
* auto_increment
* tx_table
2020-11-14 13:48:29 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values ( #1297 )
...
* aloow_delete_dupsort_values
* check canonical receipt
* exclusive lock for mdbx
* readme_load_dump_tables
* readme_load_dump_tables
* readme_load_dump_tables
* Remove IdealBatchSize change
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Alex Sharov
531d36efcb
mdbx support ( #1235 )
2020-10-28 10:18:10 +07:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx ( #1271 )
...
* squash
* add --database flag to integration
* clean
* split to 2 buckets
* split to 2 buckets
* split to 2 buckets
* split to 2 buckets
* split to 2 buckets
* save progress
* save progress
* improve test
* improve test
* save progress
* change app logic
* change app logic
* return err from rawdb package
* don't clean automatically
* don't clean automatically
* clean
* clean
* clean
* don't rely on `make clean`
* improve cbor code
* clean
* clean
* clean
* fix tests
* rebase master
* stop on error: headers stage
* make TxDb walk and multiwalk safe
* Fix panics
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader ( #1288 )
...
* protect_from_wrong_class_use
* more cases
2020-10-24 09:09:20 +01:00
Alex Sharov
5e53c79f50
stable version of grpc codegen, drop kv_remote_v1 support ( #1285 )
2020-10-24 07:54:03 +01:00
ledgerwatch
b747ab5324
[WIP] CallTraces index ( #1157 )
...
* Initial commit for CallTraces index
* Fix compilation
* fix lint, add comment
* Fix integration
* Add Close function to ethdb.Cursor, fix some compile errors
* Try to stop cursor leak in Get
* Fix compile errors in RPC daemon
* Fix compile errors
* fixing another way
* Some fixes
* More fixes
* More fixes
* More fixes
* Fixes to core/state
* Fix lint
* Fix lint
* Fixes
* Stage caching for call trace stage
* Add mem stats
* Try to stop the leak
* Turn off debug
* Chunks for 10k blocks
* Print
* Revert "Print"
This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.
* Revert "Chunks for 10k blocks"
This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.
* Trying to fix the leak
* Don't compute receipts in re-tracing
* Not compose block
* Print speed, fix receipts, bigger caches
* Fix lint
* Utilise changeset info
* Counters
* Use NoReceipts and ReadOnly
* ReadOnly is incompatible with caching
* Skip test leaking transactions
* Fix block test
* Change disable message for call-traces stage
* Use block option for call traces integration
* Fix retracing due to incarnation
2020-10-12 09:39:04 +01:00
Alex Sharov
9cf10a8f85
Rpcdaemon: Gracefull shutdown, better grpc stream close ( #1196 )
...
* save progress
* GracefulShutdown grpc server, send to server close signal before canceling context
* GracefulShutdown grpc server, send to server close signal before canceling context
* GracefulShutdown json server
* GracefulShutdown json server
* fix lint
* fix lint
* can
* clean
* test
* test
* test
* increase keepalive timeout, increase vegeta timeout, mark server-side errors
* increase keepalive timeout, increase vegeta timeout, mark server-side errors
* hack hugeFreelist
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* fix_race_condition_on_zstd_build
2020-10-10 07:06:54 +01:00
ledgerwatch
4b5a333583
Revert the context unbundling change ( #1203 )
2020-10-07 22:53:04 +01:00
ledgerwatch
70ebf59642
Context cleanup and make context independent ( #1193 )
2020-10-06 22:07:57 +01:00
Alexey Akhunov
eb5e01ff5d
Do not cancel context when closing remote cursor
2020-10-06 17:11:30 +01:00
Alex Sharov
48cf5fb43e
History-style sharding for bitmap indices ( #1159 )
2020-10-02 10:54:11 +07:00
Alex Sharov
0559d56dae
Pin protoc version and store it in ./build/bin ( #1154 )
...
* use ./build/bin folder to store protobuf binaries
* use ./build/bin folder to store protobuf binaries
2020-09-29 15:09:50 +01:00
ledgerwatch
7bb9cb4274
Remove BloomStatus from the API ( #1155 )
...
* Remove BloomStatus from the API
* Trying to find missing space
* Fix lint
2020-09-29 15:09:29 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs ( #1124 )
...
* save progress
* try now
* don't create bloom inside rlpDecode
* don't create bloom inside ApplyTransaction
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* rename method
* print timings
* print timings
* print timings
* sort before flush
* fix err lint
* clean
* move tests to transactions
* compressed version
* up bound
* up bound
* more tests
* more tests
* more tests
* more tests
* better removal
* clean
* better performance of get/put methods
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* optimize rpcdaemon
* fix test
* fix rpcdaemon
* fix test
* simplify
* simplify
* fix nil pointer
* clean
* revert some changes
* add some logs
* clean
* try without optimize
* clean
* clean
* clean
* clean
* try
* move log_index to own stage
* move log_index to own stage
* integration add log_index stage
* integration add log_index stage
* clean
* clean
* print timing
* remove duplicates at unwind
* extract truncateBitmaps func
* try detect
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* add blackList of topics
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* sharding 1
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 3
* sharded 3
* sharded 3
* speedup things by putCurrent and putReserve
* clean
* optimize trim
* clean
* remove blacklist
* add more info to err
* ?
* clean
* clean
* clean
* clean
* clean
* working version
* switch to cgo version of roaring bitmaps
* clean
* clean
* clean
* clean
* more docs
* clean
* clean
* fix logs bloom field
* Fix debug_getModifiedAccountsByNumber
* Try to fix crash
* fix problem with "absent block"
* fix problem with "absent block"
* remove optimize method call
* remove roaring iterator
* fix problem with rebuild indicess
* remove debug prints
* tests for eth_getLogs involving topics
* add tests for new stage, speparate topics into 2 buckets
* version up
* remove debug logs
* remove debug logs
* remove bloom filter implementation
* Optimisation
* Optimisatin not required, make rpctest lenient to geth errors
* Lenient to geth failures
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) ( #1113 )
...
* added tls auth
* added client side
* put --tls
* fixed flag
* Add key/cert generation instructions, turn off Common Name verification
* Add CLI arguments and Warning
* Lint
* Update the doc about Internal IP
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon ( #1089 )
...
* added eth_getStorageAt
* used uint32
* now its 256
* incarnation
* added TLS handshake
* code minified
* lint
* minified client
* Update flags.go
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Alex Sharov
62fe81e4be
IH stage speedup and lmdb custom comparators support ( #1080 )
...
* etl.Loader - allow use of custom comparator
* log timing
* try now
* try now
* more performance
* etl.Loader - allow use of custom comparator
* working version
* simplify IH cursor
* clean
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* clean
* add only unwind support
* squash
* squash
* clean
* fix test
* clean
* clean
* clean
2020-09-10 13:35:58 +01:00
