Commit Graph

557 Commits

Author SHA1 Message Date
b00ris
777882ff52
Fix corner case for KV_Snapshot.Next (#1957)
* fix next

* comment test
2021-05-18 19:32:04 +01:00
Alex Sharov
cceaf75b16
use t.TempDir() (#1955) 2021-05-18 19:13:16 +07:00
Alex Sharov
0c91bfbf3e
RPCDaemon: support Pending block (#1942)
* clean

* save

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* save

* tx pub-sub

* tx pub-sub

* clean

* clean

* save

* save

* save

* save

* save

* Squashed 'interfaces/' content from commit c469f3ae0

git-subtree-dir: interfaces
git-subtree-split: c469f3ae073b60c8821b61fed2910191080ef835

* save

* save

* save

* save

* Squashed 'interfaces/' changes from c469f3ae0..958dfc669

958dfc669 save

git-subtree-dir: interfaces
git-subtree-split: 958dfc669f8daeefe686a13aa852fb95f1537886

* save

* save

* up some deps

* up some deps

* clean

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test
2021-05-17 13:15:19 +01:00
Alex Sharov
617baac56b
move db tests to kv interface (#1946) 2021-05-17 16:54:29 +07:00
Alex Sharov
9bef3fb656
move some tests to mdbx (#1947) 2021-05-17 16:14:08 +07:00
Alex Sharov
2ff6ce7d92
move staged sync to tx (#1897) 2021-05-14 18:00:20 +07:00
b00ris
48733c31cd
test fix (#1930) 2021-05-13 21:04:47 +01:00
Alex Sharov
f3a9eb340c
mdbx_setup_debug - handle response correct way (#1931) 2021-05-13 21:01:23 +07:00
b00ris
7948eafcc2
Property tests for kv snapshot (#1928)
* model tests for snapshot kv

* cursor test

* first failing test

* fix

* fix lint

* remove fmt
2021-05-13 09:40:32 +01:00
Alex Sharov
3e1bdbb465
Mdbx: fix for MDBX_PROBLEM (#1922) 2021-05-13 14:40:44 +07:00
Alex Sharov
0163329a5a
KV and RemoteKV: to reuse stateless cursors during transaction (#1909) 2021-05-11 15:23:17 +07:00
Artem Vorotnikov
80bbbf62c2
Misc fixes (#1908)
* Fix debug levels

* try this

* remove adapter

* fix

* fix bodies stage
2021-05-10 06:59:29 +01:00
Alex Sharov
3f36e362c2
mdbx switch to release (#1906) 2021-05-09 15:35:47 +07:00
ledgerwatch
8907e95d4a
Trace filter 3 (#1904)
* Prevent the leak

* Default storage mode, fix leak again

* TopSeenHeight to be updated from any header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-08 23:07:16 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836)
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 09:45:40 +01:00
Artem Vorotnikov
09788dbc02
Pruning flag, incremental staged sync (#1899) 2021-05-08 00:07:49 +03:00
ledgerwatch
59e5502744
Initial work on RPC streaming (#1880)
* Initial work on RPC streaming

* Create the stream and pass it on

* trace_filter expressed as streamable (not finished)

* Reenable call trace index

* Add new line

* Remove storage mode override

* Tool for overriding storage mode

* Diagnostics

* Make trace_filter more streamy

* Streaming debug_traceTransaction and debug_traceCall

* Fix test

* Log error on stream flush

* Enable streaming for http

* Flush the stream too

* Make trace_filter flush too

* Stop streaming if client is not interested

* Try to quiet annoying test|

* Revert "Try to quiet annoying test|"

This reverts commit 42849257bfa52e90140aa535af34b957cd97a222.

* Debug log for test

* Proceed with handshake regardless of whether peer notification worked

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-06 18:37:38 +01:00
Alex Sharov
4323d56376
Mdbx tx limit metric (#1891) 2021-05-06 17:02:50 +07:00
Alex Sharov
49225f7319
Mdbx page ops metrics (#1885) 2021-05-06 10:51:49 +07:00
Alex Sharov
a627bc2d14
use same keepalive settings in sentry and rpcdaemon (#1883)
* use same keepalive settings in sentry and rpcdaemon

* use same keepalive settings in sentry and rpcdaemon
2021-05-05 11:32:04 +01:00
Alex Sharov
e87d4932f7
Changeset writer move to tx (#1871) 2021-05-05 14:31:40 +07:00
Alex Sharov
61c20798db
fix concurrent map write (#1882) 2021-05-05 10:16:57 +07:00
ledgerwatch
7c60ec3ec0
Fix for eth_getTransaction... returning tx from the pool (#1881)
* Fix for eth_getTransaction... returning tx from the pool

* Fix

* Fix more issues

* Fix for EOF

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-04 22:58:13 +01:00
ledgerwatch
d604bfbb4f
Fix transaction stream test (#1879)
* Fix transaction stream test

* Fix

* Update events.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-04 17:12:59 +01:00
Alex Sharov
c7eedf0091
Add kv to stage settings (because it doesn't change in runtime, tx will be passed as Stage func argument) (#1872)
* add kv to config

* move forward tx use

* move forward tx use

* gc stat

* gc stat

* changeset writer - move to tx

* clean

* clean

* clean

* clean

* clean
2021-05-04 13:36:03 +01:00
ledgerwatch
99578ec58d
Fix RPC daemon (#1873)
* Fix RPC daemon

* Skip the test to fix RPC daemon

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-04 12:31:07 +01:00
Alex Sharov
66b683a6b3
RemoteKv version up (#1874)
* clean

* clean

* clean
2021-05-04 09:56:54 +01:00
ledgerwatch
793c13e20e
trace_filter and CallTrace Index (derivative of #1626) (#1867)
* Trace Index

* trace_filter

* hrtc -> hrt

* Fix CallTraces

* wip

* wip

* Fix for incorrect gas

* fix

* Fix Tracer

* Add block and uncle coinbases to trace index

* rewind coinbase

* Commit every 100k blocks after block 3m

* Simplify

* Safe unwinds

* Safe unwind 2

* Cleanup

* Simplification

* Remove intermediate JSON transform

* Reinstate optional CallIndex stage

* Create rpctest bench for trace_filter

* Fix trace_filter generator

* Restore unwind order

* Ignore the storage mode

* Ignore the storage mode

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-04 07:23:54 +01:00
Alex Sharov
4b7873708a
clean (#1870) 2021-05-04 13:21:51 +07:00
Alex Sharov
fc196d4b9b
pending txs methods (#1869) 2021-05-04 06:51:28 +01:00
Alex Sharov
f3d2af1e33
Rpcdaemon: move tx pool to own grpc service, subscribe to pending txs (#1863) 2021-05-04 08:37:17 +07:00
Alex Sharov
f230ba1da4
Less use rawdb deprecated methods (#1861)
* less use deprecated methods

* less use deprecated methods

* auto Dp limit

* auto Dp limit

* add test
2021-05-03 21:01:01 +01:00
alex.sharov
8a74b65ba2 remove kv_remove prefix and prefetch 2021-05-02 10:53:19 +07:00
Artem Vorotnikov
2e2e3835aa
Remove MdbxCursor.prefix (#1857) 2021-05-02 10:49:00 +07:00
Alex Sharov
aacc457ea8
rawdb deprecated methods with Database interface (#1854) 2021-05-01 14:42:23 +07:00
Alex Sharov
e7849498d5
Degbug build support (for delve debugger and for getting C code profiling, traces) (#1843) 2021-04-29 21:29:58 +07:00
Alex Sharov
07a063cb8a
txn full fix - cursors leak (#1838) 2021-04-28 18:00:32 +01:00
ledgerwatch
92c20e1f67
[eip-1559] Apply gas limit elasticity (#1834)
* Apply gas limit elasticity

* Fix unmarshalling tx

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-28 15:32:48 +01:00
Alex Sharov
768bdcd850
--database.verbosity flag (#1829)
* --database.verbosity flag

* --database.verbosity flag
2021-04-27 13:31:00 +01:00
Alex Sharov
9cc4a0a97c
rebort db metrics - not guilty (#1828) 2021-04-27 15:32:41 +07:00
ledgerwatch
e8161541e2
Use RLP encoding instead of json for subscriptions (#1825)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-27 09:23:04 +01:00
ledgerwatch
b341b5aed5
Fix for RPC daemon crashing TG when pending txs are transmitted (#1819)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-26 17:56:02 +01:00
ledgerwatch
117838e213
Move compatibility check (#1814)
* Move compatibility check

* Implement Alex's suggestion

* Use root context

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-26 13:39:34 +01:00
Alex Sharov
544c597151
Pending txs part 2 (#1816) 2021-04-26 18:15:05 +07:00
Alex Sharov
4e80d6eeed
Pending txs subscription (#1810) 2021-04-26 16:53:38 +07:00
Alex Sharov
566233fee9
switch to master failed (#1812) 2021-04-26 15:40:49 +07:00
Alex Sharov
3462059ae8
mdbx rpcdaemon support, mdbx - switch to master branch (#1804) 2021-04-26 12:00:20 +07:00
Alex Sharov
d959986466
Connect TxFetcher to Sentry, Add --download.v2 option to TG (#1796) 2021-04-25 11:20:50 +07:00
ledgerwatch
e3f3dd3c9b
Integration tests 1 (#1793)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* Add architecture diagram source and picture (#10)

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Add proper golang packages, max_block into p2p sentry Status (#12)

* Add proper golang packages, max_block into p2p sentry Status

* Change EtherReply to address

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Add Rust infrastructure (#13)

* DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665

* more p2p methods (#15)

* add mining methods (#16)

* First draft of Consensus gRPC interface (#14)

* Update Rust build

* Fix interfaces in architecture diagram (#17)

* Fix KV interface provider

* Fix Consensus interface provider

* drop java attributes (#18)

* tx pool remove unused import (#19)

* ethbackend: add protocol version and client version (#20)

* Add missing ethbackend I/F (#21)

* Add interface versioning mechanism (#23)

Add versioning in KV interface

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>

* spec of tx pool method (#24)

* spec of tx pool method (#25)

* Update version.proto

* Refactor interface versioning

* Refactor interface versioning

* Testing interface

* Remove tree

* Fix

* Build testing protos

* Fix

* Fix

* Update to the newer interfaces

* Add ProtocolVersion and ClientVersion stubs

* Hook up ProtocolVersion and ClientVersion

* Remove service

* Add compatibility checks for RPC daemon

* Fix typos

* Properly update DB schema version

* Fix test

* Add test for KV compatibility|

* Info messages about compability for RPC daemon

* DB schema version to be one key

* Update release intructions

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: canepat <tullio.canepa@gmail.com>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 16:46:29 +01:00
ledgerwatch
12cde41772
Aleut support (Eip1559) (#1704)
* Where I am at

* Refactoring of transaction types

* More refactoring

* Use Homested signer in rpc daemon

* Unified signer

* Continue unified signer

* A bit more

* Fixes and down the rabbit hole...

* More tx pool fixes

* More refactoring fixes

* More fixes'

* more fixes

* More fixes

* More compile fixes

* More RLP hand-writing

* Finish RLP encoding/decoding of transactions

* Fixes to header encoding, start on protocol packets

* Transaction decoding

* Use DecodeTransaction function

* Decoding BlockBodyPacket

* Encode and decode for pool txs

* Start fixing tests

* Introduce SigningHash

* Fixes to SignHash

* RLP encoding fixes

* Fixes for encoding/decoding

* More test fixes

* Fix more tests

* More test fixes

* More test fixes

* Fix core tests

* More fixes for signer

* Fix for tx

* Fixes to string encoding/size

* Fix eip2930 test

* Fix rest of ./tests

* More fixes

* Fix compilation

* More test fixes

* More test fixes

* Test fixes

* More fixes

* Reuse EncodingSize in EncodeRLP for accessList

* Rearrange things in dynamic fee tx

* Add MarshalBinary

* More fixes

* Make V,R,S non-pointers

* More NPE fixes

* More fixes

* Receipt fixes

* Fix core/types

* Fix ./eth

* More compile fixes for tests

* More test fixes

* More test fixes

* Try to see lint errors better

* Try to see lint errors better

* Fix lint

* Debugging eip1559 test

* Fix TestEIP1559Transition test

* Fix NewBlockPacket encoding/decoding

* Fix calculation of TxHash

* Fix perf problem with senders

* Update aleut config values

* Try adding static peers

* Add staticpeers to defaul flags

* Change aleut networkID

* Fix test

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-22 18:11:37 +01:00