Commit Graph

148 Commits

Author SHA1 Message Date
Alex Sharov
781a67a4ee
Auto-Close db in tests by using t.Cleanup (#1956) 2021-05-19 10:47:28 +07:00
Alex Sharov
0c91bfbf3e
RPCDaemon: support Pending block (#1942)
* clean

* save

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* save

* tx pub-sub

* tx pub-sub

* clean

* clean

* save

* save

* save

* save

* save

* Squashed 'interfaces/' content from commit c469f3ae0

git-subtree-dir: interfaces
git-subtree-split: c469f3ae073b60c8821b61fed2910191080ef835

* save

* save

* save

* save

* Squashed 'interfaces/' changes from c469f3ae0..958dfc669

958dfc669 save

git-subtree-dir: interfaces
git-subtree-split: 958dfc669f8daeefe686a13aa852fb95f1537886

* save

* save

* up some deps

* up some deps

* clean

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test
2021-05-17 13:15:19 +01:00
Alex Sharov
3d3298cc17
Fix lint ci (#1945) 2021-05-17 09:42:24 +07:00
ledgerwatch
56dc23691c
Fix DB performance regression (#1902)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-08 12:38:32 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836)
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 09:45:40 +01:00
Evgeny Danilenko
871bb912b1
[WIP] FuzzyVM for TG (#1884)
* restore StateRoot

* empty error should be ommited

* calc state root

* logger json remove code duplication

* fix
2021-05-05 21:55:15 +01:00
Alex Sharov
f3d2af1e33
Rpcdaemon: move tx pool to own grpc service, subscribe to pending txs (#1863) 2021-05-04 08:37:17 +07:00
ledgerwatch
75ca6b8c76
Initial work on integration tests (#1797)
* Initial work on integration tests

* Delete subtree

* Squashed 'interfaces/' content from commit 41a082ba4

git-subtree-dir: interfaces
git-subtree-split: 41a082ba4bde38647325eb0416cb1da1b4ca2b12

* Add consensus interfaces

* More stuff

* comments

* Fix compile

* Squashed 'interfaces/' changes from 41a082ba4..1b13a42a7

1b13a42a7 Add chainspec to consensus interface

git-subtree-dir: interfaces
git-subtree-split: 1b13a42a7803f5464722867a71065c27a7ebe8c3

* Squashed 'interfaces/' changes from 1b13a42a7..93a072c4c

93a072c4c Add missing import

git-subtree-dir: interfaces
git-subtree-split: 93a072c4c099d169322a3a53b95e40203276820b

* New consensus interfaces

* More on clique

* Fix tests

* Squashed 'interfaces/' changes from 93a072c4c..62f4ec4b2

62f4ec4b2 Add test service for consensus engine

git-subtree-dir: interfaces
git-subtree-split: 62f4ec4b263107635ffa3aabd5d634af22e813c6

* Squashed 'interfaces/' changes from 62f4ec4b2..061a63543

061a63543 Fix

git-subtree-dir: interfaces
git-subtree-split: 061a63543babdeb51ab7e3a96dec56b2485d4389

* Configuring clique engine with toml specs - start

* More toml parsing

* Constructed rinkeby genesis

* Simplify VerifyHeaders functions

* Fix lint

* Remove concurrent verification tests

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-29 16:14:10 +01:00
Alex Sharov
e7849498d5
Degbug build support (for delve debugger and for getting C code profiling, traces) (#1843) 2021-04-29 21:29:58 +07:00
Alex Sharov
9d82456f17
remove chain indexer and core.setHead func (#1826) 2021-04-27 14:48:59 +07:00
ledgerwatch
e3f3dd3c9b
Integration tests 1 (#1793)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* Add architecture diagram source and picture (#10)

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Add proper golang packages, max_block into p2p sentry Status (#12)

* Add proper golang packages, max_block into p2p sentry Status

* Change EtherReply to address

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Add Rust infrastructure (#13)

* DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665

* more p2p methods (#15)

* add mining methods (#16)

* First draft of Consensus gRPC interface (#14)

* Update Rust build

* Fix interfaces in architecture diagram (#17)

* Fix KV interface provider

* Fix Consensus interface provider

* drop java attributes (#18)

* tx pool remove unused import (#19)

* ethbackend: add protocol version and client version (#20)

* Add missing ethbackend I/F (#21)

* Add interface versioning mechanism (#23)

Add versioning in KV interface

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>

* spec of tx pool method (#24)

* spec of tx pool method (#25)

* Update version.proto

* Refactor interface versioning

* Refactor interface versioning

* Testing interface

* Remove tree

* Fix

* Build testing protos

* Fix

* Fix

* Update to the newer interfaces

* Add ProtocolVersion and ClientVersion stubs

* Hook up ProtocolVersion and ClientVersion

* Remove service

* Add compatibility checks for RPC daemon

* Fix typos

* Properly update DB schema version

* Fix test

* Add test for KV compatibility|

* Info messages about compability for RPC daemon

* DB schema version to be one key

* Update release intructions

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: canepat <tullio.canepa@gmail.com>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-04-24 16:46:29 +01:00
Alex Sharov
ac7ece4a4d
Revert txpool grpc (#1794) 2021-04-24 10:48:06 +07:00
Alex Sharov
e7a78a2d9f
Mdbx: speedup tests (#1776) 2021-04-21 20:03:08 +07:00
Alex Sharov
93a57f46a6
minimal_golang_version_check (#1752) 2021-04-19 17:54:56 +07:00
Alex Sharov
7aabdeb829
Remote tx pool generate code (#1730) 2021-04-15 17:06:05 +07:00
Alex Sharov
acf1118300
MDBX: enable MADV_RANDOM (#1712) 2021-04-12 21:30:19 +07:00
Alex Sharov
91ce78c037
support max git version on ubuntu 18.04 (#1710) 2021-04-11 11:23:50 +07:00
alex.sharov
3a2c4913e2 remove old binaries 2021-04-11 08:03:52 +07:00
Alex Sharov
1f5c0e0dc2 drop db tool 2021-04-10 15:38:02 +07:00
Alex Sharov
ad342b27ab
Mdb no db size log (#1665) 2021-04-03 10:30:28 +07:00
Alex Sharov
ef77eb3053
tx.RwCursor() to return err (first step of removing lazy cursors) (#1656) 2021-04-02 13:36:49 +07:00
alex.sharov
2ba007b17c increase tests timeout 2021-03-31 10:48:52 +07:00
Alex Sharov
ea7425adea
tests (#1631) 2021-03-31 08:26:01 +07:00
Artem Vorotnikov
742e998dd3
Add Docker step to Actions workflow (#1630) 2021-03-30 19:16:07 +03:00
Artem Vorotnikov
3da261ba9b
Add Actions workflow (#1606) 2021-03-30 02:17:19 +03:00
Artem Vorotnikov
9b8cdc0f22
Fix lints and remove more unused code (#1621) 2021-03-29 10:58:45 +07:00
ledgerwatch
21295e475c
Update Makefile (#1583) 2021-03-22 08:31:42 +00:00
Artem Vorotnikov
c9bccbb612
Remove autogen from typedcursor (#1576) 2021-03-21 15:10:14 +07:00
Alex Sharov
ce868009d6
devel up (#1574) 2021-03-21 10:02:15 +07:00
Alex Sharov
0953c11c04
grpc version up (#1568) 2021-03-20 16:35:13 +07:00
ledgerwatch
81ea5bab78
More updates to downloader, new p2psentry protocol (#1559)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* More additions

* More additions

* Fix locking

* Intermediate

* Fix separation of phases

* Intermediate

* Fix test

* More transformations

* New simplified way of downloading headers

* Fix hard-coded header sync

* Fixed syncing near the tip of the chain

* Add architecture diagram source and picture (#10)

* More fixes

* rename tip to link

* Use preverified hashes instead of preverified headers

* Fix preverified hashes generation

* more parametrisation

* Continue parametrisation

* Fix grpc data limit, interruption of headers stage

* Add ropsten preverified hashes

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Disable penalties

* Add goerli settings, bootstrap nodes

* Try to fix goerly sync

* Remove interfaces

* Add proper golang packages, max_block into p2p sentry Status

* Prepare for proto overhaul

* Squashed 'interfaces/' content from commit ce36053c2

git-subtree-dir: interfaces
git-subtree-split: ce36053c24db2f56e48ac752808de60afa1dfb4b

* Change EtherReply to address

* Adaptations to new types

* Switch to new types

* Fixes

* Fix formatting

* Fix lint

* Lint fixes, reverse order in types

* Fix lint

* Fix lint

* Fix lint

* Fix test

* Not supporting eth/66 yet

* Fix shutdown

* Fix lint

* Fix lint

* Fix lint

* return stopped check

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
2021-03-19 21:24:49 +00:00
Igor Mandrigin
0a55895b33 Merge branch 'master' into geth-1-10-1 2021-03-17 16:50:20 +01:00
Chase Wright
487dcaa6e6
Fix Typo in Makefile (#1555)
stats -> state
2021-03-15 17:21:41 +00:00
Alex Sharov
3ffa029497
Add mdbx_drop cli tool (#1553) 2021-03-15 09:41:02 +07:00
Alexey Sharp
7fd3417f52 Fix more compile errors 2021-03-14 19:07:19 +00:00
Alex Sharov
6d0d84cda6
log git branch and git commit (#1523) 2021-03-01 11:02:06 +07:00
Alex Sharov
f556af6276
Prometheus metrics: more virtual memory visibility, more cross-platform metrics (#1520) 2021-02-28 11:11:28 +07:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500)
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
2021-02-21 18:41:59 +00:00
ledgerwatch
6e56097056
Bring in refactored p2p sentry interface and adjust downloader for it (#1497)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Use new p2psentry interface

* Remove

* Fix lint

* Fix lint

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Add header upload

* Remove

* Add upload of block headers

* Add block bodies upload

* Fix lint

* Bugfix for blockBodies

* Fix for body upload logging

* Print peerId for GetBlockHeaders

* Invoke block hashes stage to enable block body upload

* Fix test

* Fix stuck bodies download

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-15 18:54:45 +00:00
Alex Sharov
ad3b330872
lint up (#1492) 2021-02-12 19:58:12 +07:00
Alex Sharov
b77dfec055
remove debug-web-ui and restapi (#1480) 2021-02-09 19:42:14 +07:00
Alex Sharov
47b4f52db2
mdbx: C11 atomics performance fix and disable MDBX_ENV_CHECKPID (#1472) 2021-02-08 11:35:02 +07:00
Alex Sharov
f0bb3a6dfa
Mdbx new beta version with fixed blockers (#1461) 2021-02-01 20:57:41 +07:00
Alex Sharov
7fd1f19435
txdb lazy cursors creation, grpc version up (#1440)
* txdb lazy, grpc version up

* tidy

* fix lint
2021-01-15 09:38:09 +00:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00
Thomas Jay Rush
546b91f47e
Updating RPC tests in Postman (#1340) 2020-11-09 09:52:18 +01:00