Commit Graph

122 Commits

Author SHA1 Message Date
ledgerwatch
21295e475c
Update Makefile (#1583) 2021-03-22 08:31:42 +00:00
Artem Vorotnikov
c9bccbb612
Remove autogen from typedcursor (#1576) 2021-03-21 15:10:14 +07:00
Alex Sharov
ce868009d6
devel up (#1574) 2021-03-21 10:02:15 +07:00
Alex Sharov
0953c11c04
grpc version up (#1568) 2021-03-20 16:35:13 +07:00
ledgerwatch
81ea5bab78
More updates to downloader, new p2psentry protocol (#1559)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* More additions

* More additions

* Fix locking

* Intermediate

* Fix separation of phases

* Intermediate

* Fix test

* More transformations

* New simplified way of downloading headers

* Fix hard-coded header sync

* Fixed syncing near the tip of the chain

* Add architecture diagram source and picture (#10)

* More fixes

* rename tip to link

* Use preverified hashes instead of preverified headers

* Fix preverified hashes generation

* more parametrisation

* Continue parametrisation

* Fix grpc data limit, interruption of headers stage

* Add ropsten preverified hashes

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Disable penalties

* Add goerli settings, bootstrap nodes

* Try to fix goerly sync

* Remove interfaces

* Add proper golang packages, max_block into p2p sentry Status

* Prepare for proto overhaul

* Squashed 'interfaces/' content from commit ce36053c2

git-subtree-dir: interfaces
git-subtree-split: ce36053c24db2f56e48ac752808de60afa1dfb4b

* Change EtherReply to address

* Adaptations to new types

* Switch to new types

* Fixes

* Fix formatting

* Fix lint

* Lint fixes, reverse order in types

* Fix lint

* Fix lint

* Fix lint

* Fix test

* Not supporting eth/66 yet

* Fix shutdown

* Fix lint

* Fix lint

* Fix lint

* return stopped check

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
2021-03-19 21:24:49 +00:00
Igor Mandrigin
0a55895b33 Merge branch 'master' into geth-1-10-1 2021-03-17 16:50:20 +01:00
Chase Wright
487dcaa6e6
Fix Typo in Makefile (#1555)
stats -> state
2021-03-15 17:21:41 +00:00
Alex Sharov
3ffa029497
Add mdbx_drop cli tool (#1553) 2021-03-15 09:41:02 +07:00
Alexey Sharp
7fd3417f52 Fix more compile errors 2021-03-14 19:07:19 +00:00
Alex Sharov
6d0d84cda6
log git branch and git commit (#1523) 2021-03-01 11:02:06 +07:00
Alex Sharov
f556af6276
Prometheus metrics: more virtual memory visibility, more cross-platform metrics (#1520) 2021-02-28 11:11:28 +07:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500)
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
2021-02-21 18:41:59 +00:00
ledgerwatch
6e56097056
Bring in refactored p2p sentry interface and adjust downloader for it (#1497)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Use new p2psentry interface

* Remove

* Fix lint

* Fix lint

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Add header upload

* Remove

* Add upload of block headers

* Add block bodies upload

* Fix lint

* Bugfix for blockBodies

* Fix for body upload logging

* Print peerId for GetBlockHeaders

* Invoke block hashes stage to enable block body upload

* Fix test

* Fix stuck bodies download

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-15 18:54:45 +00:00
Alex Sharov
ad3b330872
lint up (#1492) 2021-02-12 19:58:12 +07:00
Alex Sharov
b77dfec055
remove debug-web-ui and restapi (#1480) 2021-02-09 19:42:14 +07:00
Alex Sharov
47b4f52db2
mdbx: C11 atomics performance fix and disable MDBX_ENV_CHECKPID (#1472) 2021-02-08 11:35:02 +07:00
Alex Sharov
f0bb3a6dfa
Mdbx new beta version with fixed blockers (#1461) 2021-02-01 20:57:41 +07:00
Alex Sharov
7fd1f19435
txdb lazy cursors creation, grpc version up (#1440)
* txdb lazy, grpc version up

* tidy

* fix lint
2021-01-15 09:38:09 +00:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00
Thomas Jay Rush
546b91f47e
Updating RPC tests in Postman (#1340) 2020-11-09 09:52:18 +01:00
Alex Sharov
1b4f721a61
fix_db_tools_build_dir (#1337) 2020-11-03 14:19:57 +07:00
ledgerwatch
cceb8c2c3a
Sharded state experiment based on call_traces stage run via integration (#1329)
* Initial work on state shards

* Fix compile errors

* Intermediate

* Implement dispatcher

* Implemented sharded reader

* No cache

* Print comms

* No prepopulation

* Print dispatcher activity

* Print more

* Print errors

* Fix dispatcher

* Remove logging

* Dispatcher to enforce initial barrier

* Don't print that much

* Turn off cache pre-seeting for state sharding

* Artificial latency, fixed number of blocks to execute

* Fix
2020-10-31 11:51:56 +00:00
Igor Mandrigin
1b2bf8bf61
fix git commit when using make docker (#1323)
* fix git commit when using `make docker`

* avoid warning about LATEST_COMMIT for linters in Docker
2020-10-30 08:43:24 +00:00
Alex Sharov
399db7ee3b
LMDB v0.9.27, and add all lmdb cli tools (#1303)
* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* clean

* clean

* clean

* clean

* clean

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:46 +00:00
ledgerwatch
ea060815ef
Remove z3 and semantics (#1314)
* Removed z3 submodule

* Remove semantics and z3 from build

* Fixes to CI
2020-10-28 17:52:00 +00:00
Igor Mandrigin
abf4fa5861
Don't require MDBX if you don't specify it (#1313)
* fix compilation

* add mdbx flag to the tg binary

* fix makefile

* fixups

* fix makefile

* fix readme

* fix linters

* fix tests
2020-10-28 12:17:18 +00:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
ledgerwatch
7554428884
POC of header downloader - splitting into 2 processes: sentry + downloader (via gRPC) (#1291)
* Splitting sentry and downloader - the beginning

* A bit more

* More on sentry

* More gRPC

* Sentry and downloader separated

* Update binding for stable version of grpc

* Better bufferSize flag

* Fix lint

* Send pelanties

* Fix lint

* Remove hard-coded tips on connect

* Tidy the logs a bit

* Deal with hardTips on Recovery

* Print hard tips

* Hide empty anchors

* Request headers after receiving a message

* Better waking up

* Print hard-coded block numbers

* Print outgoing requests

* Debug logging

* In the middle protection

* Sentry not to lose peers when core disconnects
2020-10-25 22:05:37 +00:00
Alex Sharov
9e7ed07fc6
Remove restapi and debug-ui from readme and docker-compose (#1263)
* rm_restapi_from_readme

* rm_restapi_from_readme

* rm_restapi_from_readme

* rm_restapi_from_readme
2020-10-20 09:20:24 +01:00
Alex Sharov
79815e4298
include_lmdb_stat_tool (#1206) 2020-10-08 08:11:36 +01:00
Alex Sharov
61fbe34024
Better cbor support (#1163) 2020-10-02 10:56:13 +07:00
Alex Sharov
0e62fa4d59
add return (#1158) 2020-09-30 10:28:36 +01:00
Alex Sharov
0559d56dae
Pin protoc version and store it in ./build/bin (#1154)
* use ./build/bin folder to store protobuf binaries

* use ./build/bin folder to store protobuf binaries
2020-09-29 15:09:50 +01:00
Alex Sharov
5a74fe4884
use ./build/bin folder to store protobuf binaries (#1153) 2020-09-29 10:55:41 +01:00
Igor Mandrigin
eca39e12f2
Fix reproducible builds (#1099) 2020-09-11 18:42:33 +02:00
Thomas Jay Rush
3f18b9eaa3
1074 web3 client version (#1094)
* Removed old TODOs

* Adding support for web3_clientVersion and web3_sha3

* Fixing formatting

* Adding new endpoints to README

* Responding to comments by removing date from compile-time values
2020-09-11 14:12:38 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt (#1085)
* kill bolt

* kill bolt

* fix test

* clean
2020-09-09 11:09:55 +01:00
ledgerwatch
256e4262e1
2nd batch of Headers PoC changes (#1067)
* PoC utility

* Headers cmd

* headers download command

* Fix lint

* Send handshake

* Receive Status message

* Better error messages

* Messages

* Fix

* Fix lint

* Print block number for NewBlockMsg

* Fix lint
2020-09-07 07:03:12 +01:00
Igor Mandrigin
d14e45d4d0
cmd/geth -> cmd/tg and get rid of geth implementation. (#1061) 2020-09-06 13:35:32 +02:00
Evgeny Danilenko
e4f495fa44
Get logs (#1028)
* it compiles

* after recent master

* fix linters warnings

* grpcV7

* go mod tidy

* unmarshall adresses or adress

* fix linters

* after cr

* after cr

* after cr

* after cr

* fix tests

* remove dev version

* it compiles

* mod tidy

* fix bin deps

* use stable version of grpc

* switch back to master constructor

* switch back to master constructor

* add a bit docs

* add a bit docs

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-09-03 08:51:19 +01:00
Alex Sharov
aead37a39f
Reproducible versions of binary dependencies (#1037) 2020-09-03 13:07:33 +07:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
Igor Mandrigin
a119a3a35f
Simplify build system (#955) 2020-08-21 14:38:11 +02:00
Alex Sharov
f41959f030
drop badger support (#869)
* drop badger support

* rebase master
2020-08-05 16:33:45 +01:00
Alex Sharov
67ca3fa37e
fix make prometheus (#850) 2020-08-01 22:05:48 +07:00
Igor Mandrigin
7eecbc5d2b
Fix building in Docker for some enviroments (#834)
* Fix building in Docker for some enviroments

* debug hanging test
2020-07-30 22:37:46 +07:00