Commit Graph

2487 Commits

Author SHA1 Message Date
Alex Sharov
4fe019a099
add periodical db statistic metrics (#1545) 2021-03-09 13:34:13 +07:00
alex.sharov
9d39708253 use non-app metric to parse in grafana variables 2021-03-08 20:36:56 +07:00
ledgerwatch
b743b88218
Revert "Extract validating interface (#1120)" (#1543)
* Revert "Extract validating interface (#1120)"

This reverts commit e4e36c152e.

* Fix go mod tidy

* Fix lint

* Fix lint

* fix lint

* Test fix

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-05 20:34:23 +00:00
Greg Colvin
8ec8cfcd01 Merge branch 'master' of https://github.com/ledgerwatch/turbo-geth into extractCode 2021-03-04 15:03:18 +00:00
Greg Colvin
c0d0c1885b gofmt 2021-03-04 13:46:57 +00:00
Greg Colvin
7899048eac add action to extract all contract code as hex string 2021-03-04 13:01:46 +00:00
Alex Sharov
eb7d2ad16a
Add validation of senders amount (#1539) 2021-03-04 16:34:49 +07:00
Alex Sharov
827e47e544
grafana: unstack charts (#1540) 2021-03-04 12:27:16 +07:00
alex.sharov
ceb332afdd More documentation for integration tool 2021-03-04 09:42:05 +07:00
Alex Sharov
17f99a5d3f
Grafana: to use iops units (#1537) 2021-03-03 16:18:01 +07:00
Alex Sharov
141b039b48
fix history check (#1534) 2021-03-03 10:16:37 +07:00
ledgerwatch
c00f85ab6c
Simplification of eth_estimageGas (#1531)
* Simplification of eth_estimageGas

* Set GasCap by default

* Add a test

* Fix crash

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-02 22:47:44 +00:00
alex.sharov
b7996c66d9 add more db conversion tools 2021-03-02 09:57:15 +07:00
alex.sharov
66bc517834 add more db conversion tools 2021-03-02 09:56:43 +07:00
alex.sharov
61f8a1ef0c revive 2021-03-02 09:52:05 +07:00
Alex Sharov
904a992d3c
remove old grafana boards (#1527) 2021-03-01 12:34:33 +07:00
Alex Sharov
827461039e
restore cache exec (#1525) 2021-03-01 12:03:49 +07:00
Alex Sharov
8538ec85b3
don't open db in exclusive mode if no migrations (#1524) 2021-03-01 11:02:22 +07:00
Alex Sharov
6d0d84cda6
log git branch and git commit (#1523) 2021-03-01 11:02:06 +07:00
ledgerwatch
da74bd8f38
Fixes to unwind logic for the new staged sync (#1521)
* Make DB visible

* Further fixes to unwind logic

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-28 08:47:31 +00:00
Alex Sharov
f556af6276
Prometheus metrics: more virtual memory visibility, more cross-platform metrics (#1520) 2021-02-28 11:11:28 +07:00
Evgeny Danilenko
e4e36c152e
Extract validating interface (#1120)
* interface

* generalize interface

* linters

* fix deadlock

* fix linters

* close goroutine

* fix

* debug

* id+ttl

* refactor downloader tests

* tests

* lru

* handle genesis, extract fake consensuses

* fix fake consensus

* test uncles, verify

* after a new master

* fmt

* fix close

* debug

* debug

* fix chain length

* remove test field

* use single account

* fix data race on closing channel

* remove postponed blocks queue

* miner test

* VerifyHeaderRequests

* fmt

* fmt

* fix data race

* handle validating errors

* simplify matchParents

* remove copy-paste

* move sort to constructor

* clean up

* debug for 10 parents

* debug

* debug

* batch responses

* batch requests

* works for many ancestors

* remove debug

* always Close an engine

* linters

* ancestors deduplication

* fix test

* reduce interface

* api

* clique

* green clique sync

* stable

* perpermance per second

* full sync

* linters

* gitignore

* deps

* fix panic after master merge

* init consensus

* clique tests

* fix tests

* fix tests

* clean up

* reuse snap

* store vefified snapshots

* optimize snapshots

* safe close

* cleanup loop

* after downloader

* downloader and consensus tests

* update tests

* hack.go

* clique flags

* fix cliuqe config

* review

* gitignore

* remove additional bucket

* blk/sec instead of blk/microsecond

* rename

* deps

* optimize

* debug

* test

* tests without extracted validation process

* same base performance as on master

* benchmark

* simplify ethash verification

* ethash

* ethash

* linters

* ethash

* master stats

* cleanup

* gomod

* linters

* tests

* better locks

* Fix

* Remove logging for verifyHeaders

* Verification speed in the logs

* Fix compile error

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-25 19:40:45 +00:00
ledgerwatch
d3bac21f2e
Fix combined mode for the header download (#1513)
* Fix combined mode for new headers downloader

* Fix for streams

* Return error

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-24 14:51:50 +00:00
ledgerwatch
be7b93d809
Fixes and simplifications for headers download (#1511)
* Simplify header insert, fix recovery from files

* Fix to the header processed log

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-24 07:57:02 +00:00
Alex Sharov
d4ef1f8516
blocks exec unwind - to support graceful shutdown (#1494)
* quit exec unwind

* quit exec unwind

* lint
2021-02-23 17:14:32 +00:00
Alex Sharov
6894d9b356
Remove cursor features (#1508) 2021-02-23 09:02:54 +07:00
ledgerwatch
c30ad646db
Better ergonomics for new header/bodies downloader, more test friendly, README (#1507)
* Ergonomic changes

* Add little doc and remove coreAddr

* Fix gRPC streaming

* Fix lint

* Remove peer penalty logging

* Fix lint

* Fix lint

* Fix lint

* Fix peerID in error message

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-22 21:59:23 +00:00
ledgerwatch
dc6b4c0e43
Fixes for trace_callMany (#1506)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-21 20:18:59 +00:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500)
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
2021-02-21 18:41:59 +00:00
ledgerwatch
e8501bbf43
eth_getTransactionReceipt to return nil for transactions not in the d… (#1505)
* eth_getTransactionReceipt to return nil for transactions not in the database

* Fix compile error
2021-02-21 09:38:00 +01:00
canepat
3388c1f1af
Make vegetaWrite function usable from all benchs (#1502)
Add vegetaWrite support to bench8 (disabled)
Add trace log with comparative timings in bench8
2021-02-19 13:26:28 +07:00
Alex Sharov
7aedc36baa
Tool for benchmarking execution long ranges of blocks (integration loop_exec) (#1501) 2021-02-18 20:33:59 +07:00
ledgerwatch
b9acd62280
Embed header/bodies download into a mini-staged sync (#1498)
* Embed header/bodies download into a mini-staged sync

* Move code around to prevent import cycles

* Include new files

* Fix lint

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-16 15:47:15 +00:00
ledgerwatch
6e56097056
Bring in refactored p2p sentry interface and adjust downloader for it (#1497)
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Use new p2psentry interface

* Remove

* Fix lint

* Fix lint

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Add header upload

* Remove

* Add upload of block headers

* Add block bodies upload

* Fix lint

* Bugfix for blockBodies

* Fix for body upload logging

* Print peerId for GetBlockHeaders

* Invoke block hashes stage to enable block body upload

* Fix test

* Fix stuck bodies download

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-15 18:54:45 +00:00
ledgerwatch
7c81e9198d
Fixes to trace_call, trace_callMany, dual RPC daemon mode (#1491)
* Fixes to trace_call, trace_callMany, dual RPC daemon mode

* Fix compile error

* Fix compile error

* Compile error fix

* Compile error fix

* Fix typo

* Fix discrepancy when trace is empty

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-12 16:47:32 +00:00
Evgeny Danilenko
0fdd296d34
Performance tests for ValidateHeaders on mainnet and rinkeby networks (#1489)
* test

* rename
2021-02-11 16:36:22 +03:00
alex.sharov
a49b882d1c add to_mdbx finish notification 2021-02-11 16:57:04 +07:00
ledgerwatch
bb2cc5a10e
debug_accountAt (#1487)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-10 20:43:13 +00:00
ledgerwatch
2daa71e6cb
Support for trace_callMany (#1486)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-10 17:04:47 +00:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Alex Sharov
b77dfec055
remove debug-web-ui and restapi (#1480) 2021-02-09 19:42:14 +07:00
Alex Sharov
bca54f0e52
delete old code from hack.go (#1481) 2021-02-09 17:25:36 +07:00
ledgerwatch
5834189019
Bodies download and simplify header download (#1471)
* Only insert hard-coded tips if both DB and files recovery failed

* Prevent deadlock

* Report some efficiency numbers

* Count properly unrequested bodies

* Initialise allRequests

* Increase timeout

* Fixes to scheduling

* Small fix

* Simplified scheduling

* Remove separate bodyLoop goroutine

* Update from DB at the beginnig of block bodies forward

* Timeout for repeating the request cycle

* Fix timeout

* Fix

* Increase timeout

* Increase timeout

* Try to make flow if possible

* Fix flow

* Lower timeout

* timeout for each blockNum

* Adjustable timeout

* Better log timing

* Track peers

* copy peerID

* fix scheduling

* Too much logging

* Print delivery speed

* Print committed blocks

* Fix race

* Sentry to only reset back-off timer when response to a request is received

* Print bytes/sec and wasted traffic

* Fix bandwidth accounting

* Less logging

* not to wake up on deliveries

* Spam every second

* Print peer map

* Fix npe, print requests

* Timestamps

* Improved logging

* Penalty for peers

* Log penalties only when disconnecting

* Try with smaller window

* window parameter

* Dealing with partially delivered requests

* Init bodyReq

* Fix array index

* More fix for NPE

* More NPE checks

* Print out body progress and header progress

* Fix ending condition

* Bring back waking up and penalties

* Fix duplicate tip

* Duplicate segment

* Fix lint

* Fix lint

* fix lint

* Fix lint

* Hard coded headers in the source files

* Fix lint

* Replace hix-sized header serialisation with rlp (to support clique)

* Remove anchor records from the files

* Fixeds for DB recovery

* Fix compilation

* Fix compile errors

* Fix formatting

* Fix lint

* Fix comments

* Remove headerLoop

* Properly terminate body download

* Support for StatusData p2p proto

* Fix forkid test

* Fix test

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-08 09:25:10 +00:00
Alex Sharov
bf04a6e73d
make_history_check_optional (#1467) 2021-02-05 16:46:43 +07:00
Artem Vorotnikov
91f7798c4a
Remove hashed changesets (#1464)
* Remove hashed changesets

* try this

* oops

* simplify

* better diagnostics

* don't hash account key and use bitmap 64 to check history index

* update docs of bucket

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2021-02-04 12:59:04 +03:00
Alex Sharov
f0bb3a6dfa
Mdbx new beta version with fixed blockers (#1461) 2021-02-01 20:57:41 +07:00
alex.sharov
5a9c86e552 devel up 2021-01-28 09:37:24 +07:00
Alex Sharov
e943a73ceb
improve integration (#1456) 2021-01-26 12:19:53 +07:00
Alex Sharov
58c4e03861
Revert "Remove hashed changesets (#1444)" (#1446)
This reverts commit aadb6e47cc.
2021-01-20 14:16:20 +07:00
Artem Vorotnikov
aadb6e47cc
Remove hashed changesets (#1444) 2021-01-19 17:42:26 +07:00