Commit Graph

91 Commits

Author SHA1 Message Date
ledgerwatch
7554428884
POC of header downloader - splitting into 2 processes: sentry + downloader (via gRPC) (#1291)
* Splitting sentry and downloader - the beginning

* A bit more

* More on sentry

* More gRPC

* Sentry and downloader separated

* Update binding for stable version of grpc

* Better bufferSize flag

* Fix lint

* Send pelanties

* Fix lint

* Remove hard-coded tips on connect

* Tidy the logs a bit

* Deal with hardTips on Recovery

* Print hard tips

* Hide empty anchors

* Request headers after receiving a message

* Better waking up

* Print hard-coded block numbers

* Print outgoing requests

* Debug logging

* In the middle protection

* Sentry not to lose peers when core disconnects
2020-10-25 22:05:37 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
c2f41d7723
stop on error: headers stage (#1289) 2020-10-24 18:05:12 +01:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader (#1288)
* protect_from_wrong_class_use

* more cases
2020-10-24 09:09:20 +01:00
Alex Sharov
77a4b3305d
rawdb to return errors: part 1 (#1286)
* ReadBlockByNumber and ReadChainConfig return error

* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:

ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
2020-10-24 07:57:09 +01:00
Alex Sharov
6584b43b12
Allow start read transactions from ethdb.Database interface (#1287)
* allow_read_transactions

* fix lint

* fix lint

* more cases
2020-10-24 07:55:43 +01:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274)
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
2020-10-21 18:01:40 +01:00
b00ris
e5f6412da2
Update snapshot hash (#1243)
* remove useless bucket

* update snapshot hashes

* remove log
2020-10-17 08:48:20 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00
ledgerwatch
b747ab5324
[WIP] CallTraces index (#1157)
* Initial commit for CallTraces index

* Fix compilation

* fix lint, add comment

* Fix integration

* Add Close function to ethdb.Cursor, fix some compile errors

* Try to stop cursor leak in Get

* Fix compile errors in RPC daemon

* Fix compile errors

* fixing another way

* Some fixes

* More fixes

* More fixes

* More fixes

* Fixes to core/state

* Fix lint

* Fix lint

* Fixes

* Stage caching for call trace stage

* Add mem stats

* Try to stop the leak

* Turn off debug

* Chunks for 10k blocks

* Print

* Revert "Print"

This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.

* Revert "Chunks for 10k blocks"

This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.

* Trying to fix the leak

* Don't compute receipts in re-tracing

* Not compose block

* Print speed, fix receipts, bigger caches

* Fix lint

* Utilise changeset info

* Counters

* Use NoReceipts and ReadOnly

* ReadOnly is incompatible with caching

* Skip test leaking transactions

* Fix block test

* Change disable message for call-traces stage

* Use block option for call traces integration

* Fix retracing due to incarnation
2020-10-12 09:39:04 +01:00
Alex Sharov
88a099702b
rawdb.ReadCanonicalHash to return err (#1204)
* read_canonical_hash_return_err

* hack hugeFreelist

* fix tests
2020-10-10 07:05:56 +01:00
Alex Sharov
53484d60f2
LMDB: don't search pages sequence in freelist if value is > X (#1213)
* lmdb_no_freepages_sequence_search_for_large_values

* go mod tidy
2020-10-10 07:05:18 +01:00
ledgerwatch
1af6cbe37b
Revert "Download/seed headers and block bodies via BitTorrent (#1195)" (#1202)
This reverts commit 94360212cb.
2020-10-07 21:49:44 +01:00
ledgerwatch
94360212cb
Download/seed headers and block bodies via BitTorrent (#1195)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-07 13:15:49 +01:00
ledgerwatch
bdb0155856
Revert "Add sync by bittorrent snapshots (#1160)" (#1194)
This reverts commit 4ebade3583.
2020-10-06 22:31:57 +01:00
ledgerwatch
834eaedc00
pass_root_ctx_to_p2p_server (#1192)
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-06 20:25:01 +01:00
b00ris
4ebade3583
Add sync by bittorrent snapshots (#1160)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-06 20:24:48 +01:00
ledgerwatch
b652768241
Revert "pass_root_ctx_to_p2p_server (#1179)" (#1182)
This reverts commit b8c3880254.
2020-10-04 17:18:06 +01:00
ledgerwatch
55219d7ad3
eth_call to use current state when blockNumber == latest (#1180) 2020-10-04 09:01:06 +01:00
Alex Sharov
b8c3880254
pass_root_ctx_to_p2p_server (#1179) 2020-10-03 11:15:01 +01:00
Alex Sharov
dae72a8589
(related to lmdb assert) Avoid deleteCurrent on cursor which used for iterations (#1151)
* save progress

* save progress

* avoid delete current

* undupsort_ih
2020-09-29 10:28:49 +01:00
ledgerwatch
14288b8cd0
Headers poc 5 - Intermediate (#1145)
* Add headers persistence

* Print flushBuffer

* Fix indexing problem

* Not skip hard-coded headers if the files are empty

* Fix lint

* print anchor state after init

* Properly construct file names

* Add check sub-command

* Fix lint

* Fix lint

* Fix lint

* Print more info when recovering

* Fix recovering

* Fix recovery

* Add anchors also as tips

* 2-level priority queue for tips

* Initialise tipQueue in anchor

* update maxTipHeight

* fix type

* Add anchors to the anchorQueue and rebuild anchorQueue on deletion

* Fix NPE

* fix recovery

* User buffersize, add hard coded headers to buffer

* Reinit anchorQueue

* Schedule requests after recovery

* No fix

* Remove duplicates

* Report duplicate headers

* Log buffer additions

* Fix for duplicate headers

* Try to fix duplicate headers again

* remove TODO comment

* Use LLRB instead of heap for anchors

* Print reserveTip info

* Correctly replace anchors in the tree

* Remove excessive logging

* Print tips attached to the anchor

* Print tips better

* limitTips instead of reserveTip

* Print forked headers

* Use pointers in llrb

* Print tipStretch

* Print limitTips

* Mininise AnchorItem

* Put anchors into the tree

* Fix totalDiff calculation, but is it needed?

* Remove totalDifficulty from anchors

* CheckInitiation

* Fix tests

* Fix lint, print more at check initiation

* Better output for hard-coded anchors

* Print hard-coded anchors better

* Prioritise anchors with short stretches

* Prioritise by chainSize

* Use linked list instead of heap for requestQueue

* Fix problem of no requests

* Push front

* Fix lint

* Not verify PoW for too far in the past and future

* Fix Calculation of totalDifficulty when Connect

* Fix hard tips

* Another fix for tips
2020-09-27 21:32:05 +01:00
Igor Mandrigin
65d8a12bb1
turbo-api: Add docs to some public structs, methods and fields (#1127) 2020-09-21 16:10:25 +02:00
ledgerwatch
5aab794c9b
Headers poc 4 - intermediate changes (#1125)
* Process BlockHeadersMsg

* Print invalidations

* Print reason for parent-child mismatch

* Fix off by one error

* Fix ExtendDown

* Replace tipLimiter with an LRU cache

* Retrieve 256 headers at a time

* Hard coding of tips, process new block hashes

* extract headers

* Write hard-coded headers into a file

* Add total difficulty

* Insert hard coded headers

* Log i

* Initiate download from hard-coded anchors

* Print the state of anchors

* Add hard-coded headers as tips

* Spare the peers

* Refine LRU caching of tips

* Print length of anchor trees

* Limit requests to 192 headers

* range error fix

* Print better, retain at least one tip in the LRU cache

* Skip duplicate line ends

* Throttle requests, display peer name

* Back off, intermediate tip queue

* Print block header timestamp

* Better logging

* Log less

* More logging of evictions

* Fix queue push

* Fix queue pop

* Exclude duplicate tips

* Fix tips logging

* Fix moving

* Add port flag

* Check headers

* Print extra

* Add mixDigest to serialisation

* Fix difficulty in hard-coded headers

* Avoid duplicate header requests

* Update peer characteristics on NewBlockMsg

* Remove backoff when peer responds

* Fix tests

* Fix lint

* Fix lint
2020-09-20 10:47:24 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) (#1113)
* added tls auth

* added client side

* put --tls

* fixed flag

* Add key/cert generation instructions, turn off Common Name verification

* Add CLI arguments and Warning

* Lint

* Update the doc about Internal IP

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00
Igor Mandrigin
adf52465e3
move ./trie to ./turbo/trie (#1114)
Useful for minimizing merge conflicts when rebasing new geth
2020-09-14 11:33:39 +01:00
Thomas Jay Rush
752793264b
Fixes issue #1103 (#1106) 2020-09-12 06:33:07 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon (#1089)
* added eth_getStorageAt

* used uint32

* now its 256

* incarnation

* added TLS handshake

* code minified

* lint

* minified client

* Update flags.go

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Igor Mandrigin
eca39e12f2
Fix reproducible builds (#1099) 2020-09-11 18:42:33 +02:00
ledgerwatch
ab4c456fec
Headers poc 3 - intermediate changes (#1093)
* Use log instead of Printf

* Better output

* Display NewBlockHeadersMsg

* Print tx hashes

* Separate sentry and downloader

* Fix lint

* Requesting headers

* Fix lint

* Prevent deadlock, fix NPE

* Not quit when did not restore

* Fix types

* Fix types again

* Warn on penalty

* Prints

* Avoid deadlock

* Better request logic

* Print height

* Actually send header requests

* Actually send header request

* Fix compile errors, sending

* Print BlockHeadersMsg

* Fix lint
2020-09-11 07:35:51 +01:00
Igor Mandrigin
18197b9e13
Disable IPC and remove flags for it (#1092)
* always disable IPC

* remove ipc flags from `DefaultFlags`
2020-09-10 14:01:51 +02:00
ledgerwatch
7a84a5b8cc
Increase call timeout for 5 minutes (#1082) 2020-09-08 20:40:01 +01:00
Igor Mandrigin
b41567460a
add identity flag back (#1064) 2020-09-06 17:42:19 +02:00
Igor Mandrigin
1d0ec0f0d0
turbo-api: Allow apps based on turbo-geth to add custom buckets to the chaindata (#1063) 2020-09-06 17:33:05 +02:00
Igor Mandrigin
c3ce05b5eb
Fix nousb flag missing. (#1062) 2020-09-06 14:10:57 +01:00
Igor Mandrigin
d14e45d4d0
cmd/geth -> cmd/tg and get rid of geth implementation. (#1061) 2020-09-06 13:35:32 +02:00
Igor Mandrigin
2a6478d799
turbo-api: create a second executable file with a custom stage example (#1055) 2020-09-05 18:07:27 +02:00
ledgerwatch
1a0e9275e1
Headers PoC - intermediate changes (#1016)
* First algorithm

* Update doc

* Second algorithm implemented

* Prepend (unfinished)

* More on prepend

* More fixes

* Change CalcDifficulty interface

* fix make test

* Fix formatting

* Another test

* More on Preprend

* Another test for Prepend

* More tests for Prepend

* More tests for Prepend

* Move files to turbo

* Add another test for Prepend

* Started adding Append algorithm

* More work on Append

* Start changing the terminology

* Split trees into segments

* More fixes

* Fix compilation

* Connect

* FindAnchors, FindTip

* Tip has anchor and not anchorParent field

* Remove Prepend and Append

* Add Connect

* NewAnchor algorithm

* More to recovery from files

* Add more to AddAnchor

* Request more headers

* Header fix-sizes serialisation

* Upgrade CI to 1.15

* Fix lint

* Fix lint

* replace RLP with fixed-sized serialisation

* Fix lint

* More algos

* Fix lint
2020-08-31 06:27:51 +01:00
ledgerwatch
22dc832d92
RPC method fixes (#1007)
* Fix some RPC issues

* Fudge the getBlockByNumber

* Add two more PRC methods

* Fix rpc tests
2020-08-29 22:28:09 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Alex Sharov
1dcc2b141a
Rpcdaemon as lib (#940)
* share config object

* create default config and logger

* move db connection to common func

* move server start to cli package

* clear

* clear

* rename cli to rpc

* use unified SetupLogger func

* make all root flag persistent

* use common flags in different packages

* use common flags in different packages

* move TraceTx method to eth package

* use native slice flags

* create package "turbo"

* disable geth api

* disable geth api

* move more data types to turbo/adapter package

* add support for customApiList

* run more

* run more

* run more

* dog-food

* move DoCall

* move DoCall

* fix tests

* fix test
2020-08-19 12:46:20 +01:00