Commit Graph

1096 Commits

Author SHA1 Message Date
Preston Van Loon
5345ddf686 Initial Sync: Round robin (#3538)
* first pass, step 1 works

* naive from finalized to head

* delete commented code

* checkpoint progress on tests

* passing test

* abstract code slightly

* failure cases

* chkpt

* mostly working, missing a single block and having timeout

* passing tests

* comments

* comments

* gaz

* clarify comments

* progress on a few new cases

* add back bootnode query tool

* bootstrap from DHT

* chunked responses in round robin

* fix tests and deadlines

* add basic counter, time estimation

* hello -> handshakes

* show peers in use during sync

* just one last test failure

* only request blocks starting in the finalized epoch for step 1

* revert that

* comment out test and add better commentary

* move requestBlocks out to pointer receiver

* mathutil

* Update beacon-chain/sync/initial-sync/round_robin.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* PR feedback

* PR feedback
2019-09-25 12:00:04 -05:00
Nishant Das
8ce96428b1 Fixes our Attestation Aggregation Issues (#3579) 2019-09-25 07:39:16 -07:00
Raul Jordan
5398faea44 Do Not Archive Active Indices (#3573)
* dont archive active indices

* eliminate logs
2019-09-25 17:18:37 +08:00
Nishant Das
0e329fc115 Attestation Server Fix (#3570) 2019-09-24 08:19:37 -07:00
Nishant Das
0db690df75 Chunked Responses (#3528)
* update naming

* replace with updated version

* more changes

* fixed all tests

* build and lint

* regen protos

* fix test

* remove outdated code

* prestons review

* add chunk size

* more fixes to chunked responses

* handle eof

* fix all tests

* abstract into common method

* add comment

* preston's comments

* preston's review

* preston's review

* lint

* add encoding methods

* gaz

* simplify

* simplify

* lint

* change naming

* update

* handle eof separately

* Apply suggestions from code review

Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com>

* remove def

* preston's review

* preston's review

* add unit tests

* add delay to fix test
2019-09-24 07:56:50 -07:00
Preston Van Loon
07360bcc07
Write failed blocks to disk (#3569)
* write a failed requested block to disk

* write faile blocks from pubsub too

* gaz
2019-09-23 21:46:40 -07:00
terence tsao
cbcbb487ac
Lock around validateBeaconBlockPubSub (#3567) 2019-09-23 18:43:07 -07:00
Preston Van Loon
ad47817bcd Only write interop ssz states to disk with flag ON (#3566)
* only write SSZ states to disk with flag on

* lint

* also write blocks
2019-09-23 18:36:12 -05:00
Raul Jordan
305d0299dd
Revamp GetValidators to Retrieve Historical Validators By Epoch (#3563)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* properly retrieve the validators

* revert weird change

* historical validator fetching

* resolves issues with current tests

* adding test for old epoch validators

* tests in
2019-09-23 18:00:38 -05:00
Preston Van Loon
5d33514001 Restore bootnode query tool for kaddht and fix bootstrapping (#3565)
* add back bootnode query tool

* bootstrap from DHT
2019-09-23 17:18:01 -05:00
Preston Van Loon
f5aa25821d
p2p: Relay support, CIDR whitelist, connection maintenance (#3564)
* add relay dial

* add relay support advertisement, add peer watching to maintain connections to bootstrap nodes and relay nodes

* gofmt

* double space
2019-09-23 14:43:53 -07:00
Raul Jordan
b8bdf71d5b
Allow for Fetching RPC Historical Assignments from Archived Data (#3559)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* define archived validator assignment method

* archived assign logic

* need to use compute committee next

* compute archival assignments helper func

* properly compute committee using current shard

* modify the assignments request to take in a query filter item

* more intuitive implementation of list assignments

* utilize the query filter

* complete implementation

* revamp tests

* fixing current tests before adding archive tests

* test now passes using len filtered indices for total size

* include prop index in test

* revert bad change

* use ethapis

* add necessary tests

* comments
2019-09-23 16:04:59 -05:00
terence tsao
9577e2c123 Removed attestation check for votes (#3562) 2019-09-23 15:44:13 -05:00
skillful-alex
b015dc793a HeadFetcher data race fix (#3460)
* HeadFetcher data race fix

* bazel run //:gazelle -- fix

* add the db teardown to test

* add TestChainService_SaveHead_DataRace test

* split race and norace tests

* change testset name

* test CI with 'spectest' tag instead of 'raceon'

* one more test CI with 'spectest' tag instead of 'raceon'

* bazel run //:gazelle -- fix

* set test tag to 'race_on'

* not clone the state
2019-09-23 14:24:42 -05:00
terence tsao
4432c88f73 Update beacon chain server AttestationPool (#3560)
* Run time bug

* Still failing

* Run time working

* Run time working

* Gazelle

* Fixed all the tests

* Revert config

* Revert back test configs

* Revert config

* Tested run time again, everything is good

* Implemented AttestationPoolNoVerify
2019-09-23 13:01:57 -05:00
Nishant Das
b5b10a8d35 Add Back Kademlia DHT to Prysm (#3557)
* serve nodes

* remove testing flag

* add back bootnode

* add dht

* add back dht

* gaz

* fix build

* bootnode works in runtime

* fix all references

* all tests pass

* remove feature flag

* separate out ports

* lint

* fix docker build

* use one error package
2019-09-23 10:24:16 -07:00
Raul Jordan
5294a6c5af Use Archive in Retrieving Validator Balances (#3536) 2019-09-23 09:19:18 -07:00
terence tsao
ab2d4e8ad6 Implement attestation pool in memory (#3542)
* Run time bug

* Still failing

* Run time working

* Run time working

* Gazelle

* Fixed all the tests

* Revert config

* Revert back test configs

* Revert config

* Tested run time again, everything is good
2019-09-23 11:11:44 -05:00
Raul Jordan
64795bd231
Utilize Archived Data in GetValidatorParticipation RPC Server (#3526)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* remove double negative grammar in comment

* use archive in rpc

* uses the archive!

* error if nothing found in archive

* comment

* use head fetcher and root

* tests pass

* archive active set changes appropriately

* archive committees

* archive info

* done with committee info archiving

* archived set changes stored

* fix build

* test for archive balances and active indices

* further abstractions

* only archive epoch end

* tests all pass

* tests pass now

* archive done

* test for activated validators

* tests for exited

* amend message

* use different proto

* finalization fetcher

* gaz

* use root

* use ctx

* use new ethapis

* use proper hash

* match apis compatibility

* match apis

* properly use participation

* fix tests

* use right commit
2019-09-23 10:47:34 -05:00
terence tsao
4e6ed2744d Fixed 2 cosmetic errors (#3543)
* Fixed

* Treehash

* Updated error msg
2019-09-23 08:30:39 -07:00
Nishant Das
71098b6ed8
Lookup Bootnode instead of Random Nodes (#3555)
* add workspace change

* change to lookup bootnode
2019-09-23 10:17:11 +05:30
terence tsao
041735ef54
Removed dead code (#3544) 2019-09-21 22:04:20 -07:00
terence tsao
315d4f0549 Save aggregated attestations in DB after processing (#3541)
* Added AggregatedAttestation helper

* Implemented `saveNewAttestation` for processing attestation

* Implemented `savesNewBlockAttestations` for processing block

* Tests

* Fix name

* Raul's feedback
2019-09-21 13:57:32 -05:00
terence tsao
2b2ef4f37c
Added AggregatedAttestation helper (#3539) 2019-09-21 09:43:18 -07:00
Preston Van Loon
fb8d6a4046 infrequently ping bootnode (#3540) 2019-09-21 09:21:44 -07:00
Preston Van Loon
37596ac188 Is len(map) threadsafe? (#3535) 2019-09-20 13:05:08 -05:00
Preston Van Loon
9fcc6fc201
Wait until fully synced to process pubsub messages (#3514)
* context timeout for pubsub message processing

* add syncing check

* gofmt

* use a global cache

* lint

* fmt

* fix conflicts

* revert change

* gaz
2019-09-20 10:54:32 -07:00
terence tsao
c29a7be0ec Fix justified check point mutation (#3534)
* first version of the watchtower api

* first version

* delete watchtower

* move to message loop

* roughtime

* one time

* fix test

* Fixed

* Fixed

* Revert unused lock
2019-09-20 10:44:28 -07:00
terence tsao
cf2ad1f21c Parent blocks fetching/processing (#3459)
* first version of the watchtower api

* Initial prototype of sync parent fetching/processing

* Another map to track seen block root

* Fixed fmt

* Ready to live test

* Ready to live test

* Seperate pending block queue into its own

* first version

* delete watchtower

* move to message loop

* roughtime

* one time

* fix test

* Started testing but peer list empty

* Comment

* Loggins

* Stuck at decoding non proto type

* Revert

* First take, need feedback

* Run time panics at hello

* Revert

* use reflect properly

* Fixed subscriber

* instantiate helper

* More reverts

* Revert back tests

* Cont when EOF

* Working

* Clean hello tracker on peer disconnection

* Clean hello tracker on peer disconnection

* Move to validation

* Propoer locking

* Propoer locking

* Fmt

* Nishant's feedbacke

* More feedback

* All tests passing

* fix build

* remove log

* gaz

* Added the todo
2019-09-20 10:08:32 -07:00
Raul Jordan
a2aa142b90
Archive Remaining Data at Epoch End in Archiver Service (#3531)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* remove double negative grammar in comment

* use head fetcher and root

* tests pass

* archive active set changes appropriately

* archive committees

* archive info

* done with committee info archiving

* archived set changes stored

* fix build

* test for archive balances and active indices

* further abstractions

* only archive epoch end

* tests all pass

* tests pass now

* archive done

* test for activated validators

* tests for exited

* use ctx
2019-09-20 11:51:06 -05:00
Nishant Das
44e5e5de65 Remove Error Message Type (#3533)
* proto change

* fix test

* fix error resp
2019-09-20 09:13:38 -05:00
Nishant Das
4bc2d628b1
Update Naming to Latest Networking Spec (#3519)
* update naming

* replace with updated version

* more changes

* fixed all tests

* build and lint

* regen protos

* fix test

* remove outdated code

* prestons review

* preston's comments

* preston's review

* preston's review

* lint
2019-09-20 11:57:28 +05:30
terence tsao
ac176a5078 Update validators votes from incoming block (#3530)
* first version of the watchtower api

* first version

* delete watchtower

* move to message loop

* roughtime

* one time

* fix test

* Update block attestation votes

* Clean up

* Found a bug

* Confirmed to be working in run time

* Confirmed to be working run time

* Raul's feedback

* Tests
2019-09-19 20:34:57 -05:00
Raul Jordan
4ffef61e1d
Archive Validator Participation on End of Epoch Event (#3524)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* remove double negative grammar in comment

* use head fetcher and root

* tests pass
2019-09-19 15:59:23 -05:00
Nishant Das
8179ed57b9
fix validator (#3527) 2019-09-19 13:30:47 +05:30
Raul Jordan
e8b6951591
Complete ListAttestations EthereumAPIs v1alpha (#3452)
* retrieve attestations by block root as well

* add beacon block root filter

* rem err unimpl

* add changes to list atts filter proto

* utilize the new filter attributes

* add filter types

* utilize filters in the api server impl

* tests for filter

* tests pass

* filter test done

* fix test by using head fetcher instead

* gaz

* no panic

* use new ethapis commit

* elim panic

* res panic

* ensure proto compatibility

* fixed broken test
2019-09-18 20:14:26 -05:00
terence tsao
495621e99b
Sync RPC to support none proto message (#3512) 2019-09-18 13:48:16 -07:00
Preston Van Loon
e1861bdb31 Clean hello tracker on peer disconnection (#3522)
* Clean hello tracker on peer disconnection

* Clean hello tracker on peer disconnection
2019-09-18 15:02:34 -05:00
terence tsao
f69195f211
Continue on bad attestation (#3523) 2019-09-18 12:22:26 -07:00
Raul Jordan
36e3a9f82a
Implement Archival DB Methods (#3521)
* generate archive proto

* archived committee info

* archive methods added to iface definition

* impl

* update iface

* proto comments

* implement first method

* committee info

* save committee info

* participation checked in

* fully implemented

* tests

* test defs

* db impls done
2019-09-18 13:41:47 -05:00
Raul Jordan
b919429801
Archived Data Definitions & DB API (#3510)
* generate archive proto

* archived committee info

* archive methods added to iface definition

* impl

* update iface

* proto comments
2019-09-18 12:05:24 -05:00
Nishant Das
26af4496c0 Batch Validator Performance Requests (#3520)
* change proto msg types

* change server and client

* regen protos
2019-09-18 11:47:14 -05:00
Raul Jordan
037c01f4d7
Archiver Service Definition (#3507)
* archive flags

* gaz

* create archiver

* register archiver in node

* registering the head updater feed

* add more gazelle

* cancel func

* test for service

* properly utilize the mocks

* lint

* remove extraneous log

* add back write to disk

* gaz
2019-09-18 09:30:02 -05:00
terence tsao
9ab08e6998
Remove beacon rpc service (#3515)
* first version of the watchtower api

* first version

* delete watchtower

* move to message loop

* roughtime

* one time

* fix test

* Fixed test

* Fixed proposer server

* Gaz

* gaz

* Stuck

* Tests passing

* Fixed all the tests
2019-09-18 06:34:50 -07:00
shayzluf
bdb1b472b6 Test proccess on chainstart (#3516)
* first version of the watchtower api

* first version

* delete watchtower

* move to message loop

* roughtime

* one time

* fix test

* add test to chain start

* fix test

* move logic to mock

* remove unused method

* remove imports

* gaz

* goimports

* goimport
2019-09-18 14:14:25 +05:30
shayzluf
b9f9cf0b2c Handle blocks after chain start (#3486) 2019-09-17 14:14:51 -07:00
Nishant Das
b1b76ac87c Handle Attestations in a Separate Goroutine (#3487)
* move into separate routine

* preston's review

* use opencensus
2019-09-17 11:17:21 -05:00
Nishant Das
31eae719b9
fix config (#3500) 2019-09-17 11:19:58 +05:30
terence tsao
b863004b2a
Aggregate attestations before verify and update votes (#3493) 2019-09-16 17:48:03 -07:00
Nishant Das
7eba8da9d2 Save Network Keys in Data Directory (#3488)
* change marshalling

* add networkkeys

* gaz

* fix test

* add new function

* resolve comments, rename to datadir
2019-09-16 17:09:16 -05:00