Commit Graph

340 Commits

Author SHA1 Message Date
Preston Van Loon
6bf14dedcd
Better aggregated attestations pool (#3761)
* WIP of aggregated signatures in DB

* new lines at end

* taking a nap on the plane now

* fix tests

* remove duplication of attestations. so much for that airplane nap lol

* benchmark before flight lands

* gaz

* manual gaz

* fully contained checks

* quick improvement before landing

* new bitlist with fixes

* doesn't need real signatures

* it works, mostly

* print shard too

* some refactoring

* Revert "some refactoring"

This reverts commit 377ce7fbfb02321c20522da2eb27208a20a67669.

* Revert "Revert "some refactoring""

This reverts commit b46a458898acd214da44fc20dfae920d2725b6e8.

These changes are ok, just need to update the expected values

* fix tests

* lint

* lint

* upstream changes

* fix tests

* what

* resolve TODOs

* gofmt

* revert unrelated pb

* remove debug statement
2019-10-16 23:46:07 -07:00
Preston Van Loon
4e22f52ab3 Testnet restart and hotfixes (#3736)
* hotfix for round robin, hotfix for grpc recovery

* gofmt

* break

* wrong subtraction

* lint

* testnet fork version update

* ignore relay / DHT protocol not supported error
2019-10-08 07:59:08 +08:00
terence tsao
750bc83369
Clean up feature flag namings (#3715) 2019-10-07 14:11:49 +09:00
Nishant Das
4daf62fc28 Fix Validator Activation (#3684) 2019-10-02 13:26:11 -07:00
Jim McDonald
323ee8dfac SetupInitialDeposits() now returns deposit data roots (#3683) 2019-10-02 10:50:34 +08:00
Raul Jordan
d9c0e65cef
Improve Beacon Node Logging UX (#3600)
* info logs beacon node improvements

* prom test fixes

* info logging changes

* wrapped up node info logging

* changed to debug level

* warn logs taken care of

* Terence suggestion

* warn spacing

* better logging in initial sync

* debug level standardized

* complete debug standardization

* participation at epoch end

* fix archive tests

* even more test fixes

* prom test

* ops test

* powtest

* rpc sync test

* rem part

* log formatting
2019-10-01 15:05:17 -05:00
Raul Jordan
3d37a4e038
Optimize Domain Data RPC Request (#3671)
* gaz

* fix broken build

* fix broken test

* fix broken test
2019-10-01 09:36:36 -05:00
Raul Jordan
8d215feb25
Large Prysm Performance Improvements (#3622)
* skip bls verification with a feature flag at runtime

* gazelle

* more bls mocks

* block roots efficiency

* db block roots now does not show up on the flame graphs

* save validator latest votes batch

* batch save att

* misc improvements to pprof

* import

* include validator index cache

* error if no filter criteria

* resolved comments

* build fix

* lint

* remove delay global

* attestation and block test fixes

* preston suggestions

* fix db tests

* fix missing broken tests

* tests pass
2019-09-30 15:45:53 -05:00
Preston Van Loon
571efc11d1
add error spans, interceptrs (#3641) 2019-09-29 11:48:55 -07:00
Ivan Martinez
ee837ecbb9 Reorganize State Transition Functions (#3589) 2019-09-27 09:54:03 -07:00
Preston Van Loon
f681bc6867 Change pk bytes maps to 48 bytes (#3613)
* change pk bytes maps to 48 bytes

* test
2019-09-27 14:07:36 +08:00
Raul Jordan
9dc1674417
resolve queue (#3611) 2019-09-26 22:34:51 -05:00
Raul Jordan
2690c2080d
Implement GetValidatorQueue RPC (#3574)
* changes

* active set change

* helpers for active set changes

* include the helpers in archive service

* table driven tests for helpers

* use from archive

* remove item

* properly use the keys in the response

* test for active set changes

* test passing

* test passing no archive

* archive tests completed

* add ethapis latest commit

* begin implementation get validator queue

* include queue

* finish the queue implementation

* consolidated churn  limit

* pending active testing

* pending active t est

* tests below churn in response

* pubkey

* below  churn  test

* only fetches below the churn  limit

* exit queue churn clip as needed

* full test for  pending active below churn limits

* pending exit test

* pending exit logic

* pending exit below churn test

* all tests done for queue impl

* revert some bad changes

* bug
2019-09-26 16:46:06 -05:00
Raul Jordan
3a890e70f7
Implement Active Set Changes RPC Method (#3568)
* changes

* active set change

* helpers for active set changes

* include the helpers in archive service

* table driven tests for helpers

* use from archive

* remove item

* properly use the keys in the response

* test for active set changes

* test passing

* test passing no archive

* archive tests completed

* add ethapis latest commit

* fix test
2019-09-25 18:06:02 -05:00
Nishant Das
8ce96428b1 Fixes our Attestation Aggregation Issues (#3579) 2019-09-25 07:39:16 -07:00
Nishant Das
0e329fc115 Attestation Server Fix (#3570) 2019-09-24 08:19:37 -07:00
Preston Van Loon
07360bcc07
Write failed blocks to disk (#3569)
* write a failed requested block to disk

* write faile blocks from pubsub too

* gaz
2019-09-23 21:46:40 -07:00
Raul Jordan
305d0299dd
Revamp GetValidators to Retrieve Historical Validators By Epoch (#3563)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* properly retrieve the validators

* revert weird change

* historical validator fetching

* resolves issues with current tests

* adding test for old epoch validators

* tests in
2019-09-23 18:00:38 -05:00
Raul Jordan
b8bdf71d5b
Allow for Fetching RPC Historical Assignments from Archived Data (#3559)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* define archived validator assignment method

* archived assign logic

* need to use compute committee next

* compute archival assignments helper func

* properly compute committee using current shard

* modify the assignments request to take in a query filter item

* more intuitive implementation of list assignments

* utilize the query filter

* complete implementation

* revamp tests

* fixing current tests before adding archive tests

* test now passes using len filtered indices for total size

* include prop index in test

* revert bad change

* use ethapis

* add necessary tests

* comments
2019-09-23 16:04:59 -05:00
terence tsao
4432c88f73 Update beacon chain server AttestationPool (#3560)
* Run time bug

* Still failing

* Run time working

* Run time working

* Gazelle

* Fixed all the tests

* Revert config

* Revert back test configs

* Revert config

* Tested run time again, everything is good

* Implemented AttestationPoolNoVerify
2019-09-23 13:01:57 -05:00
Raul Jordan
5294a6c5af Use Archive in Retrieving Validator Balances (#3536) 2019-09-23 09:19:18 -07:00
terence tsao
ab2d4e8ad6 Implement attestation pool in memory (#3542)
* Run time bug

* Still failing

* Run time working

* Run time working

* Gazelle

* Fixed all the tests

* Revert config

* Revert back test configs

* Revert config

* Tested run time again, everything is good
2019-09-23 11:11:44 -05:00
Raul Jordan
64795bd231
Utilize Archived Data in GetValidatorParticipation RPC Server (#3526)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* remove double negative grammar in comment

* use archive in rpc

* uses the archive!

* error if nothing found in archive

* comment

* use head fetcher and root

* tests pass

* archive active set changes appropriately

* archive committees

* archive info

* done with committee info archiving

* archived set changes stored

* fix build

* test for archive balances and active indices

* further abstractions

* only archive epoch end

* tests all pass

* tests pass now

* archive done

* test for activated validators

* tests for exited

* amend message

* use different proto

* finalization fetcher

* gaz

* use root

* use ctx

* use new ethapis

* use proper hash

* match apis compatibility

* match apis

* properly use participation

* fix tests

* use right commit
2019-09-23 10:47:34 -05:00
terence tsao
4e6ed2744d Fixed 2 cosmetic errors (#3543)
* Fixed

* Treehash

* Updated error msg
2019-09-23 08:30:39 -07:00
Preston Van Loon
9fcc6fc201
Wait until fully synced to process pubsub messages (#3514)
* context timeout for pubsub message processing

* add syncing check

* gofmt

* use a global cache

* lint

* fmt

* fix conflicts

* revert change

* gaz
2019-09-20 10:54:32 -07:00
Raul Jordan
4ffef61e1d
Archive Validator Participation on End of Epoch Event (#3524)
* archive participation begin implementation

* validator participation compute

* comments

* compute participation common func

* full test for archiving data

* gazelle

* complete tests

* gaz

* remove double negative grammar in comment

* use head fetcher and root

* tests pass
2019-09-19 15:59:23 -05:00
Nishant Das
8179ed57b9
fix validator (#3527) 2019-09-19 13:30:47 +05:30
Raul Jordan
e8b6951591
Complete ListAttestations EthereumAPIs v1alpha (#3452)
* retrieve attestations by block root as well

* add beacon block root filter

* rem err unimpl

* add changes to list atts filter proto

* utilize the new filter attributes

* add filter types

* utilize filters in the api server impl

* tests for filter

* tests pass

* filter test done

* fix test by using head fetcher instead

* gaz

* no panic

* use new ethapis commit

* elim panic

* res panic

* ensure proto compatibility

* fixed broken test
2019-09-18 20:14:26 -05:00
Nishant Das
26af4496c0 Batch Validator Performance Requests (#3520)
* change proto msg types

* change server and client

* regen protos
2019-09-18 11:47:14 -05:00
terence tsao
9ab08e6998
Remove beacon rpc service (#3515)
* first version of the watchtower api

* first version

* delete watchtower

* move to message loop

* roughtime

* one time

* fix test

* Fixed test

* Fixed proposer server

* Gaz

* gaz

* Stuck

* Tests passing

* Fixed all the tests
2019-09-18 06:34:50 -07:00
Nishant Das
b1b76ac87c Handle Attestations in a Separate Goroutine (#3487)
* move into separate routine

* preston's review

* use opencensus
2019-09-17 11:17:21 -05:00
Raul Jordan
49a529388b
Resolve Miscellaneous Prysm TODOs (#3465)
* resolve

* resolve

* return

* remove deprecated protos

* rem deprecated pbs

* resolve cache

* resolve md TODO

* node server

* resolve config todo

* resolve even more

* broken build
2019-09-16 15:45:03 -05:00
terence tsao
bf07cfcdab Clean up operation service (#3468)
* Cleaned up operation service

* Fixed all the tests

* Fixed node.go

* Review feedback

* Todo
2019-09-16 12:05:30 -05:00
terence tsao
9d62e542e5
Clean up validator server (#3466) 2019-09-14 13:31:38 -07:00
Raul Jordan
e0e3dada7b
Abstract Usage of Deposit Cache Into Interface (#3443)
* deposit cache refactor begin

* use interface for has chain  started

* use deposit fetcher interface instead

* use moar interfaces

* comment

* gaz

* fix breaking build

* lint

* implement chainstart fetcher

* allow start to work

* fix broken tests
2019-09-11 23:30:04 -05:00
terence tsao
798bbbdc82 Cold start for interop (#3437)
* coldstart flags for validator

* WIP beacon node flags

* wip beacon chain, flag fix in validator, arg fix in validator

* checkpoint

* Added interop service

* working on mock chainstart

* save the state lol

* fix tests

* Save genesis validators

* gaz

* fix validator help flags

* WaitForChainStart actually waits for genesis time

* cold start fixes

* cache

* change back

* allow for genesis state too

* remove logs

* increase mmap size

* dont process if head doesn't exist

* add 10ms tolerance

* enable libp2p debug at debug, fix pubsub

* works with checkpt

* initialize justified and finalized in db

* Removed preloadStatePath from blockchain service

* Clean up

* Write to disk for now post state

* revert b466dd536f8eadbdae2264a545a755370223d917

* Builds

* Only RPC test fails now

* use minimal config, no demo config

* clean up branch

* Lint

* resolve lint

* more lint fixes

* lint

* fix viz

* Fixing RPC test

* skip before epoch 2

* RPC time out

* Fixed ordering

* rename

* remove some dbg statements

* ensure index is correct

* fix some panics

* getting closer

* fix tests

* Fix private key

* Fixed RPC test

* Fixed beacon chain build for docker

* Add interop.go to validator go_image

* Fixed docker build

* handle errors

* skip test, skip disconnecting peers

* Fixed docker build

* tolerance for attestation processing

* revert copy

* clearer err message parse

* fix launching from dep contract
2019-09-11 13:38:35 -05:00
Raul Jordan
af07c13730
[Interop] Improve RPC Codebase + Start Beacon Chain With Mock ETH1 Values (#3407)
* add main.go

* interop readme

* proper visibility

* standardize and abstract into simpler funcs

* formatting

* no os pkg

* add test

* no panics anywhere, properly and nicely handle errors

* proper comments

* fix broken test

* readme

* comment

* recommend ssz

* install

* tool now works

* README

* build

* readme

* 64 validators

* rem print

* register the no powchain flag

* work on mock eth1 start

* common interface

* getting closer with the interface defs

* only two uses of powchain

* remove powchain dependency

* remove powchain dependency

* common powchain interface

* proper comment in case of flag

* proper args into rpc services

* rename fields

* pass in mock flag into RPC

* conforms to iface

* use client instead of block fetcher iface

* broken tests

* block fetcher

* finalized

* resolved broken build

* fix build

* comment

* fix tests

* tests pass

* resolved confs

* took them out

* rename into smaller interfaces

* resolve some confs

* ensure tests pass

* properly utilize mock instead of localized mock

* res lint

* lint

* finish test for mock eth1data

* run gazelle

* include flag again

* fix broken build

* disable powchain

* dont dial eth1 nodes

* reenable pow

* use smaller interfaces, standardize naming

* abstract mock into its own package

* faulty mock lint

* fix stutter in lint

* rpc tests all passing

* use mocks for operations

* no more mocks in the entire rpc package

* no  mock

* viz

* testonly
2019-09-09 17:13:50 -04:00
Ivan Martinez
5e939378d0 Update to spec v0.8.3 (#3355)
* Ignore latest messages in fork choice prior to latest justified

* Make sure Compact Committee Roots isn't changed by process_final_updates

* WIP add attestation bitfields length to match committee length

* Begin work on updating spec tests to 0.8.2

* WIP set up for new spec test structure

* Fix slashings

* Get mainnet tests mostly passing for attestations and attester slashings

* Fix process attestation test

* Undo change

* Complete spec tests for all operations
Still need sanity block tests

* Fix BLS sigs

* Reduce amount of reused code in core/blocks/spectests/

* Fix tests

* Update block sanity tests to 0.8.2

* Update epoch spec tests to 0.8.2

* Clean up all tests and fix shuffling/epoch tests

* WIP update bls tests to 0.8.2

* WIP update bls tests to 0.8.3

* Finish BLS spectest update to 0.8.3

* Fix shuffling spec tests

* Fix more tests

* Update proto ssz spec tests to 0.8.3

* Attempt to fix PrevEpochFFGDataMismatches test

* Goimports

* Fix documentation

* fix test

* Use custom general spec tests

* Reduce code footprint

* Remove unneeded minimal skip

* Fix for comments

* Fix for comments

* Fix test

* Small fixes

* Cleanup block spec tests a bit

* Undo change

* fix validator

* Fix validator tests

* Run gazelle

* Fix error output for epoch spec tests
2019-09-08 12:41:52 -07:00
terence tsao
4235980511 Clean up post --next (#3411)
* Delete old code

* RPC mock testing

* Fixed BUILD

* Conflict

* Lint

* More lint
2019-09-06 22:39:14 -04:00
Nishant Das
fb20fc7881 change to wrap (#3413) 2019-09-06 14:01:59 -07:00
terence tsao
14c59b2ff9 Remove deprecated services and --next (#3371)
* Save new validators in DB

* Use info

* Add total validator count

* Fixed tests

* Add new test

* Revert light client config

* Add state metrics back

* Gaz

* Mark old ones as deprecated

* Deprecate not --next services

* Fixed all operation tests

* Fixed node test

* All tests passing locally

* Add deprecated-p2p back, blocked by boostrap-query

* Revert message proto

* delete deprecated DB items

* delete all other instances of old db

* gaz

* cycle rem

* clear db
2019-09-05 11:04:06 -05:00
Preston Van Loon
b0e6d7215c
Tracing: Add additional attributes (#3404)
* Add some attributes for tracing

* gaz
2019-09-03 20:03:09 -07:00
terence tsao
314bc513af
Fixed validator pubkey -> index getter (#3361) 2019-08-30 13:50:21 -07:00
terence tsao
d8fd7e502a Fix GetChainHead for RPC (#3352)
* Fix ChainHeadQuery

* Fixed test
2019-08-29 10:17:21 -05:00
Raul Jordan
e8f030977a
wait for chainstart (#3343) 2019-08-28 12:07:58 -05:00
terence tsao
9f2c2f0197 Minor runtime fixes (#3335) 2019-08-27 22:19:47 -05:00
Nishant Das
701c70ae3b add better logging (#3329) 2019-08-27 06:27:04 -07:00
terence tsao
f188609137 Implement GetHead for RPC (#3326)
* next compatible, tests pass

* terence feedback

* skip comment

* fixes

* misc fix

* on block

* Update RPC service to use chain info

* OOps

* All tests pass, run time with 8 validators work!

* Remove saving genesis validator

* Revert gensis count

* Move redundant headstate

* Comments

* Implemented GetChainHead

* Test works

* Moved mock package

* Fixed visibility for BUILD file

* Conflict
2019-08-26 15:59:17 -05:00
terence tsao
64d0826469
Update RPC service to use chain info (#3309) 2019-08-26 13:06:16 -07:00
Raul Jordan
9ad00ffafb
Use New Attestation Receiver Method in RPC (#3287)
* next compatible, tests pass

* terence feedback

* skip comment

* fixes

* misc fix

* on block

* resolved err
2019-08-25 15:45:55 -05:00