Commit Graph

407 Commits

Author SHA1 Message Date
Raul Jordan
0326be86b5 Apply Patch Rules to Use EthereumAPIs Generated Protos in Prysm (#4112)
* starting on patch
* finish determining all required patches
* properly redefine the patch rules
* new patch
* rem double semicolon
* fix patch file
* Merge branch 'master' of github.com:prysmaticlabs/prysm into deprecate-eth-protos
* building the deps
* test target passes using ethereumapis
* compile gateway
* attempting to build everything
* e2e use ethereumapis
* more fixes for slasher
* other item
* getting closer to compiling slasher
* build slasher package
* Merge branch 'master' into deprecate-eth-protos
* Merge branch 'master' into deprecate-eth-protos
* fix benches
* lint gazelle
* Merge branch 'deprecate-eth-protos' of github.com:prysmaticlabs/prysm into deprecate-eth-protos
* proper gateway
* lint
* Merge branch 'master' into deprecate-eth-protos
* fix build
* Merge branch 'deprecate-eth-protos' of github.com:prysmaticlabs/prysm into deprecate-eth-protos
* use swag
* resolve
* ignore change
* include new patch changes
* fix test
* builds
* fix e2e
* gaz
2019-11-27 05:08:18 +00:00
terence tsao
1ced4754db Add signatures to logs (#4095)
* Enhance logging with sig
* Fixed
* Merge branch 'master' into add-sig
* Merge branch 'master' into add-sig
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into add-sig
* signature via debug
* Merge branch 'add-sig' of https://github.com/prysmaticlabs/prysm into add-sig
* Merge branch 'master' into add-sig
* Merge branch 'master' into add-sig
* Merge branch 'master' into add-sig
2019-11-26 20:36:18 +00:00
Ivan Martinez
c1c48a8af5 Create Benchmarks Package for State Transition (#3688)
* Begin benchmarks file for block processing
* Complete block processing benchmarks
* Begin epoch benchmarks
* Write most of epoch benchmarks
* Start config
* Make cases for max conditions
* Begin work on benchmarking doc
* Update benchmark numbers
* Complete epoch benchmarks
* Minor changes
* Make createFullBlock function
* Clean up block benchmarks
* Begin fixing merge issues
* Start adding 4M benchmarks
* Almost finish epoch benchmarks
* Test blocks under real life conditions
* More progress on benchmarks
* Fixes
* Fix benchmark errors
* Begin fixing benchmarks
* More progress on tests
* Complete epoch benchmarks
* More progress on block benches
* Finish epoch benchmarks, get progress on block benchmarks
* Undo unneeded changes
* Fix
* Fix block benchmarks
* Complete block benchmarks
* Finish block benchmarks
* Complete benchmarks
* Increase block benchmarks to 65536
* Fix everything
* Reset configs after benchmarks
* Fix logging and suggestions
* Fix comments
* Fix benchmarks after merge
* Fix merge issues
* Add sanity tests for benchmark
* Make sanity check simpler
* Begin fixing after merge
* Add log
* Remove extra line
* Remove unneeded change
* Finally get block benchmarks to pass
* Begin fixing epoch test
* Finetuning constants
* Revert "Finetuning constants"

This reverts commit a872790d675154e1ce213285e76fb6e08ef21d79.
* Finetuning
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Add benches for helper functions
* Abstract block generation to testutil
* Create block generation util in testutil
* Gazelle
* Fix deps
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into block-util
* Fix imports
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into block-util
* Merge branch 'master' into block-util
* Change tests to use config and fix integer division
* Merge branch 'block-util' of https://github.com/0xKiwi/prysm into block-util
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into block-util
* Remove logs
* Fix build
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Merge branch 'master' into block-util
* Add test to ensure finalization occurs
* Add check for finalization
* Merge branch 'block-util' of https://github.com/0xKiwi/prysm into block-util
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into block-util
* Add comment for incrementing the state
* Fix test
* Fix test
* Merge branch 'master' into block-util
* Fix testutil use
* Fix tests
* Change var name
* Merge branch 'master' into block-util
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Merge branch 'block-util' of https://github.com/0xKiwi/prysm into new-benchmarks
* Begin cleaning benchmarks
* Get some numbers going
* Use state saved to disk
* Remove cruft
* Cleanup
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Fix merge arrows
* Set up block util and benchmarks for 128 attestations
* Use intended config for benchmark
* Add more benchmark functions
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Add benchmark epoch and modify block gen config to exclude signing
* Cleanup
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Begin unstaleling
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Update block gen util to v0.9 changes
* Prepare benchmarks to use marshalled files
* Cleanup block gen tool some more
* split up into file generation and benchmarking
* Remove logrus
* Merge branch 'master' into new-benchmarks
* Get benchmarks work, start work on process epoch benchmark
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Merge branch 'new-benchmarks' of https://github.com/0xKiwi/prysm into new-benchmarks
* All benchmarks working
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Fix after merge
* Cleanup
* Add bazel target
* Added TestBenchmarkExecuteStateTransition_WithCache
* Change tests to use SSZ and begin making binary
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Merge branch 'new-benchmarks' of https://github.com/0xKiwi/prysm into new-benchmarks
* bazel binary
* Fully change to binary
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Create go_binary to handle benchmark files
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Gofmt
* Remove genesis state from generated files
* Fix tests
* Gazelle
* Fix tests
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Fix block util
* Allow attestations to be in future for block util
* Fix inclusion delay issue
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Finally fix test
* Add README detailing usage and results
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Change test to run with bazel test
* Fix imports
* Merge branch 'master' into new-benchmarks
* Accidentally removed  config change
* Merge branch 'new-benchmarks' of https://github.com/0xKiwi/prysm into new-benchmarks
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into new-benchmarks
* Move to core/state/
* Update readme
* Gazelle
* Remove test for cached block
2019-11-26 18:09:57 +00:00
Jim McDonald
9e45cffabc Move StateInitialized and ChainStarted to state feed (#4084)
* Migrate ChainStarted and StateInitialized to state notifier
* Provide state notifier to powchain service
* Merge remote-tracking branch 'upstream/master'
* Merge remote-tracking branch 'upstream/master'
* Remove commented line
* Merge branch 'master' into master
* Merge branch 'master' into master
* Merge remote-tracking branch 'upstream/master'
* Merge branch 'master' of github.com:mcdee/prysm
* Accept err from HeadState() as non-fatal
* Merge branch 'master' into master
* Merge branch 'master' into master
* Merge branch 'master' into master
2019-11-23 03:35:47 +00:00
Preston Van Loon
0cec0ee6c3 Release --optimize-process-epoch to all (#4080)
* Release optimize-proces-epoch to all
* Merge branch 'master' into deprecate-ff-optimize-process-epoch
* Merge branch 'master' of github.com:prysmaticlabs/prysm into deprecate-ff-optimize-process-epoch
* Merge refs/heads/master into deprecate-ff-optimize-process-epoch
* Merge refs/heads/master into deprecate-ff-optimize-process-epoch
* Merge refs/heads/master into deprecate-ff-optimize-process-epoch
* Merge refs/heads/master into deprecate-ff-optimize-process-epoch
2019-11-22 09:40:53 +00:00
terence tsao
f5cb04012e Aggregator selection from RPC to validator client (#4071)
* Config
* Updated proto
* Updated pool
* Updated RPC
* Updated validator client
* run time works
* Clean ups
* Fix tests
* Visibility
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into aggregator
* Raul's feedback
* Tests for RPC server
* Tests for validator client
* Span
* More tests
* Use go routine for SubmitAggregateAndProof
* Go routines
* Updated comments
* Use array of roles
* Fixed tests
* Build
* Update validator/client/runner.go

Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com>
* Update validator/client/runner.go

Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com>
* If
* Merge branch 'refactor-validator-roles' of https://github.com/prysmaticlabs/prysm into refactor-validator-roles
* Empty
* Feedback
* Merge branch 'master' of https://github.com/prysmaticlabs/prysm into aggregator
* Removed proto/eth/v1alpha1/shard_chain.pb.go?
* Cleaned up
* Revert
* Comments
* Lint
* Comment
* Merge branch 'master' into aggregator
2019-11-22 05:11:38 +00:00
terence tsao
835418d1e3
Fix UpdateCommitteeCache slot (#4074) 2019-11-20 21:29:43 -08:00
terence tsao
fcb663acde Implement aggregation helpers (#4062)
* Aggregation helpers

* Tests

* Config

* Faulty test cases

* Err
2019-11-19 20:24:39 -06:00
Jim McDonald
49c2dd2cfc Move the state notifier to a different module (#4058)
* Move state notifier to statefeed

* Updates to state notifier

* Create state feed in beacon node

* Formatting
2019-11-19 16:15:48 -06:00
Jim McDonald
65d920e13a Add generic state feed (#4004)
* Initial implementation of state feed

* Add instructions on adding new events

* Tidy up log messages

* Tidy up mock

* Update beacon-chain/core/statefeed/events.go

Co-Authored-By: terence tsao <terence@prysmaticlabs.com>

* Update beacon-chain/core/statefeed/events.go

Co-Authored-By: terence tsao <terence@prysmaticlabs.com>

* Remove unused BlockReceivedData

* Rename BlockHash to BlockRoot in BlockProcessedData

* Punctuation

* Use correct root for block processed event

* StateFeeder -> StateNotifier; fix up tests.

* Add Verified flag to BlockProcessed event

* Fix visibility in Bazel
2019-11-19 17:17:41 +08:00
Nishant Das
023dfebc73 Revert "Reverts the Revert (#4011)" (#4026)
This reverts commit c4ca8a47b3.
2019-11-17 22:48:22 -08:00
Raul Jordan
5169209360
Properly Archive Active Set Changes (#4007)
* archiving information properly

* tests passing

* broken test fix
2019-11-15 10:45:02 -06:00
Nishant Das
c4ca8a47b3 Reverts the Revert (#4011)
* Revert "Revert "Change BLS Library to Herumi (#3752)" (#4006)"

This reverts commit 904898e405.

* turn it on

* make all docker images with cgo deps static

* change back

* fix build

* switch back

* address gateway

* fix library again
2019-11-15 10:27:23 -06:00
Nishant Das
904898e405 Revert "Change BLS Library to Herumi (#3752)" (#4006)
This reverts commit 24583864b4.
2019-11-14 13:00:50 -05:00
Nishant Das
24583864b4 Change BLS Library to Herumi (#3752)
* change to herumi's bls

* change alias

* change to better

* add benchmark

* build

* change to bazel fork

* fix prefix

* make it work with library

* update to latest

* change again

* add import

* update to latest

* add sha commit

* new static lib with groups swapped

* using herumis new lib

* fix dep paths in c headers

* update again

* new changes

* fix commit

* fix serialization

* comment

* fix test

* fix to herumis latest version

* fix test

* fix benchmarks

* add new workspace

* change commit and remove init

* get test to pass

* remove parameter

* remove reverse byte order

* make gazelle happy

* set pure to off

* fix failing tests

* remove old ref

* use HashWithDomain functions

* update to latest version

* clean up

* gaz

* add back removed code

* switch off pure
2019-11-14 09:51:42 -06:00
Raul Jordan
48f69c0762 better comment (#3990) 2019-11-13 23:37:23 +08:00
Ivan Martinez
8ba6c84d6b Change GetValidatorParticipation to compute participation from previous epoch (#3988)
* Change ComputeValidatorParticipation to allow previous and current epoch

* Change archive saving back to previous

* Fix forkchoice usage

* whoopsies
2019-11-12 23:17:34 -06:00
terence tsao
81c53c26fb Update run time to v0.9 (#3935)
* Update seed domains (#3872)

* Remove Transfers (#3870)

* Remove active index roots and compact committee roots (#3869)

* Update inclusion reward (#3886)

* Alter proposer selection logic (#3884)

* Fix early committee bias (#3888)

* Remove shards and committees (#3896)

* Epoch spec tests v0.9 (#3907)

* Block spec test v0.9 (#3905)

* rm'ed in protobuf

* build proto

* build proto

* build proto

* fix core package

* Gazelle

* Fixed all the tests

* Fixed static test

* Comment out spec test for now

* One more skip

* fix-roundRobinSync (#3862)

* Starting but need new seed function

* Revert initial sync

* Updated Proposer Slashing

* Fixed all tests

* Lint

* Update inclusion reward

* Fill randao mixes with eth1 data hash

* Test

* Fixing test part1

* All tests passing

* One last test

* Updated config

* Build proto

* Proper skip message

* Conflict and fmt

* Removed crosslinks and shards. Built

* Format and gazelle

* Fixed all the block package tests

* Fixed all the helper tests

* All epoch package tests pass

* All core package tests pass

* Fixed operation tests

* Started fixing rpc test

* RPC tests passed!

* Fixed all init sync tests

* All tests pass

* Fixed blockchain tests

* Lint

* Lint

* Preston's feedback

* Starting

* Remove container

* Fixed block spec tests

* All passing except for block_processing test

* Failing block processing test

* Starting

* Add AggregateAndProof

* All mainnet test passes

* Update deposit contract (#3906)

* Proto spec tests v0.9 (#3908)

* Starting

* Add AggregateAndProof

* Unskip block util tests (#3910)

* rm'ed in protobuf

* build proto

* build proto

* build proto

* fix core package

* Gazelle

* Fixed all the tests

* Fixed static test

* Comment out spec test for now

* One more skip

* fix-roundRobinSync (#3862)

* Starting but need new seed function

* Revert initial sync

* Updated Proposer Slashing

* Fixed all tests

* Lint

* Update inclusion reward

* Fill randao mixes with eth1 data hash

* Test

* Fixing test part1

* All tests passing

* One last test

* Updated config

* Build proto

* Proper skip message

* Conflict and fmt

* Removed crosslinks and shards. Built

* Format and gazelle

* Fixed all the block package tests

* Fixed all the helper tests

* All epoch package tests pass

* All core package tests pass

* Fixed operation tests

* Started fixing rpc test

* RPC tests passed!

* Fixed all init sync tests

* All tests pass

* Fixed blockchain tests

* Lint

* Lint

* Preston's feedback

* Starting

* Remove container

* Fixed block spec tests

* All passing except for block_processing test

* Failing block processing test

* Starting

* Add AggregateAndProof

* All mainnet test passes

* Unskip block util tests

* Slot processing spec test V0.9 (#3912)

* Starting

* Add AggregateAndProof

* Unskip slot processing mainnet test

* Unskip minimal spec test for finalization (#3920)

* Remove outdated interop tests (#3922)

* Rm outdated interop tests

* Rm test runner

* Gazelle

* Update validator to use proposer slot (#3919)

* Fix committee assignment (#3931)

* Replace shard with committee index (#3930)

* Conflict

* Clean up (#3933)

* Remove shard filter in db (#3936)

* Remove lightouse compatibility test (#3939)

* Update Committee Cache for v0.9 (#3948)

* Updated committee cache

* Removed shuffled indices cache

* Started testing run time

* Lint

* Fixed test

* Safeguard against nil head state

* address edge case

* add test

* Fixed TestRoundRobinSync by doubling the epochs

* Unskip TestProtoCompatability (#3958)

* Unskip TestProtoCompatability

* Update WORKSPACE

* Fix minimal config (#3959)

* fix minimal configs

* fix hardcoded value in test

* Simplify verify att time (#3961)

* update readme for deposit contract, regen bindings for vyper 0.1.0b12 (#3963)

* update readme for deposit contract, regen bindings

* medium

* Check nil base state (#3964)

* Copy Block When Receiving it From Sync (#3966)

* copy block

* clone for other service methods too

* Change logging of Bitfield  (#3956)

* change logging of bits

* preston's review

* Unskip Beacon Server Test (#3962)

* run test till the end

* fix up proto message types

* fmt

* resolve broken tests

* better error handling

* fixing new logic to use archived proposer info

* fix up logic

* clip using the max effective balance

* broken build fix with num arg mismatch

* amend archive

* archival logic changed

* rename test

* archive both proposer and attester seeds

* page size 100

* further experiments

* further experimentation, archivedProposerIndex seems wrong

* test passes

* rem log

* fix broken test

* fix test

* gaz

* fix imports

* ethapis
2019-11-11 17:03:43 -05:00
Preston Van Loon
7d1633230d Feature gate all caches (#3894) 2019-11-03 17:20:08 -08:00
Nishant Das
ab92326dfb set bool correctly (#3913) 2019-11-01 09:48:24 -05:00
Nishant Das
49e0ddf861 Cache Skipped Slots (#3879)
* use LRU cache on skip slots

* add a metric

* gaz

* more metrics

* temporary log

* added more logging

* fix

* only update, if its higher than the cache

* only update, if its higher than the cache

* increase the cache size, just in case

* split cache related stuff into its own file

* add feature flag

* clean up

* add new test

* make test proper

* godoc

* preston's review
2019-10-31 13:38:51 -04:00
Preston Van Loon
b2b48c2a4d
Extract state from db (#3889) 2019-10-30 19:05:37 -04:00
Preston Van Loon
1b1e994a80
Set committee assignment deadline to 1 epoch (#3902)
* set deadline to epoch end

* fix vis
2019-10-30 18:35:02 -04:00
Preston Van Loon
2b444ea954 Add feature flag to enable shuffled index cache (#3893)
* add feature flag to enable shuffled index cache

* gaz'
2019-10-30 10:26:05 +08:00
terence tsao
3d1b69e945
Inclusion rewards are based off previous epoch (#3855) 2019-10-28 22:39:16 -07:00
Nishant Das
1c4b4c8393 fix references to use ComputeDomain (#3854) 2019-10-25 08:05:53 -07:00
terence tsao
97905c3e79
Refactor operation service.go into smaller files (#3808)
* Different files

* Preston's feedback
2019-10-21 09:37:50 -07:00
terence tsao
1433fab0d4
Use lowest included slot for precompute (#3805)
* Fix attestations

* Fix new

* Testing state transition

* Fix

* Runtime works
2019-10-18 21:47:54 -07:00
terence tsao
e42af4f11d
Check attestations bitfield overlaps (#3804) 2019-10-18 17:41:27 -07:00
terence tsao
a05dca18c7
Add spec tests for process_slashing and process_justification_finalization (#3803)
* Add precompute spec tests

* Gazelle
2019-10-18 09:09:38 -07:00
terence tsao
a41ac6b498 Implement ProcessEpochPrecompute (#3798)
* Implemented ProcessSlashingsPrecompute

* Tests for ProcessSlashingsPrecompute

* Gaz

* Lint

* Feature flags

* Hook in ProcessEpochPrecompute

* Hook it to spec test

* Seperate test into it's own package to avoid circular dependency

* Lint

* Gazelle

* Preston's feedback

* Nishant's feedback
2019-10-18 16:38:47 +08:00
terence tsao
aedd38092f Process slashing with precompute (#3797)
* Implemented ProcessSlashingsPrecompute

* Tests for ProcessSlashingsPrecompute

* Gaz

* Lint

* Removed comment
2019-10-18 09:10:41 +08:00
terence tsao
89ef6d6648
Process rewards and penalties with precompute (#3793) 2019-10-17 10:54:03 -07:00
terence tsao
921d0a6e7e
Process justification with precompute (#3792) 2019-10-17 02:24:13 -07:00
terence tsao
2d863a1e63
Methods to precompute process_epoch records (#3788) 2019-10-16 17:48:26 -07:00
terence tsao
a62ac97a35
Type for process epoch optimization (#3783) 2019-10-16 11:26:02 -07:00
Preston Van Loon
86a8ec035c
Aggregation helper improvements (#3789)
* aggregation helper

* lint
2019-10-16 11:11:58 -07:00
terence tsao
9bf55e53e7 Add context and tracing to attestation pool (#3737)
* Test case for overlapping aggregation bits

* Add ctx and tracing for attestation pool and beyond

* No nil

* Use real ctx

* Use real ctx

* Fix test

* Fix test

* Fix test

* Fix test

* Fixed imports
2019-10-09 06:42:17 +08:00
terence tsao
46ecbdc997
Tests for process att with overlapping bits (#3734) 2019-10-08 10:52:07 +09:00
Preston Van Loon
4e22f52ab3 Testnet restart and hotfixes (#3736)
* hotfix for round robin, hotfix for grpc recovery

* gofmt

* break

* wrong subtraction

* lint

* testnet fork version update

* ignore relay / DHT protocol not supported error
2019-10-08 07:59:08 +08:00
Preston Van Loon
23764c4640
Abstract verifySlot to helper package (#3731)
* Abstract verifySlot to helper package

* blocks -> slots

* fix test
2019-10-07 16:15:50 +09:00
terence tsao
750bc83369
Clean up feature flag namings (#3715) 2019-10-07 14:11:49 +09:00
Ivan Martinez
14d9a83cda Add Block Generation Util to testutil package (#3674) (#3709) 2019-10-04 16:07:46 -07:00
terence tsao
273871940c Test code in different packages (#3710)
* Moved a few packagesXXX to test_XXX

* Gaz
2019-10-04 15:41:09 +08:00
Nishant Das
4daf62fc28 Fix Validator Activation (#3684) 2019-10-02 13:26:11 -07:00
Ivan Martinez
8bab55d88e Add Block Generation Util to testutil package (#3674)
* Create block generation util in testutil

* Gazelle

* Fix deps

* Fix imports

* Change tests to use config and fix integer division

* Remove logs

* Fix build

* Add test to ensure finalization occurs

* Add check for finalization

* Add comment for incrementing the state

* Fix test

* Fix tes

* Fix testutil us

* Fix tests

* Change var name

* Add regression test for bug with large validator count

* Fix bazel test
2019-10-02 08:15:40 -07:00
Jim McDonald
323ee8dfac SetupInitialDeposits() now returns deposit data roots (#3683) 2019-10-02 10:50:34 +08:00
Raul Jordan
3d37a4e038
Optimize Domain Data RPC Request (#3671)
* gaz

* fix broken build

* fix broken test

* fix broken test
2019-10-01 09:36:36 -05:00
terence tsao
98f3efffea
Add process block with full attestations test (#3676) 2019-09-30 20:41:51 -07:00
Raul Jordan
8d215feb25
Large Prysm Performance Improvements (#3622)
* skip bls verification with a feature flag at runtime

* gazelle

* more bls mocks

* block roots efficiency

* db block roots now does not show up on the flame graphs

* save validator latest votes batch

* batch save att

* misc improvements to pprof

* import

* include validator index cache

* error if no filter criteria

* resolved comments

* build fix

* lint

* remove delay global

* attestation and block test fixes

* preston suggestions

* fix db tests

* fix missing broken tests

* tests pass
2019-09-30 15:45:53 -05:00