* Show error in logs if passing invalid flags in yaml
The YAML configs pass if an invalid flag is set, this causes frustration
because the flag could have been mispelled causing a huge mystery.
This will strictly check if the flags are deserialized, if a flag
doesn't exist, it will error out as a Fatal error with appropriate
reasoning which flag is wrong.
* Fix review comments to make some of them non-fatal
* Make yaml.TypeError a pointer
* Remove UnmarshalStrict from spectests since they use a different YAML package
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* checkpoint changes
* Update beacon-chain/rpc/validator/status.go
Co-authored-by: Potuz <potuz@potuz.net>
* Update beacon-chain/rpc/validator/status.go
Co-authored-by: Potuz <potuz@potuz.net>
* add in client side tests
* add ordering
* add all new test cases
* gate feature
* handle edge case
* add one more test case
* fatal error
* zahoor's review
* Update validator/client/validator.go
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Update validator/client/validator.go
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* doppelganger not doppleganger
* preston's review
* add in comment
* change comment
* Fix e2e to only run new flags on the current version
* Fix bug where zero byte public keys were always sent in the request when attestation history existed. Still some tests to fix due to another bug in attester protection AttestationHistoryForPubKey.
* go mod tidy, gazelle
* Increase test size
* fix timeout, change size back to small
Co-authored-by: Potuz <potuz@potuz.net>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* regression test added
* better comment
* export edge case
* add case for edge test
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Adding ability to override directory permissions for db backup
* adding new flag to usage.go
* Backup now has its own directory handling method to reduce footprint of prospective change
* removing unneeded test
* fixing build error
* switching to url param for override
* stripping flag
* adding in unit tests
* fixing a test
* minor changes for testing
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Replace github.com/dgrijalva/jwt-go with github.com/form3tech-oss/jwt-go, including security fix
* Fix to include go cast in deps.bzl and go mod
* Add gocast stub for go.mod / deps.bzl trick
* revert some changes
* Remove ignore tag
* gaz
* go mod tidy
* add build exclusion
* Go mod?
* go mod tidy
* more or less feature complete, needs unit tests
* Test support and concomitant refactors
* Resolving requested changes on PR
* minor fix before trying to figure out bazel
* bazel change
* minor changes requested in pr
* fixing build errors
* fixing build errors
* fixing PR feedback
* fixing PR feedback
* resolving nitpicks
* resolving nitpicks
* build failures REEEEE
* tests fail if its not this way, so this way it is
* getting the tests to pass
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Basic build targets for debian packages
* Add comments to config files
* Harden beacon-chain systemd service
* Add install scripts to set users and folders up
* Rename bazel target, fix service files
* Extra service hardening, cleanup install scripts
* Fix linting issues
* //shared:version_file run local only
* gazelle
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Remove gogoproto compiler
* Remove more gogoproto
* Improvements
* Fix gengo
* More scripts
* Gazelle, fix deps
* Fix version and errors
* Fix gocast for arrays
* Fix ethapis
* Fixes
* Fix compile errors
* fix go.mod
* //proto/... builds
* Update for protov2
* temp fix compilation to move on
* Change everything to emptypb.empty
* Add grpc to proto/slashings
* Fix almost all build failures
* Oher build problems
* FIX THIS FUCKING THING
* gaz literally every .bazel
* Final touches
* Final final touches
* Fix proto
* Begin moving proto.Marshal to native
* Fix site_data
* Fixes
* Fix duplicate gateway
* Fix gateway target
* Fix ethapis
* Fixes from review
* Update
* Fix
* Fix status test
* Fix fuzz
* Add isprotoslice to fun
* Change DeepEqual to DeepSSZEqual for proto arrays
* Fix build
* Fix gaz
* Update go
* Fixes
* Fixes
* Add case for nil validators after copy
* Fix cast
* Fix test
* Fix imports
* Go mod
* Only use extension where needed
* Fixes
* Split gateway from gengo
* gaz
* go mod
* Add back hydrated state
* fix hydrate
* Fix proto.clone
* Fies
* Revert "Split gateway from gengo"
This reverts commit 7298bb2054d446e427d9af97e13b8fabe8695085.
* Revert "gaz"
This reverts commit ca952565701a88727e22302d6c8d60ac48d97255.
* Merge all gateway into one target
* go mod
* Gaz
* Add generate v1_gateway files
* run pb again
* goimports
* gaz
* Fix comments
* Fix protos
* Fix PR
* Fix protos
* Update grpc-gateway and ethapis
* Update ethapis and gen-go-cast
* Go tidy
* Reorder
* Fix ethapis
* fix spec tests
* Fix script
* Remove unused import
* Fix fuzz
* Fix gomod
* Update version
* Error if the cloned result is nil
* Handle optional slots
* ADd more empty checks to clone
* Undo fuzz changes
* Fix build.bazel
* Gaz
* Redo fuzz changes
* Undo some eth1data changes
* Update go.mod
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Undo clone beacon state
* Remove gogo proto more and unused v1_gateway
* Add manual fix for nil vals
* Fix gaz
* tidy
* Tidy again
* Add detailed error
* Revert "Add detailed error"
This reverts commit 59bc053dcd59569a54c95b07739d5a379665ec5d.
* Undo varint changes
* Fix nil validators in deposit test
* Commit
* Undo
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* added Czech from go-bip39
* go mod tidy
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* moved validator/rpc/gateway under shared, validator/rpc/web is not visiable to the new shared gateway
* decided to have one lib with two methods if needed, status and stop are the same, however New and Start are not
* moved beacon's gateway to shared and moved the main func to its own /server folder
* goftm
* gofmt
* removed the extra loggin
* reduce visibility
* tighter visibilty of the shared lib for beacon and validator only for now
* fix patterns , ctx needs to be first param
* fix comments
* gofmt
* added enum for the caller id
* added unit test for gateway
* deprecated .WithDialer to .WithContextDialer
* changed the string callerId to uint8, and rearranged the Gateway struct based on the compiler
* fix 1 based on comment
* iota type
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* added the removed RPC delete account to accounts.go and the rpc in proto
* reverted the 3 unit tests namely the failed derived delete, no pub keys provided and the successful imported account with provided public keys; also brought back the createImportedWalletWithAccounts back in wallet_test.go
* strong password defined elsewhere- removed
* Update validator/rpc/accounts_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* added PublicKeys Nil test case
* changed ss to s. ss was a bad name inthe first place
* goimports -w root
* fixed the goimports before running the proto scripts, also changed the deleterequest variable to PublicKeysToDelete
* removed unneeded comment
* added test case for derived, changed delete account to be for both imported and derived
* gofmt
* unrelated files
* unrelated files
* unrelatedfiles restored
* revert unrelated files
* changed the last ss
* adding slashign endpoints
* adding the rpc export and import funcs, still need more testing and add unit tests
* added import slashing unit test
* clean up
* remove less
* Update proto/validator/accounts/v2/web_api.proto
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update proto/validator/accounts/v2/web_api.proto
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update proto/validator/accounts/v2/web_api.proto
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update proto/validator/accounts/v2/web_api.proto
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* camelCase Proto
* update slashing_protection_json
* update proto
* register in validator/rpc/gateway
* removed the server db creation, the validator cannot begin with a null db
* round trip test
* gofmt
* Update validator/rpc/slashing.go
* Update validator/rpc/slashing.go
* Update validator/rpc/slashing.go
* Update validator/rpc/slashing.go
* Update validator/rpc/slashing.go
* Update validator/rpc/slashing.go
* Update validator/rpc/slashing.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* recover wallet rpc support - first attempt
* removed redundant check
* separate createwallet into imported and derived. Recover is derived
* added unit test for recover. Recover is the createWallet for derived
* so proto does CamelCase!
* fixed issues related to unit testing
* expose ValidateMnemonic from accounts to be used by the rpc module
* added Mnemonic25Support and test to ensure it is not et is not empty
* added skipMnemonic25thword support and unit test
* Update proto/validator/accounts/v2/web_api.proto
Proper naming convention
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* ran goimports,changed variable to SkipMnemonic_25ThWord
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Update validator/rpc/wallet.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* fixed variable and text msgs naming convention as per the review
* added unit test for strong password on recover
* Update proto/validator/accounts/v2/web_api.proto
Co-authored-by: Nishant Das <nish1993@hotmail.com>
* Update validator/rpc/wallet.go
Co-authored-by: Nishant Das <nish1993@hotmail.com>
* goimports -w on root proto
* added comments after exposing ValidateMnemonic to by used by rpc.RecoverWallet
* language should be case insensitive
* need to goimports before update protobuf scripts
* Update validator/rpc/wallet.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update validator/rpc/wallet.go
comments need to be consistant
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update validator/rpc/wallet_test.go
consistent comments
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* fix comments
* remove the skipMnemonic from rpc, just check if passphrase is empyt
* defer call to set the writePassword flag on web boarding
* gofmt
* fixed the password write test after recovering a wallet. Needed to have two defers. One to set to true then one back to false
* restore powchain.pb.go and finalized_block_root_container.pb.go
* revert beacon messages.pb.go
* revert unrelated files
* revert unrelated files
* revert unrelated files
* unlreated files
* restored the imports
* Update validator/rpc/wallet_test.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Nishant Das <nish1993@hotmail.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* added the removed RPC delete account to accounts.go and the rpc in proto
* reverted the 3 unit tests namely the failed derived delete, no pub keys provided and the successful imported account with provided public keys; also brought back the createImportedWalletWithAccounts back in wallet_test.go
* strong password defined elsewhere- removed
* Update validator/rpc/accounts_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* added PublicKeys Nil test case
* changed ss to s. ss was a bad name inthe first place
* goimports -w root
* fixed the goimports before running the proto scripts, also changed the deleterequest variable to PublicKeysToDelete
* removed unneeded comment
* added test case for derived, changed delete account to be for both imported and derived
* gofmt
* unrelated files
* unrelated files
* unrelatedfiles restored
* revert unrelated files
* changed the last ss
* restore imports
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Add a tool to split a series of mnemonic keys into distinct wallets
* split func
* keysplit tool
* gaz/viz
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Use a bolt cursor instead of bucket.ForEach, exit early when applicable
* Refactor to reuse the same code
* gofmt
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Update rules_go and fix proto conflicts
* gaz
* Update generated code
* more emptypb fixes
* gaz
Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
* begin a simpler pruning process
* shorter pruning
* pass all pruning tests
* passing tests
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Add metrics per keys for next scheduled attestation and proposal
* Found a better place where to update the counters
* Wrap with emitAccountMetrics flag
* refactor / move waiting for activation updates
* Commentary
* Update test to follow the full code path
* gofmt and goimports
* manual imports fixes
* Apply suggestions from code review
typo fixes
* Remove redundant handleAccountsChanged and chan. Thanks @nisdas
* var sub = to sub :=
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* validator cmd
* imports
* more imports
* e2e viz
* alias
* use native alias
* add actual
* fix macro
* work on fix e2e
* add viz
* gaz
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Make 1 occurence of 'foo', 'bar' and 'fizz!'
* Make 1 occurence of 'merkleizing list that is too large, over limit' string
* Limit password occurrences
* Make only 1 occurence of 'strongPass'
* Limit testMnemonic occurrences
* Limit expected epoch error messages
* Rename errors and use constant
* Update bazel dependencies
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update to go 1.16
* Also update go.mod
* rm go.sum then run go mod tidy
* go mod tidy with go 1.16
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Added ordered option to graffiti file
* Updated validator to use Ordered graffiti
* Track graffiti ordered index in db
* Update `ordered` to only emit each graffiti once
Co-authored-by: pinglamb <pinglambs@gmail.com>
* initial implementation
* remove listening for changes from wallet creation
* goimports
* test fix
* more goimports
* listen for changes when initializing wallet through gRPC
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Use ValidtorIndex across Prysm. Build ok
* First take at fixing tests
* Clean up e2e, fuzz... etc
* Fix new lines
* Update beacon-chain/cache/proposer_indices_test.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update beacon-chain/core/helpers/rewards_penalties.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update beacon-chain/core/helpers/shuffle.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update validator/graffiti/parse_graffiti_test.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Raul's feedback
* Fix downcast int -> uint64
* Victor's feedback
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Use types.CommitteeIndex
* Go fmt
* Update validator pkg
* Fix e2e
* Happy fuzz tests
* Sync with upstream ethereumapi
* Go mod tidy
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* include migration and logic for stopping early in slashing protection checks
* remove commented code
* extract methods
* migration logic tested up
* migration up and down tests
* Update validator/db/kv/attester_protection.go
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* added in pruning and batched migrations
Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* Add a debug log to show duration
* merge from dev
* use safe pending attestation records struct
* fix build, use atomic bool
* Add deadline checks to CheckSlashableAttestation
* Go fmt
* Add test for in-progress log
* GoDocs
* Rename pending attestation records to queued attestation records
* rename and add commentary on log
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Revert "Rename `NewService` to `New` (#8337)"
This reverts commit d121b19145.
# Conflicts:
# beacon-chain/sync/initial-sync/round_robin_test.go
* fix name in test
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Add a debug log to show duration
* Autofix issues in 1 file
Resolved issues in validator/db/kv/attester_protection.go via DeepSource Autofix
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* add programmatic voluntary exit
* add exit all flag
* test
* lint
* add multiple exits test
* fix test
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* add in batch method
* add in new proposal history methods for efficiency and progress bars
* tests fixed to use the new methods
* add back get slot proposing history method
* add gaz
* process grpc header flag for exit
* ran bazel //:gazelle
* Add regression test
* Re-run "bazel run //:gazelle"
* whitespace
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* gofmt
Co-authored-by: Danny Joyce <djoyce@bisontrails.co>
Co-authored-by: Danny Joyce <djoyahoy@users.noreply.github.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Change LowestSignedProposal to Also Return a Boolean for Slashing Protection (#8020)
* amend to use bools
* ineff assign
* comment
* Update `LowestSignedTargetEpoch` to include exists (#8004)
* Replace highest with lowerest
* Update validator/db/kv/attestation_history_v2.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update validator/db/kv/attestation_history_v2.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Invert equality for saveLowestSourceTargetToDB
* Add eip checks to ensure epochs cant be lower than db ones
* Should be less than equal to
* Check if epoch exists in DB getters
* Revert run time checks
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Export Attesting History for Slashing Interchange Standard (#8027)
* added in att history checks
* logic for export
* export return nil
* test for export atts
* round trip passes first try!
* rem println
* fix up tests
* pass test
* Validate Proposers Are Not Slashable With Regard to Data Within Slasher Interchange JSON (#8031)
* filter slashable blocks and atts in same json stub
* add filter blocks func
* add test for filtering out the bad public keys
* Export Slashing Protection History Via CLI (#8040)
* include cli entrypoint for history exports
* builds properly
* test to confirm we export the data as expected
* abstract helpers properly
* full test suite
* gaz
* better errors
* marshal ident
* Add the additional eip-3076 attestation checks (#7966)
* Replace highest with lowerest
* Update validator/db/kv/attestation_history_v2.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update validator/db/kv/attestation_history_v2.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Invert equality for saveLowestSourceTargetToDB
* Add eip checks to ensure epochs cant be lower than db ones
* Should be less than equal to
* Check if epoch exists in DB getters
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Add EIP-3076 Invariants for Proposer Slashing Protection (#8067)
* add invariant for proposer protection
* write different test cases
* pass tests
* Add EIP-3076 Interchange JSON CLI command to validator (#7880)
* Import JSON CLI
* CLI impotr
* f
* Begin adding new commands in slashing protection
* Move testing helpers to separate packae
* Add command for importing slashing protection JSONs
* fix import cycle
* fix test
* Undo cleaning changes
* Improvements
* Add better prompts
* Fix prompt
* Fix
* Fix
* Fix
* Fix conflict
* Fix
* Fixes
* Fixes
* Fix exported func
* test func
* Fixes
* fix test
* simplify import and standardize with export
* add round trip test
* true integration test works
* fix up comments
* logrus
* better error
* fix build
* build fix
* Update validator/slashing-protection/cli_export.go
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Update validator/slashing-protection/cli_import.go
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* fmt
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
* Filter Slashable Attester Public Keys in Slashing Interchange Import (#8051)
* filter slashable attesters from the same JSON
* builds
* fix up initially broken test
* circular dep
* import fix
* giz
* added in attesting history package
* add test for filter slashable attester keys
* pass tests
* Save Slashable Keys to Disk in the Validator Client (#8082)
* begin db funcs
* add in test and bucket
* gaz
* rem changes to import
* ineff assign
* add godoc
* Properly Handle Duplicate Public Key Entries in Slashing Interchange Imports (#8089)
* Prevent Blacklisted Public Keys from Slashing Protection Imports from Having Duties at Runtime (#8084)
* tests on update duties
* ensure the slashable public keys are filtered out from update duties via test
* begin test
* attempt test
* rename for better context
* pass tests
* deep source
* ensure tests pass
* Check for Signing Root Mismatch When Submitting Proposals and Importing Proposals in Slashing Interchange (#8085)
* flexible signing root
* add test
* add tests
* fix test
* Preston's comments
* res tests
* ensure we consider the case for minimum proposals
* pass test
* tests passing
* rem unused code
* Set Empty Epochs in Between Attestations as FAR_FUTURE_EPOCH in Attesting History (#8113)
* set target data
* all tests passing
* ineff assign
* signing root
* Add Slashing Interchange, EIP-3076, Spec Tests to Prysm (#7858)
* Add interchange test framework
* add checks for attestations
* Import genesis root if necessary
* flexible signing root
* add test
* Sync
* fix up test build
* only 3 failing tests now
* two failing
* attempting to debug problems in conformity tests
* include latest changes
* protect test in validator/client passing
* pass tests
* imports
* spec tests passing with bazel
* gh archive link to spectests using tar.gz suffix
* rev
* rev more comment changes
* fix sha
* godoc
* add back save
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Implement Migration for Unattested Epochs in Attesting History Database (#8121)
* migrate attesting history backbone done
* begin migration logic
* implement migration logic
* migration test
* add test
* migration logic
* bazel
* migration to its own file
* Handle empty blocks and attestations in interchange json and sort interchange json by public key (#8132)
* Handle empty blocks and attestations in interchange json
* add test
* sort json
* easier empty arrays
* pass test
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* builds
* more tests finally build
* Align Slashing Interchange With Optimized Slashing Protection (#8268)
* attestation history should account for multiple targets per source
* attempt at some fixes
* attempt some test fixes
* experimenting with sorting
* only one more failing test
* tests now pass
* slash protect tests passing
* only few tests now failing
* only spec tests failing now
* spec tests passing
* all tests passing
* helper function for verifying double votes
* use helper
* gaz
* deep source
* tests fixed
* expect specific number of times for domain data calls
* final comments
* Batch Save Imported EIP-3076 Attestations (#8304)
* optimize save
* test added
* add test for sad path
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* revert bad find replace
* add comment to db func
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Ivan Martinez <ivanthegreatdev@gmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: Shay Zluf <thezluf@gmail.com>
* Revert "Revert New Attester Protection DB Logic (#8237)"
This reverts commit 6738fa3493.
* Batch Attestation Records and Flush All at Once in Validator DB (#8243)
* begin flushing logic
* finalize logic before starting tests
* make code DRY
* better log fields
* gaz
* tweak parameter
* rename
* clarifying comment on error handling in event feed
* comprehensive tests
* more comments
* explain parameters in comments
* renamed consts
* Apply suggestions from code review
* gaz
* simplify
* typo
* comments
* Disallow request epoch to be out of bound
* Remove debug log
* Revert "Disallow request epoch to be out of bound"
This reverts commit d2dc7db594a8565ee6a613d1ee8cd0bcf84c72ff.
* Revert "Optimize Migration for new Attester Protection DB (#8231)"
This reverts commit c4ab67832f.
* Revert "Integrate New Slashing Protection DB Methods at Runtime (#8219)"
This reverts commit 3858068201.
* Revert "DB Migration for Optimal Local Slashing Protection (#8212)"
This reverts commit dd3ac6c2ed.
* integrate at runtime and revamp tests
* historical att
* Update validator/db/iface/interface.go
* deepsource
* import
* log the slashing kind
* gaz
* create a slashutils
* integrate new slashutil
* imports
* pruning and begin test
* comprehensive pruning tests
* add pruning on startup
* also prune source epochs bucket
* more testing
* greatly simplify pruning function
* pruning logic and comprehensive tests in
* att protection test
* gaz
* fix sneaky change
* rev
* documented and tested helper func
* Show attesting and proposing counts in "Next duty" log output
* Use map to store all the counts instead of magic reset
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
* begin migration logic
* wrote migration logic
* begin test file
* test for migration working
* gaz
* progressutil
* migration works even if partial data was written
* configurable keystores debounce interval
* use a time duration flag instead
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Add new GetVersion API to Health Service
This is to support showing version information in the web ui.
Since health.go is built through validator bazel, we can use
`shared.GetVersion` directly.
Backend for: prysmaticlabs/prysm-web-ui#107
* Run goimports and update-go-pbs
* go mod tidy
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* implement validator logs stream
* fix test
* tidy
* proto regen
* add logs stream to the beacon node
* beacon logs working
* impl
* pass test
* gaz
* rem lock
* fix space
* Don't use fraction, use different fields
* Update validator/client/validator.go
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Go fmt
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* small fixes in logs and errors
* fix error message in test
* found a period hiding in an error message
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* patch beacon node
* make sure that check is executed before anything else
* fix format
* make sure that check is reusable by other executables
* debug infor
* fix typo
* more debug info
* use copydir
* cleanup
* better explanation
* gazelle
* go fmt
* debug tos permissions
* upadte copydir
* gazelle
* better check of tos acceptance
* expand path
* update validator
* move fixing func
* move fixing method
* make sure that updater works both on main and subcommands
* remove from startnode
* add copydir test
* add DirFiles method and tests
* fix test
* add and test HashDir
* update tests
* fix test
* add datadir removal
* update messages
* further update messages
* att sign validations
* eliminate old cached methods and use a simple approach in the db
* redefined db methods
* db package builds
* add multilock to attest and propose
* gaz
* removed concurrency tests that are no longer relevant
* add cache to db functions for attesting history checks
* passing
* add in feature flag --disable-attesting-history-db-cache
* remove lock
* Revert "remove lock"
This reverts commit b1a65020e406b9fa6e4f57c5cc6a5a0b2a11a240.
* comment
* gaz
The reason of why a validator input was failing was very generic,
this allows the actual implementation to state why it failed by
passing the exception forward.
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
* Update waiting validator methods to be context aware
* Enable debug logging for validator in e2e
* invert logic in deferred function, remove for loop
* return early if wait is 0 or less
* overwrite ctx
* t.Stop() chan closure is hanging. Doesnt make sense, so ignoring that cleanup task. It shouldnt happen at runtime anyway
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* restore beacon node db
* revert image name
* move restore out of the kv folder
* remove files from kv folder
* go mod tidy
* Remove usage of prometheus testutil
* add yes/no to prompt text
* restore slasher db
* organize imports
* go mod tidy
* restore validator db
* close slasher db
* defer close backup db in tests
* simplify function literal
* fix(grpcHeaders): accept values with equal signs
# What
Before this commit, it was not possible to pass in base64-encoded
content as a header value if it contained an equals sign. This commit
changes the behavior slightly. Rather than ignore key/value pairs where
the value happens to have an equals sign, we assume the first equals
sign delimits the key from the value and pass in the rest of the value
as-is.
Also, instead of printing the header name along with its value, we
print the name, so there is less risk of leaking information into logs
that shouldn't be there.
# Testing
This has not been tested in the context of the full validator client.
Instead, a small example was made to demonstrate the feasibility. The
example is shown here:
```go
package main
import (
"log"
"os"
"strings"
"github.com/davecgh/go-spew/spew"
"github.com/urfave/cli/v2"
)
type Config struct {
GrpcHeadersFlag string
}
func main() {
app := &cli.App{
Action: func(c *cli.Context) error {
for _, hdr := range strings.Split(c.String("grpc-headers"), ",") {
if hdr != "" {
ss := strings.Split(hdr, "=")
spew.Dump(ss[0])
spew.Dump(strings.Join(ss[1:], "="))
}
}
return nil
},
Flags: []cli.Flag{
&cli.StringFlag{
Name: "grpc-headers",
Usage: "A comma-separated list of key value pairs to pass as gRPC headers for all gRPC " +
"calls. Example: --grpc-headers=key=value",
},
},
}
err := app.Run(os.Args)
if err != nil {
log.Fatal(err)
}
}
```
Example invocation:
```command
❯ go run main.go --grpc-headers=key=value,Authorization="Basic $(echo -n hello:world | base64)"
(string) (len=3) "key"
(string) (len=5) "value"
(string) (len=13) "Authorization"
(string) (len=22) "Basic aGVsbG86d29ybGQ="
```
* Adds tests to new gRPC header parsing code
* bazel run //:gazelle
* Log attesting total
* Use the right library
* Go fmt
* Use fmt
* No space is better
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
