Commit Graph

36 Commits

Author SHA1 Message Date
Victor Farazdagi
59ee339497
Move TestMain into dedicated file (#8779)
* Move TestMain into dedicated file

* update db_test

* formatting
2021-04-16 16:25:28 +00:00
Raul Jordan
d2b1115f46
Feature Flag for Enabling Slashing Protection Pruning (#8632) 2021-03-19 17:54:47 +00:00
Josh Yudaken
f0eb843138
Graffiti ordered index (#8482)
* Added ordered option to graffiti file

* Updated validator to use Ordered graffiti

* Track graffiti ordered index in db

* Update `ordered` to only emit each graffiti once

Co-authored-by: pinglamb <pinglambs@gmail.com>
2021-02-24 22:50:47 +00:00
Raul Jordan
4f9752bb3e
Stop Early in Validator Surround Vote Local Protection (#8460)
* include migration and logic for stopping early in slashing protection checks

* remove commented code

* extract methods

* migration logic tested up

* migration up and down tests

* Update validator/db/kv/attester_protection.go

Co-authored-by: terence tsao <terence@prysmaticlabs.com>

* added in pruning and batched migrations

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
2021-02-17 19:23:59 +00:00
Preston Van Loon
558b16275d
Validator: Annotate attestation error spans (#8451)
* Annotate attestation spans

* Annotate attestation spans

* gaz

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2021-02-16 09:06:52 +00:00
Preston Van Loon
e2c5ae53e7
Validator: Safer pending attestation records flushing (#8433)
* Add a debug log to show duration

* merge from dev

* use safe pending attestation records struct

* fix build, use atomic bool

* Add deadline checks to CheckSlashableAttestation

* Go fmt

* Add test for in-progress log

* GoDocs

* Rename pending attestation records to queued attestation records

* rename and add commentary on log

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2021-02-12 20:19:01 +00:00
Victor Farazdagi
a8e501b3cf
ETH2 Types: Epoch (#8373)
* update deps

* update deps

* update protos/*

* update deps

* reset protos

* update protos

* update shared/params/config

* update protos

* update /shared

* update shared/slotutil and shared/testutil

* update beacon-chain/core/helpers

* updates beacon-chain/state

* update beacon-chain/forkchoice

* update beacon-chain/blockchain

* update beacon-chain/cache

* update beacon-chain/core

* update beacon-chain/db

* update beacon-chain/node

* update beacon-chain/p2p

* update beacon-chain/rpc

* update beacon-chain/sync

* go mod tidy

* make sure that beacon-chain build suceeds

* go fmt

* update e2e tests

* update slasher

* remove redundant alias

* update validator

* gazelle

* fix build errors in unit tests

* go fmt

* update deps

* update fuzz/BUILD.bazel

* fix unit tests

* more unit test fixes

* fix blockchain UTs

* more unit test fixes
2021-02-09 10:05:22 +00:00
Raul Jordan
92932ae58e
[Feature] - Slashing Interchange Support (#8024)
* Change LowestSignedProposal to Also Return a Boolean for Slashing Protection (#8020)

* amend to use bools

* ineff assign

* comment

* Update `LowestSignedTargetEpoch` to include exists (#8004)

* Replace highest with lowerest

* Update validator/db/kv/attestation_history_v2.go

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Update validator/db/kv/attestation_history_v2.go

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Invert equality for saveLowestSourceTargetToDB

* Add eip checks to ensure epochs cant be lower than db ones

* Should be less than equal to

* Check if epoch exists in DB getters

* Revert run time checks

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Export Attesting History for Slashing Interchange Standard (#8027)

* added in att history checks

* logic for export

* export return nil

* test for export atts

* round trip passes first try!

* rem println

* fix up tests

* pass test

* Validate Proposers Are Not Slashable With Regard to Data Within Slasher Interchange JSON (#8031)

* filter slashable blocks and atts in same json stub

* add filter blocks func

* add test for filtering out the bad public keys

* Export Slashing Protection History Via CLI (#8040)

* include cli entrypoint for history exports

* builds properly

* test to confirm we export the data as expected

* abstract helpers properly

* full test suite

* gaz

* better errors

* marshal ident

* Add the additional eip-3076 attestation checks (#7966)

* Replace highest with lowerest

* Update validator/db/kv/attestation_history_v2.go

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Update validator/db/kv/attestation_history_v2.go

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Invert equality for saveLowestSourceTargetToDB

* Add eip checks to ensure epochs cant be lower than db ones

* Should be less than equal to

* Check if epoch exists in DB getters

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Add EIP-3076 Invariants for Proposer Slashing Protection (#8067)

* add invariant for proposer protection

* write different test cases

* pass tests

* Add EIP-3076 Interchange JSON CLI command to validator (#7880)

* Import JSON CLI

* CLI impotr

* f

* Begin adding new commands in slashing protection

* Move testing helpers to separate packae

* Add command for importing slashing protection JSONs

* fix import cycle

* fix test

* Undo cleaning changes

* Improvements

* Add better prompts

* Fix prompt

* Fix

* Fix

* Fix

* Fix conflict

* Fix

* Fixes

* Fixes

* Fix exported func

* test func

* Fixes

* fix test

* simplify import and standardize with export

* add round trip test

* true integration test works

* fix up comments

* logrus

* better error

* fix build

* build fix

* Update validator/slashing-protection/cli_export.go

Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>

* Update validator/slashing-protection/cli_import.go

Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>

* fmt

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>

* Filter Slashable Attester Public Keys in Slashing Interchange Import (#8051)

* filter slashable attesters from the same JSON

* builds

* fix up initially broken test

* circular dep

* import fix

* giz

* added in attesting history package

* add test for filter slashable attester keys

* pass tests

* Save Slashable Keys to Disk in the Validator Client (#8082)

* begin db funcs

* add in test and bucket

* gaz

* rem changes to import

* ineff assign

* add godoc

* Properly Handle Duplicate Public Key Entries in Slashing Interchange Imports (#8089)

* Prevent Blacklisted Public Keys from Slashing Protection Imports from Having Duties at Runtime (#8084)

* tests on update duties

* ensure the slashable public keys are filtered out from update duties via test

* begin test

* attempt test

* rename for better context

* pass tests

* deep source

* ensure tests pass

* Check for Signing Root Mismatch When Submitting Proposals and Importing Proposals in Slashing Interchange (#8085)

* flexible signing root

* add test

* add tests

* fix test

* Preston's comments

* res tests

* ensure we consider the case for minimum proposals

* pass test

* tests passing

* rem unused code

* Set Empty Epochs in Between Attestations as FAR_FUTURE_EPOCH in Attesting History (#8113)

* set target data

* all tests passing

* ineff assign

* signing root

* Add Slashing Interchange, EIP-3076, Spec Tests to Prysm (#7858)

* Add interchange test framework

* add checks for attestations

* Import genesis root if necessary

* flexible signing root

* add test

* Sync

* fix up test build

* only 3 failing tests now

* two failing

* attempting to debug problems in conformity tests

* include latest changes

* protect test in validator/client passing

* pass tests

* imports

* spec tests passing with bazel

* gh archive link to spectests using tar.gz suffix

* rev

* rev more comment changes

* fix sha

* godoc

* add back save

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Implement Migration for Unattested Epochs in Attesting History Database (#8121)

* migrate attesting history backbone done

* begin migration logic

* implement migration logic

* migration test

* add test

* migration logic

* bazel

* migration to its own file

* Handle empty blocks and attestations in interchange json and sort interchange json by public key (#8132)

* Handle empty blocks and attestations in interchange json

* add test

* sort json

* easier empty arrays

* pass test

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* builds

* more tests finally build

* Align Slashing Interchange With Optimized Slashing Protection (#8268)

* attestation history should account for multiple targets per source

* attempt at some fixes

* attempt some test fixes

* experimenting with sorting

* only one more failing test

* tests now pass

* slash protect tests passing

* only few tests now failing

* only spec tests failing now

* spec tests passing

* all tests passing

* helper function for verifying double votes

* use helper

* gaz

* deep source

* tests fixed

* expect specific number of times for domain data calls

* final comments

* Batch Save Imported EIP-3076 Attestations (#8304)

* optimize save

* test added

* add test for sad path

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* revert bad find replace

* add comment to db func

Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Ivan Martinez <ivanthegreatdev@gmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: Shay Zluf <thezluf@gmail.com>
2021-01-22 17:12:22 -06:00
Raul Jordan
4c19e622cd
Implement Migration Up/Down Logic for Validator DB (#8271)
* rollback logic

* implement up down logic

* begin down migration tests

* rollback works

* unset test

* remove iface

* gaz

* add comment

* fix ineff assign

* preston comment

* add progress
2021-01-15 15:35:21 -06:00
Raul Jordan
470d5aa491
Cleanup Attester DB Protection Files and Add Back Save Lowest Epochs Functionality (#8232)
* cleanup for att protection

* rename to deprecated

* rem old item

* imports
2021-01-12 09:32:13 -06:00
Raul Jordan
d2bd954a6c
Remove Snappy from Validator Slashing Protection DB (#8248)
* no snappy

* Update validator/db/kv/migration_optimal_attester_protection.go

* fmt

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2021-01-12 01:10:11 +00:00
Raul Jordan
d97596348e
Add Back New Attester Protection DB Logic (#8242)
* Revert "Revert New Attester Protection DB Logic (#8237)"

This reverts commit 6738fa3493.

* Batch Attestation Records and Flush All at Once in Validator DB (#8243)

* begin flushing logic

* finalize logic before starting tests

* make code DRY

* better log fields

* gaz

* tweak parameter

* rename

* clarifying comment on error handling in event feed

* comprehensive tests

* more comments

* explain parameters in comments

* renamed consts

* Apply suggestions from code review

* gaz

* simplify

* typo

* comments
2021-01-11 23:59:17 +00:00
Radosław Kapka
5fd03f8fb0
Unify the pattern of using a package-level logger (#8245)
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
2021-01-11 20:03:28 +00:00
Preston Van Loon
97320a0a8e
Remove snappy compression migration for attestation history (#8238)
* Remove snappy compression migration for attestation history

* gofmt

* remove more snappy stuff

* revert validator/db/kv/historical_attestations.go
2021-01-11 09:28:34 -08:00
Raul Jordan
6738fa3493
Revert New Attester Protection DB Logic (#8237)
* Revert "Optimize Migration for new Attester Protection DB (#8231)"

This reverts commit c4ab67832f.

* Revert "Integrate New Slashing Protection DB Methods at Runtime (#8219)"

This reverts commit 3858068201.

* Revert "DB Migration for Optimal Local Slashing Protection (#8212)"

This reverts commit dd3ac6c2ed.
2021-01-09 04:00:56 +00:00
Raul Jordan
c4ab67832f
Optimize Migration for new Attester Protection DB (#8231)
* migrate using tx commit

* add improvement

* fixed up test

* rem fmt

* gaz

Co-authored-by: nisdas <nishdas93@gmail.com>
2021-01-08 17:31:54 +00:00
Raul Jordan
3858068201
Integrate New Slashing Protection DB Methods at Runtime (#8219)
* integrate at runtime and revamp tests

* historical att

* Update validator/db/iface/interface.go

* deepsource

* import

* log the slashing kind

* gaz

* create a slashutils

* integrate new slashutil

* imports
2021-01-07 23:30:25 +00:00
Raul Jordan
0d5e2cfb27
Prune Validator Attester Records Older than Weak Subjectivity Periods (#8221)
* pruning and begin test

* comprehensive pruning tests

* add pruning on startup

* also prune source epochs bucket

* more testing

* greatly simplify pruning function

* pruning logic and comprehensive tests in

* att protection test

* gaz

* fix sneaky change

* rev

* documented and tested helper func
2021-01-07 15:40:37 -06:00
Raul Jordan
dd3ac6c2ed
DB Migration for Optimal Local Slashing Protection (#8212)
* begin migration logic

* wrote migration logic

* begin test file

* test for migration working

* gaz

* progressutil

* migration works even if partial data was written
2021-01-06 20:41:31 +00:00
Raul Jordan
c354871762
Optimal Local Slashing Protection DB Schema (#8211)
* add new approach for slashing protection

* benches

* tests passing

* gaz

* all tests passing

* comment
2021-01-06 03:04:46 +00:00
Preston Van Loon
4326cbbf08
Validator database: use snappy compression on encoded attestation history (#8129)
* Remove old buckets, rename new buckets to be canonical

* Add migration and test

* Do the migrations

* Gazelle

* Clearify code, code review

* gofmt
2020-12-16 03:33:04 +00:00
Raul Jordan
1fbfd52e52
Simpler and Safer Attester Slashing Protection (#8086)
* att sign validations

* eliminate old cached methods and use a simple approach in the db

* redefined db methods

* db package builds

* add multilock to attest and propose

* gaz

* removed concurrency tests that are no longer relevant

* add cache to db functions for attesting history checks

* passing

* add in feature flag --disable-attesting-history-db-cache

* remove lock

* Revert "remove lock"

This reverts commit b1a65020e406b9fa6e4f57c5cc6a5a0b2a11a240.

* comment

* gaz
2020-12-11 12:31:35 -06:00
Raul Jordan
14e1f08208
Add Backup Webhooks to All Prysm Services With DBs (#8025)
* integrate backup webhooks

* pass slasher tests

* fix node

* cmd

* gaz

* read test passes

* radek feedback

* added comment

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
2020-12-03 22:28:57 +00:00
terence tsao
1b1b36497f
Add validator bolt db metric collector (#7982)
* Validator db: add metrics collector

* Use the right library

* Go fmt

* Fix tests
2020-11-27 20:03:44 +00:00
Raul Jordan
10857223d0
Delete Dead Code in the Validator Client's Database Package (#7970)
* delete deprecated

* mod

* deep source

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-11-26 16:50:55 +00:00
Ivan Martinez
4dc65c5787
Save GenesisValidatorsRoot from WaitForChainStart (#7855)
* Add GenValRoot dbs

* Test genvalroot

* Fix names

* Add overwrite rejection

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-11-20 18:06:12 +00:00
Victor Farazdagi
2f11e55869
Use t.TempDir() in tests (#7769)
* use t.TempDir()

* remove redundant delete

* simplify setupDB()

* simplify db/testing/setup_db

* fix tests
2020-11-10 22:45:17 +00:00
Shay Zluf
7b0ee3adfe
Use new attestation protection (#7605)
* Use new attestation protection

* tests fixes

* fix tests

* fix comment

* fix TestSetTargetData

* fix tests

* empty history handling

* fix another test

* mock domain request

* fix empty handling

* use far future epoch

* use far future epoch

* migrate data

* copy byte array to resolve sigbus error

* init validator protection on pre validation

* raul feedback

* rename fixes

* nishant feedback

* map with values

* fix tests

* lock and add test

* add and fix concurrency tests

* added tests error msg

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-11-10 14:14:11 +00:00
Raul Jordan
d4c954648c
Prevent Usage of Stdlib File/Dir Writing With Static Analysis (#7685)
* write file and mkdirall analyzers

* include analyzer in build bazel

* comments to the single entrypoint and fix validator references

* enforce 600 for files, 700 for dirs

* pass validator tests

* add to nogo

* remove references

* beaconfuzz

* docker img

* fix up kv issue

* mkdir if not exists

* radek comments

* final comments

* Try to fix file problem

Co-authored-by: Ivan Martinez <ivanthegreatdev@gmail.com>
2020-11-09 14:27:03 -06:00
Shay Zluf
acf201428e
Use new proposal protection format (#7518)
* Use new proposal protection format

* Update comments

* Split and merge with new db

* fix tests

* fix test

* optimize domain

* fix validation

* fix validation

* check import error

* fix e2e

* fix old propose tests add ign block test

* constant secret key

* static test for signing

* test domain

* fix testsplit

* gaz

* gaz

* tidy

* raul feedback

* fix tests

* tidy

* added info log for the migration

* gaz

* Update validator/client/propose_protect.go

Co-authored-by: Nishant Das <nishdas93@gmail.com>

* nishant feedback

* import fix

* fix

* remove propose protection flag

* fix block sign test

Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-10-15 19:35:31 +00:00
Shay Zluf
7b5f71229e
New attestation store for local protection (#7248)
* Update attestation with new marshal unmarshal

* Add db methods to handle new slashing protection schema

* lint

* add tests

* goimports

* Apply suggestions from code review

Co-authored-by: Ivan Martinez <ivanthegreatdev@gmail.com>

* const fix

* fix ineffectual assignments

* goimports

* fix import issue

* victor feedback fixes

* victor feedback fixes

* receiver methods

* receiver methods

* Update validator/db/kv/attestation_history_new.go

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>

* Update validator/db/kv/attestation_history_new.go

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>

* Update validator/db/kv/attestation_history_new.go

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>

* Update validator/db/kv/attestation_history_new.go

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>

* Update validator/db/kv/attestation_history_new_test.go

* type change

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Ivan Martinez <ivanthegreatdev@gmail.com>
Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
2020-10-06 19:59:36 +00:00
Raul Jordan
ba440abe2d
Store Hashed Password to Wallet Path (#7295)
* store wallet pass on creation
* store wallet hash
* save hash to wallet dir
* initialize wallet hash
* gaz
* imports spacing
* remove mentions of hashed pass from db schema
* simplify the wallet hash code
* comment removal
* Merge branch 'master' into fix-password-override
* pass tests
* test fix
* Merge branch 'fix-password-override' of github.com:prysmaticlabs/prysm into fix-password-override
* remove extraneous printfs
* tests passing
* require auth for create wallet
* gaz
* fix signup logic
* Merge refs/heads/master into fix-password-override
* Merge refs/heads/master into fix-password-override
* Merge refs/heads/master into fix-password-override
* Merge refs/heads/master into fix-password-override
* Merge branch 'master' into fix-password-override
2020-09-22 14:49:07 +00:00
Shay Zluf
fcfd828725
Local protection proposal schema update (#7197)
* Local protection schema update
* Merge branch 'master' of github.com:prysmaticlabs/prysm into proposal_protection_schema_update
* fix interface issue
* fix lint
* gaz
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Merge refs/heads/master into proposal_protection_schema_update
* Update validator/db/kv/new_proposal_history_test.go
2020-09-14 01:55:33 +00:00
Raul Jordan
16c34b627f
Add Authentication Functions to Validator RPC (#6968)
* define auth endpoints
* add intercepter with tests
* auth functions
* fix up the auth functions
* add functions for storing and saving the hashed password from the validator db
* validate strong password input and simplify jwt claims
* tests for db funcs
* comments for db funcs
* wrap up the authentication tests
* register auth srv
* use proper db iface package and check if existing password
* fix broken tests and add new test to check if password already exists
* use roughtime
* rlock to check the auth paths
* Merge refs/heads/master into auth-rpc
* Merge refs/heads/master into auth-rpc
* Merge refs/heads/master into auth-rpc
* leave out the stream interceptor
* resolve confs
* Merge branch 'master' into auth-rpc
* confs
* Merge branch 'auth-rpc' of github.com:prysmaticlabs/prysm into auth-rpc
* Merge refs/heads/master into auth-rpc
* Merge refs/heads/master into auth-rpc
* Merge refs/heads/master into auth-rpc
* Merge refs/heads/master into auth-rpc
2020-08-13 20:27:42 +00:00
Victor Farazdagi
0006377aec
Applies assertion funcs to validator tests (#6694)
* Applies assertion funcs to validator/keymanager/v1 tests
* gazelle
* Applies assertion funcs to validator/keymanager/v2 tests
* Applies assertion funcs to validator/misc tests
* Applies assertion funcs to validator/misc tests
* Merge branch 'master' into validator-apply-testutils-assertions
* gazelle
* Merge branch 'master' into validator-apply-testutils-assertions
* Merge branch 'master' into validator-apply-testutils-assertions
* Merge refs/heads/master into validator-apply-testutils-assertions
2020-07-23 01:13:52 +00:00
terence tsao
2a997828a3
Audit best practice feedback - validator DB test folder (#6426)
* Refactor validators db

* Lint

* Lint

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-06-26 21:37:43 -05:00