* `EpochFromString`: Use already defined `Uint64FromString` function.
* `Test_uint64FromString` => `Test_FromString`
This test function tests more functions than `Uint64FromString`.
* Slashing protection history: Remove unreachable code.
The function `NewKVStore` creates, via `kv.UpdatePublicKeysBuckets`,
a new item in the `proposal-history-bucket-interchange`.
IMO there is no real reason to prefer `proposal` than `attestation`
as a prefix for this bucket, but this is the way it is done right now
and renaming the bucket will probably be backward incompatible.
An `attestedPublicKey` cannot exist without
the corresponding `proposedPublicKey`.
Thus, the `else` portion of code removed in this commit is not reachable.
We raise an error if we get there.
This is also probably the reason why the removed `else` portion was not
tested.
* `NewKVStore`: Switch items in `createBuckets`.
So the order corresponds to `schema.go`
* `slashableAttestationCheck`: Fix comments and logs.
* `ValidatorClient.db`: Use `iface.ValidatorDB`.
* BoltDB database: Implement `GraffitiFileHash`.
* Filesystem database: Creates `db.go`.
This file defines the following structs:
- `Store`
- `Graffiti`
- `Configuration`
- `ValidatorSlashingProtection`
This files implements the following public functions:
- `NewStore`
- `Close`
- `Backup`
- `DatabasePath`
- `ClearDB`
- `UpdatePublicKeysBuckets`
This files implements the following private functions:
- `slashingProtectionDirPath`
- `configurationFilePath`
- `configuration`
- `saveConfiguration`
- `validatorSlashingProtection`
- `saveValidatorSlashingProtection`
- `publicKeys`
* Filesystem database: Creates `genesis.go`.
This file defines the following public functions:
- `GenesisValidatorsRoot`
- `SaveGenesisValidatorsRoot`
* Filesystem database: Creates `graffiti.go`.
This file defines the following public functions:
- `SaveGraffitiOrderedIndex`
- `GraffitiOrderedIndex`
* Filesystem database: Creates `migration.go`.
This file defines the following public functions:
- `RunUpMigrations`
- `RunDownMigrations`
* Filesystem database: Creates proposer_settings.go.
This file defines the following public functions:
- `ProposerSettings`
- `ProposerSettingsExists`
- `SaveProposerSettings`
* Filesystem database: Creates `attester_protection.go`.
This file defines the following public functions:
- `EIPImportBlacklistedPublicKeys`
- `SaveEIPImportBlacklistedPublicKeys`
- `SigningRootAtTargetEpoch`
- `LowestSignedTargetEpoch`
- `LowestSignedSourceEpoch`
- `AttestedPublicKeys`
- `CheckSlashableAttestation`
- `SaveAttestationForPubKey`
- `SaveAttestationsForPubKey`
- `AttestationHistoryForPubKey`
* Filesystem database: Creates `proposer_protection.go`.
This file defines the following public functions:
- `HighestSignedProposal`
- `LowestSignedProposal`
- `ProposalHistoryForPubKey`
- `ProposalHistoryForSlot`
- `ProposedPublicKeys`
* Ensure that the filesystem store implements the `ValidatorDB` interface.
* `slashableAttestationCheck`: Check the database type.
* `slashableProposalCheck`: Check the database type.
* `slashableAttestationCheck`: Allow usage of minimal slashing protection.
* `slashableProposalCheck`: Allow usage of minimal slashing protection.
* `ImportStandardProtectionJSON`: Check the database type.
* `ImportStandardProtectionJSON`: Allow usage of min slashing protection.
* Implement `RecursiveDirFind`.
* Implement minimal<->complete DB conversion.
3 public functions are implemented:
- `IsCompleteDatabaseExisting`
- `IsMinimalDatabaseExisting`
- `ConvertDatabase`
* `setupDB`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/node/node_test.go`.
The rest of this commit consists in minimal slashing protection testing.
* `setupWithKey`: Add `isSlashingProtectionMinimal` argument.
The feature addition is located in `validator/client/propose_test.go`.
The rest of this commit consists in tests wrapping.
* `setup`: Add `isSlashingProtectionMinimal` argument.
The added feature is located in the `validator/client/propose_test.go`
file.
The rest of this commit consists in tests wrapping.
* `initializeFromCLI` and `initializeForWeb`: Factorize db init.
* Add `convert-complete-to-minimal` command.
* Creates `--enable-minimal-slashing-protection` flag.
* `importSlashingProtectionJSON`: Check database type.
* `exportSlashingProtectionJSON`: Check database type.
* `TestClearDB`: Test with minimal slashing protection.
* KeyManager: Test with minimal slashing protection.
* RPC: KeyManager: Test with minimal slashing protection.
* `convert-complete-to-minimal`: Change option names.
Options were:
- `--source` (for source data directory), and
- `--target` (for target data directory)
However, since this command deals with slashing protection, which has
source (epochs) and target (epochs), the initial option names may confuse
the user.
In this commit:
`--source` ==> `--source-data-dir`
`--target` ==> `--target-data-dir`
* Set `SlashableAttestationCheck` as an iface method.
And delete `CheckSlashableAttestation` from iface.
* Move helpers functions in a more general directory.
No functional change.
* Extract common structs out of `kv`.
==> `filesystem` does not depend anymore on `kv`.
==> `iface` does not depend anymore on `kv`.
==> `slashing-protection` does not depend anymore on `kv`.
* Move `ValidateMetadata` in `validator/helpers`.
* `ValidateMetadata`: Test with mock.
This way, we can:
- Avoid any circular import for tests.
- Implement once for all `iface.ValidatorDB` implementations
the `ValidateMetadata`function.
- Have tests (and coverage) of `ValidateMetadata`in
its own package.
The ideal solution would have been to implement `ValidateMetadata` as
a method with the `iface.ValidatorDB`receiver.
Unfortunately, golang does not allow that.
* `iface.ValidatorDB`: Implement ImportStandardProtectionJSON.
The whole purpose of this commit is to avoid the `switch validatorDB.(type)`
in `ImportStandardProtectionJSON`.
* `iface.ValidatorDB`: Implement `SlashableProposalCheck`.
* Remove now useless `slashableProposalCheck`.
* Delete useless `ImportStandardProtectionJSON`.
* `file.Exists`: Detect directories and return an error.
Before, `Exists` was only able to detect if a file exists.
Now, this function takes an extra `File` or `Directory` argument.
It detects either if a file or a directory exists.
Before, if an error was returned by `os.Stat`, the the file was
considered as non existing.
Now, it is treated as a real error.
* Replace `os.Stat` by `file.Exists`.
* Remove `Is{Complete,Minimal}DatabaseExisting`.
* `publicKeys`: Add log if unexpected file found.
* Move `{Source,Target}DataDirFlag`in `db.go`.
* `failedAttLocalProtectionErr`: `var`==> `const`
* `signingRoot`: `32`==> `fieldparams.RootLength`.
* `validatorClientData`==> `validator-client-data`.
To be consistent with `slashing-protection`.
* Add progress bars for `import` and `convert`.
* `parseBlocksForUniquePublicKeys`: Move in `db/kv`.
* helpers: Remove unused `initializeProgressBar` function.
* avoid part path collisions with mem addr entropy
* Regression test
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Improve package documentation.
* `processAttestations`: Improve logging.
* Add `Benchmark_checkSurroundVotes` benchmark.
* Implement `saveChunksToDisk` as remplacement of `saveUpdatedChunks`.
The idea is to open only on DB transaction for all validator chunk indexes instead of
one DB transaction per validator chunk index.
It saves the overhead due to transaction start/stop of the DB.
Result of `Benchmark_checkSurroundVotes`:
- Before this commit: 133 seconds
- After this commit: 5.05 seconds
* `LoadSlasherChunks` and `SaveSlasherChunks`: Batch.
* `loadChunks` ==> `loadChunksFromDisk`
* `updatedChunkByChunkIndex`: Don't update if `latestEpochWritten == currentEpoch `.
* `updatedChunkByChunkIndex`: Load all needed chunks once.
* `latestEpochWritten` ==> `latestEpochUpdated`.
* `checkSurroundVotes`: Dump to disk at most every `25_600` chunks.
* `SaveAttestationRecordsForValidators`: Batch.
* `batchSize`: Use as package const and add comment.
* `getChunk` ==> `getChunkFromDatabase`.
* `loadChunks`: Rename variables.
* `Update`: Use explicit arguments.
* `detect_attestations.go`: Reduce abstraction layers.
* `loadAndUpdateChunks`: Change arguments order.
* `updatedChunkByChunkIndex`: Update all known validators in the chunk.
* `LastEpochWrittenForValidators`: Avoid avoidable `for`loop.
* `chunks.go`: Ensure implementations respect the interface.
* `LastEpochWrittenForValidators`: Stop considering lack of epoch as genesis epoch.
* `updatedChunkByChunkIndex`: Don't update latest updated epoch.
And add a bunch of tests.
* Improve slasher cold boot duration.
Before this commit, on a slasher cold boot (aka, without any db),
the `updatedChunkByChunkIndex` function looped for all validators
AND for all epochs between the genesis epoch and the current epoch.
This could take several dozen of minutes, and it is useless since the
min/max spans are actually a circular buffer with a limited lenght.
Cells of min/max spans can be overwritten (with the same value)
plenty of times.
After this commit, the `updatedChunkByChunkIndex` function loops
for all validators AND AT most 'historyLength' lenght.
Every cell of min/max spans are written AT MOST once.
Time needed for slasher boot goes from `O(nm)` to "only" `O(m)`, where:
- `n` is the number of epochs since the genesis.
- `m` is the number of validators.
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `LastEpochWrittenForValidators`: Use golang if.
* `SaveLastEpochsWrittenForValidators`: Refactor.
* `SaveLastEpochsWrittenForValidators`: Fix when `epochByValIndex` > `batchSize`.
Before this commit, `TestStore_LastEpochWrittenForValidators` works if `validatorsCount <= 10000`
and stops working if `validatorsCount > 10000`.
* Slasher: Detect surrounded votes in multiple batches.
Fixes https://github.com/prysmaticlabs/prysm/issues/13591.
* `Test_processAttestations`: Remove duplicated tests.
* Sort indexed attestations by data root.
* `processAttestations`: Don't return duplicate slashings anymore.
Fix https://github.com/prysmaticlabs/prysm/issues/13592.
* `AttesterDoubleVote`: Rename fields.
* Detect double votes in different batches.
In order to do that:
1. Each attestation of the batch is tested against the other attestations of the batch.
2. Each attestation of the batch is tested against the content of the database.
2. Attestations are saved into the database.
Fixes https://github.com/prysmaticlabs/prysm/issues/13590.
* `helpers.go`: Improve naming consistency.
* `detect_attestations.go`: Improve readability.
* `receive.go`: Add `attsQueueSize` in log message.
* `checkSlashableAttestations`: Improve logging.
`avgBatchProcessingTime` is not displayed any more if not batch is
processed.
* `loadChunks`: Use explicit `chunkKind` and `chunkIndices`.
* `getChunk`: Use specific `chunkIndex` and `chunkKind`.
* `validatorIndicesInChunk` -> `validatorIndexesInChunk`.
* `epochUpdateForValidator`: Use explicit arguments.
* `getChunk`: Change order of arguments.
* `latestEpochWrittenForValidator`: Use `ok` parameter.
So the default value is not any more considered as the absence of
value.
* `applyAttestationForValidator`: Use explicit arguments.
* `updateSpans`: Use explicit arguments.
* `saveUpdatedChunks`: Use explicit arguments.
* `checkSurrounds`: Use explicit arguments.
We see here that, previously, in `checkSlashableAttestations`,
`checkSurrounds` was called with the default value of `slashertypes`: `MinSpan`.
Now, we set it expliciterly at `MinSpan`, which may explicit a bug.
* `epochUpdateForValidator`: Set modified by the function argument first.
* `applyAttestationForValidator`: Set mutated argument `chunksByChunkIdx`first.
* `applyAttestationForValidator`: Rename variables.
* `Test_processQueuedAttestations`: Fix test.
Two tests were actually exactly the same.
* `updateSpans`: Keep happy path in the outer scope.
Even if in this case the "happy" path means slashing.
* `checkSurrounds`: Rename variable.
* `getChunk`: Avoid side effects.
It adds a few lines for callers, but it does not modify any more
arguments and it does what it says: getting a chunk.
* `CheckSlashable`: Flatten.
* `detect_attestations_test.go`: Simplify.
* `CheckSlashable`: Add error log in case of missing attestation.
* `processQueuedAttestations`: Extract a sub function.
So testing will be easier.
* `processAttesterSlashings` and `processProposerSlashings`: Improve.
* `processAttesterSlashings`: Return processed slashings.
* `createAttestationWrapper`: Rename variables.
* `signingRoot` ==> `headerRoot` or `dataRoot`.
Before this commit, there is two typse of `signing root`s floating around.
- The first one is a real signing root, aka a hash tree root computed from an object root and
a domain. This real signing root is the object ready to be signed.
- The second one is a "false" signing root, which is actually just the hash tree root of an object. This object is either the `Data` field of an attestation, or the `Header` field of a block.
Having 2 differents objects with the same name `signing root` is quite confusing.
This commit renames wrongly named `signing root` objects.
* `createAttestationWrapper` => `createAttestationWrapperEmptySig`.
So it's clear for the user that the created attestation wrapper has an empty signature.
* Implement `createAttestationWrapper`.
* `processAttestations`: Return processed attester slashings.
* Test `processAttestations` instead of `processQueuedAttestations`.
By testing `processAttestations` instead of `processQueuedAttestations`, we get rid of a lot of tests fixtures, including the 200 ms sleep.
The whole testing duration is shorter.
* `Test_processAttestations`: Allow multiple steps.
* `Test_processAttestations`: Add double steps tests.
Some new failing tests are commented with a corresponding github issue.
* `NextChunkStartEpoch`: Fix function comment.
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `chunks.go`: Avoid templating log messages.
* `checkSlashableAttestations`: Simplify duration computation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Slasher: Ensure all gorouting are stopped before running `Stop` actions.
Fixes#13550.
In tests, `exitChan` are now useless since waitgroup are used to wait
for all goroutines to be stopped.
* `slasher.go`: Add comments and rename some variables. - NFC
* `detect_blocks.go`: Improve. - NFC
- Rename some variables.
- Add comments.
- Use second element of `range` when possible.
* `chunks.go`: Remove `_`receivers. - NFC
* `validateAttestationIntegrity`: Improve documentation. - NFC
* `filterAttestations`: Avoid `else`and rename variable. - NFC
* `slasher.go`: Fix and add comments.
* `SaveAttestationRecordsForValidators`: Remove unused code.
* `LastEpochWrittenForValidators`: Name variables consistently. - NFC
Avoid mixes between `indice(s)`and `index(es)`.
* `SaveLastEpochsWrittenForValidators`: Name variables consistently. - NFC
* `CheckAttesterDoubleVotes`: Rename variables and add comments. - NFC
* `schema.go`: Add comments. - NFC
* `processQueuedAttestations`: Add comments. - NFC
* `checkDoubleVotes`: Rename variable. - NFC
* `Test_processQueuedAttestations`: Ensure there is no error log.
* `shouldNotBeSlashable` => `shouldBeSlashable`
* `Test_processQueuedAttestations`: Add 2 test cases:
- Same target with different signing roots
- Same target with same signing roots
* `checkDoubleVotesOnDisk` ==> `checkDoubleVotes`.
Before this commit, `checkDoubleVotes` did two tasks:
- Checking if there are any slashable double votes in the input
list of attestations with respect to each other.
- Checking if there are any slashable double votes in the input
list of attestations with respect to our database.
However, `checkDoubleVotes` is called only in
`checkSlashableAttestations`.
And `checkSlashableAttestations` is called only in:
- `processQueuedAttestations`, and in
- `IsSlashableAttestation`
Study of case `processQueuedAttestations`:
---------------------------------------------
In `processQueuedAttestations`, `checkSlashableAttestations`
is ALWAYS called after
`Database.SaveAttestationRecordsForValidators`.
It means that, when calling `checkSlashableAttestations`,
`validAtts` are ALREADY stored in the DB.
Each attestation of `validAtts` will be checked twice:
- Against the other attestations of `validAtts` (the portion of
deleted code)
- Against the content of the database.
One of those two checks is redundent.
==> We can remove the check against other attestations in `validAtts`.
Study of case `Database.SaveAttestationRecordsForValidators`:
----------------------------------------------------------------
In `Database.SaveAttestationRecordsForValidators`,
`checkSlashableAttestations` is ALWAYS called with a list of
attestations containing only ONE attestation.
This only attestaion will be checked twice:
- Against itself, and an attestation cannot conflict with itself.
- Against the content of the database.
==> We can remove the check against other attestations in `validAtts`.
=========================
In both cases, we showed that we can remove the check of attestation
against the content of `validAtts`, and the corresponding test
`Test_checkDoubleVotes_SlashableInputAttestations`.
* `Test_processQueuedBlocks_DetectsDoubleProposals`: Wrap proposals.
So we can add new proposals later.
* Fix slasher multiple proposals false negative.
If a first batch of blocks is sent with:
- validator 1 - slot 4 - signing root 1
- validator 1 - slot 5 - signing root 1
Then, if a second batch of blocks is sent with:
- validator 1 - slot 4 - signing root 2
Because we have two blocks proposed by the same validator (1) and for
the same slot (4), but with two different signing roots (1 and 2), the
validator 1 should be slashed.
This is not the case before this commit.
A new test case has been added as well to check this.
Fixes#13551
* `params.go`: Change comments. - NFC
* `CheckSlashable`: Keep the happy path without indentation.
* `detectAllAttesterSlashings` => `checkSurrounds`.
* Update beacon-chain/db/slasherkv/slasher.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* Update beacon-chain/db/slasherkv/slasher.go
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* `CheckAttesterDoubleVotes`: Keep happy path without indentation.
Well, even if, in our case, "happy path" mean slashing.
* 'SaveAttestationRecordsForValidators': Save the first attestation.
In case of multiple votes, arbitrarily save the first attestation.
Saving the first one in particular has no functional impact,
since in any case all attestations will be tested against
the content of the database. So all but the first one will be
detected as slashable.
However, saving the first one and not an other one let us not
to modify the end to end tests, since they expect the first one
to be saved in the database.
* Rename `min` => `minimum`.
Not to conflict with the new `min` built-in function.
* `couldNotSaveSlashableAtt` ==> `couldNotCheckSlashableAtt`
---------
Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
* backfill service
* fix bug where origin state is never unlocked
* support mvslice states
* use renamed interface
* refactor db code to skip block cache for backfill
* lint
* add test for verifier.verify
* enable service in service init test
* cancellation cleanup
* adding nil checks to configset juggling
* assume blocks are available by default
As long as we're sure the AvailableBlocker is initialized correctly
during node startup, defaulting to assuming we aren't in a checkpoint
sync simplifies things greatly for tests.
* block saving path refactor and bugfix
* fix fillback test
* fix BackfillStatus init tests
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* fix all cases
* update web3signer
* current progress
* fix it finally
* push it back to capella
* remove hard-coded forks
* fix failing tests
* gaz
* fix dumb bug
* fix bad test setup
* change back
* making needed changes to beacon API based on removal of blobsidecar from block contents
* fixing tests and reverting some changes to be addressed later
* fixing generated code from protos
* gaz
* fixing get blob handler and adding blob storage to the blob service
* updating unit tests
* WIP
* wip tests
* got tests passing but needs cleanup
* removing gomod and gosum changes
* fixing more tests
* fixing more tests
* fixing more tests
* gaz
* moving some proto types around
* removing unneeded unit test
* fixing proposer paths
* adding more tests
* fixing more tests
* improving more unit tests
* updating one blob only unit test
* changing arguments of buildBlobSidecar
* reverting a change based on feedback
* terence's review items
* fixing test based on new develop changes
* radek's comments
* addressed more comments from radek
* adding in blobs to test data
* fixing casing in test
* removing extra line
* fixing issue from bad merge
* Update beacon-chain/rpc/eth/beacon/handlers_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update beacon-chain/rpc/eth/beacon/handlers_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update beacon-chain/rpc/eth/beacon/handlers_test.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update beacon-chain/rpc/eth/blob/handlers.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* moving core getblob business logic to blocker based on radek's comment
* fixing mock blocker
* gaz
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Add deletion
* Gaz
* Return on removal
* Test cleanup
* Simply blob deletion
* Add test case to prove that deleting a root that doesn't exist will not return an error
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Add Save blob and tests
* Remove locks
* Remove test cleanup
* Fix go mod
* Cleanup
* Add checksum
* Add file hashing to fileutil
* Move test
* Check data when exists
* Add one more test
* Rename
* Gaz
* Add packaged level comment
* Save full sidecar + reviews
* Use path builder in test
* Use other BlobSidecar
* Cleanup
* Fix gosec
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
