* Improve package documentation.
* `processAttestations`: Improve logging.
* Add `Benchmark_checkSurroundVotes` benchmark.
* Implement `saveChunksToDisk` as remplacement of `saveUpdatedChunks`.
The idea is to open only on DB transaction for all validator chunk indexes instead of
one DB transaction per validator chunk index.
It saves the overhead due to transaction start/stop of the DB.
Result of `Benchmark_checkSurroundVotes`:
- Before this commit: 133 seconds
- After this commit: 5.05 seconds
* `LoadSlasherChunks` and `SaveSlasherChunks`: Batch.
* `loadChunks` ==> `loadChunksFromDisk`
* `updatedChunkByChunkIndex`: Don't update if `latestEpochWritten == currentEpoch `.
* `updatedChunkByChunkIndex`: Load all needed chunks once.
* `latestEpochWritten` ==> `latestEpochUpdated`.
* `checkSurroundVotes`: Dump to disk at most every `25_600` chunks.
* `SaveAttestationRecordsForValidators`: Batch.
* `batchSize`: Use as package const and add comment.
* pending broadcast err if missing blobs and 0 peers
* compute request first for len check
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* `getChunk` ==> `getChunkFromDatabase`.
* `loadChunks`: Rename variables.
* `Update`: Use explicit arguments.
* `detect_attestations.go`: Reduce abstraction layers.
* `loadAndUpdateChunks`: Change arguments order.
* `updatedChunkByChunkIndex`: Update all known validators in the chunk.
* `LastEpochWrittenForValidators`: Avoid avoidable `for`loop.
* `chunks.go`: Ensure implementations respect the interface.
* `LastEpochWrittenForValidators`: Stop considering lack of epoch as genesis epoch.
* `updatedChunkByChunkIndex`: Don't update latest updated epoch.
And add a bunch of tests.
* Improve slasher cold boot duration.
Before this commit, on a slasher cold boot (aka, without any db),
the `updatedChunkByChunkIndex` function looped for all validators
AND for all epochs between the genesis epoch and the current epoch.
This could take several dozen of minutes, and it is useless since the
min/max spans are actually a circular buffer with a limited lenght.
Cells of min/max spans can be overwritten (with the same value)
plenty of times.
After this commit, the `updatedChunkByChunkIndex` function loops
for all validators AND AT most 'historyLength' lenght.
Every cell of min/max spans are written AT MOST once.
Time needed for slasher boot goes from `O(nm)` to "only" `O(m)`, where:
- `n` is the number of epochs since the genesis.
- `m` is the number of validators.
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* retrieve and save blobs during backfill
* Update beacon-chain/sync/backfill/batch.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/sync/backfill/blobs.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update beacon-chain/sync/backfill/metrics.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* make blobSync initialization path a little safer
* use bytes.Equal to avoid extra allocation
* stop using log.Fatal and other little cleanups
* wrap up post blob sync actions in batch mutator
* unit test coverage on verifier
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `LastEpochWrittenForValidators`: Use golang if.
* `SaveLastEpochsWrittenForValidators`: Refactor.
* `SaveLastEpochsWrittenForValidators`: Fix when `epochByValIndex` > `batchSize`.
Before this commit, `TestStore_LastEpochWrittenForValidators` works if `validatorsCount <= 10000`
and stops working if `validatorsCount > 10000`.
* Slasher: Detect surrounded votes in multiple batches.
Fixes https://github.com/prysmaticlabs/prysm/issues/13591.
* `Test_processAttestations`: Remove duplicated tests.
* Sort indexed attestations by data root.
* `processAttestations`: Don't return duplicate slashings anymore.
Fix https://github.com/prysmaticlabs/prysm/issues/13592.
* `AttesterDoubleVote`: Rename fields.
* Detect double votes in different batches.
In order to do that:
1. Each attestation of the batch is tested against the other attestations of the batch.
2. Each attestation of the batch is tested against the content of the database.
2. Attestations are saved into the database.
Fixes https://github.com/prysmaticlabs/prysm/issues/13590.
* `helpers.go`: Improve naming consistency.
* `detect_attestations.go`: Improve readability.
* `receive.go`: Add `attsQueueSize` in log message.
* `checkSlashableAttestations`: Improve logging.
`avgBatchProcessingTime` is not displayed any more if not batch is
processed.
* `loadChunks`: Use explicit `chunkKind` and `chunkIndices`.
* `getChunk`: Use specific `chunkIndex` and `chunkKind`.
* `validatorIndicesInChunk` -> `validatorIndexesInChunk`.
* `epochUpdateForValidator`: Use explicit arguments.
* `getChunk`: Change order of arguments.
* `latestEpochWrittenForValidator`: Use `ok` parameter.
So the default value is not any more considered as the absence of
value.
* `applyAttestationForValidator`: Use explicit arguments.
* `updateSpans`: Use explicit arguments.
* `saveUpdatedChunks`: Use explicit arguments.
* `checkSurrounds`: Use explicit arguments.
We see here that, previously, in `checkSlashableAttestations`,
`checkSurrounds` was called with the default value of `slashertypes`: `MinSpan`.
Now, we set it expliciterly at `MinSpan`, which may explicit a bug.
* `epochUpdateForValidator`: Set modified by the function argument first.
* `applyAttestationForValidator`: Set mutated argument `chunksByChunkIdx`first.
* `applyAttestationForValidator`: Rename variables.
* `Test_processQueuedAttestations`: Fix test.
Two tests were actually exactly the same.
* `updateSpans`: Keep happy path in the outer scope.
Even if in this case the "happy" path means slashing.
* `checkSurrounds`: Rename variable.
* `getChunk`: Avoid side effects.
It adds a few lines for callers, but it does not modify any more
arguments and it does what it says: getting a chunk.
* `CheckSlashable`: Flatten.
* `detect_attestations_test.go`: Simplify.
* `CheckSlashable`: Add error log in case of missing attestation.
* `processQueuedAttestations`: Extract a sub function.
So testing will be easier.
* `processAttesterSlashings` and `processProposerSlashings`: Improve.
* `processAttesterSlashings`: Return processed slashings.
* `createAttestationWrapper`: Rename variables.
* `signingRoot` ==> `headerRoot` or `dataRoot`.
Before this commit, there is two typse of `signing root`s floating around.
- The first one is a real signing root, aka a hash tree root computed from an object root and
a domain. This real signing root is the object ready to be signed.
- The second one is a "false" signing root, which is actually just the hash tree root of an object. This object is either the `Data` field of an attestation, or the `Header` field of a block.
Having 2 differents objects with the same name `signing root` is quite confusing.
This commit renames wrongly named `signing root` objects.
* `createAttestationWrapper` => `createAttestationWrapperEmptySig`.
So it's clear for the user that the created attestation wrapper has an empty signature.
* Implement `createAttestationWrapper`.
* `processAttestations`: Return processed attester slashings.
* Test `processAttestations` instead of `processQueuedAttestations`.
By testing `processAttestations` instead of `processQueuedAttestations`, we get rid of a lot of tests fixtures, including the 200 ms sleep.
The whole testing duration is shorter.
* `Test_processAttestations`: Allow multiple steps.
* `Test_processAttestations`: Add double steps tests.
Some new failing tests are commented with a corresponding github issue.
* `NextChunkStartEpoch`: Fix function comment.
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* `chunks.go`: Avoid templating log messages.
* `checkSlashableAttestations`: Simplify duration computation.
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
