Commit Graph

8729 Commits

Author SHA1 Message Date
Nishant Das
bfaba378f6
activate deneb (#13311) 2023-12-12 08:04:55 +00:00
Sammy Rosso
3bd116db16
Blob filesystem add pruning at startup (#13253)
* Add Save blob and tests

* Remove locks

* Remove test cleanup

* Fix go mod

* Cleanup

* Add checksum

* Add file hashing to fileutil

* Move test

* Check data when exists

* Add one more test

* Rename

* Gaz

* Add packaged level comment

* Fix block proposals in the REST validator client (#13116)

* Fix block proposals in the REST validator client

* fix graffiti test

* return empty graffiti

* fallback to old endpoints

* logs

* handle 404

* everything passes

* review from James

* log undecoded value

* test fixes and additions

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* fix head slot in log (#13139)

* zig: Update zig to recent main branch commit (#13142)

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Fix Pending Queue Deadline Bug (#13145)

* rearrange deadline

* naming

* Add pruning

* Gaz

* Gaz

* Update pruning

* Cleanup

* Making a mess

* Benchmarking

* Forgot to add the file + fixes

* Fixes

* Pruning from DB fixed

* Add prune by file data

* Fix pruning

* Prune fixes

* Cleanup db blockRoot filter

* Handle file close error

* Fix deletion

* Change read at + remove retentionEpich from bs

* Gaz

* Seperate logic + add detailed comments

* Add tests

* Add retention slot when creating blobStorage

* Fix tests

* Gaz

* Fix testonly import

* Add pruning at startup

* Add nil check

* Fix merge errors

* Fix test

* Fix test

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
2023-12-11 17:08:52 +00:00
terence
7d2ddaee43
Test improvement TestValidateVoluntaryExit_ValidExit (#13313) 2023-12-11 08:01:56 +00:00
terence
122a7782ff
Initialize blob storage for initial sync service (#13312) 2023-12-11 07:52:07 +00:00
terence
9b1b6f9be6
Use verified blob for gossip checks (#13294)
* Use blob verifier for gossip rules

* Fixing tests

* Fix lint

* Mocks

* Trying Kasey's rec

* mock verifier init workaround

* Add more tests

* Reset deneb epoch for exit test

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2023-12-11 00:37:45 +00:00
Delweng
0eb08a4f96
beacon-chain/rpc: use BalanceAtIndex instead of Balances to reduce memory copy (#13279)
* beacon-chain/rpc: use BalanceAtIndex instead of Balances

Signed-off-by: jsvisa <delweng@gmail.com>

* beacon-chain/rpc: stream use BalanceAtIndex is sufficient

Signed-off-by: jsvisa <delweng@gmail.com>

* beacon-chain/rpc: fix commit review

Signed-off-by: jsvisa <delweng@gmail.com>

* beacon-chain/rpc: http2 -> httputil

Signed-off-by: jsvisa <delweng@gmail.com>

---------

Signed-off-by: jsvisa <delweng@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2023-12-10 00:04:45 +00:00
Radosław Kapka
bdfa06ed65
Simplify post-evaluation in Beacon API evaluator (#13309)
* Simplify post-evaluation in Beacon API evaluator

* revert slottime

* remove unused error
2023-12-09 07:54:31 +00:00
Manu NALEPA
a94f2b93e3
filterAndCacheActiveKeys: Stop filtering out exiting validators (#13305)
* `filterAndCacheActiveKeys`: Add test cases

- Validator is in unknown status (to be fitered out)
- Validator is in pending status,
   with activation period > current period (to be filtered out)
- Validator is in pending status,
   with activation period == current period (to be kept)

* `filterAndCacheActiveKeys`: Keep exiting keys

Initially:
-------
If a validator is in exiting (so, with status==EXITING != ACTIVE) state,
it will be filtered out by the `filterAndCacheActiveKeys` function.
The validator won't be registered to the beacon node.

If this exiting validator has to propose a block:
- the block will be proposed using local block building only.
- the fee recipient will be the one set in the beacon node.

(Additionally, if the beacon node Prysm without any
fee recipient defined at the beacon node level, the fee recipient
will default on the `0x00000...` burn address.)

This commit modifies the `filterAndCacheActiveKeys` function
by stopping filtering out exiting validators.
2023-12-09 07:53:08 +00:00
Radosław Kapka
4c47756aed
HTTP endpoints cleanup (#13251)
* remove validation package

* structs cleanup

* merge with apimiddleware removal

* more validation and Bls capitalization

* builder test fix

* use strconv for uint->str conversions

* use DecodeHexWithLength

* use exact param names

* rename http package to httputil

* change conversions to fmt.Sprintf

* handle query paramsd and route variables

* spans and receiver name

* split structs, move bytes helper

* missing ok check

* fix reference to indexed failure

* errors fixup

* add godoc to helper

* fix BLS casing and chainhead ref

* review

* fix import in tests

* gzl
2023-12-08 20:37:20 +00:00
Nishant Das
440841d565
only run it in the middle of an epoch (#13303) 2023-12-08 15:14:01 +00:00
Preston Van Loon
ff99616833
Fix staticcheck violations (#13301)
* Fix violations of sa2002

* Fix violations of sa4005

* Fix violations of sa4010

* Fix violations for sa4023

* Comment on commented static checks
2023-12-08 13:07:52 +00:00
Preston Van Loon
f537a98fcd
Add staticchecks to bazel builds (#13298)
* Update staticcheck to latest

* Add static checks while ignoring for third party / external stuff

* Added a hack to keep go mod happy.

* disable SA2002

* Pin go mod tidy checker image to golang:1.20-alpine
2023-12-08 05:42:55 +00:00
Radosław Kapka
cee38660c7
Gracefully handle unknown validator index in the REST VC (#13296)
* Gracefully handle unknown validaor index in the REST VC

* add apostrophes
2023-12-08 04:30:50 +00:00
james-prysm
481d77bfde
APIs: reusing grpc cors middleware for rest (#13284)
* reusing grpc cors middleware for rest

* addressing radek's comments

* Update api/server/middleware.go

Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>

* fixing to recommended name

* fixing naming

* fixing rename on test

---------

Co-authored-by: Sammy Rosso <15244892+saolyn@users.noreply.github.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-08 04:24:18 +00:00
Nishant Das
590317553c
Support New Subnet Backbone (#13179)
* add in changes

* fix it up

* fix test

* gaz

* lint

* add back

* fix tests

* fix it

* fix tests

* add lib

* fix it
2023-12-08 04:07:48 +00:00
terence
68b7d1009e
Update README.md (#13302) 2023-12-08 04:07:10 +00:00
james-prysm
b5b8825cc8
Beacon API: fix get blob returns 500 instead of empty (#13297)
* fix blob api, should return empty if no indicies were found

* fixing small bug with slice
2023-12-07 22:33:26 +00:00
Justin Traglia
382b8b23c2
Ensure partial blob is deleted if there's an error (#13292)
* Ensure partial blob is deleted if there's an error

* Add debug log if file is removed
2023-12-07 20:52:16 +00:00
kasey
40a3ebab91
initialize sig cache for verification.Initializer (#13295)
* initialize sig cache for verification.Initializer

* gaz

* lint

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2023-12-07 20:14:01 +00:00
james-prysm
83af9a5694
Beacon API: update Deneb endpoints after removing blob signing (#13235)
* making needed changes to beacon API based on removal of blobsidecar from block contents

* fixing tests and reverting some changes to be addressed later

* fixing generated code from protos

* gaz

* fixing get blob handler and adding blob storage to the blob service

* updating unit tests

* WIP

* wip tests

* got tests passing but needs cleanup

* removing gomod and gosum changes

* fixing more tests

* fixing more tests

* fixing more tests

* gaz

* moving some proto types around

* removing unneeded unit test

* fixing proposer paths

* adding more tests

* fixing more tests

* improving more unit tests

* updating one blob only unit test

* changing arguments of buildBlobSidecar

* reverting a change based on feedback

* terence's review items

* fixing test based on new develop changes

* radek's comments

* addressed more comments from radek

* adding in blobs to test data

* fixing casing in test

* removing extra line

* fixing issue from bad merge

* Update beacon-chain/rpc/eth/beacon/handlers_test.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update beacon-chain/rpc/eth/beacon/handlers_test.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update beacon-chain/rpc/eth/beacon/handlers_test.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update beacon-chain/rpc/eth/blob/handlers.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* moving core getblob business logic to blocker based on radek's comment

* fixing mock blocker

* gaz

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2023-12-07 17:37:11 +00:00
Nishant Das
6a45323ab7
only run metrics for canonical blocks (#13289) 2023-12-07 11:03:23 +00:00
kasey
4008ea736f
Verify roblobs (#13245)
* scaffolding for verification package

* WIP blob verification methods

* lock wrapper for safer forkchoice sharing

* more solid cache and verification designs; adding tests

* more test coverage, adding missing cache files

* clearer func name

* remove forkchoice borrower (it's in another PR)

* revert temporary interface experiment

* lint

* nishant feedback

* add comments with spec text to all verifications

* some comments on public methods

* invert confusing verification name

* deep source

* remove cache from ProposerCache + gaz

* more consistently early return on error paths

* messed up the test with the wrong config value

* terence naming feedback

* tests on BeginsAt

* lint

* deep source...

* name errors after failure, not expectation

* deep sooource

* check len()==0 instead of nil so empty lists work

* update test for EIP-7044

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2023-12-07 02:36:25 +00:00
Justin Traglia
4e4fb9ad52
Split blob pruning into two funcs (#13285) 2023-12-06 23:39:02 +00:00
kasey
737e0e0d3a
Use functional options for --blob-retention-epochs (#13283)
* blob retention period functional opts

* missed unstaged change

* missed other init after cleardb

* fix ineffassign

* fix dup import

* config failsafe for tests

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
2023-12-06 20:20:34 +00:00
Potuz
604c82626f
Allow requests for old target roots (#13281) 2023-12-06 17:11:34 +00:00
Nishant Das
e1a3852f08
push up the defaults (#13278) 2023-12-06 16:46:46 +00:00
Preston Van Loon
a40cc40edf
CI: Add merge queue events trigger for github workflows (#13282) 2023-12-06 16:13:13 +00:00
james-prysm
0e3c1d42f6
Beacon API: routes unit test (#13276)
* refactoring to add a routes unit test for rest handlers

* gaz

* updating names for functions

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-06 15:17:13 +00:00
Radosław Kapka
f41e603e5a
Simplify Beacon API evaluator (#13265)
* Simplify Beacon API evaluator

* remove comment

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-06 14:54:33 +00:00
Radosław Kapka
28c3330375
Don't fetch duties for unknown keys (#13269)
* Don't fetch duties for unknown keys

* test
2023-12-06 14:18:45 +00:00
Nishant Das
cb465086e8
Fix Optimistic Sync Evaluator (#13262)
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-06 13:41:47 +00:00
Nishant Das
cdd7958739
Fix Domain Data Caching (#13263)
* fix domain data caching

* fix locking of domain data

* preston's review
2023-12-06 12:04:19 +00:00
Nishant Das
97a522827b
Bump Up Gossip Queue Size (#13277) 2023-12-06 09:49:26 +00:00
Preston Van Loon
b84b795f23
Relax file permissions check on existing directories (#13274) 2023-12-05 19:39:45 -06:00
Sammy Rosso
f40b8583f7
Blob filesystem: prune blobs (#13147)
* Add Save blob and tests

* Remove locks

* Remove test cleanup

* Fix go mod

* Cleanup

* Add checksum

* Add file hashing to fileutil

* Move test

* Check data when exists

* Add one more test

* Rename

* Gaz

* Add packaged level comment

* Fix block proposals in the REST validator client (#13116)

* Fix block proposals in the REST validator client

* fix graffiti test

* return empty graffiti

* fallback to old endpoints

* logs

* handle 404

* everything passes

* review from James

* log undecoded value

* test fixes and additions

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* fix head slot in log (#13139)

* zig: Update zig to recent main branch commit (#13142)

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Fix Pending Queue Deadline Bug (#13145)

* rearrange deadline

* naming

* Add pruning

* Gaz

* Gaz

* Update pruning

* Cleanup

* Making a mess

* Benchmarking

* Forgot to add the file + fixes

* Fixes

* Pruning from DB fixed

* Add prune by file data

* Fix pruning

* Prune fixes

* Cleanup db blockRoot filter

* Handle file close error

* Fix deletion

* Change read at + remove retentionEpich from bs

* Gaz

* Seperate logic + add detailed comments

* Add tests

* Add retention slot when creating blobStorage

* Fix tests

* Gaz

* Fix testonly import

* Fix linter errors

* Fix retentionSlot calculation

* Move + use MaxEpochsToPersistBlobs

* Remove unused ctx

* Prestons suggestion

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>

* Rename

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
2023-12-05 21:07:34 +00:00
Preston Van Loon
a6b6a938de
blobstorage: Improve mkdirall error (#13271) 2023-12-05 19:57:08 +00:00
Brandon Liu
c78d698d89
Add --jwt-id flag (#13218)
* add jwt-id flag

* optimize unit test for jwt-id

* Add jwt-id to help text

* gofmt

---------

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
2023-12-05 19:02:25 +00:00
Nishant Das
705e98e3c3
no need to hash it again (#13261) 2023-12-05 07:59:01 -03:00
kasey
ce2344301c
forkchoice.Getter wrapper with locking wrappers (#13244)
* forkchoice.Getter wrapper with locking wrappers

* comments

* lint

* only expose fast fc getters

* potuz feedback re rlock

* update mocks for new fc method

* appease deepsource

* add missing exported func comment

* yeet errors to make the linter happy

* even more devious _discard

* rm TargetRoot

* derp

* handle nil error in _discard

* deep source

---------

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-04 21:01:39 +00:00
Manu NALEPA
1112e01c06
Make Prysm VC compatible with the version v5.3.0 of the slashing protections interchange tests. (#13232)
* `TestStore_GenesisValidatorsRoot_ReadAndWrite`: Make all test cases independents.

In a test with multiple test cases, each test case should be independents.
(aka: Removing test case `A` should not impact test case `B`)

* `SaveGenesisValidatorsRoot`: Allow to overwrite the genesis validator root if the root is the same.

* `ProposalHistoryForSlot`: Add `signingRootExists`

Currently, it is not possible with `ProposalHistoryForSlot` to know if a
proposal is stored with and `0x00000....` signing root or with an empty
signing root. Both cases result to `proposalExists == true` and
`signingRoot == 0x00000`.

This commit adds a new return boolean: `signingRootExists`.

If a proposal has been saved with a `0x00000...` signing root, then:
- `proposalExists` is set to `true`, and
- `signingRootExists` is set to `true`, and
- `signingRoot` is set to `0x00000...`

If a proposal has been saved with an empty signing root, then:
- `proposalExists` is set to `true`, and
- `signingRootExists` is set to `false`, and
- (`signingRoot` is set to `0x00000...`)

* `ImportStandardProtectionJSON`: When importing EIP-3076 Slashing Protection Interchange Format, do not filter any more slashable keys.
Note: Those keys are still saved into the black-listed public keys list.

There is two reason not to do so:
- The EIP-3076 test cases do not know about Prysm's internal black-listed public keys list.
  Tests will expect, without looking into this internal black-listed public keys list,
  to deny a further signature. If we filter these keys from the DB (even if we keep them
  into the black-listed keys list), then some tests will fail.
- If we import a interchange file containing slashable keys and we filter them, then,
  if we re-export the DB, those slashing offences won't appear in the exported interchange
  file.

* `transformSignedBlocks`: Store an 0-len byte slice

When importing an EIP-3076 interchange format, and when no
signing root is specified into the file, we currently store a
`0x00000.....` signing root.

In such a case, instead storing `0x00000...`, this commit stores
a 0-len byte array, so we can differentiate real `0x000.....` signing
root and no signing-root at all.

* `slashableProposalCheck`: Manage lack of sign root

Currently, `slashableProposalCheck` does not really make a difference
between a `0x0000.....` signing root and a missing signing root.

(Signing roots can be missing when importing an EIP-3076 interchange
file.)

This commit differentiate, for  `slashableProposalCheck`, `0x0000....`
signing root and a missing signing root.

* `AttestationRecord.SigningRoot`: ==> `[]byte`

When importing attestations from EIP-3076 interchange format,
the signing root of an attestation may be missing.

Currently, Prysm consider any missing attestation signing root as
`0x000...`.
However, it may conflict with signing root which really are equal to
`0x000...`.

This commit transforms `AttestationRecord.SigningRoot` from `[32]byte` to
`[]byte`, and change the minimal set of functions (sic) to support this
new type.

* `CheckSlashableAttestation`: Empty signing root

Regarding slashing roots, 2 attestations are slashable, if:
- both signing roots are defined and differs, or
- one attestation exists, but without a signing root

* `filterSlashablePubKeysFromAttestations`: Err sort

Rergarding `CheckSlashableAttestation`, we consider that:
- If slashable == NotSlashable and err != nil, then CheckSlashableAttestation
failed.
- If slashable != NotSlashable, then err contains the reason why the attestation
is slashable.

* `setupEIP3076SpecTests`: Update to `v5.3.0`

This commit:
- Updates the version of EIP-3076 tests to `v.5.2.1`.
- Setups on anti-slashing DB per test case, instead per step.

* `ImportStandardProtectionJSON`: Reduce cycl cmplxt

* `AttestationHistoryForPubKey`: copy signing root

BoltDB documentation specifies:
| Byte slices returned from Bolt are only valid during a transaction.
| Once the transaction has been committed or rolled back then the memory
| they point to can be reused by a new page or can be unmapped
| from virtual memory and you'll see an unexpected fault address panic
| when accessing it.
2023-12-04 17:10:32 +00:00
terence
243bcb03ce
Fix FFG LMD Consistency Check (Option 2) (#13258)
* Fix FFG LMD Consistency Check with TargetRootForSlot

* Add test, removed  implementation

* convert to epoch and fix self target

---------

Co-authored-by: Potuz <potuz@prysmaticlabs.com>
2023-12-04 15:31:18 +00:00
Radosław Kapka
a0ca4a67b0
Remove API Middleware (#13243)
* remove api/gateway/apimiddleware

* fix errors in api/gateway

* remove beacon-chain/rpc/apimiddleware

* fix errors in api/client/beacon

* fix errors in validator/client/beacon-api

* fix errors in beacon-chain/node

* fix errors in validator/node

* fix errors in cmd/prysmctl/validator

* fix errors in testing/endtoend

* fix all other code

* remove comment

* fix tests

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-04 11:55:21 +00:00
Preston Van Loon
b68a4e12aa
Update bazel and other CI improvements (#13246)
* Update bazel to 6.4.0, review flags

* Remove problematic/slow targets

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-01 22:20:54 +00:00
kasey
c010601f3b
Initialize cancellable root context in main.go (#13252)
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-01 21:16:06 +00:00
james-prysm
394bd1786a
HTTP validator API: beacon and account endpoints (#13191)
* fixing squashing changes, migrates beacon , account, and auth endpoints on validator client

* adding accounts endpoints

* fixing tests and query endpoints

* adding auth endpoint and fixing unit tests

* removing unused files and updating node file to skip gRPC

* ineffectual assignment fix

* rolling back a change to fix e2e

* fixing issues with ui

* updating with webui version 2.0.5

* updating package name flag in readme

* removing restore assets functions

* adding nomemcopy flag to see if vulenerability scan passes

* making data non compressed to avoid copy vulnerability

* Update beacon-chain/rpc/eth/shared/structs_validator.go

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* updating site_data, and skipping static analysis on file

* adding back deprecation comment notice

* updating workflows to ignore generated

* addressing radek comments

* missed a conversion

---------

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2023-12-01 20:40:09 +00:00
Potuz
461af4baa6
Add test helpers to produce commitments and proofs (#13242)
* Add test helpers to produce commitments and proofs

* go mod tidy

---------

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2023-12-01 19:51:19 +00:00
Sammy Rosso
7a70305935
Blob filesystem: delete blobs (#13233)
* Add deletion

* Gaz

* Return on removal

* Test cleanup

* Simply blob deletion

* Add test case to prove that deleting a root that doesn't exist will not return an error

---------

Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
2023-12-01 19:03:47 +00:00
Potuz
83ce7e3607
Verify lmd without ancestor (#13250) 2023-12-01 17:31:27 +00:00
Potuz
cf8e554981
track target in forkchoice (#13249) 2023-12-01 16:30:34 +00:00
Nishant Das
59aa978223
Optimize Multivalue Slice For Trie Recomputation (#13238)
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2023-12-01 12:01:28 +01:00