Michael Sproul
f93844e63b
Optimise concurrent block production ( #5368 )
...
* Optimise concurrent block production
2024-03-08 05:15:28 +00:00
João Oliveira
65c4ff0775
remove exit-future ( #5183 )
...
* remove exit-future usage,
as it is non maintained, and replace with async-channel which is already in the repo.
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
2024-02-27 22:12:44 +00:00
Pawan Dhananjay
3ab9d3a84e
Add a cli option for the snapshot cache size ( #5270 )
...
* Add a cli option for snapshot cache size
* Remove junk
* Make snapshot_cache module public
* lint
* Update docs
2024-02-26 05:19:39 +00:00
Michael Sproul
c9702cb0a1
Download checkpoint blobs during checkpoint sync ( #5252 )
...
* MVP implementation (untested)
* update store checkpoint sync test
* update cli help
* Merge pull request #5253 from realbigsean/checkpoint-blobs-sean
Checkpoint blobs sean
* Warn only if blobs are missing from server
* Merge remote-tracking branch 'origin/unstable' into checkpoint-blobs
* Verify checkpoint blobs
* Move blob verification earlier
2024-02-19 02:22:23 +00:00
realbigsean
f21472991d
check the da cache and the attester cache in responding to RPC requests ( #5138 )
...
* check the da cache and the attester cache in responding to RPC requests
* use the processing cache instead
* update comment
* add da cache metrics
* rename early attester cache method
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into check-da-cache-in-rpc-response
* make rustup update run on the runners
* Revert "make rustup update run on the runners"
This reverts commit d097e9bfa84a13b1d7813c03df38e7756fb0bfc5.
2024-02-19 02:22:15 +00:00
ethDreamer
a264afd19f
Verify Versioned Hashes During Optimistic Sync ( #4832 )
...
* Convert NewPayloadRequest to use Reference
* Refactor for Clarity
* Verify Versioned Hashes
* Added Tests for Version Hash Verification
* Added Moar Tests
* Fix Problems Caused By Merge
* Update to use Alloy Instead of Reth Crates (#14 )
* Update beacon_node/execution_layer/src/engine_api/new_payload_request.rs
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
* Faster Versioned Hash Extraction
* Update to rust 1.75 & Pin alloy-consensus
2024-02-18 12:40:45 +00:00
realbigsean
4172d9f75c
Update to consensus spec v1.4.0-beta.6 ( #5094 )
...
* get latest ef tests passing
* fix tests
* Fix invalid payload recovery tests
* Merge branch 'unstable' into update-to-spec-v1.4.0-beta.6
* Revert "fix tests"
This reverts commit 0c875b02e032ebd9bebd822183a1c3d10d333a4c.
* Fix fork choice def. tests
* Update beacon_node/beacon_chain/tests/payload_invalidation.rs
2024-02-08 18:08:21 +00:00
Jimmy Chen
39e9f7dc6b
Fix Rust beta compiler errors (1.77) ( #5180 )
...
* Lint fixes
* More fixes for beta compiler.
* Format fixes
* Move `#[allow(dead_code)]` to field level.
* Remove old comment.
* Update beacon_node/execution_layer/src/test_utils/mod.rs
Co-authored-by: João Oliveira <hello@jxs.pt>
* remove duplicate line
2024-02-05 17:54:11 +00:00
Michael Sproul
0b6416c444
Re-disable block verification tests in debug ( #5155 )
...
* Re-disable block verification tests in debug
2024-02-01 08:35:10 +00:00
Age Manning
7582da7855
Test backfill ( #5109 )
...
* Test backfill
* Revert cargo.toml
* Update beacon_node/beacon_chain/src/builder.rs
Co-authored-by: João Oliveira <hello@jxs.pt>
* Remove redundant code
2024-01-31 18:11:45 +00:00
Lion - dapplion
b035638f9b
Compute recent lightclient updates ( #4969 )
...
* Compute recent lightclient updates
* Review PR
* Merge remote-tracking branch 'upstream/unstable' into lc-prod-recent-updates
* Review PR
* consistent naming
* add metrics
* revert dropping reprocessing queue
* Update light client optimistic update re-processing logic. (#7 )
* Add light client server simulator tests. Co-authored by @dapplion.
* Merge branch 'unstable' into fork/dapplion/lc-prod-recent-updates
* Fix lint
* Enable light client server in simulator test.
* Fix test for light client optimistic updates and finality updates.
2024-01-31 05:25:51 +00:00
Michael Sproul
6f3af67362
Fix off-by-one in backfill sig verification ( #5120 )
...
* Fix off-by-one in backfill sig verification
* Add self-referential PR link
2024-01-30 00:33:01 +00:00
realbigsean
a4fcf60bcc
Increase attestation cache sizes ( #5135 )
...
* increase observed attesters and aggregates cache sizes
* fix comment
2024-01-30 00:32:57 +00:00
Sergey Kisel
64efdaf39a
#5102 Fix load_state_for_block_production metric mapping ( #5103 )
...
* #5102 Fix load_state_for_block_production metric mapping
2024-01-30 00:32:52 +00:00
realbigsean
1cebf41452
Backfill blob storage fix ( #5119 )
...
* store blobs in the correct db in backfill
* add database migration
* add migration file
* remove log info suggesting deneb isn't schedule
* add batching in blob migration
2024-01-24 09:35:02 +11:00
Michael Sproul
a403138ed0
Reduce size of futures in HTTP API to prevent stack overflows ( #5104 )
...
* Box::pin a few big futures
* Arc the blocks early in publication
* Fix more tests
2024-01-23 15:32:07 +11:00
Lion - dapplion
585124fb2f
Hold HeadTracker lock until persisting to disk ( #5084 )
...
* Fix head tracker drop order on un-ordered shutdown
* lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2024-01-22 15:14:11 +11:00
Mac L
a0b407c15d
Add Deneb readiness logging ( #5074 )
2024-01-18 15:21:38 -05:00
Paul Hauner
be79f74c6d
Fix IncorrectAttestationSource error in attn. simulator ( #5048 )
...
* Don't produce attestations when syncing
* Handle `IncorrectAttestationSource`
2024-01-10 10:47:00 +11:00
Michael Sproul
7e948eec9d
Fix block v3 reward encodings ( #5049 )
...
* Fix block v3 reward encodings
* Use crates.io version
2024-01-10 10:44:07 +11:00
realbigsean
b47e3f252e
Runtime rpc request sizes ( #4841 )
...
* add runtime variable list type
* add configs to ChainSpec
* git rid of max request blocks type
* fix tests and lints
* remove todos
* git rid of old const usage
* fix decode impl
* add new config to `Config` api struct
* add docs fix compilt
* move methods for per-fork-spec to chainspec
* get values off chain spec
* fix compile
* remove min by root size
* add tests for runtime var list
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-01-09 10:23:47 +11:00
realbigsean
f70c32ec70
create unified slashing cache ( #5033 )
...
* create unified slashing cache
* add observed slashable file
* fix broadcast validation tests
* revert block seen cache changes
* clean up slashable cache test
* check header signatures for RPC blobs
* don't throw error on RPC signature invalie
2024-01-08 10:30:57 -05:00
Eitan Seri-Levi
9c1505d082
Block v3 builder boost factor ( #5035 )
...
* builder boost factor
* default boost factor
* revert
* deprecate always_prefer_builder_payload, builder-profit-threshold, ignore_builder_override_suggestion_threshold and builder_comparison_factor flags
* revert
* set deprecated flags to no op, revert should_override_builder
* fix test, calc boosted relay value correctly, dont calculate if none
* Add deprecation warnings and restore CLI docs
2024-01-08 11:10:32 +11:00
Michael Sproul
af11e78ae1
Clean up blockv3 metadata and client ( #5015 )
...
* Improve block production v3 client
* Delete wayward line
* Overhaul JSON endpoint as well
* Rename timeout param
* Update tests
* I broke everything
* Ah this is an insane fix
* Remove unnecessary optionals
* Doc fix
2023-12-22 09:39:17 -05:00
realbigsean
c55608be10
suppress error on duplicate blobs ( #4995 )
2023-12-18 12:15:12 -05:00
Jimmy Chen
b0c374c1ca
Update dependencies to get rid of the yanked deps ( #4994 )
...
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.
* Replace `.expect()` usage with use of `const`.
* Update ahash 0.7 as well
* Remove unsafe code.
* Update `lru` to 0.12 and fix release test errors.
* Set non-blocking socket
* Bump testcontainers to 0.15.
* Fix lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-15 18:31:59 +11:00
Michael Sproul
f1113540d8
Fix off-by-one bug in missed block detection ( #5012 )
...
* Regression test
* Fix the bug
2023-12-15 14:23:54 +11:00
Pawan Dhananjay
ae4a296089
Convert a FullPayload to a BlindedPayload correctly ( #5005 )
2023-12-14 08:44:14 -05:00
Joel Rousseau
189430a45c
Add attestation simulator ( #4880 )
...
* basic scaffold
* remove unnecessary ?
* check if committee cache is init
* typed ValidatorMonitor with ethspecs + store attestations within
* nits
* process unaggregated attestation
* typo
* extract in func
* add tests
* better naming
* better naming 2
* less verbose
* use same naming as validator monitor
* use attestation_simulator
* add metrics
* remove cache
* refacto flag_indices process
* add lag
* remove copying state
* clean and lint
* extract metrics
* nits
* compare prom metrics in tests
* implement lag
* nits
* nits
* add attestation simulator service
* fmt
* return beacon_chain as arc
* nit: debug
* sed s/unaggregated/unagg.//
* fmt
* fmt
* nit: remove unused comments
* increase max unaggregated attestation hashmap to 64
* nit: sed s/clone/copied//
* improve perf: remove unecessary hashmap copy
* fix flag indices comp
* start service in client builder
* remove //
* cargo fmt
* lint
* cloned keys
* fmt
* use Slot value instead of pointer
* Update beacon_node/beacon_chain/src/attestation_simulator.rs
Co-authored-by: Paul Hauner <paul@paulhauner.com>
---------
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-12-14 11:44:56 +11:00
Pawan Dhananjay
a3a370302a
Use the block header to compute the canonical_root ( #5003 )
2023-12-14 09:24:36 +11:00
Gua00va
a3fb27c99b
add forK_choice_read_lock as parameter ( #4978 )
2023-12-13 16:06:00 +11:00
ethDreamer
78ffa378b4
Batch Verify RPC Blobs ( #4934 )
2023-12-08 16:48:03 -05:00
realbigsean
46184e5ce4
Remove delayed lookups ( #4992 )
...
* initial rip out
* fix unused imports
* delete tests and fix lint
* fix peers scoring for blobs
2023-12-08 15:42:55 -05:00
Jimmy Chen
67e0569d9b
Fix corrupted DB on networks where the first slot is skipped (Holesky) ( #4985 )
...
* Fix zero block roots on skip slots.
* Remove temporary comment, println code and unused imports.
* Remove `println!` in test.
2023-12-07 15:12:06 +11:00
ethDreamer
52117f43ba
Small Improvements ( #4980 )
...
* initial changes
* use arc<blobsidecar> in vector
* Utilize new pattern for KzgVerifiedBlob
* fmt
* Update beacon_node/beacon_chain/src/blob_verification.rs
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
* forgot to save..
* lint
* fmt.. again
---------
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-12-06 15:51:40 -05:00
Pawan Dhananjay
31044402ee
Sidecar inclusion proof ( #4900 )
...
* Refactor BlobSidecar to new type
* Fix some compile errors
* Gossip verification compiles
* Fix http api types take 1
* Fix another round of compile errors
* Beacon node crate compiles
* EF tests compile
* Remove all blob signing from VC
* fmt
* Tests compile
* Fix some tests
* Fix more http tests
* get compiling
* Fix gossip conditions and tests
* Add basic proof generation and verification
* remove unnecessary ssz decode
* add back build_sidecar
* remove default at fork for blobs
* fix beacon chain tests
* get relase tests compiling
* fix lints
* fix existing spec tests
* add new ef tests
* fix gossip duplicate rule
* lints
* add back sidecar signature check in gossip
* add finalized descendant check to blob sidecar gossip
* fix error conversion
* fix release tests
* sidecar inclusion self review cleanup
* Add proof verification and computation metrics
* Remove accidentally committed file
* Unify some block and blob errors; add slashing conditions for sidecars
* Address review comment
* Clean up re-org tests (#4957 )
* Address more review comments
* Add Comments & Eliminate Unnecessary Clones
* update names
* Update beacon_node/beacon_chain/src/metrics.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Update beacon_node/network/src/network_beacon_processor/tests.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* pr feedback
* fix test compile
* Sidecar Inclusion proof small refactor and updates (#4967 )
* Update some comments, variables and small cosmetic fixes.
* Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety.
* Update function comment.
* Update testing/ef_tests/src/cases/merkle_proof_validity.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Rename the block and blob wrapper types used in the beacon API interfaces.
* make sure gossip invalid blobs are passed to the slasher (#4970 )
* Add blob headers to slasher before adding to DA checker
* Replace Vec with HashSet in BlockQueue
* fmt
* Rename gindex -> index
* Simplify gossip condition
---------
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-12-05 11:19:59 -05:00
Michael Sproul
c8b2324880
Enable progressive balances fast mode by default ( #4971 )
...
* Enable progressive balances fast mode by default
* Fix default in chain_config
2023-12-04 14:42:49 +11:00
Michael Sproul
547ed1de63
Clone state ahead of block production ( #4925 )
...
* Clone state ahead of block production
* Add pruning and fix logging
* Don't hold 2 states in mem
2023-11-30 13:49:35 +11:00
GeemoCandama
8a599ec7dc
API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events ( #3954 )
...
* rebase and add comment
* conditional test
* test
* optimistic chould be working now
* finality should be working now
* try again
* try again
* clippy fix
* add lc bootstrap beacon api
* add lc optimistic/finality update to events
* fmt
* That error isn't occuring on my computer but I think this should fix it
* Add missing test file
* Update light client types to comply with Altair light client spec.
* Fix test compilation
* Support deserializing light client structures for the Bellatrix fork
* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.
* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses
* Add light client bootstrap API test and fix existing ones.
* Fix test for `light-client-server` http api config.
* Appease clippy
* Efficiency improvement when retrieving beacon state.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Jimmy Chen
6b63d18420
Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) ( #4932 )
2023-11-18 03:55:11 +11:00
Mac L
68e076d60a
Remove legacy database migrations ( #4919 )
...
* Remove legacy db migrations
* Fix tests
* Remove migrations to/from v15
* Remove v16 migrations
2023-11-16 22:09:05 +11:00
Michael Sproul
d04e361129
Fix missed block logs ( #4922 )
2023-11-16 22:07:58 +11:00
Zackary Scott
e181741d38
Fix for issue 4860 - Added in process_justification_and_finalization ( #4877 )
...
* Added in process_justification_and_finalization
Added in process_justification_and_finalization to compute_attestation_rewards_altair to take into account justified attestations when coming out of inactivity leak. Also added in test to check for this edge case.
* Added in justification and finalization for compute_attestation_rewards_base
* Added in test for altair rewards without inactivity leak
2023-11-16 22:07:48 +11:00
Michael Sproul
051c3e842f
Always use a separate database for blobs ( #4892 )
...
* Always use a separate blobs DB
* Add + update tests
2023-11-09 16:51:36 +11:00
ethDreamer
7818100777
Verify KZG in Bulk During Block Sync ( #4903 )
2023-11-09 15:05:44 +11:00
Joel Rousseau
ac8811afac
Add missed blocks to monitored validators ( #4731 )
...
* add missed_block metric
* init missed_block in constructor
* declare beaconproposercache in ValidatorMonitor
* refacto proposer_shuffling_decision_root to use epoch instead of current.epoch
* imple new proposer_shuffling_decision_root in callers
* push missed_blocks
* prune missed_blocks
* only add to hashmap if it's a monitored validator
* remove current_epoch dup + typos
* extract in func
* add prom metrics
* checkpoint is not only epoch but slot as well
* add safeguard if we start a new chain at slot 0
* clean
* remove unnecessary negative value for a slot
* typo in comment
* remove unused current_epoch
* share beacon_proposer_cache between validator_monitor and beacon_chain
* pass Hash256::zero()
* debug objects
* fix loop: lag is at the head
* sed s/get_slot/get_epoch
* fewer calls to cache.get_epoch
* fix typos
* remove cache first call
* export TYPICAL_SLOTS_PER_EPOCH and use it in validator_monitor
* switch to gauge & loop over missed_blocks hashset
* fix subnet_service tests
* remove unused var
* clean + fix nits
* add beacon_proposer_cache + validator_monitor in builder
* fix store_tests
* fix builder tests
* add tests
* add validator monitor set of tests
* clean tests
* nits
* optimise imports
* lint
* typo
* added self.aggregatable
* duplicate proposer_shuffling_decision_root
* remove duplication in passing beacon_proposer_cache
* remove duplication in passing beacon_proposer_cache
* using indices
* fmt
* implement missed blocks total
* nits
* avoid heap allocation
* remove recursion limit
* fix lint
* Fix valdiator monitor builder pattern
Unify validator monitor config struct
* renaming metrics
* renaming metrics in validator monitor
* add log if there's a missing validator index
* consistent log
* fix loop
* better loop
* move gauge to counter
* fmt
* add error message
* lint
* fix prom metrics
* set gauge to 0 when non-finalized epochs
* better wording
* remove hash256::zero in favour of block_root
* fix gauge total label
* fix last missed block validator
* Add `MissedBlock` struct
* Fix comment
* Refactor non-finalized block loop
* Fix off-by-one
* Avoid string allocation
* Fix compile error
* Remove non-finalized blocks metric
* fix func clojure
* remove unused variable
* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD
* remove unused DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD in builder
* add validator index depending on the fork name
* typos
---------
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-11-09 15:05:14 +11:00
Justin Traglia
f2aabe915b
Fix comment for blob sidecar observation pruning ( #4893 )
...
## Issue Addressed
The comment implies that observations for the given slot would be retained but they are not.
## Proposed Changes
I'm pretty sure the functionality is correct and the comment is slightly incorrect, so just update the comment. The comment needs to say something along the lines of "less than or equal to" rather than just "less than."
## Additional Info
It doesn't make sense to keep finalized observations since those are no longer accepted.
2023-11-03 12:38:13 +00:00
Jimmy Chen
36d8849813
Add commmand for pruning states ( #4835 )
...
## Issue Addressed
Closes #4481 .
(Continuation of #4648 )
## Proposed Changes
- [x] Add `lighthouse db prune-states`
- [x] Make it work
- [x] Ensure block roots are handled correctly (to be addressed in 4735)
- [x] Check perf on mainnet/Goerli/Gnosis (takes a few seconds max)
- [x] Run block root healing logic (#4875 ) at the beginning
- [x] Add some tests
- [x] Update docs
- [x] Add `--freezer` flag and other improvements to `lighthouse db inspect`
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Jimmy Chen <jimmy@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:19 +00:00
Eitan Seri-Levi
07f53b18fc
Block v3 endpoint ( #4629 )
...
## Issue Addressed
#4582
## Proposed Changes
Add a new v3 block fetching flow that can decide to return a Full OR Blinded payload
## Additional Info
Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-11-03 00:12:18 +00:00
Eitan Seri-Levi
4ce01ddd11
Activate clippy::manual_let_else lint ( #4889 )
...
## Issue Addressed
#4888
## Proposed Changes
Enabled `clippy::manual_let_else` lint and resolved the warning messages.
2023-10-31 10:31:02 +00:00
