lighthouse-pulse/testing
Jimmy Chen 2de3451011 Rate limiting backfill sync (#3936)
## Issue Addressed

#3212 

## Proposed Changes

- Introduce a new `rate_limiting_backfill_queue` - any new inbound backfill work events gets immediately sent to this FIFO queue **without any processing**
- Spawn a `backfill_scheduler` routine that pops a backfill event from the FIFO queue at specified intervals (currently halfway through a slot, or at 6s after slot start for 12s slots) and sends the event to `BeaconProcessor` via a `scheduled_backfill_work_tx` channel
- This channel gets polled last in the `InboundEvents`, and work event received is  wrapped in a `InboundEvent::ScheduledBackfillWork` enum variant, which gets processed immediately or queued by the `BeaconProcessor` (existing logic applies from here)

Diagram comparing backfill processing with / without rate-limiting: 
https://github.com/sigp/lighthouse/issues/3212#issuecomment-1386249922

See this comment for @paulhauner's  explanation and solution: https://github.com/sigp/lighthouse/issues/3212#issuecomment-1384674956

## Additional Info

I've compared this branch (with backfill processing rate limited to to 1 and 3 batches per slot) against the latest stable version. The CPU usage during backfill sync is reduced by ~5% - 20%, more details on this page:

https://hackmd.io/@jimmygchen/SJuVpJL3j

The above testing is done on Goerli (as I don't currently have hardware for Mainnet), I'm guessing the differences are likely to be bigger on mainnet due to block size.

### TODOs

- [x] Experiment with processing multiple batches per slot. (need to think about how to do this for different slot durations)
- [x] Add option to disable rate-limiting, enabed by default.
- [x] (No longer required now we're reusing the reprocessing queue) Complete the `backfill_scheduler` task when backfill sync is completed or not required
2023-04-03 03:02:55 +00:00
..
antithesis update antithesis dockerfile (#3883) 2023-01-20 00:46:55 +00:00
ef_tests Rate limiting backfill sync (#3936) 2023-04-03 03:02:55 +00:00
eth1_test_rig Support for Ipv6 (#4046) 2023-03-14 01:13:34 +00:00
execution_engine_integration Reconstruct Payloads using Payload Bodies Methods (#4028) 2023-03-19 23:15:59 +00:00
node_test_rig Support for Ipv6 (#4046) 2023-03-14 01:13:34 +00:00
simulator Support for Ipv6 (#4046) 2023-03-14 01:13:34 +00:00
state_transition_vectors Consensus context with proposer index caching (#3604) 2022-10-15 22:25:54 +00:00
test-test_logger Update to Rust 1.59 and 2021 edition (#3038) 2022-02-25 00:10:17 +00:00
web3signer_tests Delete Kiln and Ropsten configs (#4038) 2023-03-01 06:16:14 +00:00