prysm-pulse/beacon-chain/sync/initial-sync/service.go
Radosław Kapka 12403d249f
[Feature] - API Middleware (#8926)
* HTTP proxy server for Eth2 APIs (#8904)

* Implement API HTTP proxy server

* cleanup + more comments

* gateway will no longer be dependent on beaconv1

* handle error during ErrorJson type assertion

* simplify handling of endpoint data

* fix mux v1 route

* use URL encoding for all requests

* comment fieldProcessor

* fix failing test

* change proxy port to not interfere with e2e

* gzl

* simplify conditional expression

* Move appending custom error header to grpcutils package

* add api-middleware-port flag

* fix documentation for processField

* modify e2e port

* change field processing error message

* better error message for field processing

* simplify base64ToHexProcessor

* fix json structs

* Run several new endpoints through API middleware (#8922)

* Implement API HTTP proxy server

* cleanup + more comments

* gateway will no longer be dependent on beaconv1

* handle error during ErrorJson type assertion

* simplify handling of endpoint data

* fix mux v1 route

* use URL encoding for all requests

* comment fieldProcessor

* fix failing test

* change proxy port to not interfere with e2e

* gzl

* simplify conditional expression

* Move appending custom error header to grpcutils package

* add api-middleware-port flag

* fix documentation for processField

* modify e2e port

* change field processing error message

* better error message for field processing

* simplify base64ToHexProcessor

* fix json structs

* /eth/v1/beacon/states/{state_id}/validators

* /eth/v1/beacon/states/{state_id}/validators/{validator_id}

* /eth/v1/beacon/states/{state_id}/validator_balances

* /eth/v1/beacon/states/{state_id}/committees

* allow skipping base64-encoding for query params

* /eth/v1/beacon/pool/attestations

* replace break with continue

* Remove unused functions (#8924)

Co-authored-by: terence tsao <terence@prysmaticlabs.com>

* Process SSZ-serialized beacon state through API middleware (#8925)

* update field names

* Process SSZ-serialized beacon state through API middleware

* revert changes to go.mod and go.sum

* Revert "Merge branch '__develop' into feature/api-middleware"

This reverts commit 7c739a8fd71e2c1e3a14be85abd29a59b57ae9b5, reversing
changes made to 2d0f8e012ecb006888ed8e826b45625a3edc2eeb.

* update ethereumapis

* update validator field name

* update deps.bzl

* update json tags (#8942)

* Run `/node/syncing` through API Middleware (#8944)

* add IsSyncing field to grpc response

* run /node/syncing through the middleware

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Return HTTP status codes other than 200 and 500 from node and debug endpoints (#8937)

* error codes for node endpoints

* error codes for debug endpoints

* better comment about headers

* gzl

* review comments

* comment on return value

* update fakeChecker used for fuzz tests

* fix failing tests

* Allow to pass URL params literally, without encoding to base64 (#8938)

* Allow to pass URL params literally, without encoding to base64

* fix compile error

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Process SSZ-serialized beacon state through API middleware (#8925)

* update field names

* Process SSZ-serialized beacon state through API middleware

* revert changes to go.mod and go.sum

* Revert "Merge branch '__develop' into feature/api-middleware"

This reverts commit 7c739a8fd71e2c1e3a14be85abd29a59b57ae9b5, reversing
changes made to 2d0f8e012ecb006888ed8e826b45625a3edc2eeb.

* update ethereumapis

* update validator field name

* update deps.bzl

* update json tags (#8942)

* Run `/node/syncing` through API Middleware (#8944)

* add IsSyncing field to grpc response

* run /node/syncing through the middleware

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* Return HTTP status codes other than 200 and 500 from node and debug endpoints (#8937)

* error codes for node endpoints

* error codes for debug endpoints

* better comment about headers

* gzl

* review comments

* comment on return value

* update fakeChecker used for fuzz tests

* fix failing tests

* Allow to pass URL params literally, without encoding to base64 (#8938)

* Allow to pass URL params literally, without encoding to base64

* fix compile error

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* unused import

* Return correct status codes from beacon endpoints (#8960)

* Various API Middleware fixes (#8963)

* Return correct status codes from `/states` endpoints

* better error messages in debug and node

* better error messages in state

* returning correct error codes from validator endpoints

* correct error codes for getting a block header

* gzl

* fix err variable name

* fix nil block comparison

* test fixes

* make status enum test better

* fix ineffectual assignment

* make PR unstuck

* return proper status codes

* return uppercase keys from /config/spec

* return lowercase validator status

* convert requested enum values to uppercase

* validator fixes

* Implement `/beacon/headers` endpoint (#8966)

* Refactor API Middleware into more manageable code  (#8984)

* move endpoint registration out of shared package

* divide main function into smaller components

* return early on error

* implement hooks

* implement custom handlers and add documentation

* fix test compile error

* restrict package visibility

* remove redundant error checking

* rename file

* API Middleware unit tests (#8998)

* move endpoint registration out of shared package

* divide main function into smaller components

* return early on error

* implement hooks

* implement custom handlers and add documentation

* fix test compile error

* restrict package visibility

* remove redundant error checking

* rename file

* api_middleware_processing

* endpoints

* gzl

* remove gazelle:ignore

* merge

* Implement SSZ version of `/blocks/{block_id}` (#8970)

* Implement SSZ version of `/blocks/{block_id}`

* add dependencies back

* fix indentation in deps.bzl

* parameterize ssz functions

* get block ssz

* update ethereumapis dependency

* gzl

* Do not reuse `Endpoint` structs between API calls (#9007)

* code refactor

* implement endpoint factory

* fix test

* fmt

* include pbs

* gaz

* test naming fixes

* remove unused code

* radek comments

* revert endpoint test

* bring back bytes test case

* move `signedBeaconBlock` to `migration` package

* change `fmt.Errorf` to `errors.Wrap`

* capitalize SSZ

* capitalize URL

* more review feedback

* rename `handleGetBlockSSZ` to `handleGetBeaconBlockSSZ`

* rename `IndexOutOfRangeError` to `ValidatorIndexOutOfRangeError`

* simplify parameter names

* test header

* more corrections

* properly allocate array capacity

Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
2021-06-15 10:28:49 -05:00

228 lines
6.8 KiB
Go

// Package initialsync includes all initial block download and processing
// logic for the beacon node, using a round robin strategy and a finite-state-machine
// to handle edge-cases in a beacon node's sync status.
package initialsync
import (
"context"
"time"
"github.com/paulbellamy/ratecounter"
"github.com/pkg/errors"
"github.com/prysmaticlabs/prysm/beacon-chain/blockchain"
"github.com/prysmaticlabs/prysm/beacon-chain/core/feed"
blockfeed "github.com/prysmaticlabs/prysm/beacon-chain/core/feed/block"
statefeed "github.com/prysmaticlabs/prysm/beacon-chain/core/feed/state"
"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
"github.com/prysmaticlabs/prysm/beacon-chain/db"
"github.com/prysmaticlabs/prysm/beacon-chain/p2p"
"github.com/prysmaticlabs/prysm/cmd/beacon-chain/flags"
"github.com/prysmaticlabs/prysm/shared"
"github.com/prysmaticlabs/prysm/shared/abool"
"github.com/prysmaticlabs/prysm/shared/params"
"github.com/prysmaticlabs/prysm/shared/timeutils"
"github.com/sirupsen/logrus"
)
var _ shared.Service = (*Service)(nil)
// blockchainService defines the interface for interaction with block chain service.
type blockchainService interface {
blockchain.BlockReceiver
blockchain.ChainInfoFetcher
}
// Config to set up the initial sync service.
type Config struct {
P2P p2p.P2P
DB db.ReadOnlyDatabase
Chain blockchainService
StateNotifier statefeed.Notifier
BlockNotifier blockfeed.Notifier
}
// Service service.
type Service struct {
cfg *Config
ctx context.Context
cancel context.CancelFunc
synced *abool.AtomicBool
chainStarted *abool.AtomicBool
counter *ratecounter.RateCounter
genesisChan chan time.Time
}
// NewService configures the initial sync service responsible for bringing the node up to the
// latest head of the blockchain.
func NewService(ctx context.Context, cfg *Config) *Service {
ctx, cancel := context.WithCancel(ctx)
s := &Service{
cfg: cfg,
ctx: ctx,
cancel: cancel,
synced: abool.New(),
chainStarted: abool.New(),
counter: ratecounter.NewRateCounter(counterSeconds * time.Second),
genesisChan: make(chan time.Time),
}
go s.waitForStateInitialization()
return s
}
// Start the initial sync service.
func (s *Service) Start() {
// Wait for state initialized event.
genesis := <-s.genesisChan
if genesis.IsZero() {
log.Debug("Exiting Initial Sync Service")
return
}
if flags.Get().DisableSync {
s.markSynced(genesis)
log.WithField("genesisTime", genesis).Info("Due to Sync Being Disabled, entering regular sync immediately.")
return
}
if genesis.After(timeutils.Now()) {
s.markSynced(genesis)
log.WithField("genesisTime", genesis).Info("Genesis time has not arrived - not syncing")
return
}
currentSlot := helpers.SlotsSince(genesis)
if helpers.SlotToEpoch(currentSlot) == 0 {
log.WithField("genesisTime", genesis).Info("Chain started within the last epoch - not syncing")
s.markSynced(genesis)
return
}
s.chainStarted.Set()
log.Info("Starting initial chain sync...")
// Are we already in sync, or close to it?
if helpers.SlotToEpoch(s.cfg.Chain.HeadSlot()) == helpers.SlotToEpoch(currentSlot) {
log.Info("Already synced to the current chain head")
s.markSynced(genesis)
return
}
s.waitForMinimumPeers()
if err := s.roundRobinSync(genesis); err != nil {
if errors.Is(s.ctx.Err(), context.Canceled) {
return
}
panic(err)
}
log.Infof("Synced up to slot %d", s.cfg.Chain.HeadSlot())
s.markSynced(genesis)
}
// Stop initial sync.
func (s *Service) Stop() error {
s.cancel()
return nil
}
// Status of initial sync.
func (s *Service) Status() error {
if s.synced.IsNotSet() && s.chainStarted.IsSet() {
return errors.New("syncing")
}
return nil
}
// Syncing returns true if initial sync is still running.
func (s *Service) Syncing() bool {
return s.synced.IsNotSet()
}
// Initialized returns true if initial sync has been started.
func (s *Service) Initialized() bool {
return s.chainStarted.IsSet()
}
// Synced returns true if initial sync has been completed.
func (s *Service) Synced() bool {
return s.synced.IsSet()
}
// Resync allows a node to start syncing again if it has fallen
// behind the current network head.
func (s *Service) Resync() error {
headState, err := s.cfg.Chain.HeadState(s.ctx)
if err != nil || headState == nil || headState.IsNil() {
return errors.Errorf("could not retrieve head state: %v", err)
}
// Set it to false since we are syncing again.
s.synced.UnSet()
defer func() { s.synced.Set() }() // Reset it at the end of the method.
genesis := time.Unix(int64(headState.GenesisTime()), 0)
s.waitForMinimumPeers()
if err = s.roundRobinSync(genesis); err != nil {
log = log.WithError(err)
}
log.WithField("slot", s.cfg.Chain.HeadSlot()).Info("Resync attempt complete")
return nil
}
func (s *Service) waitForMinimumPeers() {
required := params.BeaconConfig().MaxPeersToSync
if flags.Get().MinimumSyncPeers < required {
required = flags.Get().MinimumSyncPeers
}
for {
_, peers := s.cfg.P2P.Peers().BestNonFinalized(flags.Get().MinimumSyncPeers, s.cfg.Chain.FinalizedCheckpt().Epoch)
if len(peers) >= required {
break
}
log.WithFields(logrus.Fields{
"suitable": len(peers),
"required": required,
}).Info("Waiting for enough suitable peers before syncing")
time.Sleep(handshakePollingInterval)
}
}
// waitForStateInitialization makes sure that beacon node is ready to be accessed: it is either
// already properly configured or system waits up until state initialized event is triggered.
func (s *Service) waitForStateInitialization() {
// Wait for state to be initialized.
stateChannel := make(chan *feed.Event, 1)
stateSub := s.cfg.StateNotifier.StateFeed().Subscribe(stateChannel)
defer stateSub.Unsubscribe()
log.Info("Waiting for state to be initialized")
for {
select {
case event := <-stateChannel:
if event.Type == statefeed.Initialized {
data, ok := event.Data.(*statefeed.InitializedData)
if !ok {
log.Error("Event feed data is not type *statefeed.InitializedData")
continue
}
log.WithField("starttime", data.StartTime).Debug("Received state initialized event")
s.genesisChan <- data.StartTime
return
}
case <-s.ctx.Done():
log.Debug("Context closed, exiting goroutine")
// Send a zero time in the event we are exiting.
s.genesisChan <- time.Time{}
return
case err := <-stateSub.Err():
log.WithError(err).Error("Subscription to state notifier failed")
// Send a zero time in the event we are exiting.
s.genesisChan <- time.Time{}
return
}
}
}
// markSynced marks node as synced and notifies feed listeners.
func (s *Service) markSynced(genesis time.Time) {
s.synced.Set()
s.cfg.StateNotifier.StateFeed().Send(&feed.Event{
Type: statefeed.Synced,
Data: &statefeed.SyncedData{
StartTime: genesis,
},
})
}