prysm-pulse/beacon-chain/interop-cold-start/service.go
Raul Jordan cc741ed8af
Ensure New State Type Tests Pass in Prysm (#4646)
* begin state service

* begin on the state trie idea

* created beacon state structure

* add in the full clone getter

* return by value instead

* add all setters

* new state setters are being completed

* arrays roots exposed

*  close to finishing all these headerssss

* functionality complete

* added in proto benchmark test

* test for compatibility

* add test for compat

* comments fixed

* add clone

* add clone

* remove underlying copies

* make it immutable

* integrate it into chainservice

* revert

* wrap up comments for package

* address all comments and godocs

* address all comments

* clone the pending attestation properly

* properly clone remaining items

* tests pass fixed bug

* begin using it instead of head state

* prevent nil pointer exceptions

* begin using new struct in db

* integrated new type into db package

* add proper nil checks

* using new state in archiver

* refactored much of core

* editing all the precompute functions

* done with most core refactor

* fixed up some bugs in the clone comparisons

* append current epoch atts

* add missing setters

* add new setters

* fix other core methods

* fix up transition

* main service and forkchoice

* fix rpc

* integrated to powchain

* some more changes

* fix build

* improve processing of deposits

* fix error

* prevent panic

* comment

* fix process att

* gaz

* fix up att process

* resolve existing review comments

* resolve another batch of gh comments

* resolve broken cpt state

* revise testutil to use the new state

* begin updating the state transition func to pass in more compartmentalized args

* finish editing transition function to return errors

* block operations pretty much done with refactor

* state transition fully refactored

* got epoch processing completed

* fix build in fork choice

* fixing more of the build

* fix up broken sync package

* it builds nowww it buildssss

* revert registry changes

* Recompute on Read (#4627)

* compute on read

* fix up eth1 data votes

* looking into slashings bug introduced in core/

* able to advance more slots

* add logging

* can now sync with testnet yay

* remove the leaves algorithm and other merkle imports

* expose initialize unsafe funcs

* Update beacon-chain/db/kv/state.go

* lint

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* More Optimizations for New State (#4641)

* map optimization

* more optimizations

* use a custom hasher

* comment

* block operations optimizations

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* fixed up various operations to use the validator index map access

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* archiver tests pass

* fixing cache tests

* cache tests passing

* edited validator tests

* powchain tests passing

* halfway thru sync tests

* more sync test fixes

* add in tests for state/

* working through rpc tests

* assignments tests passed

* almost done with rpc/beacon tests

* resolved painful validator test

* fixed up even more tests

* resolve tests

* fix build

* reduce a randao mixes copy

* fixes under //beacon-chain/blockchain/...

* build //beacon-chain/core/...

* fixes

* Runtime Optimizations (#4648)

* parallelize shuffling

* clean up

* lint

* fix build

* use callback to read from registry

* fix array roots and size map

* new improvements

* reduce hash allocs

* improved shuffling

* terence's review

* use different method

* raul's comment

* new array roots

* remove clone in pre-compute

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* raul's review

* lint

* fix build issues

* fix visibility

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* fix visibility

* build works for all

* fix blockchain test

* fix a few tests

* fix more tests

* update validator in slashing

* archiver passing

* fixed rpc/validator

* progress on core tests

* resolve broken rpc tests

* blockchain tests passed

* fix up some tests in core

* fix message diff

* remove unnecessary save

* Save validator after slashing

* Update validators one by one

* another update

* fix everything

* fix more precompute tests

* fix blocks tests

* more elegant fix

* more helper fixes

* change back ?

* fix test

* fix skip slot

* fix test

* reset caches

* fix testutil

* raceoff fixed

* passing

* Retrieve cached state in the beginning

* lint

* Fixed tests part 1

* Fixed rest of the tests

* Minor changes to avoid copying, small refactor to reduce deplicated code

* Handle att req for slot 0

* New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689)

* Only populate merkle layers as needed, copy merkle layers on copy/clone.

* use custom copy

* Make maps of correct size

* slightly fast, doesn't wait for lock

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Target root can't be 0x00

* Don't use cache for current slot (may not be the right fix)

* fixed up tests

* Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing...

* Align with prev logic for process slots cachedState.Slot() < slot

* Fix Initial Sync Flag (#4692)

* fixes

* fix up some test failures due to lack of nil checks

* fix up some test failures due to lack of nil checks

* fix up imports

* revert some changes

* imports

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* resolving further conflicts

* Better skip slot cache (#4694)

* Return copy of skip slot cache state, disable skip slot cache on sync

* fix

* Fix pruning

* fix up issues with broken tests

Co-authored-by: Nishant Das <nish1993@hotmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: shayzluf <thezluf@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-01-31 12:57:01 -08:00

199 lines
6.4 KiB
Go

package interopcoldstart
import (
"context"
"io/ioutil"
"math/big"
"github.com/pkg/errors"
ethpb "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
"github.com/prysmaticlabs/go-ssz"
"github.com/prysmaticlabs/prysm/beacon-chain/cache/depositcache"
"github.com/prysmaticlabs/prysm/beacon-chain/core/blocks"
"github.com/prysmaticlabs/prysm/beacon-chain/db"
"github.com/prysmaticlabs/prysm/beacon-chain/powchain"
stateTrie "github.com/prysmaticlabs/prysm/beacon-chain/state"
pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
"github.com/prysmaticlabs/prysm/shared"
"github.com/prysmaticlabs/prysm/shared/interop"
)
var _ = shared.Service(&Service{})
var _ = depositcache.DepositFetcher(&Service{})
var _ = powchain.ChainStartFetcher(&Service{})
// Service spins up an client interoperability service that handles responsibilities such
// as kickstarting a genesis state for the beacon node from cli flags or a genesis.ssz file.
type Service struct {
ctx context.Context
cancel context.CancelFunc
genesisTime uint64
numValidators uint64
beaconDB db.HeadAccessDatabase
powchain powchain.Service
depositCache *depositcache.DepositCache
genesisPath string
chainStartDeposits []*ethpb.Deposit
}
// Config options for the interop service.
type Config struct {
GenesisTime uint64
NumValidators uint64
BeaconDB db.HeadAccessDatabase
DepositCache *depositcache.DepositCache
GenesisPath string
}
// NewColdStartService is an interoperability testing service to inject a deterministically generated genesis state
// into the beacon chain database and running services at start up. This service should not be used in production
// as it does not have any value other than ease of use for testing purposes.
func NewColdStartService(ctx context.Context, cfg *Config) *Service {
log.Warn("Saving generated genesis state in database for interop testing")
ctx, cancel := context.WithCancel(ctx)
s := &Service{
ctx: ctx,
cancel: cancel,
genesisTime: cfg.GenesisTime,
numValidators: cfg.NumValidators,
beaconDB: cfg.BeaconDB,
depositCache: cfg.DepositCache,
genesisPath: cfg.GenesisPath,
}
if s.genesisPath != "" {
data, err := ioutil.ReadFile(s.genesisPath)
if err != nil {
log.Fatalf("Could not read pre-loaded state: %v", err)
}
genesisState := &pb.BeaconState{}
if err := ssz.Unmarshal(data, genesisState); err != nil {
log.Fatalf("Could not unmarshal pre-loaded state: %v", err)
}
genesisTrie, err := stateTrie.InitializeFromProto(genesisState)
if err != nil {
log.Fatalf("Could not get state trie: %v", err)
}
if err := s.saveGenesisState(ctx, genesisTrie); err != nil {
log.Fatalf("Could not save interop genesis state %v", err)
}
return s
}
// Save genesis state in db
genesisState, _, err := interop.GenerateGenesisState(s.genesisTime, s.numValidators)
if err != nil {
log.Fatalf("Could not generate interop genesis state: %v", err)
}
genesisTrie, err := stateTrie.InitializeFromProto(genesisState)
if err != nil {
log.Fatalf("Could not get state trie: %v", err)
}
if s.genesisTime == 0 {
// Generated genesis time; fetch it
s.genesisTime = genesisTrie.GenesisTime()
}
if err := s.saveGenesisState(ctx, genesisTrie); err != nil {
log.Fatalf("Could not save interop genesis state %v", err)
}
return s
}
// Start initializes the genesis state from configured flags.
func (s *Service) Start() {
}
// Stop does nothing.
func (s *Service) Stop() error {
return nil
}
// Status always returns nil.
func (s *Service) Status() error {
return nil
}
// AllDeposits mocks out the deposit cache functionality for interop.
func (s *Service) AllDeposits(ctx context.Context, beforeBlk *big.Int) []*ethpb.Deposit {
return []*ethpb.Deposit{}
}
// ChainStartDeposits mocks out the powchain functionality for interop.
func (s *Service) ChainStartDeposits() []*ethpb.Deposit {
return s.chainStartDeposits
}
// ChainStartEth1Data mocks out the powchain functionality for interop.
func (s *Service) ChainStartEth1Data() *ethpb.Eth1Data {
return &ethpb.Eth1Data{}
}
// PreGenesisState returns an empty beacon state.
func (s *Service) PreGenesisState() *stateTrie.BeaconState {
return &stateTrie.BeaconState{}
}
// ClearPreGenesisData --
func (s *Service) ClearPreGenesisData() {
//no-op
}
// DepositByPubkey mocks out the deposit cache functionality for interop.
func (s *Service) DepositByPubkey(ctx context.Context, pubKey []byte) (*ethpb.Deposit, *big.Int) {
return &ethpb.Deposit{}, big.NewInt(1)
}
// DepositsNumberAndRootAtHeight mocks out the deposit cache functionality for interop.
func (s *Service) DepositsNumberAndRootAtHeight(ctx context.Context, blockHeight *big.Int) (uint64, [32]byte) {
return 0, [32]byte{}
}
func (s *Service) saveGenesisState(ctx context.Context, genesisState *stateTrie.BeaconState) error {
s.chainStartDeposits = make([]*ethpb.Deposit, genesisState.NumValidators())
stateRoot, err := genesisState.HashTreeRoot()
if err != nil {
return err
}
genesisBlk := blocks.NewGenesisBlock(stateRoot[:])
genesisBlkRoot, err := ssz.HashTreeRoot(genesisBlk.Block)
if err != nil {
return errors.Wrap(err, "could not get genesis block root")
}
if err := s.beaconDB.SaveBlock(ctx, genesisBlk); err != nil {
return errors.Wrap(err, "could not save genesis block")
}
if err := s.beaconDB.SaveState(ctx, genesisState, genesisBlkRoot); err != nil {
return errors.Wrap(err, "could not save genesis state")
}
if err := s.beaconDB.SaveGenesisBlockRoot(ctx, genesisBlkRoot); err != nil {
return errors.Wrap(err, "could save genesis block root")
}
if err := s.beaconDB.SaveHeadBlockRoot(ctx, genesisBlkRoot); err != nil {
return errors.Wrap(err, "could not save head block root")
}
genesisCheckpoint := &ethpb.Checkpoint{Root: genesisBlkRoot[:]}
if err := s.beaconDB.SaveJustifiedCheckpoint(ctx, genesisCheckpoint); err != nil {
return errors.Wrap(err, "could save justified checkpoint")
}
if err := s.beaconDB.SaveFinalizedCheckpoint(ctx, genesisCheckpoint); err != nil {
return errors.Wrap(err, "could save finalized checkpoint")
}
for i := uint64(0); i < uint64(genesisState.NumValidators()); i++ {
pk := genesisState.PubkeyAtIndex(i)
if err := s.beaconDB.SaveValidatorIndex(ctx, pk[:], i); err != nil {
return errors.Wrapf(err, "could not save validator index: %d", i)
}
s.chainStartDeposits[i] = &ethpb.Deposit{
Data: &ethpb.Deposit_Data{
PublicKey: pk[:],
},
}
}
return nil
}