prysm-pulse/shared/testutil/block.go
Raul Jordan cc741ed8af
Ensure New State Type Tests Pass in Prysm (#4646)
* begin state service

* begin on the state trie idea

* created beacon state structure

* add in the full clone getter

* return by value instead

* add all setters

* new state setters are being completed

* arrays roots exposed

*  close to finishing all these headerssss

* functionality complete

* added in proto benchmark test

* test for compatibility

* add test for compat

* comments fixed

* add clone

* add clone

* remove underlying copies

* make it immutable

* integrate it into chainservice

* revert

* wrap up comments for package

* address all comments and godocs

* address all comments

* clone the pending attestation properly

* properly clone remaining items

* tests pass fixed bug

* begin using it instead of head state

* prevent nil pointer exceptions

* begin using new struct in db

* integrated new type into db package

* add proper nil checks

* using new state in archiver

* refactored much of core

* editing all the precompute functions

* done with most core refactor

* fixed up some bugs in the clone comparisons

* append current epoch atts

* add missing setters

* add new setters

* fix other core methods

* fix up transition

* main service and forkchoice

* fix rpc

* integrated to powchain

* some more changes

* fix build

* improve processing of deposits

* fix error

* prevent panic

* comment

* fix process att

* gaz

* fix up att process

* resolve existing review comments

* resolve another batch of gh comments

* resolve broken cpt state

* revise testutil to use the new state

* begin updating the state transition func to pass in more compartmentalized args

* finish editing transition function to return errors

* block operations pretty much done with refactor

* state transition fully refactored

* got epoch processing completed

* fix build in fork choice

* fixing more of the build

* fix up broken sync package

* it builds nowww it buildssss

* revert registry changes

* Recompute on Read (#4627)

* compute on read

* fix up eth1 data votes

* looking into slashings bug introduced in core/

* able to advance more slots

* add logging

* can now sync with testnet yay

* remove the leaves algorithm and other merkle imports

* expose initialize unsafe funcs

* Update beacon-chain/db/kv/state.go

* lint

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* More Optimizations for New State (#4641)

* map optimization

* more optimizations

* use a custom hasher

* comment

* block operations optimizations

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* fixed up various operations to use the validator index map access

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* archiver tests pass

* fixing cache tests

* cache tests passing

* edited validator tests

* powchain tests passing

* halfway thru sync tests

* more sync test fixes

* add in tests for state/

* working through rpc tests

* assignments tests passed

* almost done with rpc/beacon tests

* resolved painful validator test

* fixed up even more tests

* resolve tests

* fix build

* reduce a randao mixes copy

* fixes under //beacon-chain/blockchain/...

* build //beacon-chain/core/...

* fixes

* Runtime Optimizations (#4648)

* parallelize shuffling

* clean up

* lint

* fix build

* use callback to read from registry

* fix array roots and size map

* new improvements

* reduce hash allocs

* improved shuffling

* terence's review

* use different method

* raul's comment

* new array roots

* remove clone in pre-compute

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* raul's review

* lint

* fix build issues

* fix visibility

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* fix visibility

* build works for all

* fix blockchain test

* fix a few tests

* fix more tests

* update validator in slashing

* archiver passing

* fixed rpc/validator

* progress on core tests

* resolve broken rpc tests

* blockchain tests passed

* fix up some tests in core

* fix message diff

* remove unnecessary save

* Save validator after slashing

* Update validators one by one

* another update

* fix everything

* fix more precompute tests

* fix blocks tests

* more elegant fix

* more helper fixes

* change back ?

* fix test

* fix skip slot

* fix test

* reset caches

* fix testutil

* raceoff fixed

* passing

* Retrieve cached state in the beginning

* lint

* Fixed tests part 1

* Fixed rest of the tests

* Minor changes to avoid copying, small refactor to reduce deplicated code

* Handle att req for slot 0

* New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689)

* Only populate merkle layers as needed, copy merkle layers on copy/clone.

* use custom copy

* Make maps of correct size

* slightly fast, doesn't wait for lock

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Target root can't be 0x00

* Don't use cache for current slot (may not be the right fix)

* fixed up tests

* Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing...

* Align with prev logic for process slots cachedState.Slot() < slot

* Fix Initial Sync Flag (#4692)

* fixes

* fix up some test failures due to lack of nil checks

* fix up some test failures due to lack of nil checks

* fix up imports

* revert some changes

* imports

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* resolving further conflicts

* Better skip slot cache (#4694)

* Return copy of skip slot cache state, disable skip slot cache on sync

* fix

* Fix pruning

* fix up issues with broken tests

Co-authored-by: Nishant Das <nish1993@hotmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: shayzluf <thezluf@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-01-31 12:57:01 -08:00

472 lines
14 KiB
Go

package testutil
import (
"context"
"fmt"
"log"
"math"
"math/rand"
"github.com/pkg/errors"
ethpb "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
"github.com/prysmaticlabs/go-bitfield"
"github.com/prysmaticlabs/go-ssz"
"github.com/prysmaticlabs/prysm/beacon-chain/core/helpers"
"github.com/prysmaticlabs/prysm/beacon-chain/core/state"
stateTrie "github.com/prysmaticlabs/prysm/beacon-chain/state"
"github.com/prysmaticlabs/prysm/shared/attestationutil"
"github.com/prysmaticlabs/prysm/shared/bls"
"github.com/prysmaticlabs/prysm/shared/params"
)
// BlockGenConfig is used to define the requested conditions
// for block generation.
type BlockGenConfig struct {
NumProposerSlashings uint64
NumAttesterSlashings uint64
NumAttestations uint64
NumDeposits uint64
NumVoluntaryExits uint64
}
// DefaultBlockGenConfig returns the block config that utilizes the
// current params in the beacon config.
func DefaultBlockGenConfig() *BlockGenConfig {
return &BlockGenConfig{
NumProposerSlashings: 0,
NumAttesterSlashings: 0,
NumAttestations: 1,
NumDeposits: 0,
NumVoluntaryExits: 0,
}
}
// GenerateFullBlock generates a fully valid block with the requested parameters.
// Use BlockGenConfig to declare the conditions you would like the block generated under.
func GenerateFullBlock(
bState *stateTrie.BeaconState,
privs []*bls.SecretKey,
conf *BlockGenConfig,
slot uint64,
) (*ethpb.SignedBeaconBlock, error) {
currentSlot := bState.Slot()
if currentSlot > slot {
return nil, fmt.Errorf("current slot in state is larger than given slot. %d > %d", currentSlot, slot)
}
bState = bState.Copy()
if conf == nil {
conf = &BlockGenConfig{}
}
var err error
pSlashings := []*ethpb.ProposerSlashing{}
numToGen := conf.NumProposerSlashings
if numToGen > 0 {
pSlashings, err = generateProposerSlashings(bState, privs, numToGen)
if err != nil {
return nil, errors.Wrapf(err, "failed generating %d proposer slashings:", numToGen)
}
}
numToGen = conf.NumAttesterSlashings
aSlashings := []*ethpb.AttesterSlashing{}
if numToGen > 0 {
aSlashings, err = generateAttesterSlashings(bState, privs, numToGen)
if err != nil {
return nil, errors.Wrapf(err, "failed generating %d attester slashings:", numToGen)
}
}
numToGen = conf.NumAttestations
atts := []*ethpb.Attestation{}
if numToGen > 0 {
atts, err = GenerateAttestations(bState, privs, numToGen, slot)
if err != nil {
return nil, errors.Wrapf(err, "failed generating %d attestations:", numToGen)
}
}
numToGen = conf.NumDeposits
newDeposits, eth1Data := []*ethpb.Deposit{}, bState.Eth1Data()
if numToGen > 0 {
newDeposits, eth1Data, err = generateDepositsAndEth1Data(bState, numToGen)
if err != nil {
return nil, errors.Wrapf(err, "failed generating %d deposits:", numToGen)
}
}
numToGen = conf.NumVoluntaryExits
exits := []*ethpb.SignedVoluntaryExit{}
if numToGen > 0 {
exits, err = generateVoluntaryExits(bState, privs, numToGen)
if err != nil {
return nil, errors.Wrapf(err, "failed generating %d attester slashings:", numToGen)
}
}
newHeader := bState.LatestBlockHeader()
prevStateRoot, err := bState.HashTreeRoot()
if err != nil {
return nil, err
}
newHeader.StateRoot = prevStateRoot[:]
parentRoot, err := ssz.HashTreeRoot(newHeader)
if err != nil {
return nil, err
}
if slot == currentSlot {
slot = currentSlot + 1
}
// Temporarily incrementing the beacon state slot here since BeaconProposerIndex is a
// function deterministic on beacon state slot.
if err := bState.SetSlot(slot); err != nil {
return nil, err
}
reveal, err := RandaoReveal(bState, helpers.CurrentEpoch(bState), privs)
if err != nil {
return nil, err
}
block := &ethpb.BeaconBlock{
Slot: slot,
ParentRoot: parentRoot[:],
Body: &ethpb.BeaconBlockBody{
Eth1Data: eth1Data,
RandaoReveal: reveal,
ProposerSlashings: pSlashings,
AttesterSlashings: aSlashings,
Attestations: atts,
VoluntaryExits: exits,
Deposits: newDeposits,
},
}
if err := bState.SetSlot(currentSlot); err != nil {
return nil, err
}
signature, err := BlockSignature(bState, block, privs)
if err != nil {
return nil, err
}
return &ethpb.SignedBeaconBlock{Block: block, Signature: signature.Marshal()}, nil
}
func generateProposerSlashings(
bState *stateTrie.BeaconState,
privs []*bls.SecretKey,
numSlashings uint64,
) ([]*ethpb.ProposerSlashing, error) {
currentEpoch := helpers.CurrentEpoch(bState)
proposerSlashings := make([]*ethpb.ProposerSlashing, numSlashings)
for i := uint64(0); i < numSlashings; i++ {
proposerIndex, err := randValIndex(bState)
if err != nil {
return nil, err
}
header1 := &ethpb.SignedBeaconBlockHeader{
Header: &ethpb.BeaconBlockHeader{
Slot: bState.Slot(),
BodyRoot: []byte{0, 1, 0},
},
}
root, err := ssz.HashTreeRoot(header1.Header)
if err != nil {
return nil, err
}
domain := helpers.Domain(bState.Fork(), currentEpoch, params.BeaconConfig().DomainBeaconProposer)
header1.Signature = privs[proposerIndex].Sign(root[:], domain).Marshal()
header2 := &ethpb.SignedBeaconBlockHeader{
Header: &ethpb.BeaconBlockHeader{
Slot: bState.Slot(),
BodyRoot: []byte{0, 2, 0},
},
}
root, err = ssz.HashTreeRoot(header2.Header)
if err != nil {
return nil, err
}
header2.Signature = privs[proposerIndex].Sign(root[:], domain).Marshal()
slashing := &ethpb.ProposerSlashing{
ProposerIndex: proposerIndex,
Header_1: header1,
Header_2: header2,
}
proposerSlashings[i] = slashing
}
return proposerSlashings, nil
}
func generateAttesterSlashings(
bState *stateTrie.BeaconState,
privs []*bls.SecretKey,
numSlashings uint64,
) ([]*ethpb.AttesterSlashing, error) {
currentEpoch := helpers.CurrentEpoch(bState)
attesterSlashings := make([]*ethpb.AttesterSlashing, numSlashings)
for i := uint64(0); i < numSlashings; i++ {
committeeIndex := rand.Uint64() % params.BeaconConfig().MaxCommitteesPerSlot
committee, err := helpers.BeaconCommitteeFromState(bState, bState.Slot(), committeeIndex)
if err != nil {
return nil, err
}
committeeSize := uint64(len(committee))
randIndex := rand.Uint64() % uint64(len(committee))
valIndex := committee[randIndex]
aggregationBits := bitfield.NewBitlist(committeeSize)
aggregationBits.SetBitAt(randIndex, true)
att1 := &ethpb.Attestation{
Data: &ethpb.AttestationData{
Slot: bState.Slot(),
CommitteeIndex: committeeIndex,
Target: &ethpb.Checkpoint{
Epoch: currentEpoch,
Root: params.BeaconConfig().ZeroHash[:],
},
Source: &ethpb.Checkpoint{
Epoch: currentEpoch + 1,
Root: params.BeaconConfig().ZeroHash[:],
},
},
AggregationBits: aggregationBits,
}
dataRoot, err := ssz.HashTreeRoot(att1.Data)
if err != nil {
return nil, err
}
domain := helpers.Domain(bState.Fork(), i, params.BeaconConfig().DomainBeaconAttester)
sig := privs[valIndex].Sign(dataRoot[:], domain)
att1.Signature = bls.AggregateSignatures([]*bls.Signature{sig}).Marshal()
att2 := &ethpb.Attestation{
Data: &ethpb.AttestationData{
Slot: bState.Slot(),
CommitteeIndex: committeeIndex,
Target: &ethpb.Checkpoint{
Epoch: currentEpoch,
Root: params.BeaconConfig().ZeroHash[:],
},
Source: &ethpb.Checkpoint{
Epoch: currentEpoch,
Root: params.BeaconConfig().ZeroHash[:],
},
},
AggregationBits: aggregationBits,
}
dataRoot, err = ssz.HashTreeRoot(att2.Data)
if err != nil {
return nil, err
}
sig = privs[valIndex].Sign(dataRoot[:], domain)
att2.Signature = bls.AggregateSignatures([]*bls.Signature{sig}).Marshal()
indexedAtt1, err := attestationutil.ConvertToIndexed(context.Background(), att1, committee)
if err != nil {
return nil, err
}
indexedAtt2, err := attestationutil.ConvertToIndexed(context.Background(), att2, committee)
if err != nil {
return nil, err
}
slashing := &ethpb.AttesterSlashing{
Attestation_1: indexedAtt1,
Attestation_2: indexedAtt2,
}
attesterSlashings[i] = slashing
}
return attesterSlashings, nil
}
// GenerateAttestations creates attestations that are entirely valid, for all
// the committees of the current state slot. This function expects attestations
// requested to be cleanly divisible by committees per slot. If there is 1 committee
// in the slot, and numToGen is set to 4, then it will return 4 attestations
// for the same data with their aggregation bits split uniformly.
//
// If you request 4 attestations, but there are 8 committees, you will get 4 fully aggregated attestations.
func GenerateAttestations(
bState *stateTrie.BeaconState,
privs []*bls.SecretKey,
numToGen uint64,
slot uint64,
) ([]*ethpb.Attestation, error) {
currentEpoch := helpers.SlotToEpoch(slot)
attestations := []*ethpb.Attestation{}
generateHeadState := false
bState, err := stateTrie.InitializeFromProtoUnsafe(bState.CloneInnerState())
if err != nil {
return nil, err
}
if slot > bState.Slot() {
// Going back a slot here so there's no inclusion delay issues.
slot--
generateHeadState = true
}
targetRoot := make([]byte, 32)
headRoot := make([]byte, 32)
// Only calculate head state if its an attestation for the current slot or future slot.
if generateHeadState || slot == bState.Slot() {
headState, err := stateTrie.InitializeFromProtoUnsafe(bState.CloneInnerState())
if err != nil {
return nil, err
}
headState, err = state.ProcessSlots(context.Background(), headState, slot+1)
if err != nil {
return nil, err
}
headRoot, err = helpers.BlockRootAtSlot(headState, slot)
if err != nil {
return nil, err
}
targetRoot, err = helpers.BlockRoot(headState, currentEpoch)
if err != nil {
return nil, err
}
} else {
headRoot, err = helpers.BlockRootAtSlot(bState, slot)
if err != nil {
return nil, err
}
}
activeValidatorCount, err := helpers.ActiveValidatorCount(bState, currentEpoch)
if err != nil {
return nil, err
}
committeesPerSlot := helpers.SlotCommitteeCount(activeValidatorCount)
if numToGen < committeesPerSlot {
log.Printf(
"Warning: %d attestations requested is less than %d committees in current slot, not all validators will be attesting.",
numToGen,
committeesPerSlot,
)
} else if numToGen > committeesPerSlot {
log.Printf(
"Warning: %d attestations requested are more than %d committees in current slot, attestations will not be perfectly efficient.",
numToGen,
committeesPerSlot,
)
}
attsPerCommittee := math.Max(float64(numToGen/committeesPerSlot), 1)
if math.Trunc(attsPerCommittee) != attsPerCommittee {
return nil, fmt.Errorf(
"requested attestations %d must be easily divisible by committees in slot %d, calculated %f",
numToGen,
committeesPerSlot,
attsPerCommittee,
)
}
domain := helpers.Domain(bState.Fork(), currentEpoch, params.BeaconConfig().DomainBeaconAttester)
fmt.Printf("Justified: %d\n", bState.CurrentJustifiedCheckpoint().Epoch)
for c := uint64(0); c < committeesPerSlot && c < numToGen; c++ {
committee, err := helpers.BeaconCommitteeFromState(bState, slot, c)
if err != nil {
return nil, err
}
attData := &ethpb.AttestationData{
Slot: slot,
CommitteeIndex: c,
BeaconBlockRoot: headRoot,
Source: bState.CurrentJustifiedCheckpoint(),
Target: &ethpb.Checkpoint{
Epoch: currentEpoch,
Root: targetRoot,
},
}
dataRoot, err := ssz.HashTreeRoot(attData)
if err != nil {
return nil, err
}
committeeSize := uint64(len(committee))
bitsPerAtt := committeeSize / uint64(attsPerCommittee)
for i := uint64(0); i < committeeSize; i += bitsPerAtt {
aggregationBits := bitfield.NewBitlist(committeeSize)
sigs := []*bls.Signature{}
for b := i; b < i+bitsPerAtt; b++ {
aggregationBits.SetBitAt(b, true)
sigs = append(sigs, privs[committee[b]].Sign(dataRoot[:], domain))
}
att := &ethpb.Attestation{
Data: attData,
AggregationBits: aggregationBits,
Signature: bls.AggregateSignatures(sigs).Marshal(),
}
attestations = append(attestations, att)
}
}
return attestations, nil
}
func generateDepositsAndEth1Data(
bState *stateTrie.BeaconState,
numDeposits uint64,
) (
[]*ethpb.Deposit,
*ethpb.Eth1Data,
error,
) {
previousDepsLen := bState.Eth1DepositIndex()
currentDeposits, _, err := DeterministicDepositsAndKeys(previousDepsLen + numDeposits)
if err != nil {
return nil, nil, errors.Wrap(err, "could not get deposits")
}
eth1Data, err := DeterministicEth1Data(len(currentDeposits))
if err != nil {
return nil, nil, errors.Wrap(err, "could not get eth1data")
}
return currentDeposits[previousDepsLen:], eth1Data, nil
}
func generateVoluntaryExits(
bState *stateTrie.BeaconState,
privs []*bls.SecretKey,
numExits uint64,
) ([]*ethpb.SignedVoluntaryExit, error) {
currentEpoch := helpers.CurrentEpoch(bState)
voluntaryExits := make([]*ethpb.SignedVoluntaryExit, numExits)
for i := 0; i < len(voluntaryExits); i++ {
valIndex, err := randValIndex(bState)
if err != nil {
return nil, err
}
exit := &ethpb.SignedVoluntaryExit{
Exit: &ethpb.VoluntaryExit{
Epoch: helpers.PrevEpoch(bState),
ValidatorIndex: valIndex,
},
}
root, err := ssz.HashTreeRoot(exit.Exit)
if err != nil {
return nil, err
}
domain := helpers.Domain(bState.Fork(), currentEpoch, params.BeaconConfig().DomainVoluntaryExit)
exit.Signature = privs[valIndex].Sign(root[:], domain).Marshal()
voluntaryExits[i] = exit
}
return voluntaryExits, nil
}
func randValIndex(bState *stateTrie.BeaconState) (uint64, error) {
activeCount, err := helpers.ActiveValidatorCount(bState, helpers.CurrentEpoch(bState))
if err != nil {
return 0, err
}
return rand.Uint64() % activeCount, nil
}