prysm-pulse/shared/interop/generate_genesis_state.go
Raul Jordan cc741ed8af
Ensure New State Type Tests Pass in Prysm (#4646)
* begin state service

* begin on the state trie idea

* created beacon state structure

* add in the full clone getter

* return by value instead

* add all setters

* new state setters are being completed

* arrays roots exposed

*  close to finishing all these headerssss

* functionality complete

* added in proto benchmark test

* test for compatibility

* add test for compat

* comments fixed

* add clone

* add clone

* remove underlying copies

* make it immutable

* integrate it into chainservice

* revert

* wrap up comments for package

* address all comments and godocs

* address all comments

* clone the pending attestation properly

* properly clone remaining items

* tests pass fixed bug

* begin using it instead of head state

* prevent nil pointer exceptions

* begin using new struct in db

* integrated new type into db package

* add proper nil checks

* using new state in archiver

* refactored much of core

* editing all the precompute functions

* done with most core refactor

* fixed up some bugs in the clone comparisons

* append current epoch atts

* add missing setters

* add new setters

* fix other core methods

* fix up transition

* main service and forkchoice

* fix rpc

* integrated to powchain

* some more changes

* fix build

* improve processing of deposits

* fix error

* prevent panic

* comment

* fix process att

* gaz

* fix up att process

* resolve existing review comments

* resolve another batch of gh comments

* resolve broken cpt state

* revise testutil to use the new state

* begin updating the state transition func to pass in more compartmentalized args

* finish editing transition function to return errors

* block operations pretty much done with refactor

* state transition fully refactored

* got epoch processing completed

* fix build in fork choice

* fixing more of the build

* fix up broken sync package

* it builds nowww it buildssss

* revert registry changes

* Recompute on Read (#4627)

* compute on read

* fix up eth1 data votes

* looking into slashings bug introduced in core/

* able to advance more slots

* add logging

* can now sync with testnet yay

* remove the leaves algorithm and other merkle imports

* expose initialize unsafe funcs

* Update beacon-chain/db/kv/state.go

* lint

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* More Optimizations for New State (#4641)

* map optimization

* more optimizations

* use a custom hasher

* comment

* block operations optimizations

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* fixed up various operations to use the validator index map access

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* archiver tests pass

* fixing cache tests

* cache tests passing

* edited validator tests

* powchain tests passing

* halfway thru sync tests

* more sync test fixes

* add in tests for state/

* working through rpc tests

* assignments tests passed

* almost done with rpc/beacon tests

* resolved painful validator test

* fixed up even more tests

* resolve tests

* fix build

* reduce a randao mixes copy

* fixes under //beacon-chain/blockchain/...

* build //beacon-chain/core/...

* fixes

* Runtime Optimizations (#4648)

* parallelize shuffling

* clean up

* lint

* fix build

* use callback to read from registry

* fix array roots and size map

* new improvements

* reduce hash allocs

* improved shuffling

* terence's review

* use different method

* raul's comment

* new array roots

* remove clone in pre-compute

* Update beacon-chain/state/types.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* raul's review

* lint

* fix build issues

* fix visibility

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* fix visibility

* build works for all

* fix blockchain test

* fix a few tests

* fix more tests

* update validator in slashing

* archiver passing

* fixed rpc/validator

* progress on core tests

* resolve broken rpc tests

* blockchain tests passed

* fix up some tests in core

* fix message diff

* remove unnecessary save

* Save validator after slashing

* Update validators one by one

* another update

* fix everything

* fix more precompute tests

* fix blocks tests

* more elegant fix

* more helper fixes

* change back ?

* fix test

* fix skip slot

* fix test

* reset caches

* fix testutil

* raceoff fixed

* passing

* Retrieve cached state in the beginning

* lint

* Fixed tests part 1

* Fixed rest of the tests

* Minor changes to avoid copying, small refactor to reduce deplicated code

* Handle att req for slot 0

* New beacon state: Only populate merkle layers as needed, copy merkle layers on copy/clone. (#4689)

* Only populate merkle layers as needed, copy merkle layers on copy/clone.

* use custom copy

* Make maps of correct size

* slightly fast, doesn't wait for lock

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>

* Target root can't be 0x00

* Don't use cache for current slot (may not be the right fix)

* fixed up tests

* Remove some copy for init sync. Not sure if it is safe enough for runtime though... testing...

* Align with prev logic for process slots cachedState.Slot() < slot

* Fix Initial Sync Flag (#4692)

* fixes

* fix up some test failures due to lack of nil checks

* fix up some test failures due to lack of nil checks

* fix up imports

* revert some changes

* imports

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>

* resolving further conflicts

* Better skip slot cache (#4694)

* Return copy of skip slot cache state, disable skip slot cache on sync

* fix

* Fix pruning

* fix up issues with broken tests

Co-authored-by: Nishant Das <nish1993@hotmail.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: shayzluf <thezluf@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-01-31 12:57:01 -08:00

168 lines
6.7 KiB
Go

package interop
import (
"sync"
"github.com/pkg/errors"
ethpb "github.com/prysmaticlabs/ethereumapis/eth/v1alpha1"
"github.com/prysmaticlabs/go-ssz"
"github.com/prysmaticlabs/prysm/beacon-chain/core/state"
pb "github.com/prysmaticlabs/prysm/proto/beacon/p2p/v1"
"github.com/prysmaticlabs/prysm/shared/bls"
"github.com/prysmaticlabs/prysm/shared/hashutil"
"github.com/prysmaticlabs/prysm/shared/mputil"
"github.com/prysmaticlabs/prysm/shared/params"
"github.com/prysmaticlabs/prysm/shared/roughtime"
"github.com/prysmaticlabs/prysm/shared/trieutil"
)
var (
// This is the recommended mock eth1 block hash according to the Eth2 interop guidelines.
// https://github.com/ethereum/eth2.0-pm/blob/a085c9870f3956d6228ed2a40cd37f0c6580ecd7/interop/mocked_start/README.md
mockEth1BlockHash = []byte{66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66}
)
// GenerateGenesisState deterministically given a genesis time and number of validators.
// If a genesis time of 0 is supplied it is set to the current time.
func GenerateGenesisState(genesisTime, numValidators uint64) (*pb.BeaconState, []*ethpb.Deposit, error) {
privKeys, pubKeys, err := DeterministicallyGenerateKeys(0 /*startIndex*/, numValidators)
if err != nil {
return nil, nil, errors.Wrapf(err, "could not deterministically generate keys for %d validators", numValidators)
}
depositDataItems, depositDataRoots, err := DepositDataFromKeys(privKeys, pubKeys)
if err != nil {
return nil, nil, errors.Wrap(err, "could not generate deposit data from keys")
}
trie, err := trieutil.GenerateTrieFromItems(
depositDataRoots,
int(params.BeaconConfig().DepositContractTreeDepth),
)
if err != nil {
return nil, nil, errors.Wrap(err, "could not generate Merkle trie for deposit proofs")
}
deposits, err := GenerateDepositsFromData(depositDataItems, trie)
if err != nil {
return nil, nil, errors.Wrap(err, "could not generate deposits from the deposit data provided")
}
root := trie.Root()
if genesisTime == 0 {
genesisTime = uint64(roughtime.Now().Unix())
}
beaconState, err := state.GenesisBeaconState(deposits, genesisTime, &ethpb.Eth1Data{
DepositRoot: root[:],
DepositCount: uint64(len(deposits)),
BlockHash: mockEth1BlockHash,
})
if err != nil {
return nil, nil, errors.Wrap(err, "could not generate genesis state")
}
return beaconState.CloneInnerState(), deposits, nil
}
// GenerateDepositsFromData a list of deposit items by creating proofs for each of them from a sparse Merkle trie.
func GenerateDepositsFromData(depositDataItems []*ethpb.Deposit_Data, trie *trieutil.SparseMerkleTrie) ([]*ethpb.Deposit, error) {
deposits := make([]*ethpb.Deposit, len(depositDataItems))
results, err := mputil.Scatter(len(depositDataItems), func(offset int, entries int, _ *sync.RWMutex) (interface{}, error) {
return generateDepositsFromData(depositDataItems[offset:offset+entries], offset, trie)
})
if err != nil {
return nil, errors.Wrap(err, "failed to generate deposits from data")
}
for _, result := range results {
if depositExtent, ok := result.Extent.([]*ethpb.Deposit); ok {
copy(deposits[result.Offset:], depositExtent)
} else {
return nil, errors.New("extent not of expected type")
}
}
return deposits, nil
}
// generateDepositsFromData a list of deposit items by creating proofs for each of them from a sparse Merkle trie.
func generateDepositsFromData(depositDataItems []*ethpb.Deposit_Data, offset int, trie *trieutil.SparseMerkleTrie) ([]*ethpb.Deposit, error) {
deposits := make([]*ethpb.Deposit, len(depositDataItems))
for i, item := range depositDataItems {
proof, err := trie.MerkleProof(i + offset)
if err != nil {
return nil, errors.Wrapf(err, "could not generate proof for deposit %d", i+offset)
}
deposits[i] = &ethpb.Deposit{
Proof: proof,
Data: item,
}
}
return deposits, nil
}
// DepositDataFromKeys generates a list of deposit data items from a set of BLS validator keys.
func DepositDataFromKeys(privKeys []*bls.SecretKey, pubKeys []*bls.PublicKey) ([]*ethpb.Deposit_Data, [][]byte, error) {
type depositData struct {
items []*ethpb.Deposit_Data
roots [][]byte
}
depositDataItems := make([]*ethpb.Deposit_Data, len(privKeys))
depositDataRoots := make([][]byte, len(privKeys))
results, err := mputil.Scatter(len(privKeys), func(offset int, entries int, _ *sync.RWMutex) (interface{}, error) {
items, roots, err := depositDataFromKeys(privKeys[offset:offset+entries], pubKeys[offset:offset+entries])
return &depositData{items: items, roots: roots}, err
})
if err != nil {
return nil, nil, errors.Wrap(err, "failed to generate deposit data from keys")
}
for _, result := range results {
if depositDataExtent, ok := result.Extent.(*depositData); ok {
copy(depositDataItems[result.Offset:], depositDataExtent.items)
copy(depositDataRoots[result.Offset:], depositDataExtent.roots)
} else {
return nil, nil, errors.New("extent not of expected type")
}
}
return depositDataItems, depositDataRoots, nil
}
func depositDataFromKeys(privKeys []*bls.SecretKey, pubKeys []*bls.PublicKey) ([]*ethpb.Deposit_Data, [][]byte, error) {
dataRoots := make([][]byte, len(privKeys))
depositDataItems := make([]*ethpb.Deposit_Data, len(privKeys))
for i := 0; i < len(privKeys); i++ {
data, err := createDepositData(privKeys[i], pubKeys[i])
if err != nil {
return nil, nil, errors.Wrapf(err, "could not create deposit data for key: %#x", privKeys[i].Marshal())
}
hash, err := ssz.HashTreeRoot(data)
if err != nil {
return nil, nil, errors.Wrap(err, "could not hash tree root deposit data item")
}
dataRoots[i] = hash[:]
depositDataItems[i] = data
}
return depositDataItems, dataRoots, nil
}
// Generates a deposit data item from BLS keys and signs the hash tree root of the data.
func createDepositData(privKey *bls.SecretKey, pubKey *bls.PublicKey) (*ethpb.Deposit_Data, error) {
di := &ethpb.Deposit_Data{
PublicKey: pubKey.Marshal(),
WithdrawalCredentials: withdrawalCredentialsHash(pubKey.Marshal()),
Amount: params.BeaconConfig().MaxEffectiveBalance,
}
sr, err := ssz.SigningRoot(di)
if err != nil {
return nil, err
}
domain := bls.ComputeDomain(params.BeaconConfig().DomainDeposit)
di.Signature = privKey.Sign(sr[:], domain).Marshal()
return di, nil
}
// withdrawalCredentialsHash forms a 32 byte hash of the withdrawal public
// address.
//
// The specification is as follows:
// withdrawal_credentials[:1] == BLS_WITHDRAWAL_PREFIX_BYTE
// withdrawal_credentials[1:] == hash(withdrawal_pubkey)[1:]
// where withdrawal_credentials is of type bytes32.
func withdrawalCredentialsHash(pubKey []byte) []byte {
h := hashutil.Hash(pubKey)
return append([]byte{blsWithdrawalPrefixByte}, h[1:]...)[:32]
}