prysm-pulse/beacon-chain/state/stategen/migrate.go

117 lines
3.4 KiB
Go
Raw Normal View History

package stategen
import (
"context"
"encoding/hex"
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
"fmt"
"github.com/prysmaticlabs/prysm/v3/beacon-chain/state"
"github.com/prysmaticlabs/prysm/v3/encoding/bytesutil"
"github.com/sirupsen/logrus"
"go.opencensus.io/trace"
)
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
// MigrateToCold advances the finalized info in between the cold and hot state sections.
// It moves the recent finalized states from the hot section to the cold section and
// only preserves the ones that are on archived point.
func (s *State) MigrateToCold(ctx context.Context, fRoot [32]byte) error {
ctx, span := trace.StartSpan(ctx, "stateGen.MigrateToCold")
defer span.End()
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
s.finalizedInfo.lock.RLock()
oldFSlot := s.finalizedInfo.slot
s.finalizedInfo.lock.RUnlock()
fBlock, err := s.beaconDB.Block(ctx, fRoot)
if err != nil {
return err
}
fSlot := fBlock.Block().Slot()
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if oldFSlot > fSlot {
return nil
}
// Start at previous finalized slot, stop at current finalized slot (it will be handled in the next migration).
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
// If the slot is on archived point, save the state of that slot to the DB.
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
for slot := oldFSlot; slot < fSlot; slot++ {
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if ctx.Err() != nil {
return ctx.Err()
}
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
if slot%s.slotsPerArchivedPoint == 0 && slot != 0 {
cached, exists, err := s.epochBoundaryStateCache.getBySlot(slot)
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if err != nil {
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
return fmt.Errorf("could not get epoch boundary state for slot %d", slot)
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
}
var aRoot [32]byte
var aState state.BeaconState
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
// When the epoch boundary state is not in cache due to skip slot scenario,
// we have to regenerate the state which will represent epoch boundary.
// By finding the highest available block below epoch boundary slot, we
// generate the state for that block root.
if exists {
aRoot = cached.root
aState = cached.state
} else {
_, roots, err := s.beaconDB.HighestRootsBelowSlot(ctx, slot)
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if err != nil {
return err
}
// Given the block has been finalized, the db should not have more than one block in a given slot.
// We should error out when this happens.
if len(roots) != 1 {
return errUnknownBlock
}
aRoot = roots[0]
// There's no need to generate the state if the state already exists in the DB.
// We can skip saving the state.
if !s.beaconDB.HasState(ctx, aRoot) {
aState, err = s.StateByRoot(ctx, aRoot)
if err != nil {
return err
}
}
2020-05-22 01:15:16 +00:00
}
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if s.beaconDB.HasState(ctx, aRoot) {
// If you are migrating a state and its already part of the hot state cache saved to the db,
// you can just remove it from the hot state cache as it becomes redundant.
s.saveHotStateDB.lock.Lock()
roots := s.saveHotStateDB.blockRootsOfSavedStates
for i := 0; i < len(roots); i++ {
if aRoot == roots[i] {
s.saveHotStateDB.blockRootsOfSavedStates = append(roots[:i], roots[i+1:]...)
// There shouldn't be duplicated roots in `blockRootsOfSavedStates`.
// Break here is ok.
break
}
}
s.saveHotStateDB.lock.Unlock()
continue
}
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if err := s.beaconDB.SaveState(ctx, aState, aRoot); err != nil {
return err
}
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
log.WithFields(
logrus.Fields{
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
"slot": aState.Slot(),
"root": hex.EncodeToString(bytesutil.Trunc(aRoot[:])),
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
}).Info("Saved state in DB")
}
}
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
// Update finalized info in memory.
fInfo, ok, err := s.epochBoundaryStateCache.getByBlockRoot(fRoot)
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if err != nil {
return err
2020-05-22 01:15:16 +00:00
}
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
if ok {
s.SaveFinalizedState(fSlot, fRoot, fInfo.state)
2020-05-22 01:15:16 +00:00
}
Hot states use no DB (#6488) * Add cache to service struct * Update hot getters/setters to use cache * Update migration * Update other services to adapt * Fix initial sync get state * Update getter related tests * Update hot related tests * Update migrate related tests * New awesome tests for migration * Clean up rest of the tests * Merge refs/heads/master into hot-state-no-db * Fix block chain head tests * Fix block chain processor tests * Fixed RPC tests * Update cold getter and test * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Fix sync tests * Short cut if state is already in DB * Remove uneeded saves * Update beacon-chain/state/stategen/hot_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/getter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/service.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/state/stategen/setter_test.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Preston's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Return a copy of cache states * Remove hot state caches check in StateByRoot * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db * Merge refs/heads/master into hot-state-no-db * Raul's feedback * Merge branch 'hot-state-no-db' of github.com:prysmaticlabs/prysm into hot-state-no-db
2020-07-06 17:22:12 +00:00
return nil
2020-05-22 01:15:16 +00:00
}