prysm-pulse/beacon-chain/db/kv/migration_archived_index.go

73 lines
1.9 KiB
Go
Raw Normal View History

Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
package kv
import (
"bytes"
"context"
types "github.com/prysmaticlabs/prysm/consensus-types/primitives"
"github.com/prysmaticlabs/prysm/encoding/bytesutil"
ethpb "github.com/prysmaticlabs/prysm/proto/prysm/v1alpha1"
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
bolt "go.etcd.io/bbolt"
)
var migrationArchivedIndex0Key = []byte("archive_index_0")
func migrateArchivedIndex(ctx context.Context, db *bolt.DB) error {
if updateErr := db.Update(func(tx *bolt.Tx) error {
mb := tx.Bucket(migrationsBucket)
if b := mb.Get(migrationArchivedIndex0Key); bytes.Equal(b, migrationCompleted) {
return nil // Migration already completed.
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
}
bkt := tx.Bucket(archivedRootBucket)
if bkt == nil {
return nil
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
}
// Remove "last archived index" key before iterating over all keys.
if err := bkt.Delete(lastArchivedIndexKey); err != nil {
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
return err
}
var highest types.Slot
c := bkt.Cursor()
for k, v := c.First(); k != nil; k, v = c.Next() {
// Look up actual slot from block
b := tx.Bucket(blocksBucket).Get(v)
// Skip this key if there is no block for whatever reason.
if b == nil {
continue
}
blk := &ethpb.SignedBeaconBlock{}
if err := decode(context.TODO(), b, blk); err != nil {
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
return err
}
if err := tx.Bucket(stateSlotIndicesBucket).Put(bytesutil.SlotToBytesBigEndian(blk.Block.Slot), v); err != nil {
return err
}
if blk.Block.Slot > highest {
highest = blk.Block.Slot
}
// check if context is cancelled in between
if ctx.Err() != nil {
return ctx.Err()
}
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
}
// Delete deprecated buckets.
for _, bkt := range [][]byte{slotsHasObjectBucket, archivedRootBucket} {
if tx.Bucket(bkt) != nil {
if err := tx.DeleteBucket(bkt); err != nil {
return err
}
}
}
// Mark migration complete.
return mb.Put(migrationArchivedIndex0Key, migrationCompleted)
}); updateErr != nil {
log.WithError(updateErr).Errorf("could not migrate bucket: %s", archivedRootBucket)
return updateErr
}
return nil
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
}