prysm-pulse/beacon-chain/db/kv/migration.go

28 lines
476 B
Go
Raw Normal View History

Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
package kv
import (
"context"
bolt "go.etcd.io/bbolt"
)
var migrationCompleted = []byte("done")
type migration func(context.Context, *bolt.DB) error
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
var migrations = []migration{
migrateArchivedIndex,
migrateBlockSlotIndex,
Efficient Historical State Representation (#9155) * added migration in to new schema * gazel fix * goimports fix * construct state with validator entries from a seperate bucket * save state with validator entry indirection * fixed save and retreieve issues * fixed more test cases related to DeepEqual * added save benchmark * gazel fix * organize benchmarks * add cache and improve state construction performance * gazel fix * check type * remove cache checking from Has * fix decoding when cache is off * fix slice bugs * add migration testcases * linter fix * fix few review comments * fix review feedback * gazel fix * satisfy deepsource * added the feature flag * lint fix * fix usage of featureconfig Inti in testcases * call resetConfig method * add feature flag in migration testcases * fix formatting * change bucket name from blockRootValidatorKeysIndexBucket to blockRootValidatorHashesBucket * remove from cache when state deleted * fixed few more comments * added to devModeFlags * added cache delete under the feature flag * fix lint * change cache sizes and improve documentation * fiexed few more review coments * not using hash anymore and using a new SaveStates function * satisfu deepsource * run gazel * fix feature flag related stuff * fixing merge conflict fix * few goodies * improve UX and dont swallow error * merge fix * import format fix * fix migrationion when flag not given issue Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: Nishant Das <nishdas93@gmail.com>
2021-07-27 20:02:53 +00:00
migrateStateValidators,
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
}
// RunMigrations defined in the migrations array.
func (s *Store) RunMigrations(ctx context.Context) error {
for _, m := range migrations {
if err := m(ctx, s.db); err != nil {
Improved cold state checkpoints: migrate database index and usage (#6461) * Add database migrations, still need to update the API usage... * gofmt goimports * progress * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * use slot instead of index * rename LastArchivedIndex to LastArchivedSlot * rename LastArchivedIndexRoot to LastArchivedRoot * remove unused HighestSlotStates method * deprecate old key, include in migration * deprecate old key, include in migration * remove blocks index in migration * rename bucket variable * fix code to pass tests * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * gofmt, goimports * fix * Add state slot index * progress * lint * fix build * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration * kafka * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * remove SaveArchivedPointRoot, a few other big changes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * fix tests and lint * lint again * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * block migration, some renaming * gaz, gofmt * add tests * change index to uint bytes * Merge branch 'index-migration' of github.com:prysmaticlabs/prysm into index-migration * rm method notes * stop if the bucket doesn't exist * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * @rauljordan pr feedback * Simplify * Merge refs/heads/master into index-migration * Remove unused method, add roundtrip test * gofmt * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge refs/heads/master into index-migration * Merge branch 'master' of github.com:prysmaticlabs/prysm into index-migration
2020-07-18 18:05:04 +00:00
return err
}
}
return nil
}