Add documentation and issue template for feature flags (#6018)

* Add details around feature flag usage and recommend filing tracking issues for feature flags.

* 🦄

* 🦄

* labels

* more comments

.github/ISSUE_TEMPLATE/feature_flag.md

@@ -0,0 +1,40 @@
+---
+name: "\U0001F984Feature Flag Tracking"
+about: Track a new feature, in development, in Prysm. This issue template should only be used by 
+developers or contributors!
+labels: Tracking
+
+---
+<!--💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎
+
+Hellooo! 😄 
+
+Thanks for taking the time to file a tracking issue for your new feature. These issues really help
+us track progress of features as they work their way through development. Be sure to review 
+shared/featureconfig/README.md for the latest documentation around feature flags.
+
+💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎💎-->
+
+# 🦄 Feature Tracking
+
+**Current status** (opt-in or opt-out): 
+
+**Opt-in feature flag name**:
+
+**Opt-out feature flag name**: 
+
+### Feature Description
+
+<!-- ✍️--> A clear and concise description of the new feature. Provide links to your technical
+design document or other supporting information.
+
+### Tasks
+
+- [ ] Feature flag created as opt-in
+- [ ] Feature implemented / code complete
+- [ ] Feature tested in production for adequate amount of time
+- [ ] Feature flag is inverted to be opt-out and the opt-in flag is deprecated
+- [ ] Feature has made it to a tagged production release as an opt-out flag
+- [ ] Opt-out feature flag is deprecated and old code paths are cleaned up
+
+This issue should be closed after all of the above tasks are complete.

shared/featureconfig/README.md

@@ -0,0 +1,61 @@
+# Prysm Feature Flags
+
+Part of Prysm's feature development often involves use of feature flags which serve as a way to
+toggle new features as they are introduced. Using this methodology, you are assured that your
+feature can be safely tested in production with a fall back option if any regression were to occur.
+This reduces the likelihood of an emergency release or rollback of a given feature due to
+unforeseen issues.
+
+## When to use a feature flag?
+
+It is best to use a feature flag any time you are adding or removing functionality in an existing
+critical application path. 
+
+Examples of when to use a feature flag:
+
+- Adding a caching layer to an expensive RPC call.
+- Optimizing a particular algorithm or routine.
+- Introducing a temporary public testnet configuration.
+
+Examples of when not to use a feature flag:
+
+- Adding a new gRPC endpoint. (Unless dangerous or expensive to expose).
+- Adding new logging statements.
+- Removing dead code.
+- Adding any trivial feature with no risk of regressions to existing functionality.
+
+## How to use a feature flag?
+
+Once it has been decided that you should use a feature flag. Follow these steps to safely
+releasing your feature. In general, try to create a single PR for each step of this process.
+
+1. Add your feature flag to shared/featureconfig/flags.go, use the flag to toggle a boolean in the
+feature config in shared/featureconfig/config.go. It is a good idea to use the `enable` prefix for
+your flag since you're going to invert the flag in a later step. i.e you will use `disable` prefix
+later. For example, `--enable-my-feature`. Additionally, [create a feature flag tracking issue](https://github.com/prysmaticlabs/prysm/issues/new?template=feature_flag.md) 
+for your feature using the appropriate issue template.
+2. Use the feature throughout the application to enable your new functionality and be sure to write
+tests carefully and thoughtfully to ensure you have tested all of your new funcitonality without losing
+coverage on the existing functionality. This is considered an opt-in feature flag. Example usage:
+```go
+func someExistingMethod(ctx context.Context) error {
+    if featureconfig.Get().MyNewFeature {
+       return newMethod(ctx)
+    }
+    // Otherwise continue with the existing code path.
+}
+``` 
+3. Add the flag to the end to end tests. This set of flags can also be found in shared/featureconfig/flags.go. 
+4. Test the functionality locally and safely in production. Once you have enough confidence that
+your new function works and is safe to release then move onto the next step.
+5. Move your existing flag to the deprecated section of shared/featureconfig/flags.go. It is
+important NOT to delete your existing flag outright. Deleting a flag can be extremely frustrating
+to users as it may break their existing workflow! Marking a flag as deprecated gives users time to
+adjust their start scripts and workflow. Add another feature flag to represent the inverse of your
+flag from step 1. For example `--disable-my-feature`. Read the value of this flag to appropriately
+the config value in shared/featureconfig/config.go.
+6. After your feature has been included in a release as "opt-out" and there are no issues,
+deprecate the opt-out feature flag, delete the config field from shared/featureconfig/config.go,
+delete any deprecated / obsolete code paths.
+
+Deprecated flags are deleted upon each major semver point release. Ex: v1, v2, v3.