prysm-pulse/io/logs/logutil.go

61 lines
1.8 KiB
Go
Raw Permalink Normal View History

// Package logs creates a Multi writer instance that
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
// write all logs that are written to stdout.
package logs
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
import (
"io"
Mask credentials for logging (#8429) * Added MaskCredentialsLogging to logutil, which masks the user info, path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added MaskCredentialsLogging to logutil, which masks the user info,path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added newline based on the PR checks * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Gazelle * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * added unit tests * updated one test case * added logutil_test.go unit test cases * Refactor validator subnet subscriptions to be non-blocking (#8319) * Use response.NextEpochDuties for aggregator subnet subscriptions (credit: @KaanKC PR #8204). Make committee subnet subscriptions method non-blocking call * Fix test * Fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> * Add Ability to Specify All Public Keys When Exiting Validators (#8399) * add programmatic voluntary exit * add exit all flag * test * lint * add multiple exits test * fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Increase Validation Queue (#8431) Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Validator: add a DEBUG log to show batch attestation save duration (#8432) * Add a debug log to show duration * Autofix issues in 1 file Resolved issues in validator/db/kv/attester_protection.go via DeepSource Autofix Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Add Mutex and Block Profiling (#8435) * Implement GetStateRoot in the beacon API (#8402) * update ethereumapis dependency * span * initial implementation * introduce stategen Service interface and MockService * Include AddStateForSlot function in the mock service * return states from mock * add GenesisState to POWChain mock * populate roots in helper state * initialize Slot when creating helper state * tests * code refactor - extract helper functions * gzl * use SetSlot in tests * handle SetSlot error * use new testutil's NewBeaconState * gzl * go mod tidy * rename Service to StateManager * move regex check to helper Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * On Block Cleanup (#8438) * Beacon API: update GetStateRoot (#8437) * Address various feedbacks * Gaz * More nil check * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: Radosław Kapka <rkapka@wp.pl> * qualifying my unix user ahaddad - no real changes to the files * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nishdas93@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2021-02-15 21:54:19 +00:00
"net/url"
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
"os"
"path/filepath"
Mask credentials for logging (#8429) * Added MaskCredentialsLogging to logutil, which masks the user info, path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added MaskCredentialsLogging to logutil, which masks the user info,path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added newline based on the PR checks * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Gazelle * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * added unit tests * updated one test case * added logutil_test.go unit test cases * Refactor validator subnet subscriptions to be non-blocking (#8319) * Use response.NextEpochDuties for aggregator subnet subscriptions (credit: @KaanKC PR #8204). Make committee subnet subscriptions method non-blocking call * Fix test * Fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> * Add Ability to Specify All Public Keys When Exiting Validators (#8399) * add programmatic voluntary exit * add exit all flag * test * lint * add multiple exits test * fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Increase Validation Queue (#8431) Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Validator: add a DEBUG log to show batch attestation save duration (#8432) * Add a debug log to show duration * Autofix issues in 1 file Resolved issues in validator/db/kv/attester_protection.go via DeepSource Autofix Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Add Mutex and Block Profiling (#8435) * Implement GetStateRoot in the beacon API (#8402) * update ethereumapis dependency * span * initial implementation * introduce stategen Service interface and MockService * Include AddStateForSlot function in the mock service * return states from mock * add GenesisState to POWChain mock * populate roots in helper state * initialize Slot when creating helper state * tests * code refactor - extract helper functions * gzl * use SetSlot in tests * handle SetSlot error * use new testutil's NewBeaconState * gzl * go mod tidy * rename Service to StateManager * move regex check to helper Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * On Block Cleanup (#8438) * Beacon API: update GetStateRoot (#8437) * Address various feedbacks * Gaz * More nil check * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: Radosław Kapka <rkapka@wp.pl> * qualifying my unix user ahaddad - no real changes to the files * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nishdas93@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2021-02-15 21:54:19 +00:00
"strings"
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
"github.com/prysmaticlabs/prysm/v5/config/params"
"github.com/prysmaticlabs/prysm/v5/io/file"
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
"github.com/sirupsen/logrus"
)
func addLogWriter(w io.Writer) {
mw := io.MultiWriter(logrus.StandardLogger().Out, w)
logrus.SetOutput(mw)
}
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
// ConfigurePersistentLogging adds a log-to-file writer. File content is identical to stdout.
func ConfigurePersistentLogging(logFileName string) error {
logrus.WithField("logFileName", logFileName).Info("Logs will be made persistent")
if err := file.MkdirAll(filepath.Dir(logFileName)); err != nil {
return err
}
f, err := os.OpenFile(logFileName, os.O_CREATE|os.O_WRONLY|os.O_APPEND, params.BeaconIoConfig().ReadWritePermissions) // #nosec G304
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
if err != nil {
return err
}
addLogWriter(f)
Persistent logs (#2660) * added file log feature * moved the logic to one central location (shared/logutil/logutil.go), removed new line chars from file logs * removed a resdundant temp file refrences that went into beachon-chain/BUILD.bazel by mistake * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * Update shared/cmd/flags.go Co-Authored-By: shayzluf <thezluf@gmail.com> * manually added loguitl dep to the go image target * Manaully added the logutil dep to the go image target * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: terence tsao <terence@prysmaticlabs.com> * syntax and styling changes required by code reviewers * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * changed the return type of 'ConfigurePersistentLogging' from bool, error to error based on recommendation from code review * ran goimports in beacon-chain/main.go after tests have failed * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update beacon-chain/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Update validator/main.go Co-Authored-By: Preston Van Loon <preston@prysmaticlabs.com> * Changes requested by code reviewer * Added a mandatory comment (linter required) to the 'Fire' event * Changed the beacon-chain and validator to support only same format stdout and file logging due to complications in the outputs when using different formats. * Had to run gazelle --fix due to check failure
2019-05-22 13:22:11 +00:00
logrus.Info("File logging initialized")
return nil
}
Mask credentials for logging (#8429) * Added MaskCredentialsLogging to logutil, which masks the user info, path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added MaskCredentialsLogging to logutil, which masks the user info,path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added newline based on the PR checks * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Gazelle * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * added unit tests * updated one test case * added logutil_test.go unit test cases * Refactor validator subnet subscriptions to be non-blocking (#8319) * Use response.NextEpochDuties for aggregator subnet subscriptions (credit: @KaanKC PR #8204). Make committee subnet subscriptions method non-blocking call * Fix test * Fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> * Add Ability to Specify All Public Keys When Exiting Validators (#8399) * add programmatic voluntary exit * add exit all flag * test * lint * add multiple exits test * fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Increase Validation Queue (#8431) Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Validator: add a DEBUG log to show batch attestation save duration (#8432) * Add a debug log to show duration * Autofix issues in 1 file Resolved issues in validator/db/kv/attester_protection.go via DeepSource Autofix Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Add Mutex and Block Profiling (#8435) * Implement GetStateRoot in the beacon API (#8402) * update ethereumapis dependency * span * initial implementation * introduce stategen Service interface and MockService * Include AddStateForSlot function in the mock service * return states from mock * add GenesisState to POWChain mock * populate roots in helper state * initialize Slot when creating helper state * tests * code refactor - extract helper functions * gzl * use SetSlot in tests * handle SetSlot error * use new testutil's NewBeaconState * gzl * go mod tidy * rename Service to StateManager * move regex check to helper Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * On Block Cleanup (#8438) * Beacon API: update GetStateRoot (#8437) * Address various feedbacks * Gaz * More nil check * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: Radosław Kapka <rkapka@wp.pl> * qualifying my unix user ahaddad - no real changes to the files * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nishdas93@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2021-02-15 21:54:19 +00:00
// MaskCredentialsLogging masks the url credentials before logging for security purpose
Mask credentials for logging (#8429) * Added MaskCredentialsLogging to logutil, which masks the user info, path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added MaskCredentialsLogging to logutil, which masks the user info,path and query. It leaves the hostname and port untouched . Making it more secure during logging * Added newline based on the PR checks * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Gazelle * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> * added unit tests * updated one test case * added logutil_test.go unit test cases * Refactor validator subnet subscriptions to be non-blocking (#8319) * Use response.NextEpochDuties for aggregator subnet subscriptions (credit: @KaanKC PR #8204). Make committee subnet subscriptions method non-blocking call * Fix test * Fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> * Add Ability to Specify All Public Keys When Exiting Validators (#8399) * add programmatic voluntary exit * add exit all flag * test * lint * add multiple exits test * fix test Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Increase Validation Queue (#8431) Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * Validator: add a DEBUG log to show batch attestation save duration (#8432) * Add a debug log to show duration * Autofix issues in 1 file Resolved issues in validator/db/kv/attester_protection.go via DeepSource Autofix Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> * Add Mutex and Block Profiling (#8435) * Implement GetStateRoot in the beacon API (#8402) * update ethereumapis dependency * span * initial implementation * introduce stategen Service interface and MockService * Include AddStateForSlot function in the mock service * return states from mock * add GenesisState to POWChain mock * populate roots in helper state * initialize Slot when creating helper state * tests * code refactor - extract helper functions * gzl * use SetSlot in tests * handle SetSlot error * use new testutil's NewBeaconState * gzl * go mod tidy * rename Service to StateManager * move regex check to helper Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> * On Block Cleanup (#8438) * Beacon API: update GetStateRoot (#8437) * Address various feedbacks * Gaz * More nil check * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> * Update beacon-chain/rpc/beaconv1/state_test.go Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: Radosław Kapka <rkapka@wp.pl> * qualifying my unix user ahaddad - no real changes to the files * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go * Update shared/logutil/logutil.go Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com> Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: Raul Jordan <raul@prysmaticlabs.com> Co-authored-by: Nishant Das <nishdas93@gmail.com> Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com> Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2021-02-15 21:54:19 +00:00
// [scheme:][//[userinfo@]host][/]path[?query][#fragment] --> [scheme:][//[***]host][/***][#***]
// if the format is not matched nothing is done, string is returned as is.
func MaskCredentialsLogging(currUrl string) string {
// error if the input is not a URL
MaskedUrl := currUrl
u, err := url.Parse(currUrl)
if err != nil {
return currUrl // Not a URL, nothing to do
}
// Mask the userinfo and the URI (path?query or opaque?query ) and fragment, leave the scheme and host(host/port) untouched
if u.User != nil {
MaskedUrl = strings.Replace(MaskedUrl, u.User.String(), "***", 1)
}
if len(u.RequestURI()) > 1 { // Ignore the '/'
MaskedUrl = strings.Replace(MaskedUrl, u.RequestURI(), "/***", 1)
}
if len(u.Fragment) > 0 {
MaskedUrl = strings.Replace(MaskedUrl, u.RawFragment, "***", 1)
}
return MaskedUrl
}