Commit Graph

5533 Commits

Author SHA1 Message Date
Raul Jordan
148e7fcd59
Consolidate Required Web UI Endpoints Into Single Protobuf File (#8127)
* define required endpoints for unifying backends

* proto definitions

* impl

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-16 17:30:48 +00:00
terence tsao
20dede7532
Move state summary cache to DB (#8101) 2020-12-16 08:56:21 -08:00
Preston Van Loon
3fb49433a1
Limit prometheus requests in flight, set timeout to reasonable 30 seconds (#8130) 2020-12-16 04:57:02 +00:00
Preston Van Loon
4326cbbf08
Validator database: use snappy compression on encoded attestation history (#8129)
* Remove old buckets, rename new buckets to be canonical

* Add migration and test

* Do the migrations

* Gazelle

* Clearify code, code review

* gofmt
2020-12-16 03:33:04 +00:00
terence tsao
dc27cd7a1e
Pass context to NewKVStore (#8125)
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-15 22:07:01 +00:00
terence tsao
0449cd3450
Allow update cache at last slot of the epoch (#8094)
* Epoch boundary updates at the last slot of the epoch

* Report metric on first slot

* Remove comment

* Add locks to param config

* Remove lock for copy

* Revert "Add locks to param config"

This reverts commit 79d5130b58ae7f80c72109e29cc9ea775c91b063.

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-15 21:19:02 +00:00
terence tsao
6244163770
FC test coverage improvement (#8120)
* Test coverage improvements

* Gazelle

* Update namings

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-15 20:44:35 +00:00
Victor Farazdagi
f5c87075f2
Fix issue with roaming data dir on Wins (#8095)
* patch beacon node

* make sure that check is executed before anything else

* fix format

* make sure that check is reusable by other executables

* debug infor

* fix typo

* more debug info

* use copydir

* cleanup

* better explanation

* gazelle

* go fmt

* debug tos permissions

* upadte copydir

* gazelle

* better check of tos acceptance

* expand path

* update validator

* move fixing func

* move fixing method

* make sure that updater works both on main and subcommands

* remove from startnode

* add copydir test

* add DirFiles method and tests

* fix test

* add and test HashDir

* update tests

* fix test

* add datadir removal

* update messages

* further update messages
2020-12-15 14:01:51 -06:00
Preston Van Loon
ad7d3c74cc
Validator DB cleanup: remove obsolete buckets (#8122) 2020-12-15 18:24:56 +00:00
Nishant Das
508c5fcf2f
More Efficient Validation of Proposer Index (#8107)
* metric

* make it better

* make it better

* gaz

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-15 16:27:05 +00:00
Victor Farazdagi
a0c475671c
Add extra methods to shared/fileutil (#8117)
* add extra methods to fileutil

* Shay's suggestion

* Update shared/fileutil/fileutil.go

Co-authored-by: Shay Zluf <thezluf@gmail.com>

* Adds reference to the original implementation

Co-authored-by: Shay Zluf <thezluf@gmail.com>
2020-12-15 14:18:15 +00:00
Preston Van Loon
72a92fe708
CI: Disable remote caching of GoStdLib (#8111)
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-15 05:58:52 +00:00
terence tsao
6a5589f99e
Skip proposer indices cache update if exists (#8096)
* Check if cache is empty before update

* Add tests

* Fix tests

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-15 00:09:30 +00:00
terence tsao
70c0bb106b
Beacon node code health improvements (#8109)
* Apply code health fixes after code inspect

* Remove attestationRoot

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-14 23:41:24 +00:00
Victor Farazdagi
0f18867f08
Remove duplicate package imports (#8104)
* remove duplicate imports

* remove unused lookupLimit

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-14 22:22:55 +00:00
Cipio
630d57377a
Fix Slasher Backup DB panic on call. (#8099) 2020-12-14 21:39:26 +00:00
terence tsao
3e9d721280
Validator code health improvements (#8106) 2020-12-14 21:10:02 +00:00
Steven Allen
2428880058
Update go-libp2p to 0.12.0 (#8015)
* Update go-libp2p to 0.12.0

go-libp2p 0.12.0 made some significant changes to the stream interfaces around
stream closing:

* Close now closes in both directions and frees the stream. However, unlike
FullClose did, it doesn't _wait_ for the remote peer to respond with an EOF.
* To close for writing, call CloseWrite (like one would on a TCP connection, etc.).

This patch:

* Replaces calls to FullClose with Close where appropriate.
* Replaces calls to Close with CloseWrite where appropriate.
* Removes redundant Close calls.
* Calls Reset to where appropriate to indicate that the request/response was
  aborted. Unlike Close, this will not flush and will not cause the remote peer
  to read an EOF. Instead, the remote peer will read an ErrReset error.
* Ensures we always either close or reset streams. Send wasn't closing the
  stream on some error paths.
* Now that stream closing is async, we explicitly wait for a response when
  "hanging up" on a peer (so we don't hang up before they receive our
  response/goodbye message).

* update bazel

* Gazelle

* revert unintentional bazel workspace change

* appease an overzealous linter

* update to latest

* Refactor encoder

* gazelle

* Gazelle

Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-14 17:22:25 +00:00
Potuz
4d1f01aacc
Change block_arrival_latency buckets to exponential format (#8065)
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
2020-12-14 16:00:07 +00:00
Victor Farazdagi
b9848dc94f
Remove unused vars in tests (#8103)
* update process_block_test

* service_attester_test

* update service and rpc_status tests

* go fmt
2020-12-13 05:23:13 +00:00
Nishant Das
579335f81a
Remove Saving Of Target And Source Epoch (#8102)
* remove

* skip
2020-12-13 04:26:18 +00:00
Nishant Das
11bbf06d03
Add Inbound Peer Limit (#7942)
* add changes

* fix up

* fix

* add test

* fix test

* fix again

Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
2020-12-12 03:46:55 +00:00
Nishant Das
29804fa572
Refactor Subnet Search (#8048)
* checkpoint progress

* checkpoint

* clean up

* do better

* fix test

* fix

* fix

* preston's review

* fix

* fix

* fix

* add iterator

* go doc

* make it a config parameter

* Apply suggestions from code review

Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
2020-12-11 18:40:56 -08:00
Radosław Kapka
4ec396c025
Refetch validating keys if no keys are fetched (#8000)
* refetch validating keys every 30 seconds

* deduplicate error/log messages

* remove redundant break statement

* comment about execution flow

* move code to wait_for_activation.go

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
2020-12-11 20:55:52 +01:00
Raul Jordan
1fbfd52e52
Simpler and Safer Attester Slashing Protection (#8086)
* att sign validations

* eliminate old cached methods and use a simple approach in the db

* redefined db methods

* db package builds

* add multilock to attest and propose

* gaz

* removed concurrency tests that are no longer relevant

* add cache to db functions for attesting history checks

* passing

* add in feature flag --disable-attesting-history-db-cache

* remove lock

* Revert "remove lock"

This reverts commit b1a65020e406b9fa6e4f57c5cc6a5a0b2a11a240.

* comment

* gaz
2020-12-11 12:31:35 -06:00
Mohamed Mansour
2e18df642d
Expand Validator Input Errors (#8090)
The reason of why a validator input was failing was very generic,
this allows the actual implementation to state why it failed by
passing the exception forward.

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-11 10:57:47 -06:00
Nishant Das
99b3835f19
Add Fallback Option for Eth1 Nodes (#8062)
* fix

* fix tests and change back

* gaz

* change

* ready again

* Update beacon-chain/flags/base.go

Co-authored-by: Shay Zluf <thezluf@gmail.com>

* radek's review

* Update shared/cmd/helpers.go

* Update shared/cmd/helpers_test.go

* Update shared/cmd/helpers_test.go

* Endpoint/endpoint

Co-authored-by: Shay Zluf <thezluf@gmail.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
2020-12-11 19:15:04 +08:00
Victor Farazdagi
46d99fdc00
Attestation aggregation: remove proper set duplicates (#8063)
* fix proper-set issue

* move test into dedicated test file

* move functionality from maxcover -> proposer

* adds dedup to proposer

* fix tests

* update tests

* remove redundant proper subset test

* fix bug with identical expression
2020-12-11 11:13:14 +03:00
Preston Van Loon
923e4d3a5e
Attempt to reconnect when waiting for activation (#8057)
* Attempt to reconnect when waiting for activation

* rm fuzz/attestation_fuzz.go

* math.Min, not math.Max

* Gofmt

* resolve RVV-B0003
2020-12-10 19:26:31 -06:00
terence tsao
bb9e2ba12c
Validator client logs time left to next duty (#8088) 2020-12-10 10:56:18 -08:00
Preston Van Loon
f44b2a35e4
Update waiting validator methods to be context aware (#8078)
* Update waiting validator methods to be context aware

* Enable debug logging for validator in e2e

* invert logic in deferred function, remove for loop

* return early if wait is 0 or less

* overwrite ctx

* t.Stop() chan closure is hanging. Doesnt make sense, so ignoring that cleanup task. It shouldnt happen at runtime anyway

Co-authored-by: terence tsao <terence@prysmaticlabs.com>
2020-12-10 10:00:48 -06:00
terence tsao
00dacbd00d
Remove custom block body root and block root methods (#8069)
* Remove custom block body root and block root methods

* Add nil checks and fix tests

* Fmt

* Typo
2020-12-09 12:11:42 -06:00
Radosław Kapka
92736d0188
warn when creating an imported wallet (#8081) 2020-12-09 16:41:26 +01:00
Preston Van Loon
c96db1a122
Add spans to pre and post att signing updates (#8079)
* Add spans to pre and post att signing updates

* span on db method

* Add span to isNewAttSlashable
2020-12-08 23:47:02 -06:00
terence tsao
c5770a2e56
Add beacon block nil body checks (#8077)
* Add nil checks and fix tests

* Gaz

* Revert one minor typo

* Update test

* Remove extra space
2020-12-08 16:27:58 -06:00
terence tsao
ade3b2f2df
Add state field count to config (#8068)
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-07 21:49:28 -06:00
Radosław Kapka
9d7052796b
Restore database CLI command (#8061)
* restore beacon node db

* revert image name

* move restore out of the kv folder

* remove files from kv folder

* go mod tidy

* Remove usage of prometheus testutil

* add yes/no to prompt text

* restore slasher db

* organize imports

* go mod tidy

* restore validator db

* close slasher db

* defer close backup db in tests

* simplify function literal
2020-12-07 21:36:43 +01:00
Shay Zluf
fbbdd94fea
Resolve panic on shutdown with offline eth1 node (#8033)
* Resolve panic on shutdown with offline eth1 node

* Move fix tne right place nishant feedback

* fix log message

Co-authored-by: Nishant Das <nishdas93@gmail.com>
2020-12-07 10:16:23 +00:00
Victor Farazdagi
b51aec6981
Init sync: minor fixes (#8060)
* fixes typo

* simplify code
2020-12-07 09:33:06 +00:00
terence tsao
b4437e6cec
Load graffiti from file (#8041)
* Pass graffati file and   use it

* Visibility

* Parse test

* More proposal tests

* Gazelle

* Add sequential functionality

* fix length check

* Update priorities. Specified -> random -> default

* Log warn instead return err

* Comment

* E2e test

* Comment

Co-authored-by: Preston Van Loon <preston@prysmaticlabs.com>
2020-12-04 23:15:12 +00:00
Arthur Burkart
8ad328d9b3
fix(grpcHeaders): accept values with "=" symbols (#8047)
* fix(grpcHeaders): accept values with equal signs

# What

Before this commit, it was not possible to pass in base64-encoded
content as a header value if it contained an equals sign. This commit
changes the behavior slightly. Rather than ignore key/value pairs where
the value happens to have an equals sign, we assume the first equals
sign delimits the key from the value and pass in the rest of the value
as-is.

Also, instead of printing the header name along with its value, we
print the name, so there is less risk of leaking information into logs
that shouldn't be there.

# Testing

This has not been tested in the context of the full validator client.
Instead, a small example was made to demonstrate the feasibility. The
example is shown here:

```go
package main

import (
	"log"
	"os"
	"strings"

	"github.com/davecgh/go-spew/spew"
	"github.com/urfave/cli/v2"
)

type Config struct {
	GrpcHeadersFlag string
}

func main() {
	app := &cli.App{
		Action: func(c *cli.Context) error {
			for _, hdr := range strings.Split(c.String("grpc-headers"), ",") {
				if hdr != "" {
					ss := strings.Split(hdr, "=")
					spew.Dump(ss[0])
					spew.Dump(strings.Join(ss[1:], "="))
				}
			}
			return nil
		},
		Flags: []cli.Flag{
			&cli.StringFlag{
				Name: "grpc-headers",
				Usage: "A comma-separated list of key value pairs to pass as gRPC headers for all gRPC " +
					"calls. Example: --grpc-headers=key=value",
			},
		},
	}

	err := app.Run(os.Args)
	if err != nil {
		log.Fatal(err)
	}
}
```

Example invocation:

```command
❯ go run main.go --grpc-headers=key=value,Authorization="Basic $(echo -n hello:world | base64)"
(string) (len=3) "key"
(string) (len=5) "value"
(string) (len=13) "Authorization"
(string) (len=22) "Basic aGVsbG86d29ybGQ="
```

* Adds tests to new gRPC header parsing code

* bazel run //:gazelle
2020-12-04 21:31:15 +00:00
Victor Farazdagi
21ede7634e
Attestation aggregation: maxcover vs naive aggregation effectiveness test (#8043)
* maxcover performance test

* gazelle

Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-04 17:24:57 +00:00
Victor Farazdagi
57b74283d3
Attestation aggregation: removes redundant code from max-cover benchmarks (#8044)
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-04 16:51:31 +00:00
Victor Farazdagi
be078d6a16
Update TestMain(): do not call os.Exit() explicitly (#8046)
* update workspace

* update testmain
2020-12-04 16:10:07 +00:00
Victor Farazdagi
ccba8cfa5a
Update rules_go to v0.24.9 and golang to v1.15.6 (#8045) 2020-12-04 15:28:03 +00:00
Victor Farazdagi
afbfaedea4
Unskip fixed init-sync service test (#8042) 2020-12-04 11:57:46 +00:00
terence tsao
3ce96701de
Update attestation schedule log with total attester count (#8013)
* Log attesting total

* Use the right library

* Go fmt

* Use fmt

* No space is better

Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
2020-12-04 07:02:58 +00:00
simonatsn
d2ba45aad9
Update blst to v0.3.1 and incorporate subgroup changes (#7971)
* Update blst to v0.3.1 and incorporate subgroup changes

* go mod tidy

* gofmt

* Update bzl blst dependency

* Remove unnecessary check for nil

* Run bazel run //:gazelle -- fix

* Update blst to v0.3.2

* fix sha

Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
Co-authored-by: Raul Jordan <raul@prysmaticlabs.com>
2020-12-04 06:46:08 +00:00
Raul Jordan
14e1f08208
Add Backup Webhooks to All Prysm Services With DBs (#8025)
* integrate backup webhooks

* pass slasher tests

* fix node

* cmd

* gaz

* read test passes

* radek feedback

* added comment

Co-authored-by: Victor Farazdagi <simple.square@gmail.com>
2020-12-03 22:28:57 +00:00
Victor Farazdagi
647b4cf108
Round robin: half open interval in syncToFinalizedEpoch (#8039) 2020-12-03 21:55:42 +00:00