This change introduces additional processes to manage snapshot uploading
for E2 snapshots:
## erigon snapshots upload
The `snapshots uploader` command starts a version of erigon customized
for uploading snapshot files to
a remote location.
It breaks the stage execution process after the senders stage and then
uses the snapshot stage to send
uploaded headers, bodies and (in the case of polygon) bor spans and
events to snapshot files. Because
this process avoids execution in run signifigantly faster than a
standard erigon configuration.
The uploader uses rclone to send seedable (100K or 500K blocks) to a
remote storage location specified
in the rclone config file.
The **uploader** is configured to minimize disk usage by doing the
following:
* It removes snapshots once they are loaded
* It aggressively prunes the database once entities are transferred to
snapshots
in addition to this it has the following performance related features:
* maximizes the workers allocated to snapshot processing to improve
throughput
* Can be started from scratch by downloading the latest snapshots from
the remote location to seed processing
## snapshots command
Is a stand alone command for managing remote snapshots it has the
following sub commands
* **cmp** - compare snapshots
* **copy** - copy snapshots
* **verify** - verify snapshots
* **manifest** - manage the manifest file in the root of remote snapshot
locations
* **torrent** - manage snapshot torrent files
This is the second part of the bor milestone release it contains the
following changes:
* Initialize services
* This is a change from the initial pull request I have moved all of the
initialization to the bor engine. To facilitate this I have just passed
in the heimdall client interface, rather than the whole engine
* Stage processing
* This is also a change from the original PR - the code is contained in
the bor heimdall stage rather than in headers - the effect should be the
same, but this needs testing
---------
Co-authored-by: Mark Holt <mark@disributed.vision>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
this first major move separates the transient beacon state cache from
the underlying tree.
leaf updates are enforced in the setters, which should make programming
easier.
all exported methods of the raw.BeaconState should be safe to call
(without disrupting internal state)
changes many functions to consume *raw.BeaconState in perparation for
interface
beyond refactor it also:
adds a pool for the leaves of the validator ssz hash
adds a pool for the snappy writers
removed the parallel hash experiment (high memory use)
- lives in internal/logging
- all log flags moved to internal/logging/flags
- allows continued use of root logger via log.Info etc.
- update logger to take change allowing string to lvl for 'trace'
Verbosity flag is overridden by log.console.verbosity. Logs will be
colocated if all run as one process, only split where progs are run as
separate processes, in a future update this will be addressed so for
example rpcdeamon will always log to it's own file
Upstreams the improvement from the maticnetwork/erigon fork
to add a goreleaser workflow that will build and release
a docker image for both arm and amd64.
* Patch plumbing of docker-compose UID/GID build args
* Fallback to 1000/1000 if DOCKER_(U|G)ID not set
* Revise README.md instructions for docker further
* Fix existing typo forc 'servie' -> 'service'
* Rename PUID/GUID -> UID/GID
* Specify user in erigon docker service
* Rely on .env instead of specifying :-1000
* Polish Makefile for docker use case
* one more helpful comment
* make docker should use UID/GID --build-arg
* Fix make docker and more fail-fast if envvar set incorrect
* mv .env->.env.example to not intefere existing workflows
* Specify envvars in docker CI
* Adjust validate_docker_build_args to permit non-erigon user
* Also run docker CI target on macos-11 os
* Add DOCKER_UID, DOCKER_GID in hooks/build
* Patch docker build arg validation for macos
* Add actions-setup-docker@master for macos
* Don't run automated test for docker macos
* Cleanup Makefile
* Comments, targets for erigon users
* More Makefile cleanup, debugging still
* Typo fix
* Create subdirs before calling ls
* Get rid of flaky validation
* DOCKER_UID, DOCKER_GID init to runner if not set
* Get rid of unnecessary variable for now
* Improved README based on new changes
* Proper uid/gid `make user_*` when no envars set
* Fix typo in Makefile comment
* Fix make docker as sudo user
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* move mdbx to submodule
* fix `make grpc` on new checkouts
* update proto files
* add some stub
* prototype with fake events
* notifying about events
* pass events
* events are being sent
* transfer headers to filters
* create the “filters” struct
* implement new heads
* PoC of New Heads subscription
* fix keep alive
* fixups for the client
* add “type” to the event
* support header event type on client
* better stage refactor
* fixup for the eth backend
* fixups
* fix tests
* fix tests
* fix linters
* address comments
* remove unused log
* add docker-compose
* test.http
* make different dockerfile for each component
* make different dockerfile for each component
* fix versions
* remove dedicated docker files
* cleanup
* cleanup
* cleanup
* build: use golangci-lint
This changes build/ci.go to download and run golangci-lint instead
of gometalinter.
* core/state: fix unnecessary conversion
* p2p/simulations: fix lock copying (found by go vet)
* signer/core: fix unnecessary conversions
* crypto/ecies: remove unused function cmpPublic
* core/rawdb: remove unused function print
* core/state: remove unused function xTestFuzzCutter
* core/vm: disable TestWriteExpectedValues in a different way
* core/forkid: remove unused function checksum
* les: remove unused type proofsData
* cmd/utils: remove unused functions prefixedNames, prefixFor
* crypto/bn256: run goimports
* p2p/nat: fix goimports lint issue
* cmd/clef: avoid using unkeyed struct fields
* les: cancel context in testRequest
* rlp: delete unreachable code
* core: gofmt
* internal/build: simplify DownloadFile for Go 1.11 compatibility
* build: remove go test --short flag
* .travis.yml: disable build cache
* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement
* .golangci.yml: enable goconst and ineffassign linters
* build: print message when there are no lint issues
* internal/build: refactor download a bit
- added a case error struct that contains information about certain error cases
in which we would like to output more information to the client
- added a validation method that iterates and adds the information that is
stored in the error cases
* dashboard: footer, deep state update
* dashboard: resolve asset path
* dashboard: remove bundle.js
* dashboard: prevent state update on every reconnection
* dashboard: fix linter issue
* dashboard, cmd: minor UI fix, include commit hash
* remove geth binary
* dashboard: gitCommit renamed to commit
* dashboard: move the geth version to the right, make commit optional
* dashboard: commit limited to 7 characters
* dashboard: limit commit length on client side
* dashboard: run go generate
This commit converts the dependency management from Godeps to the vendor
folder, also switching the tool from godep to trash. Since the upstream tool
lacks a few features proposed via a few PRs, until those PRs are merged in
(if), use github.com/karalabe/trash.
You can update dependencies via trash --update.
All dependencies have been updated to their latest version.
Parts of the build system are reworked to drop old notions of Godeps and
invocation of the go vet command so that it doesn't run against the vendor
folder, as that will just blow up during vetting.
The conversion drops OpenCL (and hence GPU mining support) from ethash and our
codebase. The short reasoning is that there's noone to maintain and having
opencl libs in our deps messes up builds as go install ./... tries to build
them, failing with unsatisfied link errors for the C OpenCL deps.
golang.org/x/net/context is not vendored in. We expect it to be fetched by the
user (i.e. using go get). To keep ci.go builds reproducible the package is
"vendored" in build/_vendor.
The new build script, ci.go, replaces some of the older shell scripts.
ci.go can compile go-ethereum, run the tests, create release archives
and debian source packages.
Many people need or want to build go-ethereum from the git repository,
mostly to stay up to date with recent changes. We cannot expect that
people without Go experience grok the Go workspace concept.
With the Makefile, building from github requires only
three steps (provided that a Go toolchain is installed):
- git clone https://github.com/ethereum/go-ethereum
- ... install C libraries (libgmp, etc.) ...
- make
