From 7020946a2e80620988a3025cda17aac4a636bf33 Mon Sep 17 00:00:00 2001 From: Andrea Lanfranchi Date: Sun, 23 May 2021 16:50:44 +0200 Subject: [PATCH] Windows build automation (#1998) * PowerShell script * Patch Readme.md --- README.md | 36 ++++---- win-build.ps1 | 227 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+), 16 deletions(-) create mode 100644 win-build.ps1 diff --git a/README.md b/README.md index bfa991c8a..d2613df03 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ Erigon is an implementation of Ethereum (aka "Ethereum client"), on the efficien + [Run all components by docker-compose](#run-all-components-by-docker-compose) + [Grafana dashboard](#grafana-dashboard) - [Getting in touch](#getting-in-touch) - + [Turbo-Geth Discord Server](#turbo-geth-discord-server) + + [Erigon Discord Server](#erigon-discord-server) + [Reporting security issues/concerns](#reporting-security-issues-concerns) + [Team](#team) - [Known issues](#known-issues) @@ -53,8 +53,8 @@ Usage ### Getting Started ```sh -> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/turbo-geth.git -> cd turbo-geth +> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/erigon.git +> cd erigon > make tg > ./build/bin/tg ``` @@ -63,8 +63,8 @@ Usage If you would like to give turbo-geth a try, but do not have spare 2Tb on your driver, a good option is to start syncing one of the public testnets, Görli. It syncs much quicker, and does not take so much disk space: ```sh -> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/turbo-geth.git -> cd turbo-geth +> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/erigon.git +> cd erigon > make tg > ./build/bin/tg --datadir goerli --chain goerli ``` @@ -91,9 +91,11 @@ Support only remote-miners. ### Windows -Windows users may run turbo-geth in 3 possible ways: +Windows users may run erigon in 3 possible ways: -* Build tg binaries natively for Windows : while this method is possible we still lack a fully automated build process thus, at the moment, is not to be preferred. Besides there's also a caveat which might cause your experience with TG as native on Windows uncomfortable: data file allocation is fixed so you need to know in advance how much space you want to allocate for database file using the option `--lmdb.mapSize` +* Build executable binaries natively for Windows using provided `win-build.ps1` PowerShell script which has to be run with local Administrator privileges. +The script creates `libmdbx.dll` (MDBX is current default database for Erigon) and copies it into Windows's `system32` folder (generally `C:\Windows\system32`). +Though is still possible to run erigon with LMDB database there's a caveat which might cause your experience with LMDB on Windows uncomfortable: data file allocation is fixed so you need to know in advance how much space you want to allocate for database file using the command line option `--lmdb.mapSize` * Use Docker : see [docker-compose.yml](./docker-compose.yml) @@ -138,7 +140,7 @@ accounts and the storage. ### Faster Initial Sync -Turbo-Geth uses a rearchitected full sync algorithm from +Erigon uses a rearchitected full sync algorithm from [Go-Ethereum](https://github.com/ethereum/go-ethereum) that is split into "stages". @@ -158,15 +160,17 @@ Examples of stages are: * Downloading block bodies; +* Recovering senders' addresses; + * Executing blocks; * Validating root hashes and building intermediate hashes for the state Merkle trie; -* And more... +* [...] ### JSON-RPC daemon -In turbo-geth RPC calls are extracted out of the main binary into a separate daemon. +In Erigon RPC calls are extracted out of the main binary into a separate daemon. This daemon can use both local or remote DBs. That means, that this RPC daemon doesn't have to be running on the same machine as the main turbo-geth binary or it can run from a snapshot of a database for read-only calls. @@ -222,7 +226,7 @@ XDG_DATA_HOME=/preferred/data/folder docker-compose up Getting in touch ================ -### Turbo-Geth Discord Server +### Erigon Discord Server The main discussions are happening on our Discord server. To get an invite, send an email to `tg [at] torquem.ch` with your name, occupation, @@ -240,11 +244,11 @@ Core contributors (in alpabetical order of first names): * Alexey Akhunov ([@realLedgerwatch](https://twitter.com/realLedgerwatch)) -* Andrea Lanfranchi +* Andrea Lanfranchi([@AndreaLanfranchi](https://github.com/AndreaLanfranchi)) * Andrew Ashikhmin ([yperbasis](https://github.com/yperbasis)) -* Artem Vorotnikov +* Artem Vorotnikov ([vorot93](https://github.com/vorot93)) * Boris Petrov ([b00ris](https://github.com/b00ris)) @@ -252,7 +256,7 @@ Core contributors (in alpabetical order of first names): * Igor Mandrigin ([@mandrigin](https://twitter.com/mandrigin)) -* Giulio Rebuffo +* Giulio Rebuffo ([Giulio2002](https://github.com/Giulio2002)) * Thomas Jay Rush ([@tjayrush](https://twitter.com/tjayrush)) @@ -271,7 +275,7 @@ Known issues ### `htop` shows incorrect memory usage -TurboGeth's internal DB (LMDB) using `MemoryMap` - when OS does manage all `read, write, cache` operations instead of Application +Erigon's internal DB (LMDB) using `MemoryMap` - when OS does manage all `read, write, cache` operations instead of Application ([linux](https://linux-kernel-labs.github.io/refs/heads/master/labs/memory_mapping.html), [windows](https://docs.microsoft.com/en-us/windows/win32/memory/file-mapping)) `htop` on column `res` shows memory of "App + OS used to hold page cache for given App", @@ -286,7 +290,7 @@ Without `grep` you can see details - `section MALLOC ZONE column Resident Size` - `Prometheus` dashboard shows memory of Go app without OS pages cache (`make prometheus`, open in browser `localhost:3000`, credentials `admin/admin`) - `cat /proc//smaps` -TurboGeth uses ~4Gb of RAM during genesis sync and < 1Gb during normal work. OS pages cache can utilize unlimited amount of memory. +Erigon uses ~4Gb of RAM during genesis sync and < 1Gb during normal work. OS pages cache can utilize unlimited amount of memory. **Warning:** Multiple instances of TG on same machine will touch Disk concurrently, it impacts performance - one of main TG optimisations: "reduce Disk random access". diff --git a/win-build.ps1 b/win-build.ps1 new file mode 100644 index 000000000..16a73b39d --- /dev/null +++ b/win-build.ps1 @@ -0,0 +1,227 @@ +<# + Copyright 2021 The Erigon Authors + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +#> + +$goErrorText = @" + + Requirement Error. + You need to have Go Programming Language (aka golang) installed. + Minimum required version is 1.16 + Please visit https://golang.org/dl/ and download the appropriate + installer. + +"@ + +$chocolateyErrorText = @" + + Requirement Error. + For this script to run properly you need to install + chocolatey [https://chocolatey.org/] with the following + mandatory components: + + - cmake 3.20.2 + - make 4.3 + - mingw 10.2.0 + +"@ + +$chocolateyPathErrorText = @" + + Environment PATH Error. + Chocolatey install has been detected but the environment + variable PATH does not include a full path to its binaries + Please amend your setup and ensure the following + chocolatey directory is properly inserted into your PATH + environment variable. + +"@ + +$privilegeErrorText = @" + + Privileges Error ! + You must run this script with Administrator privileges + +"@ + +$Error.Clear() +$ErrorActionPreference = "SilentlyContinue" + +Set-Variable -Name "MyContext" -Value ([hashtable]::Synchronized(@{})) -Scope Script +$MyContext.Name = $MyInvocation.MyCommand.Name +$MyContext.Definition = $MyInvocation.MyCommand.Definition +$MyContext.Directory = (Split-Path (Resolve-Path $MyInvocation.MyCommand.Definition) -Parent) +$MyContext.StartDir = (Get-Location -PSProvider FileSystem).ProviderPath +$MyContext.WinVer = (Get-WmiObject Win32_OperatingSystem).Version.Split(".") +$MyContext.PSVer = [int]$PSVersionTable.PSVersion.Major + +# Test we have ADMIN Privileges +$myWindowsID = [System.Security.Principal.WindowsIdentity]::GetCurrent(); +$myWindowsPrincipal = New-Object System.Security.Principal.WindowsPrincipal($myWindowsID); +$adminRole = [System.Security.Principal.WindowsBuiltInRole]::Administrator; +if (!($myWindowsPrincipal.IsInRole($adminRole))) { + Write-Host $privilegeErrorText + return +} + + +# Test GO is installed and is at least 1.16 +$goInstalled = (Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-Object {$_.DisplayName -imatch "^Go\ Programming\ Language"}) +if(!$goInstalled) { + Write-Host $goErrorText + return +} +Write-Host " Found GO version $($goInstalled.DisplayVersion)" +$goVersionMajor = [int]$goInstalled.VersionMajor +$goVersionMinor = [int]$goInstalled.VersionMinor +if($goVersionMajor -lt 1 -or $goVersionMinor -lt 16) { + Write-Host $goErrorText + return +} + +# Test Chocolatey Install +if(!(Test-Path env:chocolateyInstall)) { + Write-Host $chocolateyErrorText + return +} +$chocolateyPath = $env:ChocolateyInstall + +# Test Chocolatey bin directory is actually in %PATH% +$chocolateyBinPath = (Join-Path $chocolateyPath "bin") +$chocolateyBinPathInPath = $false +$pathExpanded = $env:Path.Split(";") +for($i=0; $i -lt $pathExpanded.Count; $i++){ + $pathItem = $pathExpanded[$i] + if($pathItem -ieq $chocolateyBinPath){ + Write-Host " Found $($chocolateyBinPath) in PATH" + $chocolateyBinPathInPath = $true + } +} +if(!$chocolateyBinPathInPath) { + Write-Host $chocolateyPathErrorText + Write-Host $chocolateyBinPath + return +} + +# Test Chocolatey Components +$chocolateyHasCmake = $false +$chocolateyHasMake = $false +$chocolateyHasMingw = $false +$chocolateyComponents = @(clist -l) +for($i=0; $i -lt $chocolateyComponents.Count; $i++){ + $item = $chocolateyComponents[$i] + if($item -imatch "^cmake\ [0-9]") { + $chocolateyHasCmake = $true + Write-Host " Found Chocolatey component $($item)" + } + if($item -imatch "^make\ [0-9]") { + $chocolateyHasMake = $true + Write-Host " Found Chocolatey component $($item)" + } + if($item -imatch "^mingw\ [0-9]") { + $chocolateyHasMingw = $true + Write-Host " Found Chocolatey component $($item)" + } +} +If(!$chocolateyHasCmake -or !$chocolateyHasMake -or !$chocolateyHasMingw) { + Write-Host $chocolateyErrorText + return +} + +# Enter MDBX directory and build libmdbx.dll +Write-Host " Building libmdbx.dll ..." +Set-Location (Join-Path $MyContext.Directory "ethdb\mdbx\dist") +cmake -G "MinGW Makefiles" . -D CMAKE_MAKE_PROGRAM:PATH=$(Join-Path $chocolateyBinPath "make.exe") -D MDBX_BUILD_SHARED_LIBRARY:BOOL=ON -D MDBX_WITHOUT_MSVC_CRT:BOOOL=OFF +if($LASTEXITCODE) { + Write-Host "An error has occurred while configuring MDBX dll" + return +} +cmake --build . +if($LASTEXITCODE -or !(Test-Path "libmdbx.dll" -PathType leaf)) { + Write-Host "An error has occurred while building MDBX dll or libmdbx.dll cannot be found" + return +} + +# Copy libmdbx.dll into %windir%\System32 directory +# Note! default behavior is to overwrite +Copy-Item libmdbx.dll (Join-Path $env:SystemRoot system32) +if(!$?) { + Write-Host " Error ! Could not copy libmdbx.dll to $(Join-Path $env:SystemRoot system32)" + Write-Host " What can you do : " + Write-Host " - Check your permissions to directory " + Write-Host " - Check there's an already existing libmdbx.dll file " + Write-Host " - Check no instance of Erigon with mdbx is currently running " + return +} + +# Return to source folder +Set-Location $MyContext.Directory + +# Build erigon binaries +Set-Variable -Name "Erigon" -Value ([hashtable]::Synchronized(@{})) -Scope Script +$Erigon.Commit = [string]@(git.exe rev-list -1 HEAD) +$Erigon.Branch = [string]@(git.exe rev-parse --abbrev-ref HEAD) +$Erigon.Build = "go build -v -trimpath -tags=mdbx -ldflags ""-X main.gitCommit=$($Erigon.Commit) -X main.gitBranch=$($Erigon.Branch)""" +$Erigon.BinPath = [string](Join-Path $MyContext.StartDir "\build\bin") +$env:GO111MODULE = "on" + +Write-Host " Building tg ..." +$outExecutable = [string](Join-Path $Erigon.BinPath "tg.exe") +$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/tg" +$BuildCommand += ';$?' +$success = Invoke-Expression -Command $BuildCommand +if (-not $success) { + Write-Host "Could not build tg executable" + return +} + +Write-Host " Building rpcdaemon ..." +$outExecutable = [string](Join-Path $Erigon.BinPath "rpcdaemon.exe") +$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/rpcdaemon" +$BuildCommand += ';$?' +$success = Invoke-Expression -Command $BuildCommand +if (-not $success) { + Write-Host "Could not build rpcdaemon executable" + return +} + +Write-Host " Building integration ..." +$outExecutable = [string](Join-Path $Erigon.BinPath "integration.exe") +$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/integration" +$BuildCommand += ';$?' +$success = Invoke-Expression -Command $BuildCommand +if (-not $success) { + Write-Host "Could not build integration executable" + return +} + +Write-Host " Building rpctest ..." +$outExecutable = [string](Join-Path $Erigon.BinPath "rpctest.exe") +$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/rpctest" +$BuildCommand += ';$?' +$success = Invoke-Expression -Command $BuildCommand +if (-not $success) { + Write-Host "Could not build rpctest executable" + return +} + +Write-Host " Building state ..." +$outExecutable = [string](Join-Path $Erigon.BinPath "state.exe") +$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/state" +$BuildCommand += ';$?' +$success = Invoke-Expression -Command $BuildCommand +if (-not $success) { + Write-Host "Could not build state executable" + return +}