Windows build automation (#1998)

* PowerShell script * Patch Readme.md
2024-12-22 03:30:37 +00:00 · 2021-05-23 16:50:44 +02:00 · 2021-05-23 16:50:44 +02:00 · 7020946a2e
commit 7020946a2e
parent 450c79e3ab
2 changed files with 247 additions and 16 deletions
--- a/README.md
+++ b/README.md
@ -20,7 +20,7 @@ Erigon is an implementation of Ethereum (aka "Ethereum client"), on the efficien
    + [Run all components by docker-compose](#run-all-components-by-docker-compose)
    + [Grafana dashboard](#grafana-dashboard)
 - [Getting in touch](#getting-in-touch)
-    + [Turbo-Geth Discord Server](#turbo-geth-discord-server)
+    + [Erigon Discord Server](#erigon-discord-server)
    + [Reporting security issues/concerns](#reporting-security-issues-concerns)
    + [Team](#team)
 - [Known issues](#known-issues)
@ -53,8 +53,8 @@ Usage
 ### Getting Started

 ```sh
-> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/turbo-geth.git
-> cd turbo-geth
+> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/erigon.git
+> cd erigon
 > make tg
 > ./build/bin/tg
 ```
@ -63,8 +63,8 @@ Usage

 If you would like to give turbo-geth a try, but do not have spare 2Tb on your driver, a good option is to start syncing one of the public testnets, Görli. It syncs much quicker, and does not take so much disk space:
 ```sh
-> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/turbo-geth.git
-> cd turbo-geth
+> git clone --recurse-submodules -j8 https://github.com/ledgerwatch/erigon.git
+> cd erigon
 > make tg
 > ./build/bin/tg --datadir goerli --chain goerli
 ```
@ -91,9 +91,11 @@ Support only remote-miners.

 ### Windows

-Windows users may run turbo-geth in 3 possible ways:
+Windows users may run erigon in 3 possible ways:

-* Build tg binaries natively for Windows : while this method is possible we still lack a fully automated build process thus, at the moment, is not to be preferred. Besides there's also a caveat which might cause your experience with TG as native on Windows uncomfortable: data file allocation is fixed so you need to know in advance how much space you want to allocate for database file using the option `--lmdb.mapSize`
+* Build executable binaries natively for Windows using provided `win-build.ps1` PowerShell script which has to be run with local Administrator privileges.
+The script creates `libmdbx.dll` (MDBX is current default database for Erigon) and copies it into Windows's `system32` folder (generally `C:\Windows\system32`). 
+Though is still possible to run erigon with LMDB database there's a caveat which might cause your experience with LMDB on Windows uncomfortable: data file allocation is fixed so you need to know in advance how much space you want to allocate for database file using the command line option `--lmdb.mapSize`

 * Use Docker :  see [docker-compose.yml](./docker-compose.yml)

@ -138,7 +140,7 @@ accounts and the storage.

 ### Faster Initial Sync

-Turbo-Geth uses a rearchitected full sync algorithm from
+Erigon uses a rearchitected full sync algorithm from
 [Go-Ethereum](https://github.com/ethereum/go-ethereum) that is split into
 "stages".

@ -158,15 +160,17 @@ Examples of stages are:

 * Downloading block bodies;

+* Recovering senders' addresses;
+
 * Executing blocks;

 * Validating root hashes and building intermediate hashes for the state Merkle trie;

-* And more...
+* [...]

 ### JSON-RPC daemon

-In turbo-geth RPC calls are extracted out of the main binary into a separate daemon.
+In Erigon RPC calls are extracted out of the main binary into a separate daemon.
 This daemon can use both local or remote DBs. That means, that this RPC daemon
 doesn't have to be running on the same machine as the main turbo-geth binary or
 it can run from a snapshot of a database for read-only calls. 
@ -222,7 +226,7 @@ XDG_DATA_HOME=/preferred/data/folder docker-compose up
 Getting in touch
 ================

-### Turbo-Geth Discord Server
+### Erigon Discord Server

 The main discussions are happening on our Discord server. 
 To get an invite, send an email to `tg [at] torquem.ch` with your name, occupation, 
@ -240,11 +244,11 @@ Core contributors (in alpabetical order of first names):

 * Alexey Akhunov ([@realLedgerwatch](https://twitter.com/realLedgerwatch))

-* Andrea Lanfranchi
+* Andrea Lanfranchi([@AndreaLanfranchi](https://github.com/AndreaLanfranchi))

 * Andrew Ashikhmin ([yperbasis](https://github.com/yperbasis))

-* Artem Vorotnikov
+* Artem Vorotnikov ([vorot93](https://github.com/vorot93))

 * Boris Petrov ([b00ris](https://github.com/b00ris))

@ -252,7 +256,7 @@ Core contributors (in alpabetical order of first names):

 * Igor Mandrigin ([@mandrigin](https://twitter.com/mandrigin))

-* Giulio Rebuffo
+* Giulio Rebuffo ([Giulio2002](https://github.com/Giulio2002))

 * Thomas Jay Rush ([@tjayrush](https://twitter.com/tjayrush))

@ -271,7 +275,7 @@ Known issues

 ### `htop` shows incorrect memory usage

-TurboGeth's internal DB (LMDB) using `MemoryMap` - when OS does manage all `read, write, cache` operations instead of Application
+Erigon's internal DB (LMDB) using `MemoryMap` - when OS does manage all `read, write, cache` operations instead of Application
 ([linux](https://linux-kernel-labs.github.io/refs/heads/master/labs/memory_mapping.html), [windows](https://docs.microsoft.com/en-us/windows/win32/memory/file-mapping))

 `htop` on column `res` shows memory of "App + OS used to hold page cache for given App", 
@ -286,7 +290,7 @@ Without `grep` you can see details - `section MALLOC ZONE column Resident Size`
 - `Prometheus` dashboard shows memory of Go app without OS pages cache (`make prometheus`, open in browser `localhost:3000`, credentials `admin/admin`)
 - `cat /proc/<PID>/smaps`

-TurboGeth uses ~4Gb of RAM during genesis sync and < 1Gb during normal work. OS pages cache can utilize unlimited amount of memory. 
+Erigon uses ~4Gb of RAM during genesis sync and < 1Gb during normal work. OS pages cache can utilize unlimited amount of memory. 

 **Warning:** Multiple instances of TG on same machine will touch Disk concurrently, 
 it impacts performance - one of main TG optimisations: "reduce Disk random access". 
--- a/win-build.ps1
+++ b/win-build.ps1
@ -0,0 +1,227 @@
+<#
+   Copyright 2021 The Erigon Authors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+#>
+
+$goErrorText = @"
+
+ Requirement Error.
+ You need to have Go Programming Language (aka golang) installed.
+ Minimum required version is 1.16
+ Please visit https://golang.org/dl/ and download the appropriate
+ installer.
+
+"@
+
+$chocolateyErrorText = @"
+
+ Requirement Error.
+ For this script to run properly you need to install
+ chocolatey [https://chocolatey.org/] with the following
+ mandatory components: 
+
+ - cmake 3.20.2
+ - make 4.3
+ - mingw 10.2.0
+
+"@
+
+$chocolateyPathErrorText = @"
+
+ Environment PATH Error.
+ Chocolatey install has been detected but the environment
+ variable PATH does not include a full path to its binaries
+ Please amend your setup and ensure the following
+ chocolatey directory is properly inserted into your PATH
+ environment variable.
+
+"@
+
+$privilegeErrorText = @"
+
+ Privileges Error !
+ You must run this script with Administrator privileges
+
+"@
+
+$Error.Clear()
+$ErrorActionPreference = "SilentlyContinue"
+
+Set-Variable -Name "MyContext" -Value ([hashtable]::Synchronized(@{})) -Scope Script
+$MyContext.Name       = $MyInvocation.MyCommand.Name
+$MyContext.Definition = $MyInvocation.MyCommand.Definition
+$MyContext.Directory  = (Split-Path (Resolve-Path $MyInvocation.MyCommand.Definition) -Parent)
+$MyContext.StartDir   = (Get-Location -PSProvider FileSystem).ProviderPath
+$MyContext.WinVer     = (Get-WmiObject Win32_OperatingSystem).Version.Split(".")
+$MyContext.PSVer      = [int]$PSVersionTable.PSVersion.Major
+
+# Test we have ADMIN Privileges
+$myWindowsID = [System.Security.Principal.WindowsIdentity]::GetCurrent();
+$myWindowsPrincipal = New-Object System.Security.Principal.WindowsPrincipal($myWindowsID);
+$adminRole = [System.Security.Principal.WindowsBuiltInRole]::Administrator;
+if (!($myWindowsPrincipal.IsInRole($adminRole))) {
+   Write-Host $privilegeErrorText
+   return
+}
+
+
+# Test GO is installed and is at least 1.16
+$goInstalled = (Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-Object {$_.DisplayName -imatch "^Go\ Programming\ Language"})
+if(!$goInstalled) {
+    Write-Host $goErrorText
+    return
+}
+Write-Host " Found GO version $($goInstalled.DisplayVersion)"
+$goVersionMajor = [int]$goInstalled.VersionMajor
+$goVersionMinor = [int]$goInstalled.VersionMinor
+if($goVersionMajor -lt 1 -or $goVersionMinor -lt 16) {
+    Write-Host $goErrorText
+    return
+}
+
+# Test Chocolatey Install
+if(!(Test-Path env:chocolateyInstall)) {
+    Write-Host $chocolateyErrorText
+    return
+}
+$chocolateyPath = $env:ChocolateyInstall
+
+# Test Chocolatey bin directory is actually in %PATH%
+$chocolateyBinPath = (Join-Path $chocolateyPath "bin")
+$chocolateyBinPathInPath = $false
+$pathExpanded = $env:Path.Split(";")
+for($i=0; $i -lt $pathExpanded.Count; $i++){
+    $pathItem = $pathExpanded[$i]
+    if($pathItem -ieq $chocolateyBinPath){
+        Write-Host " Found $($chocolateyBinPath) in PATH"
+        $chocolateyBinPathInPath = $true
+    }
+}
+if(!$chocolateyBinPathInPath) {
+    Write-Host $chocolateyPathErrorText
+    Write-Host $chocolateyBinPath
+    return
+}
+
+# Test Chocolatey Components
+$chocolateyHasCmake = $false
+$chocolateyHasMake = $false
+$chocolateyHasMingw = $false
+$chocolateyComponents = @(clist -l)
+for($i=0; $i -lt $chocolateyComponents.Count; $i++){
+    $item = $chocolateyComponents[$i]
+    if($item -imatch "^cmake\ [0-9]") {
+        $chocolateyHasCmake = $true
+        Write-Host " Found Chocolatey component $($item)"
+    }
+    if($item -imatch "^make\ [0-9]") {
+        $chocolateyHasMake = $true
+        Write-Host " Found Chocolatey component $($item)"
+    }
+    if($item -imatch "^mingw\ [0-9]") {
+        $chocolateyHasMingw = $true
+        Write-Host " Found Chocolatey component $($item)"
+    }
+}
+If(!$chocolateyHasCmake -or !$chocolateyHasMake -or !$chocolateyHasMingw) {
+    Write-Host $chocolateyErrorText
+    return
+}
+
+# Enter MDBX directory and build libmdbx.dll
+Write-Host " Building libmdbx.dll ..."
+Set-Location (Join-Path $MyContext.Directory "ethdb\mdbx\dist")
+cmake -G "MinGW Makefiles" . -D CMAKE_MAKE_PROGRAM:PATH=$(Join-Path $chocolateyBinPath "make.exe") -D MDBX_BUILD_SHARED_LIBRARY:BOOL=ON -D MDBX_WITHOUT_MSVC_CRT:BOOOL=OFF
+if($LASTEXITCODE) {
+    Write-Host "An error has occurred while configuring MDBX dll"
+    return
+}
+cmake --build .
+if($LASTEXITCODE -or !(Test-Path "libmdbx.dll" -PathType leaf)) {
+    Write-Host "An error has occurred while building MDBX dll or libmdbx.dll cannot be found"
+    return
+}
+
+# Copy libmdbx.dll into %windir%\System32 directory
+# Note! default behavior is to overwrite
+Copy-Item libmdbx.dll (Join-Path $env:SystemRoot system32)
+if(!$?) {
+   Write-Host " Error ! Could not copy libmdbx.dll to $(Join-Path $env:SystemRoot system32)"
+   Write-Host " What can you do : "
+   Write-Host " - Check your permissions to directory "
+   Write-Host " - Check there's an already existing libmdbx.dll file "
+   Write-Host " - Check no instance of Erigon with mdbx is currently running "
+   return
+}
+
+# Return to source folder
+Set-Location $MyContext.Directory
+
+# Build erigon binaries
+Set-Variable -Name "Erigon" -Value ([hashtable]::Synchronized(@{})) -Scope Script
+$Erigon.Commit  = [string]@(git.exe rev-list -1 HEAD)
+$Erigon.Branch  = [string]@(git.exe rev-parse --abbrev-ref HEAD)
+$Erigon.Build   = "go build -v -trimpath -tags=mdbx -ldflags ""-X main.gitCommit=$($Erigon.Commit) -X main.gitBranch=$($Erigon.Branch)"""
+$Erigon.BinPath = [string](Join-Path $MyContext.StartDir "\build\bin")
+$env:GO111MODULE = "on"
+
+Write-Host " Building tg ..."
+$outExecutable = [string](Join-Path $Erigon.BinPath "tg.exe")
+$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/tg"
+$BuildCommand += ';$?'
+$success = Invoke-Expression -Command $BuildCommand
+if (-not $success) {
+    Write-Host "Could not build tg executable"
+    return
+}
+
+Write-Host " Building rpcdaemon ..."
+$outExecutable = [string](Join-Path $Erigon.BinPath "rpcdaemon.exe")
+$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/rpcdaemon"
+$BuildCommand += ';$?'
+$success = Invoke-Expression -Command $BuildCommand
+if (-not $success) {
+    Write-Host "Could not build rpcdaemon executable"
+    return
+}
+
+Write-Host " Building integration ..."
+$outExecutable = [string](Join-Path $Erigon.BinPath "integration.exe")
+$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/integration"
+$BuildCommand += ';$?'
+$success = Invoke-Expression -Command $BuildCommand
+if (-not $success) {
+    Write-Host "Could not build integration executable"
+    return
+}
+
+Write-Host " Building rpctest ..."
+$outExecutable = [string](Join-Path $Erigon.BinPath "rpctest.exe")
+$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/rpctest"
+$BuildCommand += ';$?'
+$success = Invoke-Expression -Command $BuildCommand
+if (-not $success) {
+    Write-Host "Could not build rpctest executable"
+    return
+}
+
+Write-Host " Building state ..."
+$outExecutable = [string](Join-Path $Erigon.BinPath "state.exe")
+$BuildCommand = "$($Erigon.Build) -o ""$($outExecutable)"" ./cmd/state"
+$BuildCommand += ';$?'
+$success = Invoke-Expression -Command $BuildCommand
+if (-not $success) {
+    Write-Host "Could not build state executable"
+    return
+}