Commit Graph

309 Commits

Author SHA1 Message Date
ledgerwatch
93151f0ae6
[erigon2.2] Reduce allocations when replaying historical txs (#4460)
* [erigon2.2] Reduce allocations when replaying historical txs

* import new API, fix code

* Add hack decompress function

* update erigon-lib

* Fix hack

* Update to latest erigon-lib

* Fix reindexing

* Enable skip analysis for tracing calls too

* Enable for eth_getLogs

* Fix skip analysis

* Optimise

* Stop grpc server in the tests

* Print

* No panic on server stop

* Update to latest erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-17 13:40:49 +01:00
ledgerwatch
2e4e7bd4de
[erigon2.2] Log stats tool (#4453)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-15 07:10:37 +01:00
ledgerwatch
e47dd15e68
[erigon2.2] Prototype of eth_getLogs (#4437)
* Initial work

* Fix compile errors

* Update

* Debug

* Cleanup

* Temp changes

* Temp

* Remove temp

* Remove print

* Upgate to erigon-lib main

* go mod tidy

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-13 21:25:17 +01:00
ledgerwatch
fd8adddce8
trace_filter implementation based on erigon 2 update 2 data (#4431)
* trace_filter implementation based on erigon 2 update 2 data

* Fix test compile, extract txNums

* Update to erigon-lib, add generic heap

* Fix getHeaderHash for RPC methods

* Missing files

* Skip tests

* Add reward traces

* Filter trace lint

* Reinstate filtering

* Print

* Print

* Fix

* Print

* Print txNums

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Remove prints

* Fix nil dereference

* Inclusive toBlock bound

* Print

* Print

* Hack

* remove some deps

* Fix lint

* Update erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-06-12 12:44:01 +01:00
ledgerwatch
96603f14da
Print changes (#4351)
* Print changes

* Comment out prints, add hack to find prefix

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-03 22:50:20 +01:00
Alex Sharov
726ef081ce
option to disable runtime.ReadMemStats (#4200) 2022-05-19 11:48:07 +07:00
Alex Sharov
e1dec529d4
Generic sort of slices (no allocs, inlinable) (#4161)
* save

* save

* save

* save

* save
2022-05-16 08:24:12 +01:00
ledgerwatch
25208964e5
Export chainspecs into json files (#4148)
* Export chainspecs into json files

* Add sepolia

* Add more chains

* Extract rest of the chains

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-15 17:37:42 +01:00
Alex Sharov
8b8c442dc6
Open shorter logs #400 2022-04-28 10:11:07 +07:00
Håvard Anda Estensen
6844e74ad2
Replace ioutil with io and os (#3946) 2022-04-23 15:43:00 +01:00
Alex Sharov
6aa4999729
Parallel indexing (#3921) 2022-04-21 10:34:10 +07:00
TBC Dev
bf06095811
hack: Allow blockTotal <= 0 as offset from head (#3818) 2022-04-05 08:07:07 +07:00
Alex Sharov
8b03582558
compressor: log lvl #408
* save

* save
2022-04-01 10:44:30 +07:00
battlmonstr
47df98a499
Use "err" key for logging errors. (#3632)
log.Warn/Error uses "err" key to log errors in most places.
This renames "error" to "err" in some places to adhere to this convention.
2022-03-01 15:40:51 +00:00
ledgerwatch
ef6013faff
Automation for gathering erigon2 history data (#3501)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-12 22:16:06 +00:00
ledgerwatch
2074b98568
Tools for gathering data on erigon2 history and bitmaps (#3488)
* Tools for gathering data on erigon2 history and bitmaps

* Always start with 0

* Always start with 0

* Clean up commitment files, trace block

* traceblock

* Add tracing to checkChangeSets

* Fixes to DeleteAccount

* Update to latest erigon-lib

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-12 15:48:38 +00:00
Alex Sharov
c9cee7a884
path -> filepath (path package is for urls) (#3493) 2022-02-12 20:33:09 +07:00
ledgerwatch
079a177614
[erigon2] Computing data dependencies between transactions (#3290)
* Use ibs per transaction instead of per block

* Remove copy functions, introduce balance increase map

* Fix test

* Fix lint

* Introduce balanceIncrease journal action

* Introduce balanceIncrease journal action

* Simplifications

* Fix

* Reset

* Simplify

* Fix

* Fixes

* Fix for reading nil accounts

* Fix edge cases with reverts

* Clean up test output

* Fix transfers

* Handle special case of ripemd

* Try with fixed compress, only commit every commfreq blocks

* Updates

* Updates

* Update to fixed erigon-lib

* Update to erigon-lib

* Remove stop

* Update to latest erigon-lib

* fixup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-24 23:12:25 +00:00
Alex Sharov
93e28ed3dd
fix some linter warns (#3310) 2022-01-20 14:00:31 +07:00
Alex Sharov
9188705849
Switch to parallel (#3285)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2022-01-18 15:01:36 +07:00
ledgerwatch
3655916eb9
Junk db util (#3284)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-18 12:03:23 +07:00
Alex Sharov
f9983edc76
Snapshots: ParallelCompressor class, DecompressedFile class (#3265) 2022-01-17 16:57:57 +07:00
Alex Sharov
359af9e49d
Snapshots: files list (#3264) 2022-01-15 11:23:37 +07:00
Dmitry Savonin
a49d409457
Full BSC support with validator mode (#3233)
* migrated consensus and chain config files for bsc support

* migrated more files from bsc

* fixed consensus crashing

* updated erigon lib for parlia snapshot prefix

* added staticpeers for bsc

* [+] added system contracts
[*] fixed bug with loading snapshot
[+] enabled gas bailout
[+] added fix to prevent syncing more than 1000 headers (for testing only)
[*] fixed bug with crashing sender recover sometimes

* migrated system contract calls

* [*] fixed bug with returning mutable balance object
[+] migrated lightclient contracts from bsc
[*] fixed parlia consensus config param

* [*] fixed tendermint deps

* [+] added some logs

* [+] enabled bsc forks
[*] fixed syscalls from coinbase
[*] more logging

* Fix call sys contract gas calculation

* [*] fixed executing system transactions

* [*] enabled receipt hash, gas and bloom filter checks

* [-] removed some logging scripts
[*] set header checkpoint to 10 million blocks (for testing forks)

* [*] fixed bug with commiting dirty inter block state state after system transaction execution
[-] removed some extra logs and comments

* [+] added chapel and rialto testnet support

* [*] fixed chapel allocs

* [-] removed 6 mil block limit for headers sync

* Fix hardforks on chapel and other testnets

* [*] fixed header sync issue after merge

* [*] tiny code cleanup

* [-] removed some comments

* [*] increased mdbx map size to 4 TB

* [*] increased max chaindata size to 6 tb

* [*] bring more compatibility with origin erigon and some code cleanup

* [+] added support of validator mode for BSC chain

* [*] enable private key load for bsc, rialto and chapel chains

* [*] fixed running BSC validator node

* Fix the branch list

* [*] tiny fixes for linter

* [*] formatted imports for core and parlia packages

* [*] fixed import rules in other files

* Revert "[*] formatted imports for core and parlia packages"

This reverts commit c764b58b34fedc2b14d69458583ba0dad114f227.

* [*] changed import rules in more packages

* [*] fixed type mismatch in hack command

* [*] fixed crash on new epoch, enabled bootstrap flags

* [*] fixed linter errors

* [*] fixed missing err check for syscalls

* [*] now BSC implementation is fully compatible with erigon original sources

* Revert "Add chain config and CLI changes for Binance Smart Chain support (#3131)"

This reverts commit 3d048b7f1a.

* Revert "Add Parlia consensus engine for Binance Smart Chain support (#3086)"

This reverts commit ee99f17fbe.

* [*] fixed several issues after merge

* [*] fixed integration compilation

* Revert "Fix the branch list"

This reverts commit 8150ca57e5f2707a84a9f6a1c5b809b7cc84547b.

* [-] removed receipt repair migration

* [*] fixed parlia fork numbers output

* [*] bring more devel compatibility, fixed bsc address list for access list calculation

* [*] fixed bug with commiting state transition for bad blocks in BSC

* [*] fixed bsc changes apply for integration command and updated config print for parlia

* [*] fixed bug with applying bsc forks for chapel and rialto testnet chains
[*] let's use finalize and assemble for mining to  let consensus know for what it's finalizing block

* Fix compilation errors in hack.go

* Fix lint

* reset changes in erigon-snapshots to devel

* Remove unrelated changes

* Fix embed

* Remove more unrelated changes

* Remove more unrelated changes

* Restore clique and aura miner config

* Refactor interfaces not to use slice pointers

* Refactor parlia functions to return tx and receipt instead of dealing with slices

* Fix for header panic

* Fix lint, restore system contract addresses

* Remove more unrelated changes, unify GatherForks

Co-authored-by: Dmitry Ivanov <convexman18@gmail.com>
Co-authored-by: j75689 <j75689@gmail.com>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-14 19:06:35 +00:00
ledgerwatch
ea6dbd49e8
[erigon2] Running with commitment (#3203)
* [erigon2] Running with commitment

* [erigon2] Running with commitment

* [erigon2] Running with commitment

* go mod tidy

* [erigon2] Running with commitment

* More

* Debug

* fix

* Fix

* state root command

* More fixes

* Fix

* Progress to 164735

* Only trace when failing

* fix for firstInsert

* Over block 1.36m

* Update

* fix to deleteAccount

* Fixes for plainKeys

* Undo printing

* No changeset generation by default

* Print block number on interrupt, fix lint

* Fix lint

* Open history DB as read only

* Print error

* Open non read only

* Readonly again

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-14 16:02:44 +00:00
Alex Sharov
2bcc40fd3f
Snapshots: create .dat in tmpdir (#3222) 2022-01-09 14:43:58 +07:00
Alex Sharov
4ef541f5dc
Snapshots: tx lookup in RPC from snapshots (#3214) 2022-01-07 20:52:38 +07:00
Alex Sharov
eed7475af0
Snapshot: move parallel compression to erigon-lib (#3211) 2022-01-06 14:13:09 +07:00
Alex Sharov
5cac9fbff5
Move snapshot creation from hack.go to "erigon snapshot create" cli sub-commmand (#3177) 2021-12-27 14:59:29 +07:00
Alex Sharov
82753a6cda
add words count in .seg (breaking change in snapshot format) (#3140) 2021-12-21 10:48:34 +07:00
ledgerwatch
9f861e1b3f
Check upgrade/downgrade of database schema version to prevent accidental upgrade, remove migrations (#3133)
* Check upgrade/downgrade

* Remove old migrations

* Fix compilation

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-12-15 21:03:04 +00:00
Alex Sharov
ecb10e8548
Snapshots download and seed (#3117)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* Squashed 'interfaces/' content from commit e5b1945d0

git-subtree-dir: interfaces
git-subtree-split: e5b1945d02da7a7f00e2289034ee90a6edd60184

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2021-12-14 10:13:17 +00:00
brendan-kelly
ee99f17fbe
Add Parlia consensus engine for Binance Smart Chain support (#3086)
* Add Parlia consensus engine for Binance Smart Chain support

* Leave RamanujanBlock as nil in params/config.go

* Run `go fmt` on files needing it

* Add comment for PoSA

* Remove empty branches and ineffectual assignments in parlia.go

* Remove commented imports

* Fix compilation error

* Remove EIP155Signer in transaction_signing.go

* Fix compilation issue

* Fix go fmt issues

* Remove Ramanujan from print statement

* Remove references to EthAPIBackend approach

* Fix Finalize method across consensus engines

* Run go fmt

* More linting

* Remove more changes

* remove a comment

* Remove unneeded hashing function

* Remove bytes check and fix actual vs expected mistake
2021-12-11 00:07:10 +00:00
Alex Sharov
178fd1931d
Genesis sync from existing snapshots (#3087) 2021-12-05 09:03:08 +07:00
TBC Dev
57d641b6f9
Avoid redundant Block.Header() deep-copy (#3050)
* Add separate Block.Nonce() and Block.NonceU64()

* Add Block.Seal()

* Avoid redundant Block.Header() deep-copy

* Add warning comment for Block.Header()
2021-11-29 08:52:36 +00:00
Alex Sharov
8104a90993
Snapshot: exec and trie to support snapshot files (#3033) 2021-11-29 10:43:19 +07:00
Alex Sharov
b3e8a1a02a
hack: remove createIdx and txLookup methods (#3000) 2021-11-21 12:06:36 +07:00
Alex Sharov
e55256296b
snapshots: read block from snapshots, add sender to txs file (#2996) 2021-11-21 10:32:14 +07:00
Alex Sharov
d4850b6adc
Hack: dump bodies and headers (#2994) 2021-11-20 09:41:01 +07:00
Alex Sharov
2a08dbd3d9
Recsplit: move files read/write helpers to erigon-lib (#2993) 2021-11-19 12:12:25 +07:00
alex.sharov
c1cbb21f3d save 2021-11-11 10:39:44 +07:00
Alex Sharov
f31a76baea
Compress: add maxPatternLen=64 (#2943) 2021-11-10 22:10:01 +07:00
Alex Sharov
e670f3930f
Lcp must be int32 (#2934) 2021-11-09 17:47:12 +07:00
Alex Sharov
3842351293
move DictAggregator to erigon-lib (#2933) 2021-11-09 10:12:25 +07:00
Alex Sharov
deed48c01a
Small step towards torrent downloader (#2929) 2021-11-08 20:40:56 +07:00
Alex Sharov
9ab2bbee1b
recsplit: single offset bucket (#2928) 2021-11-08 14:27:47 +07:00
Alex Sharov
194330fa4b
Recsplit: preserve immutability threshold (#2926) 2021-11-07 13:01:33 +07:00
Alex Sharov
49ff8ff166
Recsplit: collision typed error (#2925) 2021-11-07 09:58:20 +07:00
alex.sharov
e802e43fa8 add pprof http to hack 2021-11-07 09:16:25 +07:00
Alex Sharov
a5bbe82a59
ETL: use logPrefix as suffix of tmp files (#2921) 2021-11-05 17:19:44 +07:00