Commit Graph

564 Commits

Author SHA1 Message Date
Giulio rebuffo
6713abb420
fixed exec22 panic (#5045)
Co-authored-by: giuliorebuffo <giuliorebuffo@system76-pc.localdomain>
2022-08-13 18:54:32 +01:00
Alex Sharov
52fd0d0e8b
Aggregator22.Unwind() (#5039)
* save

* save
2022-08-13 18:51:25 +07:00
Alex Sharov
64bc837ccf
erigon22: integration, to support parallel execution (#5037)
* save

* save
2022-08-13 16:53:10 +07:00
Alex Sharov
de3a602a24
erigon22: bsc fixes 2022-08-13 16:09:52 +07:00
Alex Sharov
4f007ae43e
erigon22: use chainDB instead of db22 for everything (#5034) 2022-08-13 12:08:15 +07:00
Alex Sharov
e27f669a3f
erigon2.2: save stage progress to chainDB, can run "integration stage_exec" and "state erigon22" on same datadir (#5033) 2022-08-13 11:17:24 +07:00
Giulio rebuffo
8ade569cc5
Save PoW blocks for processing received through P2P (#5015)
* save terminal blocks

* ops

* save less PoW blocks

Co-authored-by: giuliorebuffo <giuliorebuffo@system76-pc.localdomain>
2022-08-12 21:54:20 +02:00
Alex Sharov
25d85d88c5
state erigon22: use right db (#5028) 2022-08-12 21:45:09 +07:00
Alex Sharov
4594ce5ef7
erigon22: history.v2 flag, align rpcdaemon22 (#5016)
* save

* save

* save

* save

* save
2022-08-12 16:13:14 +07:00
Alex Sharov
a7d07ad04d
erigon22 add to exec stage (#5008)
* save

* save

* save
2022-08-11 21:06:41 +07:00
Alex Sharov
61d3b6c297
erigon22 use initialSync variable (#5004) 2022-08-11 12:08:27 +07:00
Alex Sharov
d643da3db8
erigon22 move exec to own package (#5003)
* save

* save
2022-08-11 11:36:13 +07:00
Alex Sharov
e786cbac24
E22 step2 (#5002) 2022-08-11 11:19:59 +07:00
Alex Sharov
f83032533f
Revert "move exec22 to package, call it from stage_exec (#5000)" (#5001)
This reverts commit efa6dfd8ce.
2022-08-11 11:18:23 +07:00
Alex Sharov
efa6dfd8ce
move exec22 to package, call it from stage_exec (#5000) 2022-08-11 11:17:03 +07:00
Alex Sharov
351cd49c21
go1.19 gofmt (#4988) 2022-08-10 19:04:13 +07:00
Alex Sharov
0beb295745
torrent: try burst limit fix panic (#4983)
* save

* save
2022-08-10 18:52:36 +07:00
Alex Sharov
bd721b9fd7
Erigon22: reduce amount of variables. progress object and use of db.update (#4967)
* save

* save

* save

* save

* save
2022-08-09 11:46:57 +07:00
Alex Sharov
b717ff5df5
erigon22: move worker to package (#4966)
* save

* save

* save

* save

* save
2022-08-09 10:39:34 +07:00
Leonard Chinonso
592fe32217
Refactoring To Increase Code Quality (#4842)
* General cleanup and fixes

* Cleaning part 2

* cleanup part 3

* Refactored staterunner to have a handler function

* Fixed linting

* Cleanup part 4

* Changed the initialization of the require package and updated .golangci.yml

* Fixed lint errors from golangci-lint

* deleted state_recon_1.go

* fixed build error

* Added ReadHeaderTimeouts for httpServer

* made changes to golangci

* Reset golangci.yml
2022-08-04 20:23:00 +07:00
ledgerwatch
e56491c601
Erigon22 prototype without parallel execution, fix for BSC upgrade contract (#4884)
* No parallelisation in erigon22 prototype

* Remove paralell

* tx/s

* commit

* Fix overlap

* workers argument

* Print

* Print

* Stable sort

* Print

* Init

* Remove print

* Remove print

* Remove print

* No sorting

* Cleanup

* Fix hangs

* Fix reconstitution for contract upgrades

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-08-04 13:00:46 +01:00
Alex Sharov
35878b29c4
erigon22: auto-create dir (#4912) 2022-08-03 15:10:28 +07:00
Alex Sharov
0d21da4a01
erigon22: auto create dir (#4911) 2022-08-03 14:43:47 +07:00
Leonard Chinonso
ec67e80a8a
Fixed lint errors for gocritic linter (#4904) 2022-08-03 00:35:22 +07:00
Leonard Chinonso
6061690f61
Fixed lint errors for unused linter (#4902) 2022-08-02 22:01:53 +07:00
Leonard Chinonso
8774e1d8a2
Fixed lint errors from gosimple linter (#4900) 2022-08-02 21:09:00 +07:00
ledgerwatch
2b517b7630
Erigon22 prototype to support BSC (#4867)
* Reduce workers and queue size

* Add traces and event logs

* Add chain flag

* Print

* Open consensus db as read only

* Use only one engine instance

* chainDb

* Display palia error

* Not to panic

* Not to exit

* Add syscall

* Print

* Pointer to TxTask

* Not register senders

* Print

* Relax check

* Skip system transactions

* increase workerCount again

* Remove print

* Change recon

* Fix types

* Skip system txs

* Print

* No rules for genesis commit

* Print genesis

* Print|

* Print

* Print

* Update

* Fix

* Remove print

* No print

* Support contract upgrades

* Use the same dif

* Print create2

* Fix

* Fix

* Print

* Hack iterate

* Print empty vals

* Fix hack

* Code hashes

* Code hashes

* Print

* No panic

* No panic

* Fix?

* another incarnation fix

* Remove print

* Remove print

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-07-31 21:54:23 +01:00
ledgerwatch
167de529a8
Fixes for erigon22 prototypes (#4851)
* Fixes for erigon22 prototypes

* Add waiting/applying stats

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-07-28 13:26:48 +01:00
ledgerwatch
09c104803a
Split erigon2.2 and erigon2.3 prototypes (#4811)
* Split erigon2.2 and erigon2.3 prototypes

* Renaming

* Interruptible and resumable erigon22

* Always regenerate trie

* Introduce aggregator

* Fixes

* cleanup

* Fix lint

* Update to erigon-lib main

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-07-28 12:16:37 +01:00
Alex Sharov
aee4b53788
save list of snapshots in db (#4777)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2022-07-28 16:57:38 +07:00
ledgerwatch
81d106bc9d
Experiment in parallel execution (#4652)
* Restructure tx execution

* fixes

* Fixes and traces

* Tracing

* More tracing

* Drain the result channel

* Intermediate

* more efficient parallel exec

* Sorted buffer

* Fix results size

* fix for the recon

* Fix compilation

* Sort keys in Write and Read sets, fix compilation in rpcdaemon22

* Update to latest erigon-lib

* Update to erigon-lib

* Remove go.mod replace

* Update erigon-lib

* Update to erigon-lib main

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-07-23 18:39:08 +01:00
Alex Sharov
66758c7960
RetireBlocks: less arguments (#4785)
* save

* save
2022-07-22 13:44:42 +07:00
Artem Tsebrovskiy
9637b25a42
[erigon2] print trie variant derived from trie instead switch log (#4672)
* [erigon2] print trie variant derived from trie instead switch log

* updated erigon-lib version

* bumped erigon-lib version
2022-07-11 15:58:08 +01:00
sudeep
e13a318e0b
evm t8n to use ExecuteBlockEphemerally api (#4642)
* evm t8n tool to use ExecuteBlockEphemerally api (#4512)

* fix to set V, R, S in legacy transaction

* fix to dump post-execution alloc for evm t8n

* close tx in evm t8n

* populate current difficulty and gas used in output result

- update the ExecutionResult to include corresponding info (like
  Difficulty/GasUsed)

* initial attempt at migrating 'evm t8n' to use ExecuteBlockEphemerally

* using ExecutionResult in ExecuteBlockEphemerally

* bypass validations and integrate with EphemeralExecResult

* fixing output of 'evm t8n'

- remaining bits are "stateRoot" in results.txt and "balance" field for one account in
  alloc.txt (for testdata=1)

* get ExecuteBlockEphemerally to accept getTracer lambda

* fix build failure

* test cases for evm t8n

* more test cases for evm t8n

* fix stateRoot computation in evm t8n

* remove reward argument, as EBE itself takes care of it

* final cleanups for migration to using ExecuteBlockEphemerally

* change EBEforBSC to match EBE

* fix linter issues

* manually revert an unwanted diff

* avoid calculating ReceiptHash twice

* linter check

* minor correction

* remove unnecessary logic in EBEforBsc

* fix integration tests

* fix build
2022-07-07 12:47:00 +01:00
ledgerwatch
73b0659bb7
[erigon2.2] Optimisation of state reconstitution (#4621)
* Init

* Optimise ReconState flushes

* Updates

* Update

* Updates

* Updates

* More on parallel execution

* More on parallel exec

* Fix lint

* Improvements to parallel exec

* comment

* Fix history access, include incarnation

* Close work channel

* TxTask

* more fixes to parallel exec

* Update to latest erigon-lib

* Fix compilation

* Cleanup and timing

* Fixes to recon1

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-07-06 07:49:00 +01:00
Giulio rebuffo
d83b7a4ae0
Proper PoS Error reporting (#4631)
* better reporting

* removed debug log

* proper error reporting
2022-07-05 01:31:16 +02:00
ledgerwatch
77dc35bb85
Revert "evm t8n tool to use ExecuteBlockEphemerally api (#4512)" (#4610)
* Revert "evm t8n tool to use ExecuteBlockEphemerally api (#4512)"

This reverts commit db93d2ea37.

* Fix compilation

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-07-02 23:24:20 +01:00
ledgerwatch
8599dceec7
[erigon2.2] State reconstitution prototype (#4508)
* reconstitution

* Add history access without state function

* More on state reconstitution

* More on state recon

* More on state recon

* More

* More

* support dao fork

* More on state reconstitution

* Update to erigon-lib

* More

* Added genesis block and filling with history

* update

* Genesis works

* Start on parallel

* Preparation for parallel reconstitution, stats for EfSearch

* continue with parallel work

* Fix history reader

* Remove time measurements

* Fixes

* Fixes and UX improvements

* Fixes

* More tracing

* More fixes

* More fixes

* Fix code size

* Update to latest erigon-lib

* Fix for dao fork

* Remove hacks

* Update to erigon-lib, fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-07-02 20:48:42 +01:00
sudeep
db93d2ea37
evm t8n tool to use ExecuteBlockEphemerally api (#4512)
* fix to set V, R, S in legacy transaction

* fix to dump post-execution alloc for evm t8n

* close tx in evm t8n

* populate current difficulty and gas used in output result

- update the ExecutionResult to include corresponding info (like
  Difficulty/GasUsed)

* initial attempt at migrating 'evm t8n' to use ExecuteBlockEphemerally

* using ExecutionResult in ExecuteBlockEphemerally

* bypass validations and integrate with EphemeralExecResult

* fixing output of 'evm t8n'

- remaining bits are "stateRoot" in results.txt and "balance" field for one account in
  alloc.txt (for testdata=1)

* get ExecuteBlockEphemerally to accept getTracer lambda

* fix build failure

* test cases for evm t8n

* more test cases for evm t8n

* fix stateRoot computation in evm t8n

* remove reward argument, as EBE itself takes care of it

* final cleanups for migration to using ExecuteBlockEphemerally

* change EBEforBSC to match EBE

* fix linter issues

* manually revert an unwanted diff

* avoid calculating ReceiptHash twice

* linter check

* minor correction

* remove unnecessary logic in EBEforBsc
2022-07-02 06:52:23 +01:00
Alex Sharov
1f4f850b89
pass context around hased state stage (#4537)
* save

* save

* save
2022-06-25 20:34:42 +06:00
ledgerwatch
79830adefa
[erigon2.2] collecting read indices (#4499)
* [erigon2.2] collecting read indices

* Fix compile issue

* Update to latest erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-20 09:00:45 +01:00
ledgerwatch
e47dd15e68
[erigon2.2] Prototype of eth_getLogs (#4437)
* Initial work

* Fix compile errors

* Update

* Debug

* Cleanup

* Temp changes

* Temp

* Remove temp

* Remove print

* Upgate to erigon-lib main

* go mod tidy

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-13 21:25:17 +01:00
ledgerwatch
fd8adddce8
trace_filter implementation based on erigon 2 update 2 data (#4431)
* trace_filter implementation based on erigon 2 update 2 data

* Fix test compile, extract txNums

* Update to erigon-lib, add generic heap

* Fix getHeaderHash for RPC methods

* Missing files

* Skip tests

* Add reward traces

* Filter trace lint

* Reinstate filtering

* Print

* Print

* Fix

* Print

* Print txNums

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Fix

* Remove prints

* Fix nil dereference

* Inclusive toBlock bound

* Print

* Print

* Hack

* remove some deps

* Fix lint

* Update erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-06-12 12:44:01 +01:00
ledgerwatch
8e3ac8a21c
Erigon2 upgrade 2 prototype (#4341)
* Erigon2 upgrade 2 prototype

* Latest erigon-lib

* Fixes

* Fix print

* Fix maxSpan

* Reduce maxSpan

* Remove duplicate joins

* TxNum

* Fix resuming

* first draft of history22

* Introduce historical reads

* Update to erigon-lib

* Update erigon-lib

* Update erigon-lib

* Fixes and tracing for checkChangeSets

* More trace

* Print account details

* fix getHeader

* Update to erigon-lib main

* Add tracer indices and event log indices

* Fix calltracer

* Fix calltracer

* Duplicate rpcdaemon into rpcdaemon22

* Fix tests

* Fix tests

* Fix tests

* Update to latest erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-06-10 16:18:43 +01:00
Enrique Jose Avila Asapche
9f1cc40c89
changed syncmode flag to snapshots flag (#4353)
* changed syncmode flag to snapshots flag

* using boolT

* Remove syncmode concept

* Add mumbai

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-05 07:33:55 +01:00
ledgerwatch
ca2b0f19c1
Fix checkChangeSets and RPC daemon commands accessing history (#4365)
* Simplifications

* Print

* Fix

* Clean up

* Fixed RPC daemon command accessing history

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-04 23:06:30 +01:00
ledgerwatch
ee4ffd190d
Update skip analysis (#4364)
* Skip analysis advance, fix for change sets

* Optimisations for integration

* Print

* Print

* Fix

* Fix

* Fix

* Remove prints

* No panic

* More checks

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-04 16:54:22 +01:00
Enrique Jose Avila Asapche
bc21ab9d97
syncmode changes (#4334)
* log syncmode

* little comment

* ops

* calling log in correct place

* got rid of error instead we warn

* no return
2022-06-02 14:15:00 +01:00
Alex Sharov
0795bcb865
integration: validate syncmode flag (#4300) 2022-05-30 09:59:38 +07:00
Andrew Ashikhmin
0c76286509
Add rules to AsMessage to vet transaction types (#4276)
* Add rules to AsMessage to vet transaction types

* Fix TestState
2022-05-26 18:20:34 +02:00
Alex Sharov
66248c4bfb
Torrent: verify all files (#4270) 2022-05-26 12:27:44 +07:00
Alex Sharov
fc1a37e6dc
create turbo/services pkg (#4269) 2022-05-26 10:31:06 +07:00
Alex Sharov
c2e0956e3e
return err on invalid syncmode (#4219)
* save

* save

* save
2022-05-20 16:04:38 +07:00
Artem Tsebrovskiy
f4c8f44f61
removed obsolete trie variant (#4172) 2022-05-20 15:03:27 +07:00
Alex Sharov
726ef081ce
option to disable runtime.ReadMemStats (#4200) 2022-05-19 11:48:07 +07:00
ledgerwatch
0e457ba9d7
Restore skip analysis (#4146)
* Restore skip analysis optimisation

* Make checkChangeSets work with snapshots

* Add syncmode to checkChangeSets

* Add block number for BSC

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-14 20:17:23 +01:00
ledgerwatch
14557a2348
[erigon2] Support for binary tree commitments (#4077)
* binary tree

* Binary commitment tree

* [erigon2] Bin tree support

* Point to latest erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-05 13:54:29 +01:00
ledgerwatch
285c782b1b
More efficient header verification of headers for Parlia when snapshots are used (#3998)
* Update stageloop.go

* Print

* Consider snapshot headers as parlia checkpoints

* Not fail after not loading snapshot

* Lazy snapshots

* Print number of validators

* More printing

* Use epoch instead of checkpoint interval

* Reduce logging

* Fix compilation

* Remove trace jump dest

* Fix lint

* Not store snapshots every epoch

* Separate snapshot for verification and finalisation

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-04-29 07:10:31 +01:00
Alex Sharov
831da9ba41
reduce downloader deps (#4011) 2022-04-28 12:42:14 +07:00
Alex Sharov
9b47daee44
Default syncmode=snap for Mainnet, Bsc, Goerli (#3861)
* change syncmode default to snap sync

* save
2022-04-10 07:20:20 +01:00
alex.sharov
10f195578b fix linter 2022-04-01 15:11:02 +07:00
Alex Sharov
d1218181b0
Snapshots: geth compatibility, use --syncmode=snap flag (#3808) 2022-04-01 15:00:50 +07:00
Alex Sharov
85e47d83d2
Snapshots: remove "experimental" prefix from cli flag (#3760) 2022-03-24 12:17:31 +07:00
ledgerwatch
f8668da905
[erigon2] Bring back MDBX as storage for recent data (#3756)
* fixes

* Update

* Update to erigon-lib main

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-03-23 16:00:06 +00:00
ledgerwatch
c47cf1fd49
[erigon2] Optimise Huffman decoding (#3727)
* Update to erigon-lib

* Fix for block number

* Update

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-03-18 09:55:38 +00:00
Alex Sharov
f314222180
Snapshots: start seed new large .seg files (#3724) 2022-03-18 11:12:18 +07:00
Artem Tsebrovskiy
f14df3faa3
added support of Clique consensus for block processing for erigon2 (#3705)
* added support of Clique consensus for block processing for erigon2

* erigon2: remove block reader which doesn't use system pre and post block transactions
2022-03-17 07:30:47 +00:00
Alex Sharov
469b75c3d3
Snapshots: retire blocks by default (#3707) 2022-03-16 09:57:48 +07:00
ledgerwatch
8b00636e21
[erigon2] history indices based on Elias Fano (#3679)
* [erigon2] history indices based on Elias Fano

* Tidy

* Update

* tx lookup to support blocTo in integration

* Update to fix

* Update

* Fix

* Update to erigon-lib main

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-03-13 23:11:49 +00:00
Artem Tsebrovskiy
d300c2821b
erigon2: implemented chain switching by --chain flag (#3668)
erigon2: fix linter
2022-03-10 14:01:37 +00:00
battlmonstr
e6c056e666
Rename Datadir/dataDir to follow conventions. (#3576)
* rename from Datadir to DataDir (functions and public vars)
* rename dataDir to datadir (local variables)
2022-02-22 17:39:48 +00:00
Alex Sharov
f5da8833b6
erigon2: exec metric (#3567) 2022-02-22 08:55:49 +07:00
Alex Sharov
27b87f44d6
Erigon2: support block snapshots (#3557)
* save

* erigon

* save
2022-02-22 08:42:12 +07:00
ledgerwatch
e784f2b809
[erigon2] ETL optimisations (#3550) 2022-02-21 09:31:06 +07:00
ledgerwatch
80d7734ea8
[erigon2] fix history creation (txNum confusion) (#3547)
* More tracing

* More tracing

* More tracing

* Keep correct txNum

* Keep correct txNum

* Update to tracing

* Off by one

* Print memory info

* Quicker startup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-19 08:15:35 +00:00
ledgerwatch
c5b4d3e352
[erigon2] Thin commitment (2nd attempt) (#3515)
* Another fix for history files

* Fix NPE

* Another fix

* Proper closing sequence

* Remove first byte insert flag

* Fixes

* Update

* Add traceblock to erigon2 command

* Update

* Update

* Update

* Update

* print panic

* Fix history reads

* Update

* Update locks

* Update

* Fixes

* Set changesets and commitments upfront

* Logging instead of printing

* Fixes in aggregator and recsplit

* Update to erigon-lib

* Remove empty commitments

* Update

* Update

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-17 10:56:20 +07:00
Andrew Ashikhmin
3663f8c709
Explicit error for purged change sets (#3528) 2022-02-16 16:44:16 +01:00
Andrew Ashikhmin
7bad3a9c59
state.PlainState is now at the beginning of blockNbr, not the end (#3517)
* state.PlainState is now at the beginning of blockNbr, not the end (fix discprepancy with GetAsOf)

* Fix tests
2022-02-16 08:38:12 +00:00
ledgerwatch
c865290502
Add traceblock to erigon2 command (#3516)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-14 23:15:24 +00:00
ledgerwatch
dffcc57ecc
[erigon2] Fix history bug, optimise bitmap search (#3510)
* Fix history bug, optimise bitmap search

* Optimise SeekInBitmap64

* Update erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-13 20:02:18 +00:00
ledgerwatch
2074b98568
Tools for gathering data on erigon2 history and bitmaps (#3488)
* Tools for gathering data on erigon2 history and bitmaps

* Always start with 0

* Always start with 0

* Clean up commitment files, trace block

* traceblock

* Add tracing to checkChangeSets

* Fixes to DeleteAccount

* Update to latest erigon-lib

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-12 15:48:38 +00:00
Alex Sharov
c9cee7a884
path -> filepath (path package is for urls) (#3493) 2022-02-12 20:33:09 +07:00
ledgerwatch
eb6a168603
[erigon2] Chain history and bitmap indices (#3437)
* First update

* Update to latest erigon-lib

* Fix lint

* Update

* Update

* Update

* Add history2 command

* Updates

* Update erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-12 00:06:57 +00:00
ledgerwatch
093bfbad39
[erigon2] Optimisations (#3412)
* Update

* [erigon2] optimisations

* Update to latest erigon-lib

* Update to latest erigon-lib, fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-03 09:36:27 +00:00
ledgerwatch
4ef14215b4
[erigon2] three threads, better stats in the logs (#3378)
* [erigon2] Use compact keys in commitments

* Better stats in the logs

* Update

* Update

* Update

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-31 23:40:51 +00:00
ledgerwatch
079a177614
[erigon2] Computing data dependencies between transactions (#3290)
* Use ibs per transaction instead of per block

* Remove copy functions, introduce balance increase map

* Fix test

* Fix lint

* Introduce balanceIncrease journal action

* Introduce balanceIncrease journal action

* Simplifications

* Fix

* Reset

* Simplify

* Fix

* Fixes

* Fix for reading nil accounts

* Fix edge cases with reverts

* Clean up test output

* Fix transfers

* Handle special case of ripemd

* Try with fixed compress, only commit every commfreq blocks

* Updates

* Updates

* Update to fixed erigon-lib

* Update to erigon-lib

* Remove stop

* Update to latest erigon-lib

* fixup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-24 23:12:25 +00:00
Alex Sharov
1e3228124a
Move cli root context to erigon-lib (#3294)
* save

* save

* save
2022-01-19 10:49:07 +07:00
ledgerwatch
1af72faf71
[erigon2] optional writing of "before" value for changes, decouple commitment from blocks (#3272)
* Sepatate commitment calculation from blocks

* Fix

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-16 21:24:54 +00:00
ledgerwatch
17aff6090f
[erigon2] Resumable prototype (#3263)
* [erigon2] Resumable prototype

* Fix compile error

* Resumable prototype

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-16 08:40:16 +00:00
ledgerwatch
c774bff61e
Update erigon2.go (#3266) 2022-01-15 09:41:56 +00:00
Dmitry Savonin
a49d409457
Full BSC support with validator mode (#3233)
* migrated consensus and chain config files for bsc support

* migrated more files from bsc

* fixed consensus crashing

* updated erigon lib for parlia snapshot prefix

* added staticpeers for bsc

* [+] added system contracts
[*] fixed bug with loading snapshot
[+] enabled gas bailout
[+] added fix to prevent syncing more than 1000 headers (for testing only)
[*] fixed bug with crashing sender recover sometimes

* migrated system contract calls

* [*] fixed bug with returning mutable balance object
[+] migrated lightclient contracts from bsc
[*] fixed parlia consensus config param

* [*] fixed tendermint deps

* [+] added some logs

* [+] enabled bsc forks
[*] fixed syscalls from coinbase
[*] more logging

* Fix call sys contract gas calculation

* [*] fixed executing system transactions

* [*] enabled receipt hash, gas and bloom filter checks

* [-] removed some logging scripts
[*] set header checkpoint to 10 million blocks (for testing forks)

* [*] fixed bug with commiting dirty inter block state state after system transaction execution
[-] removed some extra logs and comments

* [+] added chapel and rialto testnet support

* [*] fixed chapel allocs

* [-] removed 6 mil block limit for headers sync

* Fix hardforks on chapel and other testnets

* [*] fixed header sync issue after merge

* [*] tiny code cleanup

* [-] removed some comments

* [*] increased mdbx map size to 4 TB

* [*] increased max chaindata size to 6 tb

* [*] bring more compatibility with origin erigon and some code cleanup

* [+] added support of validator mode for BSC chain

* [*] enable private key load for bsc, rialto and chapel chains

* [*] fixed running BSC validator node

* Fix the branch list

* [*] tiny fixes for linter

* [*] formatted imports for core and parlia packages

* [*] fixed import rules in other files

* Revert "[*] formatted imports for core and parlia packages"

This reverts commit c764b58b34fedc2b14d69458583ba0dad114f227.

* [*] changed import rules in more packages

* [*] fixed type mismatch in hack command

* [*] fixed crash on new epoch, enabled bootstrap flags

* [*] fixed linter errors

* [*] fixed missing err check for syscalls

* [*] now BSC implementation is fully compatible with erigon original sources

* Revert "Add chain config and CLI changes for Binance Smart Chain support (#3131)"

This reverts commit 3d048b7f1a.

* Revert "Add Parlia consensus engine for Binance Smart Chain support (#3086)"

This reverts commit ee99f17fbe.

* [*] fixed several issues after merge

* [*] fixed integration compilation

* Revert "Fix the branch list"

This reverts commit 8150ca57e5f2707a84a9f6a1c5b809b7cc84547b.

* [-] removed receipt repair migration

* [*] fixed parlia fork numbers output

* [*] bring more devel compatibility, fixed bsc address list for access list calculation

* [*] fixed bug with commiting state transition for bad blocks in BSC

* [*] fixed bsc changes apply for integration command and updated config print for parlia

* [*] fixed bug with applying bsc forks for chapel and rialto testnet chains
[*] let's use finalize and assemble for mining to  let consensus know for what it's finalizing block

* Fix compilation errors in hack.go

* Fix lint

* reset changes in erigon-snapshots to devel

* Remove unrelated changes

* Fix embed

* Remove more unrelated changes

* Remove more unrelated changes

* Restore clique and aura miner config

* Refactor interfaces not to use slice pointers

* Refactor parlia functions to return tx and receipt instead of dealing with slices

* Fix for header panic

* Fix lint, restore system contract addresses

* Remove more unrelated changes, unify GatherForks

Co-authored-by: Dmitry Ivanov <convexman18@gmail.com>
Co-authored-by: j75689 <j75689@gmail.com>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-14 19:06:35 +00:00
ledgerwatch
ea6dbd49e8
[erigon2] Running with commitment (#3203)
* [erigon2] Running with commitment

* [erigon2] Running with commitment

* [erigon2] Running with commitment

* go mod tidy

* [erigon2] Running with commitment

* More

* Debug

* fix

* Fix

* state root command

* More fixes

* Fix

* Progress to 164735

* Only trace when failing

* fix for firstInsert

* Over block 1.36m

* Update

* fix to deleteAccount

* Fixes for plainKeys

* Undo printing

* No changeset generation by default

* Print block number on interrupt, fix lint

* Fix lint

* Open history DB as read only

* Print error

* Open non read only

* Readonly again

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-14 16:02:44 +00:00
Alex Sharov
4ef541f5dc
Snapshots: tx lookup in RPC from snapshots (#3214) 2022-01-07 20:52:38 +07:00
Frojdi Dymylja
499c27d2e1
change: port js tracer changes (#3128)
* change: expose callCtx as ScopeContext

* change: update tracer API

* change: logger API
update: jst tracer

* add: port jst tracer 2

* add: port jst tracer 3

* add: access list tracer

* chore: lint
2021-12-15 13:19:58 +00:00
brendan-kelly
ee99f17fbe
Add Parlia consensus engine for Binance Smart Chain support (#3086)
* Add Parlia consensus engine for Binance Smart Chain support

* Leave RamanujanBlock as nil in params/config.go

* Run `go fmt` on files needing it

* Add comment for PoSA

* Remove empty branches and ineffectual assignments in parlia.go

* Remove commented imports

* Fix compilation error

* Remove EIP155Signer in transaction_signing.go

* Fix compilation issue

* Fix go fmt issues

* Remove Ramanujan from print statement

* Remove references to EthAPIBackend approach

* Fix Finalize method across consensus engines

* Run go fmt

* More linting

* Remove more changes

* remove a comment

* Remove unneeded hashing function

* Remove bytes check and fix actual vs expected mistake
2021-12-11 00:07:10 +00:00
Alexandr Borodulin
bbb3cc978f
Starknet getcode (#3038)
* deploy_cairo_smartcontract

* deploy_cairo_smartcontract / 2

Add new transaction type for cairo and vm factory

* starknet_getcode

* deploy_cairo_smartcontract / 3

* deploy_cairo_smartcontract / 4

* deploy_cairo_smartcontract / 5

Co-authored-by: Aleksandr Borodulin <a.borodulin@axioma.lv>
2021-12-06 14:58:53 +00:00
Alex Sharov
b7aa5be149
Remove snapshot migrator class (#3088) 2021-12-05 16:13:40 +07:00
Alex Sharov
178fd1931d
Genesis sync from existing snapshots (#3087) 2021-12-05 09:03:08 +07:00
ledgerwatch
b1b68e05ea
Erigon2 state runner - option to compare with historical state (#3061)
* Comment out comparisons

* Introduce check option

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-11-30 16:46:43 +00:00
TBC Dev
ae4819ddad
Prefer Block.NumberU64() over Block.Number.Uint64() (#3051) 2021-11-29 11:32:41 +00:00
TBC Dev
57d641b6f9
Avoid redundant Block.Header() deep-copy (#3050)
* Add separate Block.Nonce() and Block.NonceU64()

* Add Block.Seal()

* Avoid redundant Block.Header() deep-copy

* Add warning comment for Block.Header()
2021-11-29 08:52:36 +00:00
ledgerwatch
5a09c1c8c6
Erigon2 state runner (#3034)
* erigon 2 state experiment

* Some boilerplate

* More boilerplate

* More boilerplate

* Introduce comparison

* trace merges

* Fix

* Update erigon-lib dependency

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-11-27 09:43:25 +00:00
Alex Sharov
702631bf6f
prohibit change prune mode (#2908) 2021-11-03 20:05:52 +07:00
Alex Sharov
5ca558c667
Move lengths to erigon-lib other packages depend on it (#2799)
* move lengths to erigon-lib

* move lengths to erigon-lib
2021-10-08 10:20:45 +07:00
Andrea Lanfranchi
c913f35c2e
Inner errors (#2774) 2021-10-04 22:16:52 +07:00
Alex Sharov
3de50637cd
remove changeset.Walk func (#2716) 2021-09-22 07:54:29 +07:00
ledgerwatch
15b4095718
Move ETL to erigon-lib (#2667)
* Move ETL to erigon-lib

* Update link in the readme

* go mod tidy

* Use common/chan.go from erigon-lib

* Clean up

* Fix lint

* Fix test

* Fix compilation

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-09-12 08:50:17 +01:00
Andrea Lanfranchi
00d627bdb0
Remove config.name (const "erigon") from paths (#2531)
* Remove config.name (const "erigon") from paths

* Fix unit test

* Safety check

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-08-18 16:24:03 +07:00
Alex Sharov
c7c3a5932a
rename ReadBody to ReadBodyWithTransactions to make it obvious that it's heavier then just ReadBody (#2534)
* rename ReadBody to ReadBodyWithTransactions

* rename ReadBody to ReadBodyWithTransactions
2021-08-15 17:08:28 +07:00
Evgeny Danilenko
4cd72c8328
Keep readonly value while changing interpreters back and forth (#2508)
* restore TEVM

* fuzzing and property based

* comment

* lint

* stack callback into defer ater checking an error

* sequential tests
2021-08-10 09:48:56 +07:00
ledgerwatch
350d7fe7a9
Adding support for vmTrace into trace_ routines (#2497)
* VmTrace

* Fixes to gasCall, beginning of vmTrace

* Fix opcode tracer

* Add mem and store, enable vmTrace for all methods

* Fix lint

* More nuances and debugging|

* More fixes

* Fix for trace_callMany

* Fix for trace_callMany

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-08-08 13:28:03 +01:00
Alex Sharov
6bd44eb26c
move kv to erigon-lib (#2467) 2021-07-29 18:53:13 +07:00
Alex Sharov
5069558752
Apache licensed logger (#2460) 2021-07-29 17:23:23 +07:00
Alex Sharov
21cb7befa4
finish remove bucket suffix (#2458) 2021-07-28 10:43:51 +07:00
Alex Sharov
838e5f9ef2
Move bucket constants into kv package, move kv interface to kv package (#2455) 2021-07-28 09:47:38 +07:00
Alex Sharov
6801208497
move some tests to kv interface (#2438) 2021-07-24 14:14:11 +07:00
Alex Sharov
d2552196ad
Sokol v0: support for first epoch-set transition (#2411) 2021-07-21 18:13:26 +07:00
Alex Sharov
c3e1cfdac8
Pruning for: exec, log_index, tx_lookup, history stages (#2399)
* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* add tvm flag

* save

* db migration for storage mode
add flag --prune=
remove flag --storage-mode=
add flag --experiments=tevm,...
rename integration set_storage_mode to set_prune

* fix

* forward move of stages must skip everything before PruneTo

* keep in db progress of prune method

* keep in db progress of prune method

* simplify logs

* simplify logs

* simplify logs

* fix test

* simplify logs

* simplify logs

* simplify logs

* simplify logs

* remove callTraceSet as dupsort
use etl transform for txlookup prune
remove some logs

* cleanup tests a bit

* print_stages and eth_sync to show prune progress

* fix print_stages

* add readme about --prune.to flag

* more docs

* add --prune.history.older and other flags support

* fix migration on empty db

* better toString

* better toString
2021-07-20 21:03:19 +01:00
Alex Sharov
9f6ef74adc
Linter: to check tx.Rollback() by ruleguard (#2383) 2021-07-16 20:23:54 +07:00
Alex Sharov
557d082c51
Sokol v0: epoch support (#2348)
* remove ctx from state writer interface

* call me baby

* save

* save

* can process block 1

* can process block 1

* can process block 1

* can process block 1

* save

* save

* parse logs logic

* cleans

* cleans

* cleans
2021-07-12 16:27:25 +01:00
Alex Sharov
a78cb54132
step towards kv interface (#2311) 2021-07-07 23:15:49 +07:00
b00ris
57473175ff
Body snapshot (#2100)
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* save state

* body snapshot test

* unique tx

* body snapshot generation using walk

* move methods out of test

* block data verification added

* fix lint

* test with remove works correctly

* fix lint

* remove experiment test

* fix test

* add comment

* add second layer of remove test

* rename test

* fix typos

* fix lint

* revert testdata

* body snapshot migration save state

* fix body snapshot migration

* fix after merge

* remove debug test

* debug windows build

* fix build

* fix

* fix lint

* debug

* fix

* fix windows build

* simplify snapshot management&&get rid of lazy tx

* fix lint

* fix windows path

* debug

* debug

* debug

* debug

* remove geometry experiments

* skip windows tests

* clean

* fix

* fix ;int
2021-07-06 23:33:26 +01:00
Alex Sharov
a68b5ba361
Replace ChainConfig.WithEIPsFlags by go-ethereum's ChainConfig.Rules (#2304)
* use chainRules

* use chainRules

* use chainRules

* use chainRules

* use chainRules
2021-07-05 19:52:50 +01:00
ledgerwatch
f5e42b1e7a
Fixes in rpctest and rpcdaemon for debug_traceTransaction, and opcode tracer, tool for repairing broken receipts (#2284)
* Fixes in rpctest and rpcdaemon for debug_traceTransaction

* Fix for opcode tracer

* Tool to fix receipts

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-07-03 14:34:23 +01:00
Alex Sharov
605d7a8d59
Sokol v0 (#2228)
* handle chain flag in integration

* handle chain flag in integration

* handle chain flag in integration

* save

* save

* merge devel

* save

* noopWriter - one is enough

* chain spec parser

* chain spec parser

* embed

* embed

* embed

* embed

* embed

* embed

* embed

* clean

* clean

* correct alloc after reset state

* correct alloc after reset state

* correct alloc after reset state

* integration reset state now does re-apply genesis and chainConfig

* eips summary

* eips summary

* eips summary
2021-06-25 19:13:40 +01:00
Alex Sharov
59d05dc5fe
hide file exists err (#2218) 2021-06-22 11:09:45 +01:00
Artem Vorotnikov
d8a009837b
Nuke preimage remnants (#2203) 2021-06-20 13:00:22 +07:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
BitBaseBit
7ed337cdcc
Implemented panic handling, graceful shutdown and reporting for all goroutines that don't explicitly handle them. (#2153)
* implemented crash reporting for all goroutine panics that aren't handled explicitly

* implemented crash reporting for all goroutine panics that aren't handled explicitly

* changed node defaults back to originals after testing

* implemented panic handling for all goroutines that don't explicitly handle them, outputting the stack trace to a file in crashreports

* handling panics on all goroutines gracefully

* updated missing call

* error assignment

* implemented suggestions

* path.Join added

* implemented Evgeny's suggestions

* changed path.Join to filepath.Join for cross-platform

* added err check

* updated RecoverStackTrace to LogPanic

* updated closures

* removed call of common.Go to some goroutines

* updated scope capture

* removed testing files

* reverted back to original method, I feel like its less intrusive

* update filename for clarity
2021-06-13 17:41:39 +01:00
Alex Sharov
925d17a238
remove lmdb from hack, integration, state packages. remove lmdb dbtools (#2146) 2021-06-11 20:31:37 +07:00
Evgeny Danilenko
de26634d8d
TEVM status pruning (#2034)
* use trace for TEVM status

* restore testdata

* fix getting contract from plain state

* unwind

* linters

* reproducible build debug

* debug

* debug

* remove debug
2021-06-11 09:34:25 +01:00
Alex Sharov
5ba3ea162a
Simulated backend and genesis to kv (#2104)
* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* merge

* merge

* merge

* merge

* merge

* merge

* merge

* merge
2021-06-05 16:17:04 +01:00
Evgeny Danilenko
7352b81122
Tevm stage 1 (#1845)
* convert contracts after block execution

* check if has tevm code

* after review-1

* handle ErrNotFound

* typo

* tests

* tevm code bucket

* testdata

* execute pre-stage

* after merge

* test fix

* test fix

* fix test after merge

* disable translation stage

* after merge

* rename params

* rename to Erigon

* parallelize EVM translation

* fix

* logging and fixes

* fix

* todos

* cleanup

* revert erigorn renaming

* unwind

* tevm unwind

* fix AppData

* non-parallel version

* comments
2021-05-27 14:54:55 +01:00
ledgerwatch
d5c2e5fd9a
checkChangeSet to work with MDBX, load senders for better performance (#2024)
* CheckchangeSets switch to MDBX

* Load senders

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-27 08:37:23 +01:00
Evgeny Danilenko
74847d77e6
Rename to Erigon (#2018)
* turbo-geth to erigon

* tg, turbo to erigon
2021-05-26 11:35:39 +01:00
Alex Sharov
545fe32f68
Mdbx - make it default db. Lazy buckets renaming. (#1982) 2021-05-22 10:03:02 +07:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Artem Vorotnikov
66a8c23c37
Remove TrieDbState + other things (#1925)
* Remove old table from script

* Simplify ChangeSetWriter

* Move state tests to KV

* Simplify TrieCfg

* Port state tests away from TrieDbState

* Remove old MakePreState

* Upgrade t8ntool's MakePreState

* MakePreState2 -> MakePreState

* Remove TrieDbState
2021-05-16 06:21:29 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836)
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 09:45:40 +01:00
Alex Sharov
c7eedf0091
Add kv to stage settings (because it doesn't change in runtime, tx will be passed as Stage func argument) (#1872)
* add kv to config

* move forward tx use

* move forward tx use

* gc stat

* gc stat

* changeset writer - move to tx

* clean

* clean

* clean

* clean

* clean
2021-05-04 13:36:03 +01:00
Artem Vorotnikov
2148885637
Simplify ChangeSet implementation and tests (#1878)
* Move storage changeset code into one file

* Simplify storage changeset tests

* simplify more

* more

* more

* more

* move account changeset into one file

* simplify account changeset test

* NewStorageChangesetPlain -> NewStorageChangeSet

* EncodeStoragePlain -> EncodeStorage

* StorageChangeSetPlain -> StorageChangeSet

* AccountChangeSetPlain -> AccountChangeSet

* more

* more

* more

* Simplify EncodeStorage

* more

* Simplify FromDBFormat

* more rename

* rename changeset buckets (constant names only)
2021-05-04 13:34:08 +01:00
Alex Sharov
f230ba1da4
Less use rawdb deprecated methods (#1861)
* less use deprecated methods

* less use deprecated methods

* auto Dp limit

* auto Dp limit

* add test
2021-05-03 21:01:01 +01:00
Alex Sharov
aacc457ea8
rawdb deprecated methods with Database interface (#1854) 2021-05-01 14:42:23 +07:00
ledgerwatch
117838e213
Move compatibility check (#1814)
* Move compatibility check

* Implement Alex's suggestion

* Use root context

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-26 13:39:34 +01:00
Alex Sharov
472e161e33
fininish to remove chain context (#1807) 2021-04-26 14:23:21 +07:00
Alex Sharov
eee9438625
deprecate PlainDbState in favor of PlainKvState (#1805) 2021-04-26 12:37:48 +07:00
Alex Sharov
3af0b9b591
add hashstate, trie, history, logIndex stages (#1799)
* add hashstate and trie stages

* add history stages

* add history stages

* add history stages

* add logindex stage

* add logindex stage

* add other stages
2021-04-25 09:29:39 +01:00
Alex Sharov
922c66f7b8
check_change_sets_use_tx_obj (#1781) 2021-04-22 18:11:55 +01:00
ledgerwatch
12cde41772
Aleut support (Eip1559) (#1704)
* Where I am at

* Refactoring of transaction types

* More refactoring

* Use Homested signer in rpc daemon

* Unified signer

* Continue unified signer

* A bit more

* Fixes and down the rabbit hole...

* More tx pool fixes

* More refactoring fixes

* More fixes'

* more fixes

* More fixes

* More compile fixes

* More RLP hand-writing

* Finish RLP encoding/decoding of transactions

* Fixes to header encoding, start on protocol packets

* Transaction decoding

* Use DecodeTransaction function

* Decoding BlockBodyPacket

* Encode and decode for pool txs

* Start fixing tests

* Introduce SigningHash

* Fixes to SignHash

* RLP encoding fixes

* Fixes for encoding/decoding

* More test fixes

* Fix more tests

* More test fixes

* More test fixes

* Fix core tests

* More fixes for signer

* Fix for tx

* Fixes to string encoding/size

* Fix eip2930 test

* Fix rest of ./tests

* More fixes

* Fix compilation

* More test fixes

* More test fixes

* Test fixes

* More fixes

* Reuse EncodingSize in EncodeRLP for accessList

* Rearrange things in dynamic fee tx

* Add MarshalBinary

* More fixes

* Make V,R,S non-pointers

* More NPE fixes

* More fixes

* Receipt fixes

* Fix core/types

* Fix ./eth

* More compile fixes for tests

* More test fixes

* More test fixes

* Try to see lint errors better

* Try to see lint errors better

* Fix lint

* Debugging eip1559 test

* Fix TestEIP1559Transition test

* Fix NewBlockPacket encoding/decoding

* Fix calculation of TxHash

* Fix perf problem with senders

* Update aleut config values

* Try adding static peers

* Add staticpeers to defaul flags

* Change aleut networkID

* Fix test

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-22 18:11:37 +01:00
Evgeny Danilenko
17c07c50a5
Move clique buckets to separate DB (#1703)
* debug

* debug

* it works

* rename clique bucket

* remove genesis special case

* copy snapshot

* remove debug

* migration

* debug

* regenerate snapshots

* simplify

* regeneration

* after merge

* tests

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-04-19 22:58:05 +01:00
Alex Sharov
00f700675b
add --datadir parameter to integration, snapshot generator, header downloader (#1749) 2021-04-19 14:25:26 +07:00
Alex Sharov
5703059f29
[merge after release] Move history stage to rwtx (#1727) 2021-04-15 17:06:30 +07:00
Alex Sharov
4ef5e4b2f6
Continue move to rwtx (#1722) 2021-04-15 12:23:10 +07:00
Alex Sharov
9527671e5e
Remove blockchain object step3 (#1684) 2021-04-07 12:38:43 +07:00
Alex Sharov
a913ae8847
Remove chainContext object (#1683) 2021-04-06 16:52:53 +07:00
Alex Sharov
cc539ce266
Remove blockchain.InsertChain method (#1681) 2021-04-06 12:06:07 +07:00
Artem Vorotnikov
d66555268d
Harmonize DB APIs (#1669) 2021-04-03 09:26:00 +03:00
Alex Sharov
9a9c5b298a
bitmapdb to use rwtx (#1661) 2021-04-03 08:52:45 +07:00
Alex Sharov
fbfc43da7e
etl to use rwtx (#1660) 2021-04-02 18:22:25 +07:00
Alex Sharov
ef77eb3053
tx.RwCursor() to return err (first step of removing lazy cursors) (#1656) 2021-04-02 13:36:49 +07:00
Artem Vorotnikov
9b8cdc0f22
Fix lints and remove more unused code (#1621) 2021-03-29 10:58:45 +07:00
Artem Vorotnikov
e67be2b1be
Remove unused code (#1604)
* Remove graphql, legacy options

* Remove CODEOWNERS

* Remove EWASM and custom EVM interpreter
2021-03-26 09:08:01 +07:00
Artem Vorotnikov
a6742a4cf8
Kill cmd/state/stateless (#1592)
* Move stuff around

* Remove

* tidy

* Fix for writeReceipt printing

* fix lint typo

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-23 16:33:58 +00:00
ledgerwatch
0986ce43fb
Fix some conformance tests broken after rebase (#1587)
* Uncomment

* Try to fix

* Try again

* Compile fix

* Compile fix

* Compile fix

* Compile fix

* Compile fix

* fix lint

* Enable more tests

* Fix core tests

* Remove comment outs

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-23 07:25:10 +00:00
Alex Sharov
91e4126ca7
db commit - remove unused return param (#1584) 2021-03-22 19:41:52 +07:00
Alex Sharov
40dd00fee7
remove ideal batch size (#1582) 2021-03-22 13:47:01 +07:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions (#1577) 2021-03-21 20:15:25 +07:00
alex.sharov
e7034c0eba revert 2021-03-21 17:50:05 +07:00
alex.sharov
970c7e93a8 remove old flags 2021-03-21 17:05:42 +07:00
Alex Sharov
ba37706681
move remote eth backend from ethdb to core package (#1567) 2021-03-20 10:52:00 +07:00
b00ris
c03da8fb8e
Split header prefix bucket to headers, td and canonical (#1556)
* split headers prefix

* migration

* fix downloader bug

* test for migration

* fix lint

* uncomment t.Prallel

* fix postprocessing test
2021-03-19 12:54:47 +00:00
Alexey Sharp
7fd3417f52 Fix more compile errors 2021-03-14 19:07:19 +00:00
ledgerwatch
b743b88218
Revert "Extract validating interface (#1120)" (#1543)
* Revert "Extract validating interface (#1120)"

This reverts commit e4e36c152e.

* Fix go mod tidy

* Fix lint

* Fix lint

* fix lint

* Test fix

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-05 20:34:23 +00:00
Evgeny Danilenko
e4e36c152e
Extract validating interface (#1120)
* interface

* generalize interface

* linters

* fix deadlock

* fix linters

* close goroutine

* fix

* debug

* id+ttl

* refactor downloader tests

* tests

* lru

* handle genesis, extract fake consensuses

* fix fake consensus

* test uncles, verify

* after a new master

* fmt

* fix close

* debug

* debug

* fix chain length

* remove test field

* use single account

* fix data race on closing channel

* remove postponed blocks queue

* miner test

* VerifyHeaderRequests

* fmt

* fmt

* fix data race

* handle validating errors

* simplify matchParents

* remove copy-paste

* move sort to constructor

* clean up

* debug for 10 parents

* debug

* debug

* batch responses

* batch requests

* works for many ancestors

* remove debug

* always Close an engine

* linters

* ancestors deduplication

* fix test

* reduce interface

* api

* clique

* green clique sync

* stable

* perpermance per second

* full sync

* linters

* gitignore

* deps

* fix panic after master merge

* init consensus

* clique tests

* fix tests

* fix tests

* clean up

* reuse snap

* store vefified snapshots

* optimize snapshots

* safe close

* cleanup loop

* after downloader

* downloader and consensus tests

* update tests

* hack.go

* clique flags

* fix cliuqe config

* review

* gitignore

* remove additional bucket

* blk/sec instead of blk/microsecond

* rename

* deps

* optimize

* debug

* test

* tests without extracted validation process

* same base performance as on master

* benchmark

* simplify ethash verification

* ethash

* ethash

* linters

* ethash

* master stats

* cleanup

* gomod

* linters

* tests

* better locks

* Fix

* Remove logging for verifyHeaders

* Verification speed in the logs

* Fix compile error

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-25 19:40:45 +00:00
Alex Sharov
6894d9b356
Remove cursor features (#1508) 2021-02-23 09:02:54 +07:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500)
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
2021-02-21 18:41:59 +00:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Artem Vorotnikov
91f7798c4a
Remove hashed changesets (#1464)
* Remove hashed changesets

* try this

* oops

* simplify

* better diagnostics

* don't hash account key and use bitmap 64 to check history index

* update docs of bucket

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2021-02-04 12:59:04 +03:00
Alex Sharov
58c4e03861
Revert "Remove hashed changesets (#1444)" (#1446)
This reverts commit aadb6e47cc.
2021-01-20 14:16:20 +07:00
Artem Vorotnikov
aadb6e47cc
Remove hashed changesets (#1444) 2021-01-19 17:42:26 +07:00
Alex Sharov
e6f1e43e6f
lmdb: move fsync inside commit (#1433) 2021-01-04 08:17:18 +00:00
b00ris
8db5790838
State snapshot sync (#1417)
* move experiments to new branch&reorganise kv_snapshot

* walk&modify tests

* added delete from snapshot tests

* fmt

* state snapshot debug

* snapshot validation passed. copy state snapshot

* debug

* snapshot cursor.Prev test

* Prev works correct. Added Current check

* add err check

* added walk forward and backward test

* before refactoring

* refactoring

* execution with snapshot debug

* fix

* remove useless test

* before dupcursor implimentation

* tests with prev and delete works

* execution based on state snapshot passed

* remove useless tests

* blocks to 1140000 passed

* clean verifier

* cleanup state generation

* clean verify && seeder

* remove debug code

* tests passed

* fix lint

* save state

* test passed

* fix lint

* add state hash

* fix lint
2021-01-02 19:28:37 +00:00
Alex Sharov
311fb591c4
fix check index (#1424) 2020-12-26 09:01:00 +07:00
Artem Vorotnikov
8c0ecaeea9
Remove StageData (#1419) 2020-12-21 11:01:37 +00:00
ledgerwatch
0953fd42cb
WIP trace_call implementation (#1399)
* trace_call initial

* Fix tests

* More tracing

* Add more fields to the action

* Completed first example query

* Add initial bench11 to compare trace_call with OpenEthereum

* Exclude precompile calls from call traces

* Add self-destruct, call types, more comparison in rpctest

* Support for execution errors

* Stack underflow error and delegatecall value

* Fix lint

* Fix suicide traceAddress, Bad instruction error

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-14 11:27:52 +00:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00
Horacio Mijail Antón Quiles
ed1819ec58
Opcode tracer (#1320)
* OpcodeTracer

* Cleanup, rename to opcode_tracer

* Bring back check_change_sets, remove repeated vars

* Cleanup, gofmt

* Linter, cleanup

* Rename "segments" to "basic blocks", "bblocks"

* gofmt

* Lintci madness
2020-10-30 19:44:00 +00:00
Horacio Mijail Antón Quiles
f8477e29ab
Move batch management into the writeReceipts case (#1321) 2020-10-30 08:41:35 +00:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
Alex Sharov
6f27e3043b
lmdb_readonly_dont_create_folder (#1293) 2020-10-25 08:39:09 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader (#1288)
* protect_from_wrong_class_use

* more cases
2020-10-24 09:09:20 +01:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274)
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
2020-10-21 18:01:40 +01:00
b00ris
e5f6412da2
Update snapshot hash (#1243)
* remove useless bucket

* update snapshot hashes

* remove log
2020-10-17 08:48:20 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00
ledgerwatch
b747ab5324
[WIP] CallTraces index (#1157)
* Initial commit for CallTraces index

* Fix compilation

* fix lint, add comment

* Fix integration

* Add Close function to ethdb.Cursor, fix some compile errors

* Try to stop cursor leak in Get

* Fix compile errors in RPC daemon

* Fix compile errors

* fixing another way

* Some fixes

* More fixes

* More fixes

* More fixes

* Fixes to core/state

* Fix lint

* Fix lint

* Fixes

* Stage caching for call trace stage

* Add mem stats

* Try to stop the leak

* Turn off debug

* Chunks for 10k blocks

* Print

* Revert "Print"

This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.

* Revert "Chunks for 10k blocks"

This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.

* Trying to fix the leak

* Don't compute receipts in re-tracing

* Not compose block

* Print speed, fix receipts, bigger caches

* Fix lint

* Utilise changeset info

* Counters

* Use NoReceipts and ReadOnly

* ReadOnly is incompatible with caching

* Skip test leaking transactions

* Fix block test

* Change disable message for call-traces stage

* Use block option for call traces integration

* Fix retracing due to incarnation
2020-10-12 09:39:04 +01:00
Alex Sharov
88a099702b
rawdb.ReadCanonicalHash to return err (#1204)
* read_canonical_hash_return_err

* hack hugeFreelist

* fix tests
2020-10-10 07:05:56 +01:00
ledgerwatch
1af6cbe37b
Revert "Download/seed headers and block bodies via BitTorrent (#1195)" (#1202)
This reverts commit 94360212cb.
2020-10-07 21:49:44 +01:00
ledgerwatch
94360212cb
Download/seed headers and block bodies via BitTorrent (#1195)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-07 13:15:49 +01:00
ledgerwatch
bdb0155856
Revert "Add sync by bittorrent snapshots (#1160)" (#1194)
This reverts commit 4ebade3583.
2020-10-06 22:31:57 +01:00
b00ris
4ebade3583
Add sync by bittorrent snapshots (#1160)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-06 20:24:48 +01:00
Alex Sharov
61fbe34024
Better cbor support (#1163) 2020-10-02 10:56:13 +07:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124)
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Horacio Mijail Antón Quiles
252863fedf
Use the passed VMConfig (#1141)
* Use the passed VMConfig

* Explicitly pass the vmConfig into runBlock
2020-09-27 21:31:49 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) (#1113)
* added tls auth

* added client side

* put --tls

* fixed flag

* Add key/cert generation instructions, turn off Common Name verification

* Add CLI arguments and Warning

* Lint

* Update the doc about Internal IP

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00
Igor Mandrigin
adf52465e3
move ./trie to ./turbo/trie (#1114)
Useful for minimizing merge conflicts when rebasing new geth
2020-09-14 11:33:39 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon (#1089)
* added eth_getStorageAt

* used uint32

* now its 256

* incarnation

* added TLS handshake

* code minified

* lint

* minified client

* Update flags.go

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Alex Sharov
62fe81e4be
IH stage speedup and lmdb custom comparators support (#1080)
* etl.Loader - allow use of custom comparator

* log timing

* try now

* try now

* more performance

* etl.Loader - allow use of custom comparator

* working version

* simplify IH cursor

* clean

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* clean

* add only unwind support

* squash

* squash

* clean

* fix test

* clean

* clean

* clean
2020-09-10 13:35:58 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt (#1085)
* kill bolt

* kill bolt

* fix test

* clean
2020-09-09 11:09:55 +01:00
Alex Sharov
5ce73d438a
fix integration (#999) 2020-08-29 15:27:20 +07:00
ledgerwatch
335aedc7e3
Cut down the stateGrowth generation code (#973)
* Cut down the stateGrowth generation

* Fix NPE

* Fix for lastTimestamp

* Adjust to the current state

* Potential fix

* Fix for shorter keys in the state

* Extract code

* Iterate over code
2020-08-28 10:31:37 +01:00
Alex Sharov
d73b036cff
python_bublic_print_cli (#994) 2020-08-28 14:19:34 +07:00
Alex Sharov
9091ce62bb
KV reduce features amount (#975)
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize

* remove bolt_db.go file

* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Giulio rebuffo
0ffe32ae16
State growth restore (#951)
* new state growth

* lint
2020-08-21 07:32:11 +01:00
Alex Sharov
f827f86c11
Rpcdaemon as lib 2 (#943) 2020-08-20 10:52:27 +07:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction (#928)
* drop Bucket object from KV abstraction

* add some non-public methods

* rebase master

* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
Alex Sharov
3aa5526d5a
fix miss-type (#916) 2020-08-13 12:31:17 +07:00
Alex Sharov
9047f4fea0
python: don't hardcode buckets list (#915) 2020-08-13 11:40:30 +07:00
Alex Sharov
760fe0f0b6
DupSort of hash state (#896) 2020-08-12 09:57:55 +07:00
ledgerwatch
6f88e4058f
Fix (#908) 2020-08-11 20:48:37 +01:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
81c487803e
add python docs (#900) 2020-08-10 16:36:46 +07:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) (#854)
* return error from rawdb

* squash

* v14

* improve performance of put

* clean

* clean

* hide feature behind ENV variable

* cleanup

* cleanup

* disable ipc and make Readme less confusing (people thought points are depend on each-other)

* fix test

* cleanup

* cleanup
2020-08-02 12:53:08 +01:00
Igor Mandrigin
3aedcbcd7d Use datadir for temp files when generating indexes 2020-08-02 12:32:41 +02:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb (#788) 2020-07-27 19:15:48 +07:00
Alex Sharov
95365c8f37
remove ReadLastBlockNumber method = read execution stage status instead (#780)
* remove ReadLastBlockNumber method = read execution stage status instead

* make linter happy
2020-07-24 14:46:49 +01:00
Alex Sharov
a7e5c4b453
Remove remote_chain package, fix rpc methods: eth_blockNumber and eth_getBlockByNumber (#769) 2020-07-21 15:19:04 +07:00
Alex Sharov
1faa9dbcd4
add senders bucket to py (#763) 2020-07-20 16:11:47 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
b00ris
12aa55bcbb
Remove incarnation inversion (#755)
* remove incarnation inversion

* fix lint

* remove incarnation inversion

* remove comment

* remove inversion from new code

* revert for unwind
2020-07-17 15:18:31 +01:00
Evgeny Danilenko
5ead20e288
close tx_cache and peer.handle goruotines properly (#748)
* close goruotines

* fix using defer

* close pm managers a bit later

* fmt

* finalizer for in-mem db

* after cr

* revert finalizer

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-07-16 14:27:24 +01:00
ledgerwatch
3359ba7a04
Make cmd/tester initiate staged sync, remove some pools and caches (#746)
* no pools

* Remove excessive logging

* Revert

* Initial commit

* Cleanup

* Fix some compile errors

* Fix core tests

* Fix more tests

* Fix eth/downloader

* go mod tidy

* Fix core/vm

* Fix formatting

* Fix formatting

* Fix lint

* Fix lint

* Switch tests to StagedSync

* Cleanup

* Reuse cache in stage4

* Fix

* Fix formatting

* Try to fix test
2020-07-15 07:15:48 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes (#716)
* Fixing history generation

* Try fixes

* Reset from 0

* Cleanup
2020-07-06 07:34:24 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692)
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
2020-06-28 07:10:27 +01:00
Igor Mandrigin
69a20de5cc
Fix broken export file usage. (#680) 2020-06-20 21:07:39 +01:00