Commit Graph

875 Commits

Author SHA1 Message Date
Alex Sharov
36b9cc2982
Snapshots: initial sync fix (#3750) 2022-03-22 10:33:15 +07:00
Alex Sharov
7bf87e9597
Snapshots: grpc event (#3732) 2022-03-18 15:06:23 +07:00
primal_concrete_sledge
7d168884fe
Logs sub (#3666)
* save

* Add onLogs

* Fix lint

* Add proper logs

* Update go.mod

* goimports

* Add unwind
2022-03-17 07:40:18 +00:00
Andrew Ashikhmin
2785a57d1c
More Merge logging (#3712)
* More Merge logging

* Even more logging

* Marshall incoming payload json

* Fix json formatting

* Revert json marshalling

* Some extra logs
2022-03-16 06:55:21 +01:00
Alex Sharov
f03d08c5ce
Snapshots: retire testing tool (#3684) 2022-03-12 17:26:11 +07:00
Alex Sharov
0b2ce6f845
rpcdaemon: limit amount of read transactions (#3637)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* a

* a

* a

* a

* a

* a

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* fefefe

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2022-03-08 10:43:14 +07:00
yperbasis
d470349830 Sleepier stageLoopIsBusy + more logging 2022-03-03 17:51:25 +01:00
Giulio rebuffo
99f688c3ec
PrevRandao Renaming (#3616)
* PrevRandao Renaming

* lint
2022-02-24 18:16:05 +01:00
Andrew Ashikhmin
37ccbc53d6
Refactor payload building (#3592)
* Bump MaxPendingPayloads

* UnknownPayload error

* Improve ExchangeTransitionConfigurationV1 comment

* Extract MakeEmptyHeader

* Move MakeEmptyHeader to chain_makers

* Start with empty block as pending payload

* Only build payloads with the right parent hash

* Double check parent hash in SpawnMiningCreateBlockStage

* Simpler error formatting

* ReadHeadBlockHash instead of ReadHeadHeaderHash

* Avoid rebuilding already built blocks

* getPayload should stop the build process

* Tweaking stageLoopIsBusy

* Don't update pending payload after engine_getPayload
2022-02-23 21:27:38 +01:00
Andrew Ashikhmin
980276c322
Fix coinbase in stage_mining_exec (#3564)
* Simplify TerminalBlockHash & TerminalBlockNumber

* Simplify FeedHeaderPoS

* More accurate comment

* Relax TerminalBlockHash check

* Extract the sleep hack into stageLoopIsBusy

* Several short naps instead of a single longish sleep

* Fix coinbase in stage_mining_exec

* More orthodox loop

* typo in comment
2022-02-21 18:05:39 +01:00
Giulio rebuffo
7f596e16b3
added sleep (#3561) 2022-02-21 14:45:10 +01:00
Alex Sharov
3de83e2d96
RPCDaemon: to start with disabled txpool (#3539) 2022-02-18 09:45:20 +07:00
Andrew Ashikhmin
0701c83222
Fix stall on unwind in handleForkChoice (#3536)
* ReadHeader instead of ReadHeaderByHash

* Add Engine API to rpcdaemon README

* Fix nil pointer dereference in HandlePendingBlock

* Fix latestValidHash JSON

* Delete GetPayloadBodiesV1

* Break infinte unwind loop in handleForkChoice

* More logging
2022-02-17 18:30:57 +01:00
ledgerwatch
dffcc57ecc
[erigon2] Fix history bug, optimise bitmap search (#3510)
* Fix history bug, optimise bitmap search

* Optimise SeekInBitmap64

* Update erigon-lib

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-13 20:02:18 +00:00
Andrew Ashikhmin
fe9f841172
Add Beacon payloads to body downloader prefetch (#3482)
* Block instead of PayloadMessage

* tx.MarshalBinary instead of rlp.Encode in StartProposer

* Add Beacon payloads to body downloader prefetch

* Restore PayloadMessage
2022-02-11 13:24:29 +01:00
Alex Sharov
e695a6e58b
state and newHead: 1 goroutine per subscriber (#3452) 2022-02-10 08:25:58 +07:00
Andrew Ashikhmin
8466cb1b7b
New Engine API semantics (Kiln v1) (#3340)
* Disable PoS sync temporarily

* Resore PoS sync

* Handle Ctrl^C in HeadersPOS

* Consistent naming

* Extract verifyAndSavePoSHeader & downloadMissingPoSHeaders

* Preparation for EngineForkChoiceUpdated re-orgs

* Extract ForkingPoint

* Comments

* New proto for Engine API

* EngineExecutePayload -> EngineNewPayload

* Return INVALID_BLOCK_HASH if block hash is invalid

* Return EngineStatus_ACCEPTED for side chain blocks

* Update erigon-lib (PR 268)

* Fix payload2Hash

* reverseDownloadCh -> beaconPayloadCh

* Update erigon-lib

* Engine API updated

* ExecutionStatus -> PayloadStatus

* Introduce forkChoiceCh

* Mock ForkChoiceMessage/PayloadStatus

* Add ValidationError to PayloadStatus

* Small clean-ups

* Add INVALID_TERMINAL_BLOCK to EngineStatus

* Add a comment

* Extract handleNewPayload & handleForkChoice

* Partially implement handleForkChoice

* Update erigon-lib

* short vs long re-org

* Move header insertion out of downloadMissingPoSHeaders

* Update erigon-lib

* Refactor ProcessSegmentPOS

* Fix imports

* Wire downloadMissingPoSHeaders into handleForkChoice

* evictOldPendingPayloads

* nolint:unused for assertSegment

* Try nolint instead of nolint:unused

* Small comment improvements

* HeadHeaderHash/StageProgress in handleForkChoice

* TODO: bodyDownloader.AddToPrefetch(block)

* Review call suggestions

* Don't use ReadHeaderNumber in ProcessSegmentPOS

* Don't leave ethbackend waiting when server is stopping

* Update erigon-lib

* More explicit signature of downloadMissingPoSHeaders
2022-02-09 08:33:22 +01:00
Giulio rebuffo
1602842a63
fixed api.blockWithSenders (#3435)
* fix

* fix 2

* fixed api.blockWithSenders

* removed white space

* pretty
2022-02-05 19:11:56 +00:00
Alex Sharov
304b40dd0a
Snapshot: prohibit to change cli flags (#3311) 2022-01-22 11:18:24 +07:00
Enrique Jose Avila Asapche
ffedd6ac3f
Enabled proposer by default (#3319) 2022-01-21 22:50:34 +03:00
e-danko
d0123f52fd
Starknet grpc (#3312)
* created starknet_call method

* updated erigon lib

* updated erigon lib

* lint issues
2022-01-20 15:34:00 +00:00
Andrew Ashikhmin
c2d0f0082a
Report Valid ExecutionStatus only after all stages (#3212)
* Stricter transition check

* EthBackendServer.numberSent was unused

* Take account of PoS blocks in TopSeenHeight

* Report Valid ExecutionStatus only after all stages, not straight in headers

* Print TTD on startup

* No need to check parent.Hash() again

* core: change baseFee into baseFeePerGas in genesis json

* fix typo

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-01-07 16:54:26 +01:00
Alex Sharov
4ef541f5dc
Snapshots: tx lookup in RPC from snapshots (#3214) 2022-01-07 20:52:38 +07:00
Giulio rebuffo
195eb9abaa
Added mining for POS (#3187)
* added comunication with channels

* added final mining

* removed trash

* bug fixing

* async calls

* one thread

* tests

* better comments

* no lock

* better placing of skipCycleHack

* removed long timer

* moved transitioned block processing

* better naming

* disabled updates on blocks

* sync.Cond

* 2 sync.Cond

* better locking

* Use single sync.Cond. Proposer shutdown

Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
2022-01-04 18:37:36 +01:00
Giulio rebuffo
864b744ed2
Support for syncing on mergemock (#3174)
* block proposing

* standard finalized

* mergemock execution

* private chain can now be ran yay

* perfectioned

* polished

* more polishing

* resize PR

* resize PR

* resize PR

* simplifications

* fixed tests

* better syncronous communication

* better syncronous once again

* clean

* Re-enabled headers verification

* mining finish

* mining finish

* cleaned hash computation

* fixed evm bug

* go.mod

* Update .gitignore

* Update .gitignore

* Update .gitignore

* removed new line from .gitignore

* added go.mod and go.sum

* feeRecipient into preset

* useExternal

* todo

* fixed comment for forkchoiceUpdateV1

* smaller

* smaller

* Revert changes to miner frequency

* Restore useExternalTx

* Fix headerLoadFunc

* do not reset payloadId

* rename

* extra is empty

Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
2021-12-29 17:25:13 +01:00
Andrew Ashikhmin
445ac05220
Verify PoS headers before inserting them into DB (#3151)
* Remove TotalDifficulty stage

* Introduce FeedHeaderFuncType

* Some TODOs

* Don't gossip PoS headers

* FeedHeaderPoS

* Call header.Hash once

* Use FeedHeaderPoS for to write PoS header

* Support errors in EngineExecutePayloadV1

* useExternalTx boilerplate

* Call VerifyHeader on PoS header

* latestValidHash must be nil when status is SYNCING

* fixCanonicalChain instead of canonicalHeadersCollector

* engine.VerifyHeader + FeedHeaderPoS as headerLoadFunc

* fix daemon

Co-authored-by: Giulio Rebuffo <giulio.rebuffo@gmail.com>
2021-12-23 15:06:10 +01:00
Andrew Ashikhmin
cfb35633ad
Make waitingPosHeaders atomic since it's accessed concurrently (#3157) 2021-12-22 17:12:35 +01:00
Giulio rebuffo
23b3c1d9bc
Added reversed for proof-of-stake syncing (#3092)
* reverse sync slow but working

* progress

* p

* close request

* backwards refactoring

* cleanup 1

* added RequestAssembler

* remove trash code and spaghetti

* efficient

* fix

* refactor

* tf

* refact

* final refactoring

* headers forward restored

* test fixed

* make CI happy

* resolved comments

* not using insertList anymore

* oops

* better collectors

* removed debug config

* avoid pointers

* added sleep

* use of channels

* sleeping

* added logs for ETL

* added more cleanup

* correct glacier

* some refactoring

* maxRequests

* tweaks

* config.go

* config conflicts

* renamed functions
2021-12-13 17:46:15 +01:00
primal_concrete_sledge
d89e5d2603
Issue/2710 add grpc health check (#3091)
* ISSUE-2710: Add standard grpc health check to services with grpc server

* Go import changed files

* Add flags for healthcheck

* Add grpc healthcheck option to rpcdaemon

* Remove grpc port info if grpc is not enabled

* Resolve merge issues
2021-12-06 12:03:46 +00:00
Giulio rebuffo
3c95db0078
Fixed Proof-of-stake transition (#3075)
* added download channel to stage headers

* added direct execution

* now channel of headers rather than bodies

* simplifications

* fixed ttd headerForward

* garbage removed

* added some more stepping stones

* better transition

* covered edge case

* added names to returned values

* clean
2021-12-03 11:55:00 +01:00
ledgerwatch
b01df0f795
Bump EthBackend API to 3.0.0 in devel branch only to make it explicitly incompatible with stable (#3077) 2021-12-02 13:40:48 +00:00
Andrew Ashikhmin
094ab5e77d
[EIP-4399] Remove newly added Header.Random (reuse mixHash instead) (#3069)
* [EIP-4399] Remove newly added Header.Random (reuse mixHash instead)

* add a TODO

* Update mock payload hashes
2021-12-01 13:41:31 +01:00
Dmitry Savelev
d4f6ceff1c
Add NodeInfo implementation (#3046)
* Add NodeInfo implementation

* replace magic value with a constant.

* update dependencies

* bump minor version

* add nodes deduplication logic.

* shuffle values in test cases a little.
2021-11-30 22:42:12 +00:00
Giulio Rebuffo
8ee29ee0d1 clear test 2021-11-30 18:43:46 +01:00
Giulio Rebuffo
50dbf1fc33 lint 2021-11-30 18:24:20 +01:00
Giulio Rebuffo
b9018db353 comment 2021-11-30 18:14:34 +01:00
Giulio Rebuffo
5e9bf095c8 enum 2021-11-30 18:13:34 +01:00
Giulio Rebuffo
021aa1da08 lock 2021-11-29 15:58:46 +01:00
Giulio Rebuffo
e76bc807bb better api 2021-11-29 15:47:08 +01:00
Giulio Rebuffo
0c9e5cd513 fixed unit tests 2021-11-27 16:29:46 +01:00
Giulio Rebuffo
6866eec31f added transition checks 2021-11-26 23:03:47 +01:00
Giulio Rebuffo
6bffeb8f5f better notation 2021-11-26 19:47:23 +01:00
Giulio Rebuffo
dbfa51960c added downloader unit tests 2021-11-26 14:51:32 +01:00
Giulio Rebuffo
7c60126347 added unit tests for simple execution 2021-11-26 11:59:05 +01:00
Giulio Rebuffo
f1ca526e33 lint 2021-11-24 23:37:26 +01:00
Giulio Rebuffo
768617f412 added proper errors 2021-11-24 23:19:40 +01:00
Giulio Rebuffo
435d1d1519 channels implemented 2021-11-24 23:00:00 +01:00
Giulio Rebuffo
ddb01b6314 ? 2021-11-22 18:39:23 +01:00
Giulio Rebuffo
bac85f854b added getPayloadV1 2021-11-22 16:12:34 +01:00
Giulio Rebuffo
c05e3a9fa1 connected to daemon 2021-11-22 11:36:52 +01:00
Giulio Rebuffo
21cbc19f21 using batch over tx 2021-11-21 23:11:50 +01:00
Giulio Rebuffo
1267d540b1 added engine_executePayloadV1 2021-11-21 22:47:33 +01:00
Giulio Rebuffo
23883d9104 merge 2021-11-21 21:55:41 +01:00
Giulio Rebuffo
4eb650fb0c payload manipulation 2021-11-21 21:52:36 +01:00
Alex Sharov
e55256296b
snapshots: read block from snapshots, add sender to txs file (#2996) 2021-11-21 10:32:14 +07:00
Giulio Rebuffo
6dd2d67121 Added ExecutePayloadV1 method to engine_ namespace 2021-11-20 22:53:03 +01:00
Giulio Rebuffo
224441cb07 initial commit2 2021-11-20 17:27:11 +01:00
Alex Sharov
aad0d0c777
grpc GetBlock api (#2955) 2021-11-14 11:08:52 +07:00
Alex Sharov
c7db8a2e8d
prohibit change prune mode (#2907) 2021-11-03 18:05:02 +07:00
Alex Sharov
da00e949c4
RemoteRPC: coherentCache for kv.Code; LocalRPC: enable small blocksLRU (#2815) 2021-10-12 12:04:04 +07:00
Alex Sharov
5ca558c667
Move lengths to erigon-lib other packages depend on it (#2799)
* move lengths to erigon-lib

* move lengths to erigon-lib
2021-10-08 10:20:45 +07:00
Andrea Lanfranchi
3bc3be4b5f
Clean up DEBUG category logs (#2776)
- move many DEBUG logs into TRACE category
2021-10-05 08:14:04 +07:00
Alexandr Borodulin
d7159cd8bd
More convenient pruning for non-PoW consensus: add --prune.*.before flags (#2714) 2021-09-23 09:13:19 +07:00
Alex Sharov
3de50637cd
remove changeset.Walk func (#2716) 2021-09-22 07:54:29 +07:00
Alex Sharov
133eec0c7e
Pool: correct new pending txs notifications (#2715) 2021-09-21 17:35:28 +07:00
Alex Sharov
b30ccd2ddf
Revert "Add prune before parameters" (#2701)
This reverts commit 49a98ad4b6.
2021-09-18 08:37:39 +07:00
Alexandr Borodulin
49a98ad4b6
Add prune before parameters 2021-09-17 19:01:15 +07:00
Alex Sharov
400c71b7ce
Pool: txID-based cache (#2697) 2021-09-17 10:31:20 +07:00
Alex Sharov
ad1682cf40
Expose mdbx's txID to remote_kv (#2686) 2021-09-15 14:22:57 +07:00
Alex Sharov
77db993cfc
Pool: add coherent cache (#2678) 2021-09-13 14:58:25 +07:00
ledgerwatch
15b4095718
Move ETL to erigon-lib (#2667)
* Move ETL to erigon-lib

* Update link in the readme

* go mod tidy

* Use common/chan.go from erigon-lib

* Clean up

* Fix lint

* Fix test

* Fix compilation

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-09-12 08:50:17 +01:00
Alex Sharov
28480a36ec
Pool: add --txpool.v2 flag to rpcdaemon (#2624) 2021-09-03 11:19:35 +07:00
Alex Sharov
4c1dbdf50e
Pool: add --txpool.v2 flag to erigon (it just disabling existing machinery for now) (#2620) 2021-09-02 17:04:30 +07:00
Alex Sharov
953f348d1e
Pool: add grpc server (#2615) 2021-09-02 12:55:04 +07:00
Alex Sharov
f3021dd2af
RPC: small optimizations (#2564) 2021-08-24 08:28:58 +01:00
Alex Sharov
6c7edb0d34
state diff send to txpool (#2532) 2021-08-17 15:52:55 +07:00
Evgeny Danilenko
4cd72c8328
Keep readonly value while changing interpreters back and forth (#2508)
* restore TEVM

* fuzzing and property based

* comment

* lint

* stack callback into defer ater checking an error

* sequential tests
2021-08-10 09:48:56 +07:00
Alex Sharov
d617fca739
remove devp2p cmd - if we will support it then need do it in another repo (because it depends on account manager). (#2496)
* up cobra lib

* remove devp2p cmd - if we will support it then need do it in another repo (because it depends on account manager).
2021-08-08 09:17:08 +07:00
Alex Sharov
6e24a83065
extract kv interface from remoteKv client (#2494) 2021-08-07 12:03:12 +07:00
Richard Patel
7767b4df88
rpcdaemon: add txpool_status() (#2407)
perfecto
2021-08-06 09:45:44 +07:00
Alex Sharov
3ed0f5c83d
better prune error message (#2475) 2021-07-31 12:24:20 +07:00
Alex Sharov
6bd44eb26c
move kv to erigon-lib (#2467) 2021-07-29 18:53:13 +07:00
Alex Sharov
69dc2776dc
MIT licensed metrics lib (#2462) 2021-07-29 17:27:46 +07:00
Alex Sharov
5069558752
Apache licensed logger (#2460) 2021-07-29 17:23:23 +07:00
Alex Sharov
21cb7befa4
finish remove bucket suffix (#2458) 2021-07-28 10:43:51 +07:00
Alex Sharov
838e5f9ef2
Move bucket constants into kv package, move kv interface to kv package (#2455) 2021-07-28 09:47:38 +07:00
Alex Sharov
63a6c45ef1
Remove some db interfaces (#2452) 2021-07-27 19:35:32 +07:00
Alex Sharov
6bf54c951a
Sokol v0: unwind support (#2439) 2021-07-24 16:50:42 +07:00
Alex Sharov
9398e2ff93
Breaking change: Docker to follow XDG spec (#2437)
change default dir inside docker from /var/lib/erigon to /home/erigon/.local/share/erigon
2021-07-24 16:14:46 +07:00
Alex Sharov
6801208497
move some tests to kv interface (#2438) 2021-07-24 14:14:11 +07:00
alex.sharov
b56e500aab remove debug 2021-07-24 12:26:36 +07:00
Alex Sharov
a36a613e61
move migrations to kv interfaces (#2430) 2021-07-24 11:28:05 +07:00
Alex Sharov
f735997fa1
Sokol v0: remove unused parameters and methods (#2435) 2021-07-24 11:04:07 +07:00
b00ris
0c5f3df8a0
Temp db for state snapshot generation (#2416)
* save state

* tmp db integration

* fmt

* temp db for state snapshot

* fmt
2021-07-23 13:06:37 +01:00
b00ris
7f81f9858b
proper handling of deleted values for snapshot (#2387) 2021-07-21 16:36:09 +03:00
Alex Sharov
c3e1cfdac8
Pruning for: exec, log_index, tx_lookup, history stages (#2399)
* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* add tvm flag

* save

* db migration for storage mode
add flag --prune=
remove flag --storage-mode=
add flag --experiments=tevm,...
rename integration set_storage_mode to set_prune

* fix

* forward move of stages must skip everything before PruneTo

* keep in db progress of prune method

* keep in db progress of prune method

* simplify logs

* simplify logs

* simplify logs

* fix test

* simplify logs

* simplify logs

* simplify logs

* simplify logs

* remove callTraceSet as dupsort
use etl transform for txlookup prune
remove some logs

* cleanup tests a bit

* print_stages and eth_sync to show prune progress

* fix print_stages

* add readme about --prune.to flag

* more docs

* add --prune.history.older and other flags support

* fix migration on empty db

* better toString

* better toString
2021-07-20 21:03:19 +01:00
Alex Sharov
031b0b177e
remove mutexes use (#2406) 2021-07-20 15:34:11 +07:00
Alex Sharov
a79b3cd6eb
unify stage params (#2384) 2021-07-17 09:09:56 +07:00
Alex Sharov
9f6ef74adc
Linter: to check tx.Rollback() by ruleguard (#2383) 2021-07-16 20:23:54 +07:00
alex.sharov
bc65759c84 clien id in rpc 2021-07-15 16:11:39 +07:00
Alex Sharov
c9e4e0bd3d
broadcast local txs (#2354) 2021-07-15 11:17:48 +07:00
Alex Sharov
04465e04df
increase read transactions limit to 32K (#2366) 2021-07-14 21:48:20 +07:00
Alex Sharov
42b776ebb3
increase max msg limit for grpc (#2361) 2021-07-14 09:43:58 +07:00
Alex Sharov
b6b42930ad
simpify db logger (#2359)
* logger simplify db

* logger simplify db
2021-07-13 20:15:37 +07:00
Andrea Lanfranchi
f909be7157
Update kv_interface.go (#2338)
Amend comment
2021-07-10 18:15:31 +07:00
Alex Sharov
5ceaac53f6
batch cancel (#2314) 2021-07-08 19:40:49 +07:00
Alex Sharov
7c60cc0fee
step towards kv interface (#2313) 2021-07-08 09:56:09 +07:00
Alex Sharov
a78cb54132
step towards kv interface (#2311) 2021-07-07 23:15:49 +07:00
b00ris
57473175ff
Body snapshot (#2100)
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* save state

* body snapshot test

* unique tx

* body snapshot generation using walk

* move methods out of test

* block data verification added

* fix lint

* test with remove works correctly

* fix lint

* remove experiment test

* fix test

* add comment

* add second layer of remove test

* rename test

* fix typos

* fix lint

* revert testdata

* body snapshot migration save state

* fix body snapshot migration

* fix after merge

* remove debug test

* debug windows build

* fix build

* fix

* fix lint

* debug

* fix

* fix windows build

* simplify snapshot management&&get rid of lazy tx

* fix lint

* fix windows path

* debug

* debug

* debug

* debug

* remove geometry experiments

* skip windows tests

* clean

* fix

* fix ;int
2021-07-06 23:33:26 +01:00
Alex Sharov
ea5abbe0b5
mdbx: add trace info to open error (#2305)
* add stack info to open error

* add stack info to open error
2021-07-05 19:53:11 +01:00
Alex Sharov
c029bbe7fa
fix metrics (#2292) 2021-07-04 10:25:07 +07:00
ledgerwatch
a05d1be531
Pull grpc bindings into erigon-lib (#2265)
* Pull grpc bindings into erigon-lib

* Remove mdbx dist

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-07-01 22:31:14 +01:00
Alex Sharov
3d6d45a82f
remove subtrie loader and dcmp (#2258) 2021-06-30 10:31:36 +07:00
Alex Sharov
8f6aa2f7ae
propagate new tx to p2p (#2254) 2021-06-29 17:00:22 +07:00
Alex Sharov
a6f4b08716
Revert "Propagate new txs to P2P (#2249)" (#2253)
This reverts commit e9ae6d688f.
2021-06-29 15:46:00 +07:00
Alex Sharov
e9ae6d688f
Propagate new txs to P2P (#2249) 2021-06-29 14:40:23 +07:00
Alex Sharov
2b59b45216
Do log tables size, db metrics - avoid concurrency, check stale readers hourly (#2216)
* add table size logs and metrics

* log stale readers

* - don't collect db metrics in background
- do check stale readers once an hour

* execution to update metrics when print logs

* hide file exists err

* hide file exists err
2021-06-22 11:08:47 +01:00
Artem Vorotnikov
72e9a34365
Remove unused tables, soft rename, take 3 (#2207)
* Remove unused tables, soft rename, take 3

* Bump db schema version

* nil check in migration
2021-06-20 20:46:57 +01:00
Artem Vorotnikov
504c31e97a
Revert "Remove unused tables, soft rename" (#2205) 2021-06-20 15:08:48 +01:00
Alex Sharov
a535e8f30c
switch to mdbx-go module, and db-tools to libmdbx submodule (#2199)
* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule
2021-06-19 21:30:12 +01:00
Artem Vorotnikov
a9baf37ea1
Remove unused tables, soft rename (#2201) 2021-06-19 21:36:21 +03:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
mempirate
ca926ce94d
net_peerCount for remote sentries (#2192)
* net_peerCount for remote sentries

* cleanup

* Simplify NetPeerCount using SentryClient

* Refactor id -> count
2021-06-18 14:34:15 +01:00
Alex Sharov
e9f510c2b7
erigon as lib (#2186) 2021-06-18 10:35:11 +07:00
mempirate
e51b09da0d
Implements net_peerCount RPC call for internal sentries (#2172)
* Implement API and protobuf

* Squashed 'interfaces/' content from commit cd3eca28e

git-subtree-dir: interfaces
git-subtree-split: cd3eca28e50367af9de52fca863b1d7528bff774

* Implement net_peerCount RPC call

* Squashed 'interfaces/' content from commit cd3eca28e

git-subtree-dir: interfaces
git-subtree-split: cd3eca28e50367af9de52fca863b1d7528bff774

* Fixing merge conflicts

* Remove p2p.Server from Ethereum backend

* Fix peerCount calculation for internal sentries

* Bump eth backend API version

* Update rpcdaemon README

* Document version change
2021-06-17 22:55:20 +01:00
Artem Vorotnikov
af836a6200
Nuke LMDB (#2167) 2021-06-16 13:57:58 +03:00
Alex Sharov
dcf850dd75
fix fieldalignment linter warnings (#2171) 2021-06-16 11:00:35 +07:00
Alex Sharov
b79823d1b9
create dir for node keyfile (#2161) 2021-06-14 07:35:22 +01:00
Alex Sharov
e7b56358d3
Mdbx reproducible build (#2145)
* save

* save

* save
2021-06-11 14:04:12 +01:00
Alex Sharov
eedabfca85
txpool_content (#2127)
* txpool_content

* txpool_content

* txpool_content

* txpool_content

* Squashed 'interfaces/' content from commit 1e916d013

git-subtree-dir: interfaces
git-subtree-split: 1e916d0137aff7bd9e9356cc203cea5f0ea9c7f1

* add test

* add test

* add test

* merge devel

* save
2021-06-11 09:34:47 +01:00
Alex Sharov
ba56e7a139
extract p2p config out of sentry (#2126)
* extract p2p config out of sentry

* extract p2p config out of sentry

* extract p2p config out of sentry

* rebase devel

* rebase devel

* add bootnodes to sentry

* add bootnodes to sentry
2021-06-11 09:34:37 +01:00
Evgeny Danilenko
de26634d8d
TEVM status pruning (#2034)
* use trace for TEVM status

* restore testdata

* fix getting contract from plain state

* unwind

* linters

* reproducible build debug

* debug

* debug

* remove debug
2021-06-11 09:34:25 +01:00
ledgerwatch
73bb98f686
Move tests 6 (#2107)
* Move tests 6

* Compile fixes

* Fix lint

* Compile fixes

* Fix compile

* compile fixes

* Compile fix

* Fix consesus/clique

* Cleanup

* Add gas limit

* Print

* Prints

* More print

* Fix

* Reinstate TestGetBlockReceipts66

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-06-06 22:44:14 +01:00
Alex Sharov
5ba3ea162a
Simulated backend and genesis to kv (#2104)
* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* merge

* merge

* merge

* merge

* merge

* merge

* merge

* merge
2021-06-05 16:17:04 +01:00
Alex Sharov
eee871aedf
Add db label (later will add txpool db) (#2081)
* add db label - later will add txpool db

* move commit marker to kv_mdbx behind Chain label

* Update migrations.go

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 15:56:49 +01:00
Alex Sharov
ccac3c6566
Add tx.Append method for ergonomic (#2088)
* close cursor

* stateless append

* stateless append

* faster pruning del and close cursor

* faster pruning del and close cursor

* less wrappers

* less wrappers

* less wrappers

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 14:32:48 +01:00
b00ris
66c7c669fb
Snapshot builder simplification (#2064)
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* fix
2021-06-04 13:30:02 +01:00
Alex Sharov
52d35930e8
simplify receipts reading (#2094)
* simplify receipts reading

* test
2021-06-04 13:28:18 +01:00
Alex Sharov
bebaebb13a
mdbx 0.10.1 (#2083)
* mdbx 0.10.1

* remove -tags mdbx

* remove -tags mdbx

* remove -tags mdbx
2021-06-04 13:26:15 +01:00
ledgerwatch
43915a73a8
Migration to fix trace_filter (#2095)
* Migration to fix trace_filter

* Fix to db/tx

* Fix to db/tx

* Error fixes

* Bump KV version

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-06-04 12:45:06 +01:00
ledgerwatch
67d0a2d3ab
Reproduce and fix trace_filter index problem (#2084)
* Intermediate

* Test for trace_filter

* Clean up

* Clean up

* clean up

* More to the test

* Hack

* Block parameter

* Block parameter

* use mdbx

* Print number of records

* Print keys

* Print index records

* Print block numbers

* Print values in tracecallset

* Print some more

* More printing

* Print froms and tos

* Print startBlock and endBlock

* Tests and fixes

* Cleanup

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-06-04 09:09:57 +01:00
alex.sharov
c9b2d71333 add config2.h to gitignore 2021-06-03 14:17:42 +07:00
Alex Sharov
29fa1aa35a
sentry join streams (#2059) 2021-06-01 17:41:10 +07:00
Alex Sharov
017fb69da1
Fix GC chart (#2071)
* save

* save
2021-06-01 16:53:28 +07:00
Alex Sharov
a4ff299afb
Change set docs (#2062) 2021-05-31 15:29:46 +07:00
Alex Sharov
3016b96e88
1 Sentry per protocol (#2046) 2021-05-30 09:53:30 +07:00
Andrea Lanfranchi
cdcf9a10be
Mdbx wsl2 (#2048)
* wsl2

* Handle PS crash for garbage keys

* More strict error checking

* Update readme for reintroduced WSL support

* Wrong htop display only for LMDB

* Wrong link in Readme

* A further note on WSL IP address

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2021-05-29 14:16:53 +01:00
Alex Sharov
60e8a31fe3
genesis switch db (#2038)
* genesis switch db

* genesis switch db
2021-05-28 13:55:48 +01:00
alex.sharov
6b826174c1 use default dynamic mdbx dpLimit 2021-05-28 14:08:24 +07:00
Evgeny Danilenko
7352b81122
Tevm stage 1 (#1845)
* convert contracts after block execution

* check if has tevm code

* after review-1

* handle ErrNotFound

* typo

* tests

* tevm code bucket

* testdata

* execute pre-stage

* after merge

* test fix

* test fix

* fix test after merge

* disable translation stage

* after merge

* rename params

* rename to Erigon

* parallelize EVM translation

* fix

* logging and fixes

* fix

* todos

* cleanup

* revert erigorn renaming

* unwind

* tevm unwind

* fix AppData

* non-parallel version

* comments
2021-05-27 14:54:55 +01:00
ledgerwatch
c15735d7a0
Make c in storage-mode default (#2021) 2021-05-26 16:00:43 +01:00
Alex Sharov
ed64380226
Up db schema for mdbx (for all) and up kv version: to 2.0.0 (#2019)
* mdbx db schema version up

* Remove migration to simplify

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-26 12:51:44 +01:00
Evgeny Danilenko
74847d77e6
Rename to Erigon (#2018)
* turbo-geth to erigon

* tg, turbo to erigon
2021-05-26 11:35:39 +01:00
Alex Sharov
450c79e3ab
move integration to kv (#1996) 2021-05-23 12:41:42 +07:00
Artem Vorotnikov
5655409616
Do not write old block changesets in pruning storage mode (#1964)
* Remove incremental staged sync

* Do not write old block changesets in pruning storage mode

* Remove pruning flag, put behaviour behind !StorageMode.History
2021-05-22 11:57:10 +01:00
Alex Sharov
14c15cba43
Check version of remote services (#1989)
* save

* save

* Squashed 'interfaces/' content from commit 08c32a09e

git-subtree-dir: interfaces
git-subtree-split: 08c32a09e40b1e6fcb5922e723191c9477545356

* Revert "Squashed 'interfaces/' content from commit 08c32a09e"

This reverts commit 8393d9fd

* save

* seve

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* ensure versions compatibility of all remote services

* Revert "Squashed 'interfaces/' content from commit dd6a42724"

This reverts commit 2a764bf9

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* Revert "Squashed 'interfaces/' content from commit dd6a42724"

This reverts commit 52621846

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* a

* a

* a

* a

* a

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 11:00:13 +01:00
ledgerwatch
729bfab988
Split DBSchema version depending on the database type (#1990)
* Split DBSchema version depending on the database type

* Fix test

* Fix test

* Fix lint

* Update migrations.go

* Update bucket.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 10:20:43 +01:00
Alex Sharov
c4e20896fa
mdbx ensure lib version match (for .dll and .o compatibility) (#1988) 2021-05-22 11:18:04 +07:00
Alex Sharov
545fe32f68
Mdbx - make it default db. Lazy buckets renaming. (#1982) 2021-05-22 10:03:02 +07:00
Alex Sharov
4dcfd24af5
Fix mdbx_problem error (#1979) 2021-05-21 09:45:08 +07:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
215623da80
Windows support (#1944) 2021-05-20 23:46:12 +07:00
Alex Sharov
50934fa5d3
Move chain maiker to kv (#1972) 2021-05-20 18:49:33 +07:00
Alex Sharov
9d66aa0fe3
better support for mdbx in snapshots (#1971) 2021-05-20 18:12:07 +07:00
ledgerwatch
79084ede47
[download.v2] Make default (#1963)
* [download.v2] Make default

* Update server.go

* Fix for tmp files appearing in the current dir

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2021-05-19 16:19:29 +01:00
Alex Sharov
369b41dfc2
clean (#1962) 2021-05-19 15:55:10 +07:00
Alex Sharov
ba185eff83
tx.Cursor() ignored error (maybe it's fix for MDBX_PROBLEM) (#1961) 2021-05-19 11:21:17 +07:00
Alex Sharov
781a67a4ee
Auto-Close db in tests by using t.Cleanup (#1956) 2021-05-19 10:47:28 +07:00
b00ris
777882ff52
Fix corner case for KV_Snapshot.Next (#1957)
* fix next

* comment test
2021-05-18 19:32:04 +01:00
Alex Sharov
cceaf75b16
use t.TempDir() (#1955) 2021-05-18 19:13:16 +07:00
Alex Sharov
0c91bfbf3e
RPCDaemon: support Pending block (#1942)
* clean

* save

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* save

* tx pub-sub

* tx pub-sub

* clean

* clean

* save

* save

* save

* save

* save

* Squashed 'interfaces/' content from commit c469f3ae0

git-subtree-dir: interfaces
git-subtree-split: c469f3ae073b60c8821b61fed2910191080ef835

* save

* save

* save

* save

* Squashed 'interfaces/' changes from c469f3ae0..958dfc669

958dfc669 save

git-subtree-dir: interfaces
git-subtree-split: 958dfc669f8daeefe686a13aa852fb95f1537886

* save

* save

* up some deps

* up some deps

* clean

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test
2021-05-17 13:15:19 +01:00
Alex Sharov
617baac56b
move db tests to kv interface (#1946) 2021-05-17 16:54:29 +07:00
Alex Sharov
9bef3fb656
move some tests to mdbx (#1947) 2021-05-17 16:14:08 +07:00
Alex Sharov
2ff6ce7d92
move staged sync to tx (#1897) 2021-05-14 18:00:20 +07:00
b00ris
48733c31cd
test fix (#1930) 2021-05-13 21:04:47 +01:00
Alex Sharov
f3a9eb340c
mdbx_setup_debug - handle response correct way (#1931) 2021-05-13 21:01:23 +07:00
b00ris
7948eafcc2
Property tests for kv snapshot (#1928)
* model tests for snapshot kv

* cursor test

* first failing test

* fix

* fix lint

* remove fmt
2021-05-13 09:40:32 +01:00
Alex Sharov
3e1bdbb465
Mdbx: fix for MDBX_PROBLEM (#1922) 2021-05-13 14:40:44 +07:00
Alex Sharov
0163329a5a
KV and RemoteKV: to reuse stateless cursors during transaction (#1909) 2021-05-11 15:23:17 +07:00
Artem Vorotnikov
80bbbf62c2
Misc fixes (#1908)
* Fix debug levels

* try this

* remove adapter

* fix

* fix bodies stage
2021-05-10 06:59:29 +01:00
Alex Sharov
3f36e362c2
mdbx switch to release (#1906) 2021-05-09 15:35:47 +07:00
ledgerwatch
8907e95d4a
Trace filter 3 (#1904)
* Prevent the leak

* Default storage mode, fix leak again

* TopSeenHeight to be updated from any header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-08 23:07:16 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836)
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 09:45:40 +01:00
Artem Vorotnikov
09788dbc02
Pruning flag, incremental staged sync (#1899) 2021-05-08 00:07:49 +03:00
ledgerwatch
59e5502744
Initial work on RPC streaming (#1880)
* Initial work on RPC streaming

* Create the stream and pass it on

* trace_filter expressed as streamable (not finished)

* Reenable call trace index

* Add new line

* Remove storage mode override

* Tool for overriding storage mode

* Diagnostics

* Make trace_filter more streamy

* Streaming debug_traceTransaction and debug_traceCall

* Fix test

* Log error on stream flush

* Enable streaming for http

* Flush the stream too

* Make trace_filter flush too

* Stop streaming if client is not interested

* Try to quiet annoying test|

* Revert "Try to quiet annoying test|"

This reverts commit 42849257bfa52e90140aa535af34b957cd97a222.

* Debug log for test

* Proceed with handshake regardless of whether peer notification worked

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-06 18:37:38 +01:00
Alex Sharov
4323d56376
Mdbx tx limit metric (#1891) 2021-05-06 17:02:50 +07:00
Alex Sharov
49225f7319
Mdbx page ops metrics (#1885) 2021-05-06 10:51:49 +07:00
Alex Sharov
a627bc2d14
use same keepalive settings in sentry and rpcdaemon (#1883)
* use same keepalive settings in sentry and rpcdaemon

* use same keepalive settings in sentry and rpcdaemon
2021-05-05 11:32:04 +01:00
Alex Sharov
e87d4932f7
Changeset writer move to tx (#1871) 2021-05-05 14:31:40 +07:00
Alex Sharov
61c20798db
fix concurrent map write (#1882) 2021-05-05 10:16:57 +07:00
ledgerwatch
7c60ec3ec0
Fix for eth_getTransaction... returning tx from the pool (#1881)
* Fix for eth_getTransaction... returning tx from the pool

* Fix

* Fix more issues

* Fix for EOF

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-04 22:58:13 +01:00
ledgerwatch
d604bfbb4f
Fix transaction stream test (#1879)
* Fix transaction stream test

* Fix

* Update events.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-04 17:12:59 +01:00
Alex Sharov
c7eedf0091
Add kv to stage settings (because it doesn't change in runtime, tx will be passed as Stage func argument) (#1872)
* add kv to config

* move forward tx use

* move forward tx use

* gc stat

* gc stat

* changeset writer - move to tx

* clean

* clean

* clean

* clean

* clean
2021-05-04 13:36:03 +01:00
ledgerwatch
99578ec58d
Fix RPC daemon (#1873)
* Fix RPC daemon

* Skip the test to fix RPC daemon

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-04 12:31:07 +01:00
Alex Sharov
66b683a6b3
RemoteKv version up (#1874)
* clean

* clean

* clean
2021-05-04 09:56:54 +01:00
ledgerwatch
793c13e20e
trace_filter and CallTrace Index (derivative of #1626) (#1867)
* Trace Index

* trace_filter

* hrtc -> hrt

* Fix CallTraces

* wip

* wip

* Fix for incorrect gas

* fix

* Fix Tracer

* Add block and uncle coinbases to trace index

* rewind coinbase

* Commit every 100k blocks after block 3m

* Simplify

* Safe unwinds

* Safe unwind 2

* Cleanup

* Simplification

* Remove intermediate JSON transform

* Reinstate optional CallIndex stage

* Create rpctest bench for trace_filter

* Fix trace_filter generator

* Restore unwind order

* Ignore the storage mode

* Ignore the storage mode

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-04 07:23:54 +01:00
Alex Sharov
4b7873708a
clean (#1870) 2021-05-04 13:21:51 +07:00
Alex Sharov
fc196d4b9b
pending txs methods (#1869) 2021-05-04 06:51:28 +01:00
Alex Sharov
f3d2af1e33
Rpcdaemon: move tx pool to own grpc service, subscribe to pending txs (#1863) 2021-05-04 08:37:17 +07:00
Alex Sharov
f230ba1da4
Less use rawdb deprecated methods (#1861)
* less use deprecated methods

* less use deprecated methods

* auto Dp limit

* auto Dp limit

* add test
2021-05-03 21:01:01 +01:00