mirror/erigon-pulse
1
0
Fork 0
mirror of https://gitlab.com/pulsechaincom/erigon-pulse.git synced 2024-12-22 03:30:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
20,764 Commits 5 Branches 14 Tags 294 MiB
master
Commit Graph

876 Commits

Author SHA1 Message Date
Giulio Rebuffo
c05e3a9fa1 connected to daemon 2021-11-22 11:36:52 +01:00
Giulio Rebuffo
21cbc19f21 using batch over tx 2021-11-21 23:11:50 +01:00
Giulio Rebuffo
1267d540b1 added engine_executePayloadV1 2021-11-21 22:47:33 +01:00
Giulio Rebuffo
23883d9104 merge 2021-11-21 21:55:41 +01:00
Giulio Rebuffo
4eb650fb0c payload manipulation 2021-11-21 21:52:36 +01:00
Alex Sharov
e55256296b
snapshots: read block from snapshots, add sender to txs file (#2996) 2021-11-21 10:32:14 +07:00
Giulio Rebuffo
6dd2d67121 Added ExecutePayloadV1 method to engine_ namespace 2021-11-20 22:53:03 +01:00
Giulio Rebuffo
224441cb07 initial commit2 2021-11-20 17:27:11 +01:00
Alex Sharov
aad0d0c777
grpc GetBlock api (#2955) 2021-11-14 11:08:52 +07:00
Alex Sharov
c7db8a2e8d
prohibit change prune mode (#2907) 2021-11-03 18:05:02 +07:00
Alex Sharov
da00e949c4
RemoteRPC: coherentCache for kv.Code; LocalRPC: enable small blocksLRU (#2815) 2021-10-12 12:04:04 +07:00
Alex Sharov
5ca558c667
Move lengths to erigon-lib other packages depend on it (#2799) 
* move lengths to erigon-lib

* move lengths to erigon-lib
 2021-10-08 10:20:45 +07:00
Andrea Lanfranchi
3bc3be4b5f
Clean up DEBUG category logs (#2776) 
- move many DEBUG logs into TRACE category
 2021-10-05 08:14:04 +07:00
Alexandr Borodulin
d7159cd8bd
More convenient pruning for non-PoW consensus: add --prune.*.before flags (#2714) 2021-09-23 09:13:19 +07:00
Alex Sharov
3de50637cd
remove changeset.Walk func (#2716) 2021-09-22 07:54:29 +07:00
Alex Sharov
133eec0c7e
Pool: correct new pending txs notifications (#2715) 2021-09-21 17:35:28 +07:00
Alex Sharov
b30ccd2ddf
Revert "Add prune before parameters" (#2701) 
This reverts commit 49a98ad4b6.
 2021-09-18 08:37:39 +07:00
Alexandr Borodulin
49a98ad4b6
Add prune before parameters 2021-09-17 19:01:15 +07:00
Alex Sharov
400c71b7ce
Pool: txID-based cache (#2697) 2021-09-17 10:31:20 +07:00
Alex Sharov
ad1682cf40
Expose mdbx's txID to remote_kv (#2686) 2021-09-15 14:22:57 +07:00
Alex Sharov
77db993cfc
Pool: add coherent cache (#2678) 2021-09-13 14:58:25 +07:00
ledgerwatch
15b4095718
Move ETL to erigon-lib (#2667) 
* Move ETL to erigon-lib

* Update link in the readme

* go mod tidy

* Use common/chan.go from erigon-lib

* Clean up

* Fix lint

* Fix test

* Fix compilation

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-09-12 08:50:17 +01:00
Alex Sharov
28480a36ec
Pool: add --txpool.v2 flag to rpcdaemon (#2624) 2021-09-03 11:19:35 +07:00
Alex Sharov
4c1dbdf50e
Pool: add --txpool.v2 flag to erigon (it just disabling existing machinery for now) (#2620) 2021-09-02 17:04:30 +07:00
Alex Sharov
953f348d1e
Pool: add grpc server (#2615) 2021-09-02 12:55:04 +07:00
Alex Sharov
f3021dd2af
RPC: small optimizations (#2564) 2021-08-24 08:28:58 +01:00
Alex Sharov
6c7edb0d34
state diff send to txpool (#2532) 2021-08-17 15:52:55 +07:00
Evgeny Danilenko
4cd72c8328
Keep readonly value while changing interpreters back and forth (#2508) 
* restore TEVM

* fuzzing and property based

* comment

* lint

* stack callback into defer ater checking an error

* sequential tests
 2021-08-10 09:48:56 +07:00
Alex Sharov
d617fca739
remove devp2p cmd - if we will support it then need do it in another repo (because it depends on account manager). (#2496) 
* up cobra lib

* remove devp2p cmd - if we will support it then need do it in another repo (because it depends on account manager).
 2021-08-08 09:17:08 +07:00
Alex Sharov
6e24a83065
extract kv interface from remoteKv client (#2494) 2021-08-07 12:03:12 +07:00
Richard Patel
7767b4df88
rpcdaemon: add txpool_status() (#2407) 
perfecto
 2021-08-06 09:45:44 +07:00
Alex Sharov
3ed0f5c83d
better prune error message (#2475) 2021-07-31 12:24:20 +07:00
Alex Sharov
6bd44eb26c
move kv to erigon-lib (#2467) 2021-07-29 18:53:13 +07:00
Alex Sharov
69dc2776dc
MIT licensed metrics lib (#2462) 2021-07-29 17:27:46 +07:00
Alex Sharov
5069558752
Apache licensed logger (#2460) 2021-07-29 17:23:23 +07:00
Alex Sharov
21cb7befa4
finish remove bucket suffix (#2458) 2021-07-28 10:43:51 +07:00
Alex Sharov
838e5f9ef2
Move bucket constants into kv package, move kv interface to kv package (#2455) 2021-07-28 09:47:38 +07:00
Alex Sharov
63a6c45ef1
Remove some db interfaces (#2452) 2021-07-27 19:35:32 +07:00
Alex Sharov
6bf54c951a
Sokol v0: unwind support (#2439) 2021-07-24 16:50:42 +07:00
Alex Sharov
9398e2ff93
Breaking change: Docker to follow XDG spec (#2437) 
change default dir inside docker from /var/lib/erigon to /home/erigon/.local/share/erigon
 2021-07-24 16:14:46 +07:00
Alex Sharov
6801208497
move some tests to kv interface (#2438) 2021-07-24 14:14:11 +07:00
alex.sharov
b56e500aab remove debug 2021-07-24 12:26:36 +07:00
Alex Sharov
a36a613e61
move migrations to kv interfaces (#2430) 2021-07-24 11:28:05 +07:00
Alex Sharov
f735997fa1
Sokol v0: remove unused parameters and methods (#2435) 2021-07-24 11:04:07 +07:00
b00ris
0c5f3df8a0
Temp db for state snapshot generation (#2416) 
* save state

* tmp db integration

* fmt

* temp db for state snapshot

* fmt
 2021-07-23 13:06:37 +01:00
b00ris
7f81f9858b
proper handling of deleted values for snapshot (#2387) 2021-07-21 16:36:09 +03:00
Alex Sharov
c3e1cfdac8
Pruning for: exec, log_index, tx_lookup, history stages (#2399) 
* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* Pruning for: exec, log_index, tx_lookup, history stages

* add tvm flag

* save

* db migration for storage mode
add flag --prune=
remove flag --storage-mode=
add flag --experiments=tevm,...
rename integration set_storage_mode to set_prune

* fix

* forward move of stages must skip everything before PruneTo

* keep in db progress of prune method

* keep in db progress of prune method

* simplify logs

* simplify logs

* simplify logs

* fix test

* simplify logs

* simplify logs

* simplify logs

* simplify logs

* remove callTraceSet as dupsort
use etl transform for txlookup prune
remove some logs

* cleanup tests a bit

* print_stages and eth_sync to show prune progress

* fix print_stages

* add readme about --prune.to flag

* more docs

* add --prune.history.older and other flags support

* fix migration on empty db

* better toString

* better toString
 2021-07-20 21:03:19 +01:00
Alex Sharov
031b0b177e
remove mutexes use (#2406) 2021-07-20 15:34:11 +07:00
Alex Sharov
a79b3cd6eb
unify stage params (#2384) 2021-07-17 09:09:56 +07:00
Alex Sharov
9f6ef74adc
Linter: to check tx.Rollback() by ruleguard (#2383) 2021-07-16 20:23:54 +07:00
alex.sharov
bc65759c84 clien id in rpc 2021-07-15 16:11:39 +07:00
Alex Sharov
c9e4e0bd3d
broadcast local txs (#2354) 2021-07-15 11:17:48 +07:00
Alex Sharov
04465e04df
increase read transactions limit to 32K (#2366) 2021-07-14 21:48:20 +07:00
Alex Sharov
42b776ebb3
increase max msg limit for grpc (#2361) 2021-07-14 09:43:58 +07:00
Alex Sharov
b6b42930ad
simpify db logger (#2359) 
* logger simplify db

* logger simplify db
 2021-07-13 20:15:37 +07:00
Andrea Lanfranchi
f909be7157
Update kv_interface.go (#2338) 
Amend comment
 2021-07-10 18:15:31 +07:00
Alex Sharov
5ceaac53f6
batch cancel (#2314) 2021-07-08 19:40:49 +07:00
Alex Sharov
7c60cc0fee
step towards kv interface (#2313) 2021-07-08 09:56:09 +07:00
Alex Sharov
a78cb54132
step towards kv interface (#2311) 2021-07-07 23:15:49 +07:00
b00ris
57473175ff
Body snapshot (#2100) 
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* save state

* body snapshot test

* unique tx

* body snapshot generation using walk

* move methods out of test

* block data verification added

* fix lint

* test with remove works correctly

* fix lint

* remove experiment test

* fix test

* add comment

* add second layer of remove test

* rename test

* fix typos

* fix lint

* revert testdata

* body snapshot migration save state

* fix body snapshot migration

* fix after merge

* remove debug test

* debug windows build

* fix build

* fix

* fix lint

* debug

* fix

* fix windows build

* simplify snapshot management&&get rid of lazy tx

* fix lint

* fix windows path

* debug

* debug

* debug

* debug

* remove geometry experiments

* skip windows tests

* clean

* fix

* fix ;int
 2021-07-06 23:33:26 +01:00
Alex Sharov
ea5abbe0b5
mdbx: add trace info to open error (#2305) 
* add stack info to open error

* add stack info to open error
 2021-07-05 19:53:11 +01:00
Alex Sharov
c029bbe7fa
fix metrics (#2292) 2021-07-04 10:25:07 +07:00
ledgerwatch
a05d1be531
Pull grpc bindings into erigon-lib (#2265) 
* Pull grpc bindings into erigon-lib

* Remove mdbx dist

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-07-01 22:31:14 +01:00
Alex Sharov
3d6d45a82f
remove subtrie loader and dcmp (#2258) 2021-06-30 10:31:36 +07:00
Alex Sharov
8f6aa2f7ae
propagate new tx to p2p (#2254) 2021-06-29 17:00:22 +07:00
Alex Sharov
a6f4b08716
Revert "Propagate new txs to P2P (#2249)" (#2253) 
This reverts commit e9ae6d688f.
 2021-06-29 15:46:00 +07:00
Alex Sharov
e9ae6d688f
Propagate new txs to P2P (#2249) 2021-06-29 14:40:23 +07:00
Alex Sharov
2b59b45216
Do log tables size, db metrics - avoid concurrency, check stale readers hourly (#2216) 
* add table size logs and metrics

* log stale readers

* - don't collect db metrics in background
- do check stale readers once an hour

* execution to update metrics when print logs

* hide file exists err

* hide file exists err
 2021-06-22 11:08:47 +01:00
Artem Vorotnikov
72e9a34365
Remove unused tables, soft rename, take 3 (#2207) 
* Remove unused tables, soft rename, take 3

* Bump db schema version

* nil check in migration
 2021-06-20 20:46:57 +01:00
Artem Vorotnikov
504c31e97a
Revert "Remove unused tables, soft rename" (#2205) 2021-06-20 15:08:48 +01:00
Alex Sharov
a535e8f30c
switch to mdbx-go module, and db-tools to libmdbx submodule (#2199) 
* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule
 2021-06-19 21:30:12 +01:00
Artem Vorotnikov
a9baf37ea1
Remove unused tables, soft rename (#2201) 2021-06-19 21:36:21 +03:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
mempirate
ca926ce94d
net_peerCount for remote sentries (#2192) 
* net_peerCount for remote sentries

* cleanup

* Simplify NetPeerCount using SentryClient

* Refactor id -> count
 2021-06-18 14:34:15 +01:00
Alex Sharov
e9f510c2b7
erigon as lib (#2186) 2021-06-18 10:35:11 +07:00
mempirate
e51b09da0d
Implements net_peerCount RPC call for internal sentries (#2172) 
* Implement API and protobuf

* Squashed 'interfaces/' content from commit cd3eca28e

git-subtree-dir: interfaces
git-subtree-split: cd3eca28e50367af9de52fca863b1d7528bff774

* Implement net_peerCount RPC call

* Squashed 'interfaces/' content from commit cd3eca28e

git-subtree-dir: interfaces
git-subtree-split: cd3eca28e50367af9de52fca863b1d7528bff774

* Fixing merge conflicts

* Remove p2p.Server from Ethereum backend

* Fix peerCount calculation for internal sentries

* Bump eth backend API version

* Update rpcdaemon README

* Document version change
 2021-06-17 22:55:20 +01:00
Artem Vorotnikov
af836a6200
Nuke LMDB (#2167) 2021-06-16 13:57:58 +03:00
Alex Sharov
dcf850dd75
fix fieldalignment linter warnings (#2171) 2021-06-16 11:00:35 +07:00
Alex Sharov
b79823d1b9
create dir for node keyfile (#2161) 2021-06-14 07:35:22 +01:00
Alex Sharov
e7b56358d3
Mdbx reproducible build (#2145) 
* save

* save

* save
 2021-06-11 14:04:12 +01:00
Alex Sharov
eedabfca85
txpool_content (#2127) 
* txpool_content

* txpool_content

* txpool_content

* txpool_content

* Squashed 'interfaces/' content from commit 1e916d013

git-subtree-dir: interfaces
git-subtree-split: 1e916d0137aff7bd9e9356cc203cea5f0ea9c7f1

* add test

* add test

* add test

* merge devel

* save
 2021-06-11 09:34:47 +01:00
Alex Sharov
ba56e7a139
extract p2p config out of sentry (#2126) 
* extract p2p config out of sentry

* extract p2p config out of sentry

* extract p2p config out of sentry

* rebase devel

* rebase devel

* add bootnodes to sentry

* add bootnodes to sentry
 2021-06-11 09:34:37 +01:00
Evgeny Danilenko
de26634d8d
TEVM status pruning (#2034) 
* use trace for TEVM status

* restore testdata

* fix getting contract from plain state

* unwind

* linters

* reproducible build debug

* debug

* debug

* remove debug
 2021-06-11 09:34:25 +01:00
ledgerwatch
73bb98f686
Move tests 6 (#2107) 
* Move tests 6

* Compile fixes

* Fix lint

* Compile fixes

* Fix compile

* compile fixes

* Compile fix

* Fix consesus/clique

* Cleanup

* Add gas limit

* Print

* Prints

* More print

* Fix

* Reinstate TestGetBlockReceipts66

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-06-06 22:44:14 +01:00
Alex Sharov
5ba3ea162a
Simulated backend and genesis to kv (#2104) 
* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* merge

* merge

* merge

* merge

* merge

* merge

* merge

* merge
 2021-06-05 16:17:04 +01:00
Alex Sharov
eee871aedf
Add db label (later will add txpool db) (#2081) 
* add db label - later will add txpool db

* move commit marker to kv_mdbx behind Chain label

* Update migrations.go

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
 2021-06-04 15:56:49 +01:00
Alex Sharov
ccac3c6566
Add tx.Append method for ergonomic (#2088) 
* close cursor

* stateless append

* stateless append

* faster pruning del and close cursor

* faster pruning del and close cursor

* less wrappers

* less wrappers

* less wrappers

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
 2021-06-04 14:32:48 +01:00
b00ris
66c7c669fb
Snapshot builder simplification (#2064) 
* fix test

* get rid of ObjectDatabase

* sn_builder_prototype2

* save state

* save state

* integration step1

* fix lint

* fix

* fix test

* integrate migrator.finish

* fix lint

* fix build

* fix typo

* fix
 2021-06-04 13:30:02 +01:00
Alex Sharov
52d35930e8
simplify receipts reading (#2094) 
* simplify receipts reading

* test
 2021-06-04 13:28:18 +01:00
Alex Sharov
bebaebb13a
mdbx 0.10.1 (#2083) 
* mdbx 0.10.1

* remove -tags mdbx

* remove -tags mdbx

* remove -tags mdbx
 2021-06-04 13:26:15 +01:00
ledgerwatch
43915a73a8
Migration to fix trace_filter (#2095) 
* Migration to fix trace_filter

* Fix to db/tx

* Fix to db/tx

* Error fixes

* Bump KV version

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-06-04 12:45:06 +01:00
ledgerwatch
67d0a2d3ab
Reproduce and fix trace_filter index problem (#2084) 
* Intermediate

* Test for trace_filter

* Clean up

* Clean up

* clean up

* More to the test

* Hack

* Block parameter

* Block parameter

* use mdbx

* Print number of records

* Print keys

* Print index records

* Print block numbers

* Print values in tracecallset

* Print some more

* More printing

* Print froms and tos

* Print startBlock and endBlock

* Tests and fixes

* Cleanup

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-06-04 09:09:57 +01:00
alex.sharov
c9b2d71333 add config2.h to gitignore 2021-06-03 14:17:42 +07:00
Alex Sharov
29fa1aa35a
sentry join streams (#2059) 2021-06-01 17:41:10 +07:00
Alex Sharov
017fb69da1
Fix GC chart (#2071) 
* save

* save
 2021-06-01 16:53:28 +07:00
Alex Sharov
a4ff299afb
Change set docs (#2062) 2021-05-31 15:29:46 +07:00
Alex Sharov
3016b96e88
1 Sentry per protocol (#2046) 2021-05-30 09:53:30 +07:00
Andrea Lanfranchi
cdcf9a10be
Mdbx wsl2 (#2048) 
* wsl2

* Handle PS crash for garbage keys

* More strict error checking

* Update readme for reintroduced WSL support

* Wrong htop display only for LMDB

* Wrong link in Readme

* A further note on WSL IP address

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
 2021-05-29 14:16:53 +01:00
Alex Sharov
60e8a31fe3
genesis switch db (#2038) 
* genesis switch db

* genesis switch db
 2021-05-28 13:55:48 +01:00
alex.sharov
6b826174c1 use default dynamic mdbx dpLimit 2021-05-28 14:08:24 +07:00
Evgeny Danilenko
7352b81122
Tevm stage 1 (#1845) 
* convert contracts after block execution

* check if has tevm code

* after review-1

* handle ErrNotFound

* typo

* tests

* tevm code bucket

* testdata

* execute pre-stage

* after merge

* test fix

* test fix

* fix test after merge

* disable translation stage

* after merge

* rename params

* rename to Erigon

* parallelize EVM translation

* fix

* logging and fixes

* fix

* todos

* cleanup

* revert erigorn renaming

* unwind

* tevm unwind

* fix AppData

* non-parallel version

* comments
 2021-05-27 14:54:55 +01:00
ledgerwatch
c15735d7a0
Make c in storage-mode default (#2021) 2021-05-26 16:00:43 +01:00
Alex Sharov
ed64380226
Up db schema for mdbx (for all) and up kv version: to 2.0.0 (#2019) 
* mdbx db schema version up

* Remove migration to simplify

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-26 12:51:44 +01:00
Evgeny Danilenko
74847d77e6
Rename to Erigon (#2018) 
* turbo-geth to erigon

* tg, turbo to erigon
 2021-05-26 11:35:39 +01:00
Alex Sharov
450c79e3ab
move integration to kv (#1996) 2021-05-23 12:41:42 +07:00
Artem Vorotnikov
5655409616
Do not write old block changesets in pruning storage mode (#1964) 
* Remove incremental staged sync

* Do not write old block changesets in pruning storage mode

* Remove pruning flag, put behaviour behind !StorageMode.History
 2021-05-22 11:57:10 +01:00
Alex Sharov
14c15cba43
Check version of remote services (#1989) 
* save

* save

* Squashed 'interfaces/' content from commit 08c32a09e

git-subtree-dir: interfaces
git-subtree-split: 08c32a09e40b1e6fcb5922e723191c9477545356

* Revert "Squashed 'interfaces/' content from commit 08c32a09e"

This reverts commit 8393d9fd

* save

* seve

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* ensure versions compatibility of all remote services

* Revert "Squashed 'interfaces/' content from commit dd6a42724"

This reverts commit 2a764bf9

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* Revert "Squashed 'interfaces/' content from commit dd6a42724"

This reverts commit 52621846

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* a

* a

* a

* a

* a

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-22 11:00:13 +01:00
ledgerwatch
729bfab988
Split DBSchema version depending on the database type (#1990) 
* Split DBSchema version depending on the database type

* Fix test

* Fix test

* Fix lint

* Update migrations.go

* Update bucket.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-22 10:20:43 +01:00
Alex Sharov
c4e20896fa
mdbx ensure lib version match (for .dll and .o compatibility) (#1988) 2021-05-22 11:18:04 +07:00
Alex Sharov
545fe32f68
Mdbx - make it default db. Lazy buckets renaming. (#1982) 2021-05-22 10:03:02 +07:00
Alex Sharov
4dcfd24af5
Fix mdbx_problem error (#1979) 2021-05-21 09:45:08 +07:00
Alex Sharov
0be3044b7e
rename (#1978) 
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
 2021-05-20 19:25:53 +01:00
Alex Sharov
215623da80
Windows support (#1944) 2021-05-20 23:46:12 +07:00
Alex Sharov
50934fa5d3
Move chain maiker to kv (#1972) 2021-05-20 18:49:33 +07:00
Alex Sharov
9d66aa0fe3
better support for mdbx in snapshots (#1971) 2021-05-20 18:12:07 +07:00
ledgerwatch
79084ede47
[download.v2] Make default (#1963) 
* [download.v2] Make default

* Update server.go

* Fix for tmp files appearing in the current dir

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
 2021-05-19 16:19:29 +01:00
Alex Sharov
369b41dfc2
clean (#1962) 2021-05-19 15:55:10 +07:00
Alex Sharov
ba185eff83
tx.Cursor() ignored error (maybe it's fix for MDBX_PROBLEM) (#1961) 2021-05-19 11:21:17 +07:00
Alex Sharov
781a67a4ee
Auto-Close db in tests by using t.Cleanup (#1956) 2021-05-19 10:47:28 +07:00
b00ris
777882ff52
Fix corner case for KV_Snapshot.Next (#1957) 
* fix next

* comment test
 2021-05-18 19:32:04 +01:00
Alex Sharov
cceaf75b16
use t.TempDir() (#1955) 2021-05-18 19:13:16 +07:00
Alex Sharov
0c91bfbf3e
RPCDaemon: support Pending block (#1942) 
* clean

* save

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* save

* tx pub-sub

* tx pub-sub

* clean

* clean

* save

* save

* save

* save

* save

* Squashed 'interfaces/' content from commit c469f3ae0

git-subtree-dir: interfaces
git-subtree-split: c469f3ae073b60c8821b61fed2910191080ef835

* save

* save

* save

* save

* Squashed 'interfaces/' changes from c469f3ae0..958dfc669

958dfc669 save

git-subtree-dir: interfaces
git-subtree-split: 958dfc669f8daeefe686a13aa852fb95f1537886

* save

* save

* up some deps

* up some deps

* clean

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test
 2021-05-17 13:15:19 +01:00
Alex Sharov
617baac56b
move db tests to kv interface (#1946) 2021-05-17 16:54:29 +07:00
Alex Sharov
9bef3fb656
move some tests to mdbx (#1947) 2021-05-17 16:14:08 +07:00
Alex Sharov
2ff6ce7d92
move staged sync to tx (#1897) 2021-05-14 18:00:20 +07:00
b00ris
48733c31cd
test fix (#1930) 2021-05-13 21:04:47 +01:00
Alex Sharov
f3a9eb340c
mdbx_setup_debug - handle response correct way (#1931) 2021-05-13 21:01:23 +07:00
b00ris
7948eafcc2
Property tests for kv snapshot (#1928) 
* model tests for snapshot kv

* cursor test

* first failing test

* fix

* fix lint

* remove fmt
 2021-05-13 09:40:32 +01:00
Alex Sharov
3e1bdbb465
Mdbx: fix for MDBX_PROBLEM (#1922) 2021-05-13 14:40:44 +07:00
Alex Sharov
0163329a5a
KV and RemoteKV: to reuse stateless cursors during transaction (#1909) 2021-05-11 15:23:17 +07:00
Artem Vorotnikov
80bbbf62c2
Misc fixes (#1908) 
* Fix debug levels

* try this

* remove adapter

* fix

* fix bodies stage
 2021-05-10 06:59:29 +01:00
Alex Sharov
3f36e362c2
mdbx switch to release (#1906) 2021-05-09 15:35:47 +07:00
ledgerwatch
8907e95d4a
Trace filter 3 (#1904) 
* Prevent the leak

* Default storage mode, fix leak again

* TopSeenHeight to be updated from any header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-05-08 23:07:16 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836) 
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-08 09:45:40 +01:00
Artem Vorotnikov
09788dbc02
Pruning flag, incremental staged sync (#1899) 2021-05-08 00:07:49 +03:00
ledgerwatch
59e5502744
Initial work on RPC streaming (#1880) 
* Initial work on RPC streaming

* Create the stream and pass it on

* trace_filter expressed as streamable (not finished)

* Reenable call trace index

* Add new line

* Remove storage mode override

* Tool for overriding storage mode

* Diagnostics

* Make trace_filter more streamy

* Streaming debug_traceTransaction and debug_traceCall

* Fix test

* Log error on stream flush

* Enable streaming for http

* Flush the stream too

* Make trace_filter flush too

* Stop streaming if client is not interested

* Try to quiet annoying test|

* Revert "Try to quiet annoying test|"

This reverts commit 42849257bfa52e90140aa535af34b957cd97a222.

* Debug log for test

* Proceed with handshake regardless of whether peer notification worked

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-06 18:37:38 +01:00
Alex Sharov
4323d56376
Mdbx tx limit metric (#1891) 2021-05-06 17:02:50 +07:00
Alex Sharov
49225f7319
Mdbx page ops metrics (#1885) 2021-05-06 10:51:49 +07:00
Alex Sharov
a627bc2d14
use same keepalive settings in sentry and rpcdaemon (#1883) 
* use same keepalive settings in sentry and rpcdaemon

* use same keepalive settings in sentry and rpcdaemon
 2021-05-05 11:32:04 +01:00
Alex Sharov
e87d4932f7
Changeset writer move to tx (#1871) 2021-05-05 14:31:40 +07:00
Alex Sharov
61c20798db
fix concurrent map write (#1882) 2021-05-05 10:16:57 +07:00
ledgerwatch
7c60ec3ec0
Fix for eth_getTransaction... returning tx from the pool (#1881) 
* Fix for eth_getTransaction... returning tx from the pool

* Fix

* Fix more issues

* Fix for EOF

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-04 22:58:13 +01:00
ledgerwatch
d604bfbb4f
Fix transaction stream test (#1879) 
* Fix transaction stream test

* Fix

* Update events.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-05-04 17:12:59 +01:00
Alex Sharov
c7eedf0091
Add kv to stage settings (because it doesn't change in runtime, tx will be passed as Stage func argument) (#1872) 
* add kv to config

* move forward tx use

* move forward tx use

* gc stat

* gc stat

* changeset writer - move to tx

* clean

* clean

* clean

* clean

* clean
 2021-05-04 13:36:03 +01:00
ledgerwatch
99578ec58d
Fix RPC daemon (#1873) 
* Fix RPC daemon

* Skip the test to fix RPC daemon

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-04 12:31:07 +01:00
Alex Sharov
66b683a6b3
RemoteKv version up (#1874) 
* clean

* clean

* clean
 2021-05-04 09:56:54 +01:00
ledgerwatch
793c13e20e
trace_filter and CallTrace Index (derivative of #1626) (#1867) 
* Trace Index

* trace_filter

* hrtc -> hrt

* Fix CallTraces

* wip

* wip

* Fix for incorrect gas

* fix

* Fix Tracer

* Add block and uncle coinbases to trace index

* rewind coinbase

* Commit every 100k blocks after block 3m

* Simplify

* Safe unwinds

* Safe unwind 2

* Cleanup

* Simplification

* Remove intermediate JSON transform

* Reinstate optional CallIndex stage

* Create rpctest bench for trace_filter

* Fix trace_filter generator

* Restore unwind order

* Ignore the storage mode

* Ignore the storage mode

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-05-04 07:23:54 +01:00
Alex Sharov
4b7873708a
clean (#1870) 2021-05-04 13:21:51 +07:00
Alex Sharov
fc196d4b9b
pending txs methods (#1869) 2021-05-04 06:51:28 +01:00
Alex Sharov
f3d2af1e33
Rpcdaemon: move tx pool to own grpc service, subscribe to pending txs (#1863) 2021-05-04 08:37:17 +07:00
Alex Sharov
f230ba1da4
Less use rawdb deprecated methods (#1861) 
* less use deprecated methods

* less use deprecated methods

* auto Dp limit

* auto Dp limit

* add test
 2021-05-03 21:01:01 +01:00
alex.sharov
8a74b65ba2 remove kv_remove prefix and prefetch 2021-05-02 10:53:19 +07:00
Artem Vorotnikov
2e2e3835aa
Remove MdbxCursor.prefix (#1857) 2021-05-02 10:49:00 +07:00
Alex Sharov
aacc457ea8
rawdb deprecated methods with Database interface (#1854) 2021-05-01 14:42:23 +07:00
Alex Sharov
e7849498d5
Degbug build support (for delve debugger and for getting C code profiling, traces) (#1843) 2021-04-29 21:29:58 +07:00
Alex Sharov
07a063cb8a
txn full fix - cursors leak (#1838) 2021-04-28 18:00:32 +01:00
ledgerwatch
92c20e1f67
[eip-1559] Apply gas limit elasticity (#1834) 
* Apply gas limit elasticity

* Fix unmarshalling tx

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-04-28 15:32:48 +01:00
Alex Sharov
768bdcd850
--database.verbosity flag (#1829) 
* --database.verbosity flag

* --database.verbosity flag
 2021-04-27 13:31:00 +01:00
Alex Sharov
9cc4a0a97c
rebort db metrics - not guilty (#1828) 2021-04-27 15:32:41 +07:00
ledgerwatch
e8161541e2
Use RLP encoding instead of json for subscriptions (#1825) 
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-04-27 09:23:04 +01:00
ledgerwatch
b341b5aed5
Fix for RPC daemon crashing TG when pending txs are transmitted (#1819) 
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-04-26 17:56:02 +01:00
ledgerwatch
117838e213
Move compatibility check (#1814) 
* Move compatibility check

* Implement Alex's suggestion

* Use root context

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-04-26 13:39:34 +01:00
Alex Sharov
544c597151
Pending txs part 2 (#1816) 2021-04-26 18:15:05 +07:00
Alex Sharov
4e80d6eeed
Pending txs subscription (#1810) 2021-04-26 16:53:38 +07:00
Alex Sharov
566233fee9
switch to master failed (#1812) 2021-04-26 15:40:49 +07:00
Alex Sharov
3462059ae8
mdbx rpcdaemon support, mdbx - switch to master branch (#1804) 2021-04-26 12:00:20 +07:00
Alex Sharov
d959986466
Connect TxFetcher to Sentry, Add --download.v2 option to TG (#1796) 2021-04-25 11:20:50 +07:00
ledgerwatch
e3f3dd3c9b
Integration tests 1 (#1793) 
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* Add architecture diagram source and picture (#10)

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Add proper golang packages, max_block into p2p sentry Status (#12)

* Add proper golang packages, max_block into p2p sentry Status

* Change EtherReply to address

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Add Rust infrastructure (#13)

* DB stats methods removed by https://github.com/ledgerwatch/turbo-geth/pull/1665

* more p2p methods (#15)

* add mining methods (#16)

* First draft of Consensus gRPC interface (#14)

* Update Rust build

* Fix interfaces in architecture diagram (#17)

* Fix KV interface provider

* Fix Consensus interface provider

* drop java attributes (#18)

* tx pool remove unused import (#19)

* ethbackend: add protocol version and client version (#20)

* Add missing ethbackend I/F (#21)

* Add interface versioning mechanism (#23)

Add versioning in KV interface

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>

* spec of tx pool method (#24)

* spec of tx pool method (#25)

* Update version.proto

* Refactor interface versioning

* Refactor interface versioning

* Testing interface

* Remove tree

* Fix

* Build testing protos

* Fix

* Fix

* Update to the newer interfaces

* Add ProtocolVersion and ClientVersion stubs

* Hook up ProtocolVersion and ClientVersion

* Remove service

* Add compatibility checks for RPC daemon

* Fix typos

* Properly update DB schema version

* Fix test

* Add test for KV compatibility|

* Info messages about compability for RPC daemon

* DB schema version to be one key

* Update release intructions

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: canepat <tullio.canepa@gmail.com>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
 2021-04-24 16:46:29 +01:00
ledgerwatch
12cde41772
Aleut support (Eip1559) (#1704) 
* Where I am at

* Refactoring of transaction types

* More refactoring

* Use Homested signer in rpc daemon

* Unified signer

* Continue unified signer

* A bit more

* Fixes and down the rabbit hole...

* More tx pool fixes

* More refactoring fixes

* More fixes'

* more fixes

* More fixes

* More compile fixes

* More RLP hand-writing

* Finish RLP encoding/decoding of transactions

* Fixes to header encoding, start on protocol packets

* Transaction decoding

* Use DecodeTransaction function

* Decoding BlockBodyPacket

* Encode and decode for pool txs

* Start fixing tests

* Introduce SigningHash

* Fixes to SignHash

* RLP encoding fixes

* Fixes for encoding/decoding

* More test fixes

* Fix more tests

* More test fixes

* More test fixes

* Fix core tests

* More fixes for signer

* Fix for tx

* Fixes to string encoding/size

* Fix eip2930 test

* Fix rest of ./tests

* More fixes

* Fix compilation

* More test fixes

* More test fixes

* Test fixes

* More fixes

* Reuse EncodingSize in EncodeRLP for accessList

* Rearrange things in dynamic fee tx

* Add MarshalBinary

* More fixes

* Make V,R,S non-pointers

* More NPE fixes

* More fixes

* Receipt fixes

* Fix core/types

* Fix ./eth

* More compile fixes for tests

* More test fixes

* More test fixes

* Try to see lint errors better

* Try to see lint errors better

* Fix lint

* Debugging eip1559 test

* Fix TestEIP1559Transition test

* Fix NewBlockPacket encoding/decoding

* Fix calculation of TxHash

* Fix perf problem with senders

* Update aleut config values

* Try adding static peers

* Add staticpeers to defaul flags

* Change aleut networkID

* Fix test

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-04-22 18:11:37 +01:00
Alex Sharov
e7a78a2d9f
Mdbx: speedup tests (#1776) 2021-04-21 20:03:08 +07:00
Alex Sharov
41fa29d421
Mining fixes (#1771) 2021-04-21 12:01:25 +07:00
Alex Sharov
bf1c219531
Move txPool.Get method behind direct-grpc (#1760) 2021-04-20 21:41:46 +07:00
Alex Sharov
a95bc0e34d
Mdbx devel up 3 (#1763) 2021-04-20 16:01:28 +07:00
Alex Sharov
00f700675b
add --datadir parameter to integration, snapshot generator, header downloader (#1749) 2021-04-19 14:25:26 +07:00
Alex Sharov
22b9522112
RPCDaemon: add --datadir flag, use --chaindata only if wanna redefine default chaindata folder (#1747) 2021-04-19 12:44:14 +07:00
Alex Sharov
07913a4d96
Mdbx force madv random (#1733) 2021-04-16 08:12:45 +07:00
Alex Sharov
5703059f29
[merge after release] Move history stage to rwtx (#1727) 2021-04-15 17:06:30 +07:00
Alex Sharov
acf1118300
MDBX: enable MADV_RANDOM (#1712) 2021-04-12 21:30:19 +07:00
Alex Sharov
2a3dca2fdb
mdbx devel version up (#1689) 2021-04-08 09:45:15 +07:00
Alex Sharov
cc539ce266
Remove blockchain.InsertChain method (#1681) 2021-04-06 12:06:07 +07:00
Artem Vorotnikov
b71c6323a0
KVGetter, replace old geth interfaces with it (#1676) 2021-04-05 20:04:58 +07:00
Artem Vorotnikov
70181c4af8
Mutation: implement StatelessRwTx (#1670) 2021-04-05 13:00:35 +07:00
Artem Vorotnikov
19924040bc
KV: Unified Has (#1671) 2021-04-03 19:04:58 +07:00
Artem Vorotnikov
d66555268d
Harmonize DB APIs (#1669) 2021-04-03 09:26:00 +03:00
Artem Vorotnikov
07b1959d91
Stateless Tx interface (#1667) 2021-04-03 13:14:08 +07:00
Artem Vorotnikov
e0a2d47139
Remove ancients and some old skipped tests (#1668) 2021-04-03 13:08:35 +07:00
Alex Sharov
ad342b27ab
Mdb no db size log (#1665) 2021-04-03 10:30:28 +07:00
Alex Sharov
9a9c5b298a
bitmapdb to use rwtx (#1661) 2021-04-03 08:52:45 +07:00
Alex Sharov
6179933872
more (#1658) 2021-04-02 16:15:41 +07:00
Alex Sharov
ef77eb3053
tx.RwCursor() to return err (first step of removing lazy cursors) (#1656) 2021-04-02 13:36:49 +07:00
Alex Sharov
ea7425adea
tests (#1631) 2021-03-31 08:26:01 +07:00
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface (#1627) 2021-03-30 12:53:54 +03:00
Alex Sharov
86ccfd0338
RPC: Pending transactions/blocks/logs (#1625) 2021-03-30 14:09:00 +07:00
Artem Vorotnikov
9b8cdc0f22
Fix lints and remove more unused code (#1621) 2021-03-29 10:58:45 +07:00
Alex Sharov
416a69ecb3
Genesis.toBlock remove db from arguments (#1611) 2021-03-28 16:27:06 +07:00
Artem Vorotnikov
e67be2b1be
Remove unused code (#1604) 
* Remove graphql, legacy options

* Remove CODEOWNERS

* Remove EWASM and custom EVM interpreter
 2021-03-26 09:08:01 +07:00
alex.sharov
d39c7fa121 mdbx: switch to master branch 2021-03-26 07:18:35 +07:00
Alex Sharov
8e385d9648
Mining: use noop writer when applyTx, pending logs subscription (#1600) 2021-03-25 13:42:45 +07:00
Artem Vorotnikov
7968637d42
ObjectDatabase cleanup (#1598) 
* ObjectDatabase cleanup

* Remove ObjectDatabase::WarmUp
 2021-03-24 22:48:08 +00:00
Alex Sharov
aff859edc0
Mining stage (#1554) 2021-03-23 16:00:07 +07:00
Alex Sharov
47c6bdce9d
RPCDaemon: increase throughput by allowing more parallel grpc streams and not lockthread for read tx (#1580) 2021-03-23 14:28:04 +07:00
Alex Sharov
4c82b15c36
no_panic_be_happy (#1586) 2021-03-22 22:45:24 +07:00
Alex Sharov
91e4126ca7
db commit - remove unused return param (#1584) 2021-03-22 19:41:52 +07:00
Alex Sharov
40dd00fee7
remove ideal batch size (#1582) 2021-03-22 13:47:01 +07:00
Alex Sharov
f7c122417c
MDBX bindings: remove finalizers (write tx require to be closed from same thread) (#1579) 2021-03-21 20:16:04 +07:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions (#1577) 2021-03-21 20:15:25 +07:00
alex.sharov
b49717e48f revert 2021-03-21 17:53:50 +07:00
Artem Vorotnikov
c9bccbb612
Remove autogen from typedcursor (#1576) 2021-03-21 15:10:14 +07:00
Artem Vorotnikov
72ff17e0b7
Hide TxDb (#1575) 2021-03-21 12:20:44 +07:00
Alex Sharov
ce868009d6
devel up (#1574) 2021-03-21 10:02:15 +07:00
Alex Sharov
97eec4288a
Tests: use transaction instead of db.MemCopy in chain_makers.go (#1570) 
* mdbx tests speedup

* mdbx tests speedup

* devel up

* devel up

* devel up

* lint

* less db copy

* don't loose error in test

* clean
 2021-03-20 19:33:40 +00:00
Artem Vorotnikov
b5f9cdd7b2
KV: split Sequence into ReadSequence and MakeSequence (#1569) 2021-03-20 21:12:54 +07:00
Andrew Ashikhmin
e303aeb5ee
Simple test for layout of TrieOfAccountsBucket (#1563) 2021-03-20 16:38:45 +07:00
Alex Sharov
0953c11c04
grpc version up (#1568) 2021-03-20 16:35:13 +07:00
Artem Vorotnikov
01f6542729
KV: Remove reserve, putcurrent, cmp, dcmp (#1552) 2021-03-20 16:35:02 +07:00
Alex Sharov
ba37706681
move remote eth backend from ethdb to core package (#1567) 2021-03-20 10:52:00 +07:00
ledgerwatch
81ea5bab78
More updates to downloader, new p2psentry protocol (#1559) 
* Initial commit

* Add sentry gRPC interface

* p2psentry directory

* Update README.md

* Update README.md

* Update README.md

* Add go package

* Correct syntax

* add external downloader interface (#2)

* Add txpool (#3)

* Add private API (#4)

* Invert control.proto, add PeerMinBlock, Separare incoming Tx message into a separate stream (#5)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Separate upload messages into its own stream (#6)

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>

* Only send changed accounts to listeners (#7)

* Txpool interface doc (#9)

* More additions

* More additions

* Fix locking

* Intermediate

* Fix separation of phases

* Intermediate

* Fix test

* More transformations

* New simplified way of downloading headers

* Fix hard-coded header sync

* Fixed syncing near the tip of the chain

* Add architecture diagram source and picture (#10)

* More fixes

* rename tip to link

* Use preverified hashes instead of preverified headers

* Fix preverified hashes generation

* more parametrisation

* Continue parametrisation

* Fix grpc data limit, interruption of headers stage

* Add ropsten preverified hashes

* Typed hashes (#11)

* Typed hashes

* Fix PeerId

* 64-bit tx nonce

* Disable penalties

* Add goerli settings, bootstrap nodes

* Try to fix goerly sync

* Remove interfaces

* Add proper golang packages, max_block into p2p sentry Status

* Prepare for proto overhaul

* Squashed 'interfaces/' content from commit ce36053c2

git-subtree-dir: interfaces
git-subtree-split: ce36053c24db2f56e48ac752808de60afa1dfb4b

* Change EtherReply to address

* Adaptations to new types

* Switch to new types

* Fixes

* Fix formatting

* Fix lint

* Lint fixes, reverse order in types

* Fix lint

* Fix lint

* Fix lint

* Fix test

* Not supporting eth/66 yet

* Fix shutdown

* Fix lint

* Fix lint

* Fix lint

* return stopped check

Co-authored-by: Artem Vorotnikov <artem@vorotnikov.me>
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: canepat <16927169+canepat@users.noreply.github.com>
 2021-03-19 21:24:49 +00:00
ledgerwatch
a583f7f6ca
Try to fix multiple head subscriptions (#1565) 
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-03-19 17:36:38 +00:00
b00ris
c03da8fb8e
Split header prefix bucket to headers, td and canonical (#1556) 
* split headers prefix

* migration

* fix downloader bug

* test for migration

* fix lint

* uncomment t.Prallel

* fix postprocessing test
 2021-03-19 12:54:47 +00:00
Artem Vorotnikov
0b2e4df0ff
Align Cursor API with LMDB/MDBX (#1558) 2021-03-19 14:45:01 +07:00
Igor Mandrigin
7e7ec69428 use lmdb for in-mem database and restore zk tests 2021-03-17 21:51:24 +01:00
Igor Mandrigin
0a55895b33 Merge branch 'master' into geth-1-10-1 2021-03-17 16:50:20 +01:00
Igor Mandrigin
fd37289665 fixes to core tests (compilation) 2021-03-16 12:44:38 +01:00
Alex Sharov
3ffa029497
Add mdbx_drop cli tool (#1553) 2021-03-15 09:41:02 +07:00
Alex Sharov
3f305673aa
Trie: add invariant - first level of trie must be in DB (to ensure having 100% trie structure in trie_account table) (#1522) 2021-03-13 09:30:54 +07:00
Alex Sharov
879112f347
TxDb: Walk method - to reuse cursors (#1548) 2021-03-12 16:27:04 +07:00
Alex Sharov
060f3b9a60
Grafana: add Btree metrics (#1547) 2021-03-10 10:58:57 +07:00
Alex Sharov
4fe019a099
add periodical db statistic metrics (#1545) 2021-03-09 13:34:13 +07:00
alex.sharov
5ae602e9a4 less metrics performance impact 2021-03-08 18:58:44 +07:00
Alex Sharov
cdc090ebfd
Delete MultiWalk and GetIndexChunk funcs from Database interface (#1544) 2021-03-08 12:08:37 +07:00
ledgerwatch
b743b88218
Revert "Extract validating interface (#1120)" (#1543) 
* Revert "Extract validating interface (#1120)"

This reverts commit e4e36c152e.

* Fix go mod tidy

* Fix lint

* Fix lint

* fix lint

* Test fix

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-03-05 20:34:23 +00:00
Alex Sharov
2bdf7fde30
mdbx: remove custom logger (#1536) 2021-03-03 17:35:49 +07:00
ledgerwatch
c00f85ab6c
Simplification of eth_estimageGas (#1531) 
* Simplification of eth_estimageGas

* Set GasCap by default

* Add a test

* Fix crash

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-03-02 22:47:44 +00:00
alex.sharov
61f8a1ef0c revive 2021-03-02 09:52:05 +07:00
Alex Sharov
d284f4ede0
clear bucket must not change DBI handle (#1526) 2021-03-01 11:15:59 +07:00
Alex Sharov
f556af6276
Prometheus metrics: more virtual memory visibility, more cross-platform metrics (#1520) 2021-02-28 11:11:28 +07:00
Evgeny Danilenko
e4e36c152e
Extract validating interface (#1120) 
* interface

* generalize interface

* linters

* fix deadlock

* fix linters

* close goroutine

* fix

* debug

* id+ttl

* refactor downloader tests

* tests

* lru

* handle genesis, extract fake consensuses

* fix fake consensus

* test uncles, verify

* after a new master

* fmt

* fix close

* debug

* debug

* fix chain length

* remove test field

* use single account

* fix data race on closing channel

* remove postponed blocks queue

* miner test

* VerifyHeaderRequests

* fmt

* fmt

* fix data race

* handle validating errors

* simplify matchParents

* remove copy-paste

* move sort to constructor

* clean up

* debug for 10 parents

* debug

* debug

* batch responses

* batch requests

* works for many ancestors

* remove debug

* always Close an engine

* linters

* ancestors deduplication

* fix test

* reduce interface

* api

* clique

* green clique sync

* stable

* perpermance per second

* full sync

* linters

* gitignore

* deps

* fix panic after master merge

* init consensus

* clique tests

* fix tests

* fix tests

* clean up

* reuse snap

* store vefified snapshots

* optimize snapshots

* safe close

* cleanup loop

* after downloader

* downloader and consensus tests

* update tests

* hack.go

* clique flags

* fix cliuqe config

* review

* gitignore

* remove additional bucket

* blk/sec instead of blk/microsecond

* rename

* deps

* optimize

* debug

* test

* tests without extracted validation process

* same base performance as on master

* benchmark

* simplify ethash verification

* ethash

* ethash

* linters

* ethash

* master stats

* cleanup

* gomod

* linters

* tests

* better locks

* Fix

* Remove logging for verifyHeaders

* Verification speed in the logs

* Fix compile error

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-02-25 19:40:45 +00:00
Alex Sharov
25393f46ea
mdbx: fix master (#1517) 2021-02-25 13:20:57 +07:00
Alex Sharov
303d697619
mdbx: same dirty list size as lmdb (#1515) 2021-02-25 10:00:51 +07:00
Alex Sharov
de1806c173
same options as lmdb (#1514) 2021-02-25 09:41:57 +07:00
Alex Sharov
6894d9b356
Remove cursor features (#1508) 2021-02-23 09:02:54 +07:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500) 
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
 2021-02-21 18:41:59 +00:00
Alex Sharov
101c7d78c0
compatibility fix (#1495) 2021-02-14 11:38:28 +07:00
ledgerwatch
6a1f000cf7
Bring kv interfaces (#1493) 
* Remove interfaces

* Squashed 'interfaces/' content from commit 0941b0992

git-subtree-dir: interfaces
git-subtree-split: 0941b09926db64934ba7dd161fb9ca2a20ba25b8

* Change generation accordingly

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
 2021-02-13 07:41:47 +00:00
lightclient
8777899573
support multiple new head subscribers (#1490) 2021-02-12 16:46:50 +00:00
alex.sharov
9b1ec50b97 increase mdbx OptRpAugmentLimit 2021-02-11 12:54:21 +07:00
Alex Sharov
d69f9bd4b2
revert mdbx bindings (#1488) 2021-02-11 11:53:14 +07:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Artem Vorotnikov
739c0d30a7
Remove DupFixed (#1482) 2021-02-10 16:56:44 +07:00
Artem Vorotnikov
123710adcd
ChangeSet: remove Walker::Walk (#1476) 2021-02-09 17:26:00 +07:00