Commit Graph

208 Commits

Author SHA1 Message Date
Alex Sharov
2b59b45216
Do log tables size, db metrics - avoid concurrency, check stale readers hourly (#2216)
* add table size logs and metrics

* log stale readers

* - don't collect db metrics in background
- do check stale readers once an hour

* execution to update metrics when print logs

* hide file exists err

* hide file exists err
2021-06-22 11:08:47 +01:00
Alex Sharov
a535e8f30c
switch to mdbx-go module, and db-tools to libmdbx submodule (#2199)
* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule

* move mdbx to submodule
2021-06-19 21:30:12 +01:00
Alex Sharov
4736bac190
save (#2197) 2021-06-19 21:29:02 +01:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
Alex Sharov
de24b427bf
save (#2196) 2021-06-19 13:32:35 +07:00
Alex Sharov
f63d12405a
Sender stage hangs (#2177)
* - add context for goroutines shutdown
- use 1 select for all channels to avoid deadlock

* - add context for goroutines shutdown
- use 1 select for all channels to avoid deadlock
- integration: senders to use right chainconfig
2021-06-16 14:04:44 +01:00
Artem Vorotnikov
af836a6200
Nuke LMDB (#2167) 2021-06-16 13:57:58 +03:00
Alex Sharov
372ab9060a
add chain support to (#2176) 2021-06-16 11:31:25 +07:00
Alex Sharov
925d17a238
remove lmdb from hack, integration, state packages. remove lmdb dbtools (#2146) 2021-06-11 20:31:37 +07:00
Evgeny Danilenko
de26634d8d
TEVM status pruning (#2034)
* use trace for TEVM status

* restore testdata

* fix getting contract from plain state

* unwind

* linters

* reproducible build debug

* debug

* debug

* remove debug
2021-06-11 09:34:25 +01:00
Alex Sharov
338fc0df26
move snapshot config to constructor (#2121) 2021-06-08 09:43:41 +07:00
Alex Sharov
40eaf77eb8
Sokol genesis and chainconfig (#2116)
* sokol init

* test

* precompiles

* precompiles

* add code

* sokol test

* save

* save

* save

* save

* save

* save
2021-06-07 12:01:54 +01:00
Alex Sharov
dea4378927
move (#2113) 2021-06-06 18:17:03 +07:00
Alex Sharov
35fd972a9e
move sync.Run() to kv (#2112)
* less objdb in tests

* move sync.Run to tx

* move

* move

* move
2021-06-06 10:32:55 +01:00
Artem Vorotnikov
231a130e2d
Remove remnants of silkworm integration (#2110) 2021-06-06 09:06:06 +01:00
Alex Sharov
5ba3ea162a
Simulated backend and genesis to kv (#2104)
* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* move to kv

* merge

* merge

* merge

* merge

* merge

* merge

* merge

* merge
2021-06-05 16:17:04 +01:00
ledgerwatch
f7fc4d66ec
Move tests 5 (#2069)
* Remove silkworm and reader builders

* Switch tests

* Stop from hanging

* Update tests to 9.0.0

* Move tests back

* Moving code around

* Fix lint

* More fixes

* Intermediate

* Fix compile

* Fix compile errors

* Compile errors

* Fix lint

* Comment out more printing

* More simplifications

* Compile fixes

* Compile fix

* More replacements

* More fixes

* More muddling through

* lint

* Exclude dao split test

* Restore sidechains after unwind

* Errors in senders stage, stopped in bodies stage

* Fix compile

* Fix compile

* Fix tests

* More fixes

* More fixes

* Code cleanup

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2021-06-05 11:00:21 +01:00
Alex Sharov
eee871aedf
Add db label (later will add txpool db) (#2081)
* add db label - later will add txpool db

* move commit marker to kv_mdbx behind Chain label

* Update migrations.go

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 15:56:49 +01:00
Alex Sharov
50dd17f36c
switch from baikal to calaveras (#2078) 2021-06-02 08:42:52 +01:00
Alex Sharov
e34d73f162
Dont exec all blocks in 1 tx (#2076)
* save

* save

* save

* save
2021-06-02 08:15:17 +01:00
ledgerwatch
26904a1851
Remove silkworm and reader builders (#2068)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-31 18:13:10 +01:00
Alex Sharov
a6dfafd5cd
rename headers to sentry (#2060) 2021-05-31 17:20:56 +07:00
Alex Sharov
3016b96e88
1 Sentry per protocol (#2046) 2021-05-30 09:53:30 +07:00
ledgerwatch
acc562e815
Test refactoring to move to new staged sync (#2047)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-29 12:46:49 +01:00
alex.sharov
e4ee6a1f5b allow lmdb_to_mdbx create buckets 2021-05-28 08:54:23 +07:00
Evgeny Danilenko
7352b81122
Tevm stage 1 (#1845)
* convert contracts after block execution

* check if has tevm code

* after review-1

* handle ErrNotFound

* typo

* tests

* tevm code bucket

* testdata

* execute pre-stage

* after merge

* test fix

* test fix

* fix test after merge

* disable translation stage

* after merge

* rename params

* rename to Erigon

* parallelize EVM translation

* fix

* logging and fixes

* fix

* todos

* cleanup

* revert erigorn renaming

* unwind

* tevm unwind

* fix AppData

* non-parallel version

* comments
2021-05-27 14:54:55 +01:00
Evgeny Danilenko
74847d77e6
Rename to Erigon (#2018)
* turbo-geth to erigon

* tg, turbo to erigon
2021-05-26 11:35:39 +01:00
Alex Sharov
e98c1d217b
Sentry mock: mine block with 1 tx (#2017) 2021-05-26 14:02:19 +07:00
ledgerwatch
03cb54a606
Activate refund reduction EIP, simplify code (#2016)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-25 23:26:25 +01:00
ledgerwatch
835b954f5c
State stream to RPC daemon (#1986)
* Remove interfaces

* Squashed 'interfaces/' content from commit e06631eb4

git-subtree-dir: interfaces
git-subtree-split: e06631eb4d0926c2d6a4f552497b920b4ed8d1bd

* Update KV interface

* Squashed 'interfaces/' changes from e06631eb4..014677ffe

014677ffe Merge remote-tracking branch 'origin/master' into stream
08c32a09e add version method to txPool and ethbackend
5b6bf70b9 Update README.md
7712cb267 Update README.md
f895ece4c save (#37)

git-subtree-dir: interfaces
git-subtree-split: 014677ffe5bff0dee1a333f06c92e6110d791468

* Fix old interfaces

* Squashed 'interfaces/' changes from 014677ffe..df31e1146

df31e1146 remove action from storage change

git-subtree-dir: interfaces
git-subtree-split: df31e1146c368eda2e2b15ab252b78fba7a0a6f3

* add Accumulator

* add location

* Squashed 'interfaces/' changes from df31e1146..472584639

472584639 Merge remote-tracking branch 'origin/master' into stream
dd6a42724 Refactor of consensus interface (#28)

git-subtree-dir: interfaces
git-subtree-split: 472584639f637189dfb906ef1ed03665f98d55d2

* Fix compilation in cons

* Pass accumulator to Execution stage

* Fix test

* Pass accumulator to unwind and plain writer

* Add accumulator use to plain writer

* Squashed 'interfaces/' changes from 472584639f..5c36f038b8

5c36f038b8 State change stream to KV (#38)
REVERT: 472584639f Merge remote-tracking branch 'origin/master' into stream
REVERT: df31e1146c remove action from storage change
REVERT: 014677ffe5 Merge remote-tracking branch 'origin/master' into stream
REVERT: e06631eb4d Fix
REVERT: 9c10d79d2d Fix
REVERT: 61ae9307de Fix
REVERT: 4fcf34ecc5 State change stream to KV

git-subtree-dir: interfaces
git-subtree-split: 5c36f038b87096ffb6b07e90c6762c21b864cd3b

* Add state.stream flag

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-23 15:53:18 +01:00
Alex Sharov
450c79e3ab
move integration to kv (#1996) 2021-05-23 12:41:42 +07:00
Artem Vorotnikov
5655409616
Do not write old block changesets in pruning storage mode (#1964)
* Remove incremental staged sync

* Do not write old block changesets in pruning storage mode

* Remove pruning flag, put behaviour behind !StorageMode.History
2021-05-22 11:57:10 +01:00
ledgerwatch
729bfab988
Split DBSchema version depending on the database type (#1990)
* Split DBSchema version depending on the database type

* Fix test

* Fix test

* Fix lint

* Update migrations.go

* Update bucket.go

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 10:20:43 +01:00
Alex Sharov
545fe32f68
Mdbx - make it default db. Lazy buckets renaming. (#1982) 2021-05-22 10:03:02 +07:00
Alex Sharov
1df1b85d64
move_integration_to_replacement_stages (#1980) 2021-05-21 12:21:06 +07:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
9d66aa0fe3
better support for mdbx in snapshots (#1971) 2021-05-20 18:12:07 +07:00
ledgerwatch
79084ede47
[download.v2] Make default (#1963)
* [download.v2] Make default

* Update server.go

* Fix for tmp files appearing in the current dir

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2021-05-19 16:19:29 +01:00
Alex Sharov
0c91bfbf3e
RPCDaemon: support Pending block (#1942)
* clean

* save

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* pubsub

* save

* tx pub-sub

* tx pub-sub

* clean

* clean

* save

* save

* save

* save

* save

* Squashed 'interfaces/' content from commit c469f3ae0

git-subtree-dir: interfaces
git-subtree-split: c469f3ae073b60c8821b61fed2910191080ef835

* save

* save

* save

* save

* Squashed 'interfaces/' changes from c469f3ae0..958dfc669

958dfc669 save

git-subtree-dir: interfaces
git-subtree-split: 958dfc669f8daeefe686a13aa852fb95f1537886

* save

* save

* up some deps

* up some deps

* clean

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test
2021-05-17 13:15:19 +01:00
Alex Sharov
2ff6ce7d92
move staged sync to tx (#1897) 2021-05-14 18:00:20 +07:00
Alex Sharov
47d22773b9
[wip] dv2: Validate body (#1913)
* validate body

* validate body

* save

* save

* body deliveries messages handling into staged sync

* body deliveries messages handling into staged sync

* body deliveries messages handling into staged sync

* body deliveries messages handling into staged sync

* body deliveries messages handling into staged sync

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch

* penalize right after verifyUncles and stop, don't batch
2021-05-12 14:57:35 +01:00
Alex Sharov
921c6a53c9
set default mdbx log level to warning (#1912) 2021-05-11 15:22:34 +07:00
ledgerwatch
a99d22c313
Trace filter 4 + fixes for download.v2 (#1907)
* Introduce TraceCallSet bucket and fill it up

* Fix compile

* Reset TraceCallSet

* Add database.verbosity

* Warn about dropped messages

* Unwind for call traces

* Fixes for headers download

* Fix for the slow stage 3

* Remove wakeup channels

* Permit system for peer load balancing

* Reduce aggressiveness

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-10 16:07:22 +01:00
ledgerwatch
5c512fbddb
Trace filter 2 (#1898)
* Fix error message about write protection

* Clean up of trace_filter

* Add CallTraces to Execution Stage

* Fix compile

* Say something

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 16:17:17 +01:00
ledgerwatch
4a214d38e4
Fix integration after snapshot PR (#1901)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-05-08 11:42:34 +01:00
b00ris
565a4250d6
Snapshot sync headers stage (#1836)
* save state

* snapshot update works

* save state

* snapshot migrator

* tx test

* save state

* migrations stages refactor

* refactor snapshot migrator

* compilation fixed

* integrate snapshot migrator

* goerli sync headers

* debug async snapshotter on goerly

* move verify headers, remove experiments, fix remove old snapshot

* save state

* refactor snapshotsync injection

* fix deadlock

* replace snapshot generation stage logic to migrate method

* change done for body snapshot

* clean

* clean&&change deleted value

* clean

* fix hash len

* fix hash len

* remove one of wrap methods, add remove snapshots on start

* add err check

* fix shadowing

* stages unwind order debug

* matryoshka experiments

* steam test

* fix build

* fix test

* fix lint

* fix test

* fix test datarace

* add get test

* return timeout

* fix mdbx overlap

* fix after merge

* change epoch size

* clean todo

* fix

* return testdata

* added return from sndownloader gorutine

* fix review comments

* Fix

* More info

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-08 09:45:40 +01:00
Alex Sharov
06cec6d535
Move exec to tx (#1887) 2021-05-07 09:06:44 +07:00
ledgerwatch
59e5502744
Initial work on RPC streaming (#1880)
* Initial work on RPC streaming

* Create the stream and pass it on

* trace_filter expressed as streamable (not finished)

* Reenable call trace index

* Add new line

* Remove storage mode override

* Tool for overriding storage mode

* Diagnostics

* Make trace_filter more streamy

* Streaming debug_traceTransaction and debug_traceCall

* Fix test

* Log error on stream flush

* Enable streaming for http

* Flush the stream too

* Make trace_filter flush too

* Stop streaming if client is not interested

* Try to quiet annoying test|

* Revert "Try to quiet annoying test|"

This reverts commit 42849257bfa52e90140aa535af34b957cd97a222.

* Debug log for test

* Proceed with handshake regardless of whether peer notification worked

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-06 18:37:38 +01:00
Alex Sharov
e87d4932f7
Changeset writer move to tx (#1871) 2021-05-05 14:31:40 +07:00
Alex Sharov
c7eedf0091
Add kv to stage settings (because it doesn't change in runtime, tx will be passed as Stage func argument) (#1872)
* add kv to config

* move forward tx use

* move forward tx use

* gc stat

* gc stat

* changeset writer - move to tx

* clean

* clean

* clean

* clean

* clean
2021-05-04 13:36:03 +01:00