Commit Graph

441 Commits

Author SHA1 Message Date
Alex Sharov
f556af6276
Prometheus metrics: more virtual memory visibility, more cross-platform metrics (#1520) 2021-02-28 11:11:28 +07:00
Evgeny Danilenko
e4e36c152e
Extract validating interface (#1120)
* interface

* generalize interface

* linters

* fix deadlock

* fix linters

* close goroutine

* fix

* debug

* id+ttl

* refactor downloader tests

* tests

* lru

* handle genesis, extract fake consensuses

* fix fake consensus

* test uncles, verify

* after a new master

* fmt

* fix close

* debug

* debug

* fix chain length

* remove test field

* use single account

* fix data race on closing channel

* remove postponed blocks queue

* miner test

* VerifyHeaderRequests

* fmt

* fmt

* fix data race

* handle validating errors

* simplify matchParents

* remove copy-paste

* move sort to constructor

* clean up

* debug for 10 parents

* debug

* debug

* batch responses

* batch requests

* works for many ancestors

* remove debug

* always Close an engine

* linters

* ancestors deduplication

* fix test

* reduce interface

* api

* clique

* green clique sync

* stable

* perpermance per second

* full sync

* linters

* gitignore

* deps

* fix panic after master merge

* init consensus

* clique tests

* fix tests

* fix tests

* clean up

* reuse snap

* store vefified snapshots

* optimize snapshots

* safe close

* cleanup loop

* after downloader

* downloader and consensus tests

* update tests

* hack.go

* clique flags

* fix cliuqe config

* review

* gitignore

* remove additional bucket

* blk/sec instead of blk/microsecond

* rename

* deps

* optimize

* debug

* test

* tests without extracted validation process

* same base performance as on master

* benchmark

* simplify ethash verification

* ethash

* ethash

* linters

* ethash

* master stats

* cleanup

* gomod

* linters

* tests

* better locks

* Fix

* Remove logging for verifyHeaders

* Verification speed in the logs

* Fix compile error

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-25 19:40:45 +00:00
Alex Sharov
25393f46ea
mdbx: fix master (#1517) 2021-02-25 13:20:57 +07:00
Alex Sharov
303d697619
mdbx: same dirty list size as lmdb (#1515) 2021-02-25 10:00:51 +07:00
Alex Sharov
de1806c173
same options as lmdb (#1514) 2021-02-25 09:41:57 +07:00
Alex Sharov
6894d9b356
Remove cursor features (#1508) 2021-02-23 09:02:54 +07:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500)
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
2021-02-21 18:41:59 +00:00
Alex Sharov
101c7d78c0
compatibility fix (#1495) 2021-02-14 11:38:28 +07:00
ledgerwatch
6a1f000cf7
Bring kv interfaces (#1493)
* Remove interfaces

* Squashed 'interfaces/' content from commit 0941b0992

git-subtree-dir: interfaces
git-subtree-split: 0941b09926db64934ba7dd161fb9ca2a20ba25b8

* Change generation accordingly

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-13 07:41:47 +00:00
lightclient
8777899573
support multiple new head subscribers (#1490) 2021-02-12 16:46:50 +00:00
alex.sharov
9b1ec50b97 increase mdbx OptRpAugmentLimit 2021-02-11 12:54:21 +07:00
Alex Sharov
d69f9bd4b2
revert mdbx bindings (#1488) 2021-02-11 11:53:14 +07:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Artem Vorotnikov
739c0d30a7
Remove DupFixed (#1482) 2021-02-10 16:56:44 +07:00
Artem Vorotnikov
123710adcd
ChangeSet: remove Walker::Walk (#1476) 2021-02-09 17:26:00 +07:00
alex.sharov
f88607c7f4 revert: enable lifo 2021-02-09 16:05:04 +07:00
alex.sharov
b9aae2bfcb enable lifo 2021-02-09 16:04:36 +07:00
Alex Sharov
7464b7924d
Print tables size during execution stage (#1479) 2021-02-09 12:02:00 +07:00
Alex Sharov
679d70fca8
Print tables size at end of cycle if table>10Gb, and print freelist size (#1478) 2021-02-09 11:42:10 +07:00
alex.sharov
fe193723f7 make lmdb default 2021-02-09 09:36:59 +07:00
Alex Sharov
5ef9a5a9f0
mdbx: switch CI to mdbx, fix SeekExact, print slow/big transactions info (#1475)
DEBUG_BIG_RO_TX_KB - print logs with info about large read-only transactions
DEBUG_BIG_RW_TX_KB - print logs with info about large read-write transactions
DEBUG_SLOW_COMMIT_MS - print logs with commit timing details if commit is slower than this threshold
2021-02-09 09:31:37 +07:00
Alex Sharov
47b4f52db2
mdbx: C11 atomics performance fix and disable MDBX_ENV_CHECKPID (#1472) 2021-02-08 11:35:02 +07:00
Alex Sharov
f351eea439
Mdbx increase reclaim limit (#1465)
* mdbx_increase_reclaim_limit

* readable
2021-02-05 14:11:42 +07:00
alex.sharov
25213a51b1 mdbx 0.9.3 2021-02-03 10:06:42 +07:00
Alex Sharov
f0bb3a6dfa
Mdbx new beta version with fixed blockers (#1461) 2021-02-01 20:57:41 +07:00
Alex Sharov
7fd1f19435
txdb lazy cursors creation, grpc version up (#1440)
* txdb lazy, grpc version up

* tidy

* fix lint
2021-01-15 09:38:09 +00:00
Alex Sharov
b2c60a6f56
increase concurrent streams cap (#1438) 2021-01-15 09:37:24 +00:00
Artem Vorotnikov
f7c5ac42d2
Remove unused fields from remoteCursor (#1437) 2021-01-09 12:29:04 +07:00
Alex Sharov
e6f1e43e6f
lmdb: move fsync inside commit (#1433) 2021-01-04 08:17:18 +00:00
b00ris
8db5790838
State snapshot sync (#1417)
* move experiments to new branch&reorganise kv_snapshot

* walk&modify tests

* added delete from snapshot tests

* fmt

* state snapshot debug

* snapshot validation passed. copy state snapshot

* debug

* snapshot cursor.Prev test

* Prev works correct. Added Current check

* add err check

* added walk forward and backward test

* before refactoring

* refactoring

* execution with snapshot debug

* fix

* remove useless test

* before dupcursor implimentation

* tests with prev and delete works

* execution based on state snapshot passed

* remove useless tests

* blocks to 1140000 passed

* clean verifier

* cleanup state generation

* clean verify && seeder

* remove debug code

* tests passed

* fix lint

* save state

* test passed

* fix lint

* add state hash

* fix lint
2021-01-02 19:28:37 +00:00
Alex Sharov
75cb938980
rpcdaemon performance improve - less reading blocks, less allocs on server (#1426)
* rpcdaemon performance improve - less reading blocks, less allocs on server

* rpcdaemon performance improve - less reading blocks, less allocs on server

* rpcdaemon performance improve - less reading blocks, less allocs on server

* rpcdaemon performance improve - less reading blocks, less allocs on server

* don't use global variables

* df

* avoid use non-transactional db instance to reduce amount of cursors
2021-01-02 19:28:22 +00:00
Alex Sharov
6538df5e08
fix_rpcdaemon_panic (#1428) 2020-12-30 00:34:29 +03:00
Alex Sharov
c3f812ae35
switch tests to lmdb (#1415) 2020-12-17 13:27:39 +07:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
ledgerwatch
0953fd42cb
WIP trace_call implementation (#1399)
* trace_call initial

* Fix tests

* More tracing

* Add more fields to the action

* Completed first example query

* Add initial bench11 to compare trace_call with OpenEthereum

* Exclude precompile calls from call traces

* Add self-destruct, call types, more comparison in rpctest

* Support for execution errors

* Stack underflow error and delegatecall value

* Fix lint

* Fix suicide traceAddress, Bad instruction error

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-14 11:27:52 +00:00
Alex Sharov
c514f9a214
subscription_doesnt_preserve_shutdown (#1391) 2020-12-04 21:17:13 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
ledgerwatch
8d2db43e2b
Fix for slow stage3 (#1377)
* Fix for slow stage3

* Cache sequence

* Small optimisation

* mutation sequence lock and sequence(0) use read transaction

* tx

* Fixes

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-11-30 11:49:52 +00:00
Alex Sharov
ed2b36ff04
log db size if stage took longer than 1Minute (#1378)
* log db size

* log db size
2020-11-30 08:58:06 +00:00
Andrew Ashikhmin
8a877c227a
Integrate Silkworm execution (#1344)
* Expose C Handle of lmdbTx

* LoadExecutionFunctionPointer

* silkworm ExecuteBlocks

* linter

* CLI flag for SilkwormExecutionFunc

* linter

* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage

* Commit transaction after Silkworm execution

* Fix batch initialization

* Make batch_size uint64_t instead of size_t

* max_block in silkworm_execute_blocks

* More accurate logProgress

* Use CHandle exposed by lmdb-go

* Enable Silkworm on Linux only

* Update lmdb-go

* Error when attempting to use Silkworm not on Linux

* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go

* Integrate Silkworm into cmd/integration

* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution

* Refactor SpawnExecuteBlocksStage

* linters

* Small clean-ups

* Move ChangeSetHook inside executeBlockWithGo

* No need to tamper with batch size in executeBlocksWithSilkworm

* fix

* Fix

* Fixup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-28 15:08:02 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Alex Sharov
342c5fee89
[to discuss] More compact append implementation (#1372)
* more compact append implementation

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* fix tests

* fix tests
2020-11-28 14:24:47 +00:00
Alex Sharov
ccd1ad5832
Sequence in to mdbx (#1370) 2020-11-23 11:15:43 +07:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
ledgerwatch
1a0e78a2e2
Remove changeset-based decorators from WalkAsOf (#1356)
* Remove decorators

* Current changes

* Fix and skip some tests for now

* Fix lint

* Specialise WalkAsOf

* Fix compile errors

* Fix lint

* Scan index files for key sizes

* More cleanup

* Fix to remote server

* Fix for walkasofstorage

* Fix/skip tests

* Fix tests

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-18 14:58:52 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Alex Sharov
1dd283efe2
Sequence for each table (#1354)
* history_early_stop

* auto_increment

* auto_increment

* tx_table
2020-11-14 13:48:29 +00:00
Alex Sharov
6ede598ed6
less ops on autodupsort feature (#1335) 2020-11-02 21:07:58 +00:00
Alex Sharov
73358fab1b
up master (#1332) 2020-11-01 08:28:11 +00:00