Commit Graph

1548 Commits

Author SHA1 Message Date
Andrei Ivasko
022bdeaf1b
hashstate-shutdown (#1452) 2021-01-24 14:04:37 +07:00
Alex Sharov
58c4e03861
Revert "Remove hashed changesets (#1444)" (#1446)
This reverts commit aadb6e47cc.
2021-01-20 14:16:20 +07:00
Artem Vorotnikov
aadb6e47cc
Remove hashed changesets (#1444) 2021-01-19 17:42:26 +07:00
ledgerwatch
820216b77f
Block body download - more recent updates (#1441)
* Combined

* Fix lint

* Fix lint

* Initial bodydownload package

* Updates

* fix lint and process incoming body

* Fix compile errors

* Use deliveries

* Ready for initial testing

* Better handling of missing temp files dir

* Intermediate testing

* First working code

* Fix races, more efficient request scheduling

* Remove excessive logging

* Fix lint

* Interruptibility, fix lint

* Remove more logging

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-01-15 10:58:04 +00:00
b00ris
8db5790838
State snapshot sync (#1417)
* move experiments to new branch&reorganise kv_snapshot

* walk&modify tests

* added delete from snapshot tests

* fmt

* state snapshot debug

* snapshot validation passed. copy state snapshot

* debug

* snapshot cursor.Prev test

* Prev works correct. Added Current check

* add err check

* added walk forward and backward test

* before refactoring

* refactoring

* execution with snapshot debug

* fix

* remove useless test

* before dupcursor implimentation

* tests with prev and delete works

* execution based on state snapshot passed

* remove useless tests

* blocks to 1140000 passed

* clean verifier

* cleanup state generation

* clean verify && seeder

* remove debug code

* tests passed

* fix lint

* save state

* test passed

* fix lint

* add state hash

* fix lint
2021-01-02 19:28:37 +00:00
Alex Sharov
d7d9ce0a8e
fix go vet (#1425) 2020-12-26 10:18:16 +07:00
Artem Vorotnikov
8c0ecaeea9
Remove StageData (#1419) 2020-12-21 11:01:37 +00:00
ledgerwatch
3287b44c2e
Bad unwind of recreation of CREATE2 contract - unit test and potentia… (#1412)
* Bad unwind of recreation of CREATE2 contract - unit test and potential fix

* Fix unit test

* Fix lint

* Fix lint

* Fixup hack

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-16 17:16:37 +00:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
Alex Sharov
aebb2d23c8
Grpc graceful shutdown deadline (#1414)
* grpc_graceful_shutdown_deadline

* grpc_graceful_shutdown_deadline
2020-12-16 14:14:31 +01:00
ledgerwatch
0953fd42cb
WIP trace_call implementation (#1399)
* trace_call initial

* Fix tests

* More tracing

* Add more fields to the action

* Completed first example query

* Add initial bench11 to compare trace_call with OpenEthereum

* Exclude precompile calls from call traces

* Add self-destruct, call types, more comparison in rpctest

* Support for execution errors

* Stack underflow error and delegatecall value

* Fix lint

* Fix suicide traceAddress, Bad instruction error

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-14 11:27:52 +00:00
ledgerwatch
2cc36dac87
further cleanup, unit tests for RPC daemon and tracing improvements (#1392)
* Cleanup and rpcdaemon unit tests

* Fix

* Fix

* Fix lint

* Test for debug_traceTransaction

* Add NoRefunds option

* Compile fix, test for no refunds

* Fix compile

* Add poly contract, fix compile errors

* No refunds now work

* Fix NPE in rpcdaemon

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-09 18:24:08 +00:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Martin Holst Swende
ca3d54a32c all: implement EIP-2929 (gas cost increases for state access opcodes) + yolo-v2 (#21509)
* core/vm, core/state: implement EIP-2929 + YOLOv2

* core/state, core/vm: fix some review concerns

* core/state, core/vm: address review concerns

* core/vm: address review concerns

* core/vm: better documentation

* core/vm: unify sload cost as fully dynamic

* core/vm: fix typo

* core/vm/runtime: fix compilation flaw

* core/vm/runtime: fix renaming-err leftovers

* core/vm: renaming

* params/config: use correct yolov2 chainid for config

* core, params: use a proper new genesis for yolov2

* core/state/tests: golinter nitpicks
# Conflicts:
#	cmd/evm/internal/t8ntool/execution.go
#	cmd/geth/chaincmd.go
#	cmd/geth/consolecmd.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	core/state/statedb.go
#	core/state/statedb_test.go
#	core/vm/evm.go
#	core/vm/interpreter.go
#	core/vm/logger.go
#	core/vm/runtime/runtime.go
#	eth/api_tracer.go
2020-12-03 17:37:38 +01:00
aaronbuchwald
07aefb98c3 eth/api: fix potential nil deref in AccountRange (#21710)
* Fix potential nil pointer error when neither block number nor hash is specified to accountRange

* Update error description
# Conflicts:
#	eth/api.go
2020-12-03 17:37:38 +01:00
ledgerwatch
8d2db43e2b
Fix for slow stage3 (#1377)
* Fix for slow stage3

* Cache sequence

* Small optimisation

* mutation sequence lock and sequence(0) use read transaction

* tx

* Fixes

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-11-30 11:49:52 +00:00
Andrew Ashikhmin
8a877c227a
Integrate Silkworm execution (#1344)
* Expose C Handle of lmdbTx

* LoadExecutionFunctionPointer

* silkworm ExecuteBlocks

* linter

* CLI flag for SilkwormExecutionFunc

* linter

* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage

* Commit transaction after Silkworm execution

* Fix batch initialization

* Make batch_size uint64_t instead of size_t

* max_block in silkworm_execute_blocks

* More accurate logProgress

* Use CHandle exposed by lmdb-go

* Enable Silkworm on Linux only

* Update lmdb-go

* Error when attempting to use Silkworm not on Linux

* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go

* Integrate Silkworm into cmd/integration

* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution

* Refactor SpawnExecuteBlocksStage

* linters

* Small clean-ups

* Move ChangeSetHook inside executeBlockWithGo

* No need to tamper with batch size in executeBlocksWithSilkworm

* fix

* Fix

* Fixup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-28 15:08:02 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
ledgerwatch
987f02de90
Revert "ih_clear_bucket_on_regen (#1357)" (#1361)
This reverts commit d55ccb7133.
2020-11-18 10:05:14 +00:00
Alex Sharov
d55ccb7133
ih_clear_bucket_on_regen (#1357) 2020-11-18 08:37:19 +00:00
Igor Mandrigin
b23feb46e7
fix an off-by-1 error that caused double headers in a subscription (#1360) 2020-11-17 20:46:47 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
ledgerwatch
f20a278b51
Possible fix for tx pool hanging (#1352)
* Possible fix for tx pool hanging

* Fix for mutex

* Fix

* Unlock before waiting for response to top header request

* Add comment
2020-11-16 12:05:53 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00
ledgerwatch
cceb8c2c3a
Sharded state experiment based on call_traces stage run via integration (#1329)
* Initial work on state shards

* Fix compile errors

* Intermediate

* Implement dispatcher

* Implemented sharded reader

* No cache

* Print comms

* No prepopulation

* Print dispatcher activity

* Print more

* Print errors

* Fix dispatcher

* Remove logging

* Dispatcher to enforce initial barrier

* Don't print that much

* Turn off cache pre-seeting for state sharding

* Artificial latency, fixed number of blocks to execute

* Fix
2020-10-31 11:51:56 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Alex Sharov
a139f31dcd
Exclusive lock manual (#1302)
* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* mdb tools

* resolve abs path

* rpcdaemon to check exclusive lock

* rpcdaemon to check exclusive lock

* exclusive lock for mdbx

* exclusive lock for mdbx
2020-10-28 09:52:15 +00:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
Igor Mandrigin
9ad6978259
Refactor flags + fix logs (#1306) 2020-10-27 16:53:49 +01:00
mr_franklin
4d59c11d20 all: replace RWMutex with Mutex in places where RLock is not used (#21622)
# Conflicts:
#	les/txrelay.go
#	miner/unconfirmed.go
2020-10-26 17:16:00 +01:00
Marius van der Wijden
e40950ebd2 eth/downloader: cache parent hash instead of recomputing (#21678) 2020-10-26 17:16:00 +01:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
c2f41d7723
stop on error: headers stage (#1289) 2020-10-24 18:05:12 +01:00
Alex Sharov
77a4b3305d
rawdb to return errors: part 1 (#1286)
* ReadBlockByNumber and ReadChainConfig return error

* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:

ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
2020-10-24 07:57:09 +01:00
Alex Sharov
6584b43b12
Allow start read transactions from ethdb.Database interface (#1287)
* allow_read_transactions

* fix lint

* fix lint

* more cases
2020-10-24 07:55:43 +01:00
Alex Sharov
76f1b05cb2
Move tmpdir definition to app-start, move migrations folder inside tmpdir (#1282)
* extract tmpdir to app-level-code

* extract tmpdir to app-level-code

* save progresss
2020-10-23 12:18:45 +01:00
Alex Sharov
dd5258d524
save progresss (#1283) 2020-10-23 11:52:18 +01:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274)
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
2020-10-21 18:01:40 +01:00
Evgeny Danilenko
44562fd07a
Jump tables data race (#1245)
* initial

* fix

* fmt

* linters

* linters

* debug

* fix data races

* safe close

* remove jump table clonning

* revert fix
2020-10-20 22:23:56 +01:00
Giulio rebuffo
0a4386737e
Uses number instead of others to repressenting progressing in accord to block number (#1260)
* logs fixed with number=

* better logs

* removed lol
2020-10-19 22:46:26 +01:00
Giulio rebuffo
f359527356
better logs (#1259) 2020-10-19 20:17:21 +01:00
Alex Sharov
5dbe1724be
Bitmap etl (#1227)
* save progress

* save progress

* save progress

* save progress

* etl bimaps

* etl bimaps

* etl bimaps

* etl bimaps

* a

* a

* a

* a

* a

* a

* a

* save progress

* save progress

* save progress

* a

* fix_race_condition_on_zstd_build

* clean

* clean

* clean

* clean

* clean

* go mod tidy

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* fix corner case

* commit every

* call traces etl

* call traces etl

* call traces etl

* return missed error

* go mod tidy

* go mod tidy

* go mod tidy

* rebase to master

* remove trash

* print full key if it has 8 leading zeroes
2020-10-19 14:11:01 +01:00
b00ris
e5f6412da2
Update snapshot hash (#1243)
* remove useless bucket

* update snapshot hashes

* remove log
2020-10-17 08:48:20 +01:00
Evgeny Danilenko
946d4fac5b
ForkID api (#1244)
* forkID api

* merge fork calls

* optional param
2020-10-16 15:15:10 +03:00
Alex Sharov
a8f38ba1c3
fix db opening (#1240) 2020-10-14 13:05:54 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00
Igor Mandrigin
ef2353c0e0 turbo-api: custom state writers 2020-10-13 13:35:48 +02:00