Commit Graph

2021 Commits

Author SHA1 Message Date
Marius van der Wijden
bcc51ea117 core, all: split vm.Context into BlockContext and TxContext (#21672)
* all: core: split vm.Config into BlockConfig and TxConfig

* core: core/vm: reset EVM between tx in block instead of creating new

* core/vm: added docs
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/evm/internal/t8ntool/execution.go
#	cmd/geth/retesteth.go
#	core/evm.go
#	core/state_prefetcher.go
#	core/state_processor.go
#	core/state_transition.go
#	core/vm/evm.go
#	core/vm/gas_table_test.go
#	core/vm/instructions.go
#	core/vm/instructions_test.go
#	eth/api_tracer.go
#	eth/tracers/tracer_test.go
#	eth/tracers/tracers_test.go
#	les/api_backend.go
#	les/odr_test.go
#	light/odr_test.go
#	tests/vm_test_util.go
2021-03-08 12:17:35 +01:00
Martin Holst Swende
6fb6cbf27c core/vm, protocol_params: implement eip-2565 modexp repricing (#21607)
* core/vm, protocol_params: implement eip-2565 modexp repricing

* core/vm: fix review concerns
2021-03-08 11:44:39 +01:00
Alex Sharov
cdc090ebfd
Delete MultiWalk and GetIndexChunk funcs from Database interface (#1544) 2021-03-08 12:08:37 +07:00
ledgerwatch
b743b88218
Revert "Extract validating interface (#1120)" (#1543)
* Revert "Extract validating interface (#1120)"

This reverts commit e4e36c152e.

* Fix go mod tidy

* Fix lint

* Fix lint

* fix lint

* Test fix

* Fix test

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-05 20:34:23 +00:00
ledgerwatch
4decdc293d
Update skip analysis for release (#1541)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-04 13:13:12 +00:00
Alex Sharov
eb7d2ad16a
Add validation of senders amount (#1539) 2021-03-04 16:34:49 +07:00
Alex Sharov
8538ec85b3
don't open db in exclusive mode if no migrations (#1524) 2021-03-01 11:02:22 +07:00
ledgerwatch
da74bd8f38
Fixes to unwind logic for the new staged sync (#1521)
* Make DB visible

* Further fixes to unwind logic

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-28 08:47:31 +00:00
Evgeny Danilenko
e4e36c152e
Extract validating interface (#1120)
* interface

* generalize interface

* linters

* fix deadlock

* fix linters

* close goroutine

* fix

* debug

* id+ttl

* refactor downloader tests

* tests

* lru

* handle genesis, extract fake consensuses

* fix fake consensus

* test uncles, verify

* after a new master

* fmt

* fix close

* debug

* debug

* fix chain length

* remove test field

* use single account

* fix data race on closing channel

* remove postponed blocks queue

* miner test

* VerifyHeaderRequests

* fmt

* fmt

* fix data race

* handle validating errors

* simplify matchParents

* remove copy-paste

* move sort to constructor

* clean up

* debug for 10 parents

* debug

* debug

* batch responses

* batch requests

* works for many ancestors

* remove debug

* always Close an engine

* linters

* ancestors deduplication

* fix test

* reduce interface

* api

* clique

* green clique sync

* stable

* perpermance per second

* full sync

* linters

* gitignore

* deps

* fix panic after master merge

* init consensus

* clique tests

* fix tests

* fix tests

* clean up

* reuse snap

* store vefified snapshots

* optimize snapshots

* safe close

* cleanup loop

* after downloader

* downloader and consensus tests

* update tests

* hack.go

* clique flags

* fix cliuqe config

* review

* gitignore

* remove additional bucket

* blk/sec instead of blk/microsecond

* rename

* deps

* optimize

* debug

* test

* tests without extracted validation process

* same base performance as on master

* benchmark

* simplify ethash verification

* ethash

* ethash

* linters

* ethash

* master stats

* cleanup

* gomod

* linters

* tests

* better locks

* Fix

* Remove logging for verifyHeaders

* Verification speed in the logs

* Fix compile error

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-25 19:40:45 +00:00
Igor Mandrigin
569264c140 update skip_analysis 2021-02-25 09:00:29 +01:00
Alex Sharov
d4ef1f8516
blocks exec unwind - to support graceful shutdown (#1494)
* quit exec unwind

* quit exec unwind

* lint
2021-02-23 17:14:32 +00:00
Alex Sharov
f1ee8d507e
New Trie db layout - store trie structure info and multiple hashes per record (#1500)
* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE

* SE
2021-02-21 18:41:59 +00:00
Igor Mandrigin
5bcab8fc07 update skip_analysis 2021-02-18 12:47:41 +01:00
ledgerwatch
7c81e9198d
Fixes to trace_call, trace_callMany, dual RPC daemon mode (#1491)
* Fixes to trace_call, trace_callMany, dual RPC daemon mode

* Fix compile error

* Fix compile error

* Compile error fix

* Compile error fix

* Fix typo

* Fix discrepancy when trace is empty

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-12 16:47:32 +00:00
Igor Mandrigin
8f3e606426 update skip_analysis 2021-02-11 14:54:46 +01:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Artem Vorotnikov
123710adcd
ChangeSet: remove Walker::Walk (#1476) 2021-02-09 17:26:00 +07:00
ledgerwatch
5834189019
Bodies download and simplify header download (#1471)
* Only insert hard-coded tips if both DB and files recovery failed

* Prevent deadlock

* Report some efficiency numbers

* Count properly unrequested bodies

* Initialise allRequests

* Increase timeout

* Fixes to scheduling

* Small fix

* Simplified scheduling

* Remove separate bodyLoop goroutine

* Update from DB at the beginnig of block bodies forward

* Timeout for repeating the request cycle

* Fix timeout

* Fix

* Increase timeout

* Increase timeout

* Try to make flow if possible

* Fix flow

* Lower timeout

* timeout for each blockNum

* Adjustable timeout

* Better log timing

* Track peers

* copy peerID

* fix scheduling

* Too much logging

* Print delivery speed

* Print committed blocks

* Fix race

* Sentry to only reset back-off timer when response to a request is received

* Print bytes/sec and wasted traffic

* Fix bandwidth accounting

* Less logging

* not to wake up on deliveries

* Spam every second

* Print peer map

* Fix npe, print requests

* Timestamps

* Improved logging

* Penalty for peers

* Log penalties only when disconnecting

* Try with smaller window

* window parameter

* Dealing with partially delivered requests

* Init bodyReq

* Fix array index

* More fix for NPE

* More NPE checks

* Print out body progress and header progress

* Fix ending condition

* Bring back waking up and penalties

* Fix duplicate tip

* Duplicate segment

* Fix lint

* Fix lint

* fix lint

* Fix lint

* Hard coded headers in the source files

* Fix lint

* Replace hix-sized header serialisation with rlp (to support clique)

* Remove anchor records from the files

* Fixeds for DB recovery

* Fix compilation

* Fix compile errors

* Fix formatting

* Fix lint

* Fix comments

* Remove headerLoop

* Properly terminate body download

* Support for StatusData p2p proto

* Fix forkid test

* Fix test

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-08 09:25:10 +00:00
Artem Vorotnikov
91f7798c4a
Remove hashed changesets (#1464)
* Remove hashed changesets

* try this

* oops

* simplify

* better diagnostics

* don't hash account key and use bitmap 64 to check history index

* update docs of bucket

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2021-02-04 12:59:04 +03:00
Igor Mandrigin
aa6db643b9 update skip_analysis 2021-02-04 09:27:38 +01:00
Alex Sharov
6faec7e834
remove blocks compression (#1453) 2021-01-27 20:11:34 +07:00
Igor Mandrigin
f2d4a15644 update skip_analysis 2021-01-21 12:18:52 +01:00
Alex Sharov
58c4e03861
Revert "Remove hashed changesets (#1444)" (#1446)
This reverts commit aadb6e47cc.
2021-01-20 14:16:20 +07:00
Artem Vorotnikov
aadb6e47cc
Remove hashed changesets (#1444) 2021-01-19 17:42:26 +07:00
Igor Mandrigin
cc6647193e update skip_analysis 2021-01-14 16:05:14 +01:00
Alex Sharov
d7d9ce0a8e
fix go vet (#1425) 2020-12-26 10:18:16 +07:00
Igor Mandrigin
142e397423 update skip_analysis 2020-12-17 10:16:24 +01:00
ledgerwatch
3287b44c2e
Bad unwind of recreation of CREATE2 contract - unit test and potentia… (#1412)
* Bad unwind of recreation of CREATE2 contract - unit test and potential fix

* Fix unit test

* Fix lint

* Fix lint

* Fixup hack

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-16 17:16:37 +00:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
ledgerwatch
0953fd42cb
WIP trace_call implementation (#1399)
* trace_call initial

* Fix tests

* More tracing

* Add more fields to the action

* Completed first example query

* Add initial bench11 to compare trace_call with OpenEthereum

* Exclude precompile calls from call traces

* Add self-destruct, call types, more comparison in rpctest

* Support for execution errors

* Stack underflow error and delegatecall value

* Fix lint

* Fix suicide traceAddress, Bad instruction error

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-14 11:27:52 +00:00
Igor Mandrigin
8ea049aec6 update skip_analysis 2020-12-10 10:18:18 +01:00
ledgerwatch
2cc36dac87
further cleanup, unit tests for RPC daemon and tracing improvements (#1392)
* Cleanup and rpcdaemon unit tests

* Fix

* Fix

* Fix lint

* Test for debug_traceTransaction

* Add NoRefunds option

* Compile fix, test for no refunds

* Fix compile

* Add poly contract, fix compile errors

* No refunds now work

* Fix NPE in rpcdaemon

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-09 18:24:08 +00:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Igor Mandrigin
63ec2ec9f1 post-rebase fixes 2020-12-03 18:59:17 +01:00
Péter Szilágyi
233570c5ae core/state/snapshot: update generator marker in sync with flushes
# Conflicts:
#	core/state/snapshot/generate.go
#	core/state/snapshot/journal.go
#	core/state/snapshot/snapshot.go
2020-12-03 18:59:17 +01:00
gary rong
d474b21f4b core/state/snapshot: fix journal recovery from generating old journal (#21775)
* core/state/snapshot: print warning if failed to resolve journal

* core/state/snapshot: fix snapshot recovery

When we meet the snapshot journal consisted with:
- disk layer generator with new-format
- diff layer journal with old-format

The base layer should be returned without error.
The broken diff layer can be reconstructed later
but we definitely don't want to reconstruct the
huge diff layer.

* core: add tests
# Conflicts:
#	core/state/snapshot/journal.go
2020-12-03 18:59:17 +01:00
gary rong
89b2ff6097 core: improve snapshot journal recovery (#21594)
* core/state/snapshot: introduce snapshot journal version

* core: update the disk layer in an atomic way

* core: persist the disk layer generator periodically

* core/state/snapshot: improve logging

* core/state/snapshot: forcibly ensure the legacy snapshot is matched

* core/state/snapshot: add debug logs

* core, tests: fix tests and special recovery case

* core: polish

* core: add more blockchain tests for snapshot recovery

* core/state: fix comment

* core: add recovery flag for snapshot

* core: add restart after start-after-crash tests

* core/rawdb: fix imports

* core: fix tests

* core: remove log

* core/state/snapshot: fix snapshot

* core: avoid callbacks in SetHead

* core: fix setHead cornercase where the threshold root has state

* core: small docs for the test cases

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/blockchain.go
#	core/blockchain_repair_test.go
#	core/blockchain_sethead_test.go
#	core/rawdb/accessors_snapshot.go
#	core/rawdb/schema.go
#	core/state/snapshot/disklayer_test.go
#	core/state/snapshot/generate.go
#	core/state/snapshot/journal.go
#	core/state/snapshot/snapshot.go
#	tests/state_test_util.go
2020-12-03 18:59:17 +01:00
Igor Mandrigin
5d98f8d40d post-rebase. fix imports 2020-12-03 17:37:38 +01:00
Martin Holst Swende
ca3d54a32c all: implement EIP-2929 (gas cost increases for state access opcodes) + yolo-v2 (#21509)
* core/vm, core/state: implement EIP-2929 + YOLOv2

* core/state, core/vm: fix some review concerns

* core/state, core/vm: address review concerns

* core/vm: address review concerns

* core/vm: better documentation

* core/vm: unify sload cost as fully dynamic

* core/vm: fix typo

* core/vm/runtime: fix compilation flaw

* core/vm/runtime: fix renaming-err leftovers

* core/vm: renaming

* params/config: use correct yolov2 chainid for config

* core, params: use a proper new genesis for yolov2

* core/state/tests: golinter nitpicks
# Conflicts:
#	cmd/evm/internal/t8ntool/execution.go
#	cmd/geth/chaincmd.go
#	cmd/geth/consolecmd.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	core/state/statedb.go
#	core/state/statedb_test.go
#	core/vm/evm.go
#	core/vm/interpreter.go
#	core/vm/logger.go
#	core/vm/runtime/runtime.go
#	eth/api_tracer.go
2020-12-03 17:37:38 +01:00
hwanjo
7929173dfd core: fix blockchain insert report time interval calculation (#21723) 2020-12-03 17:37:38 +01:00
Marius van der Wijden
f68ac9d1c3 core/vm: marshall returnData as hexstring in trace logs (#21715)
* core/vm: marshall returnData as hexstring in trace logs

* core/vm: marshall returnData as hexstring in trace logs
# Conflicts:
#	core/vm/gen_structlog.go
2020-12-03 17:37:38 +01:00
Igor Mandrigin
7026c4fefe update skip_analysis 2020-12-03 09:55:00 +01:00
ledgerwatch
8d2db43e2b
Fix for slow stage3 (#1377)
* Fix for slow stage3

* Cache sequence

* Small optimisation

* mutation sequence lock and sequence(0) use read transaction

* tx

* Fixes

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-11-30 11:49:52 +00:00
Alex Sharov
342c5fee89
[to discuss] More compact append implementation (#1372)
* more compact append implementation

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* fix tests

* fix tests
2020-11-28 14:24:47 +00:00
Igor Mandrigin
b723f8915e update skip_analysis 2020-11-26 08:49:33 +01:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
Igor Mandrigin
449097d857 update skip_analysis 2020-11-19 11:39:32 +01:00
ledgerwatch
1a0e78a2e2
Remove changeset-based decorators from WalkAsOf (#1356)
* Remove decorators

* Current changes

* Fix and skip some tests for now

* Fix lint

* Specialise WalkAsOf

* Fix compile errors

* Fix lint

* Scan index files for key sizes

* More cleanup

* Fix to remote server

* Fix for walkasofstorage

* Fix/skip tests

* Fix tests

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-18 14:58:52 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
583fada5ad
history_last_but_one_chunk_dissapear (#1355) 2020-11-16 14:49:11 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
ledgerwatch
f55cb44001
Update skip analysis (#1351) 2020-11-13 08:35:14 +00:00
Igor Mandrigin
96a1648b83 update skip_analysis 2020-11-05 10:37:22 +01:00
Suhabe Bugrara
b18727911f
CFG analysis (#1327)
* First

* More on SA interpreter

* Fixup

* Add cfg action to hack binary that invokes the SaInterpreter. Added an operation handler for PUSH1

* refactor cfg tests into separate file

* Move cfg tests into separate file

* More refactoring into new file

* dataflow interpreter

* work on cfg0

* finish cfg0

* df works on base examples

* refactor into dataflow spec

* add bounded stack

* add harder example

* fix switch pass thru

* fix switch pass thru

* bug fix, and better printing

* manual merge

* restore call to test gencfg

* abstract interpretation based cfg analysis

* fix post signature

* use uint256 instead uint64, add post function

* preprocess stmts

* initial implementation of resolve

* fix resolve

* fix resolve

* print stmts for edges

* print stmts for edges

* print states

* print states

* bug fixes, debugging

* fix jumpi dest - first working impl

* reachability analysis to filter out dead edges

* add all transfer functions

* larger contract bytecodes from solc compiler

* simple solidity contract goes thru

* add deposit contract bytecode

* rename deposit contract test

* fix new contract arg

* Address non-determinism leading to imprecise results

* improve debugging output

* improve debugging output

* improve debugging output

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* add more test cases

* fix coverage bug

* debugging for non-termination

* fix bad fixpoint check

* fix data inference

* fix transfer function for halting stmts

* switch to deposit contract test, disable debugging

* add anly counter to viz, fix stmt.valid check

* show all preds, adjust anlycounter behavior

* dfs instead of bfs to fail earlier

* viz improvements

* add worklist size to viz

* add test case for private functions

* valueset analysis

* add more checks to fail earlier in the analysis to help debugging, improve debugging output, catch additional bad jumps

* delete old code

* delete old code

* delete old code

* fix up minor changes to jump table

* copy over comments from cgf-1 branch

* remove minor diffs

* add recompiled deposit contract

* graph viz

* cleanup/refactoring

* initial impl of viz

* script to run cfg anly and generate dot file

* div example

* accept bytecode from cmd line

* add minimal deposit contract example

* replace valueset analysis with stackset analysis

* get in sync with master

* sync with master

* fix linting

* fix linting

* fix linting

* reformatting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* harness for running over all contracts

* refactor anly, track coverage metrics

* breakdown unresolved into different types, fix bad opcode bug

* sort programs by frequency

* ingest used contracts from bigquery

* performance, concurrency, bug fixes

* more test cases, handle invalid jumps differently, remove duplicate edges, report analytics limit

* simplify concurrency

* correctly track short stack

* add new transfer function, fix stack len

* variable stack length, perf opts, inc anly count limit

* profiling

* test case for large state size

* use custom hash function for control

* timeouts

* cfg.sh

* increase to 5 min timeout

* track elpased time

* use ptr

* increase limits

* increase limits

* fix mem leak

* debug mem leak

* debug mem leak

* lower resource limits

* fix nil error

* add new lattice element

* re-enable

* cut down limits

* preliminary proof checker

* refactor batch mode to run cfg in subprocess,put memory limit

* remove hard wiring

* adjust limits

* update metrics tracking

* more succinct proof checker

* rewrite checker

* bug fixes on checker

* bug fix

* remove print stmts

* track proof size

* print proof size

* don't panic on process error

* compress proof

* go mody tidy

* code formatting

* fix capitalization

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* remove unnecessary files

* fix typo

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-30 12:24:14 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Horacio Mijail Antón Quiles
ad2ff23e6d
Add the missing CaptureStart and CaptureEnd (#1316)
* Add the missing CaptureStart and CaptureEnd

* gofmt
2020-10-29 12:44:11 +00:00
ledgerwatch
05c36c308a
Update skip_analysis (#1318) 2020-10-29 13:02:11 +01:00
Alex Sharov
60e9ea187a
pics: set IH comparator (#1300)
* set IH comparator

* set IH comparator

* Fix chain config

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-27 10:35:32 +00:00
Igor Mandrigin
0f1b2f3595 geth-1.9.23: post-rebase fixups 2020-10-26 17:16:00 +01:00
mr_franklin
4d59c11d20 all: replace RWMutex with Mutex in places where RLock is not used (#21622)
# Conflicts:
#	les/txrelay.go
#	miner/unconfirmed.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
f488346129 core: fix txpool off-by-one error (#21683) 2020-10-26 17:16:00 +01:00
Guillaume Ballet
6ae9473675 core/vm: dedup config check in markdown logger (#21655)
* core/vm: dedup config check

* review feedback: reuse buffer
# Conflicts:
#	core/vm/logger.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
96d0e0ff9c core/types: optimize bloom filters (#21624)
* core/types: tests for bloom

* core/types: refactored bloom filter for receipts, added tests

core/types: replaced old bloom implementation

core/types: change interface of bloom add+test

* core/types: refactor bloom

* core/types: minor tweak on LogsBloom

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
# Conflicts:
#	core/types/bloom9.go
#	core/types/bloom9_test.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
0338641873 core/bloombits: faster generator (#21625)
* core/bloombits: add benchmark

* core/bloombits: optimize inserts
2020-10-26 17:16:00 +01:00
aaronbuchwald
527155fc34 core: free pointer from slice after popping element from price heap (#21572)
* Fix potential memory leak in price heap

* core: nil free pointer slice (alternative version)

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-10-26 17:16:00 +01:00
Martin Holst Swende
618f9bedfa core/vm, params: make 2200 in line with spec (#21605)
# Conflicts:
#	core/vm/gas_table.go
2020-10-26 17:16:00 +01:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
c2f41d7723
stop on error: headers stage (#1289) 2020-10-24 18:05:12 +01:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader (#1288)
* protect_from_wrong_class_use

* more cases
2020-10-24 09:09:20 +01:00
Alex Sharov
77a4b3305d
rawdb to return errors: part 1 (#1286)
* ReadBlockByNumber and ReadChainConfig return error

* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:

ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
2020-10-24 07:57:09 +01:00
Alex Sharov
76f1b05cb2
Move tmpdir definition to app-start, move migrations folder inside tmpdir (#1282)
* extract tmpdir to app-level-code

* extract tmpdir to app-level-code

* save progresss
2020-10-23 12:18:45 +01:00
ledgerwatch
fc5138c0e2
Update skip analysis (#1277) 2020-10-22 11:16:39 +01:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274)
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
2020-10-21 18:01:40 +01:00
Evgeny Danilenko
44562fd07a
Jump tables data race (#1245)
* initial

* fix

* fmt

* linters

* linters

* debug

* fix data races

* safe close

* remove jump table clonning

* revert fix
2020-10-20 22:23:56 +01:00
Giulio rebuffo
f359527356
better logs (#1259) 2020-10-19 20:17:21 +01:00
Alex Sharov
5dbe1724be
Bitmap etl (#1227)
* save progress

* save progress

* save progress

* save progress

* etl bimaps

* etl bimaps

* etl bimaps

* etl bimaps

* a

* a

* a

* a

* a

* a

* a

* save progress

* save progress

* save progress

* a

* fix_race_condition_on_zstd_build

* clean

* clean

* clean

* clean

* clean

* go mod tidy

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* fix corner case

* commit every

* call traces etl

* call traces etl

* call traces etl

* return missed error

* go mod tidy

* go mod tidy

* go mod tidy

* rebase to master

* remove trash

* print full key if it has 8 leading zeroes
2020-10-19 14:11:01 +01:00
ledgerwatch
bdef07f5f2
Fix cmd/state checkChangeSets and calltraces (#1252)
* Fix to checkChangeSets

* Fix lint

* Update skip analysis

* Fix

* Update skip analysis again
2020-10-18 08:31:23 +01:00
Evgeny Danilenko
946d4fac5b
ForkID api (#1244)
* forkID api

* merge fork calls

* optional param
2020-10-16 15:15:10 +03:00
Alex Sharov
a8f38ba1c3
fix db opening (#1240) 2020-10-14 13:05:54 +01:00
ledgerwatch
b747ab5324
[WIP] CallTraces index (#1157)
* Initial commit for CallTraces index

* Fix compilation

* fix lint, add comment

* Fix integration

* Add Close function to ethdb.Cursor, fix some compile errors

* Try to stop cursor leak in Get

* Fix compile errors in RPC daemon

* Fix compile errors

* fixing another way

* Some fixes

* More fixes

* More fixes

* More fixes

* Fixes to core/state

* Fix lint

* Fix lint

* Fixes

* Stage caching for call trace stage

* Add mem stats

* Try to stop the leak

* Turn off debug

* Chunks for 10k blocks

* Print

* Revert "Print"

This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.

* Revert "Chunks for 10k blocks"

This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.

* Trying to fix the leak

* Don't compute receipts in re-tracing

* Not compose block

* Print speed, fix receipts, bigger caches

* Fix lint

* Utilise changeset info

* Counters

* Use NoReceipts and ReadOnly

* ReadOnly is incompatible with caching

* Skip test leaking transactions

* Fix block test

* Change disable message for call-traces stage

* Use block option for call traces integration

* Fix retracing due to incarnation
2020-10-12 09:39:04 +01:00
Alex Sharov
88a099702b
rawdb.ReadCanonicalHash to return err (#1204)
* read_canonical_hash_return_err

* hack hugeFreelist

* fix tests
2020-10-10 07:05:56 +01:00
rene
8a12d561f2 cmd/devp2p: add eth protocol test suite (#21598)
This change adds a test framework for the "eth" protocol and some basic
tests. The tests can be run using the './devp2p rlpx eth-test' command.
# Conflicts:
#	core/forkid/forkid.go
#	eth/discovery.go
#	eth/handler.go
#	eth/helper_test.go
#	eth/protocol_test.go
2020-10-06 14:12:09 +02:00
Guillaume Ballet
822244dcd4 core: fix a typo in comment (#21439)
# Conflicts:
#	core/block_validator.go
2020-10-06 14:12:09 +02:00
Martin Holst Swende
e9b1492b0f core/vm: fix benchmark overflow + prep for precompile repricings (#21530)
* core/vm/testdata: add gascost expectations to testcases

* core/vm: verify expected gas in tests for precompiles

* core/vm: fix overflow flaw in gas/s calculation
2020-10-06 14:12:09 +02:00
ledgerwatch
086952df16
Fix wrong log error messages (#1186) 2020-10-05 16:56:25 +01:00
ledgerwatch
82193a3670
Not swallow errors in accessor_chain (#1185) 2020-10-05 15:18:51 +01:00
Evgeny Danilenko
ec6be820ac
Don't force sync in case InMemDb (#1174)
* initial

* fix

* fix chain length
2020-10-02 14:08:28 +01:00
Alex Sharov
809d79b15f
Switch to cbor (#1172)
* switch receipts to cbor

* switch receipts to cbor

* rpcdaemon to cache chainconfig

* rpcdaemon to cache chainconfig

* rpcdaemon to cache chainconfig

* rpcdaemon to cache chainconfig
2020-10-02 13:51:20 +01:00
Alex Sharov
61fbe34024
Better cbor support (#1163) 2020-10-02 10:56:13 +07:00
ledgerwatch
96d9639570
Update skip analysis (#1164) 2020-10-01 09:25:44 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124)
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
ledgerwatch
4be6174a15
Add comment about ContractAddress in receipts, followed by some simplification (#1137)
* Add comment about receipts

* Panic on error

* Ignore errors in the simulated backend

* Fix lint
2020-09-26 13:38:10 +01:00
Igor Mandrigin
ab18d19743 update skip_analysis 2020-09-24 10:30:37 +02:00
Igor Mandrigin
f956550671 update skip_analysis 2020-09-17 14:24:04 +02:00
Alex Sharov
a112183ceb
fix logidx (#1122)
* fix logidx

* derive fields already dose set index

* start from zero
2020-09-17 13:19:11 +02:00
Igor Mandrigin
adf52465e3
move ./trie to ./turbo/trie (#1114)
Useful for minimizing merge conflicts when rebasing new geth
2020-09-14 11:33:39 +01:00
ledgerwatch
a98469c6cd
Remove changeset processing in FindHistory (#1105) 2020-09-12 08:24:13 +01:00
Igor Mandrigin
0299475a3c update skip_analysis 2020-09-10 12:53:12 +02:00
Alex Sharov
da31494cc3
transactional migrations (#1079) 2020-09-08 20:39:43 +01:00