Commit Graph

1959 Commits

Author SHA1 Message Date
Igor Mandrigin
5bcab8fc07 update skip_analysis 2021-02-18 12:47:41 +01:00
ledgerwatch
7c81e9198d
Fixes to trace_call, trace_callMany, dual RPC daemon mode (#1491)
* Fixes to trace_call, trace_callMany, dual RPC daemon mode

* Fix compile error

* Fix compile error

* Compile error fix

* Compile error fix

* Fix typo

* Fix discrepancy when trace is empty

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-12 16:47:32 +00:00
Igor Mandrigin
8f3e606426 update skip_analysis 2021-02-11 14:54:46 +01:00
Artem Vorotnikov
5c34713e71
Remove nested transactions (#1483) 2021-02-10 17:04:22 +00:00
Artem Vorotnikov
123710adcd
ChangeSet: remove Walker::Walk (#1476) 2021-02-09 17:26:00 +07:00
ledgerwatch
5834189019
Bodies download and simplify header download (#1471)
* Only insert hard-coded tips if both DB and files recovery failed

* Prevent deadlock

* Report some efficiency numbers

* Count properly unrequested bodies

* Initialise allRequests

* Increase timeout

* Fixes to scheduling

* Small fix

* Simplified scheduling

* Remove separate bodyLoop goroutine

* Update from DB at the beginnig of block bodies forward

* Timeout for repeating the request cycle

* Fix timeout

* Fix

* Increase timeout

* Increase timeout

* Try to make flow if possible

* Fix flow

* Lower timeout

* timeout for each blockNum

* Adjustable timeout

* Better log timing

* Track peers

* copy peerID

* fix scheduling

* Too much logging

* Print delivery speed

* Print committed blocks

* Fix race

* Sentry to only reset back-off timer when response to a request is received

* Print bytes/sec and wasted traffic

* Fix bandwidth accounting

* Less logging

* not to wake up on deliveries

* Spam every second

* Print peer map

* Fix npe, print requests

* Timestamps

* Improved logging

* Penalty for peers

* Log penalties only when disconnecting

* Try with smaller window

* window parameter

* Dealing with partially delivered requests

* Init bodyReq

* Fix array index

* More fix for NPE

* More NPE checks

* Print out body progress and header progress

* Fix ending condition

* Bring back waking up and penalties

* Fix duplicate tip

* Duplicate segment

* Fix lint

* Fix lint

* fix lint

* Fix lint

* Hard coded headers in the source files

* Fix lint

* Replace hix-sized header serialisation with rlp (to support clique)

* Remove anchor records from the files

* Fixeds for DB recovery

* Fix compilation

* Fix compile errors

* Fix formatting

* Fix lint

* Fix comments

* Remove headerLoop

* Properly terminate body download

* Support for StatusData p2p proto

* Fix forkid test

* Fix test

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-02-08 09:25:10 +00:00
Artem Vorotnikov
91f7798c4a
Remove hashed changesets (#1464)
* Remove hashed changesets

* try this

* oops

* simplify

* better diagnostics

* don't hash account key and use bitmap 64 to check history index

* update docs of bucket

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2021-02-04 12:59:04 +03:00
Igor Mandrigin
aa6db643b9 update skip_analysis 2021-02-04 09:27:38 +01:00
Alex Sharov
6faec7e834
remove blocks compression (#1453) 2021-01-27 20:11:34 +07:00
Igor Mandrigin
f2d4a15644 update skip_analysis 2021-01-21 12:18:52 +01:00
Alex Sharov
58c4e03861
Revert "Remove hashed changesets (#1444)" (#1446)
This reverts commit aadb6e47cc.
2021-01-20 14:16:20 +07:00
Artem Vorotnikov
aadb6e47cc
Remove hashed changesets (#1444) 2021-01-19 17:42:26 +07:00
Igor Mandrigin
cc6647193e update skip_analysis 2021-01-14 16:05:14 +01:00
Alex Sharov
d7d9ce0a8e
fix go vet (#1425) 2020-12-26 10:18:16 +07:00
Igor Mandrigin
142e397423 update skip_analysis 2020-12-17 10:16:24 +01:00
ledgerwatch
3287b44c2e
Bad unwind of recreation of CREATE2 contract - unit test and potentia… (#1412)
* Bad unwind of recreation of CREATE2 contract - unit test and potential fix

* Fix unit test

* Fix lint

* Fix lint

* Fixup hack

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-16 17:16:37 +00:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
ledgerwatch
0953fd42cb
WIP trace_call implementation (#1399)
* trace_call initial

* Fix tests

* More tracing

* Add more fields to the action

* Completed first example query

* Add initial bench11 to compare trace_call with OpenEthereum

* Exclude precompile calls from call traces

* Add self-destruct, call types, more comparison in rpctest

* Support for execution errors

* Stack underflow error and delegatecall value

* Fix lint

* Fix suicide traceAddress, Bad instruction error

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-14 11:27:52 +00:00
Igor Mandrigin
8ea049aec6 update skip_analysis 2020-12-10 10:18:18 +01:00
ledgerwatch
2cc36dac87
further cleanup, unit tests for RPC daemon and tracing improvements (#1392)
* Cleanup and rpcdaemon unit tests

* Fix

* Fix

* Fix lint

* Test for debug_traceTransaction

* Add NoRefunds option

* Compile fix, test for no refunds

* Fix compile

* Add poly contract, fix compile errors

* No refunds now work

* Fix NPE in rpcdaemon

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-09 18:24:08 +00:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Igor Mandrigin
63ec2ec9f1 post-rebase fixes 2020-12-03 18:59:17 +01:00
Péter Szilágyi
233570c5ae core/state/snapshot: update generator marker in sync with flushes
# Conflicts:
#	core/state/snapshot/generate.go
#	core/state/snapshot/journal.go
#	core/state/snapshot/snapshot.go
2020-12-03 18:59:17 +01:00
gary rong
d474b21f4b core/state/snapshot: fix journal recovery from generating old journal (#21775)
* core/state/snapshot: print warning if failed to resolve journal

* core/state/snapshot: fix snapshot recovery

When we meet the snapshot journal consisted with:
- disk layer generator with new-format
- diff layer journal with old-format

The base layer should be returned without error.
The broken diff layer can be reconstructed later
but we definitely don't want to reconstruct the
huge diff layer.

* core: add tests
# Conflicts:
#	core/state/snapshot/journal.go
2020-12-03 18:59:17 +01:00
gary rong
89b2ff6097 core: improve snapshot journal recovery (#21594)
* core/state/snapshot: introduce snapshot journal version

* core: update the disk layer in an atomic way

* core: persist the disk layer generator periodically

* core/state/snapshot: improve logging

* core/state/snapshot: forcibly ensure the legacy snapshot is matched

* core/state/snapshot: add debug logs

* core, tests: fix tests and special recovery case

* core: polish

* core: add more blockchain tests for snapshot recovery

* core/state: fix comment

* core: add recovery flag for snapshot

* core: add restart after start-after-crash tests

* core/rawdb: fix imports

* core: fix tests

* core: remove log

* core/state/snapshot: fix snapshot

* core: avoid callbacks in SetHead

* core: fix setHead cornercase where the threshold root has state

* core: small docs for the test cases

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/blockchain.go
#	core/blockchain_repair_test.go
#	core/blockchain_sethead_test.go
#	core/rawdb/accessors_snapshot.go
#	core/rawdb/schema.go
#	core/state/snapshot/disklayer_test.go
#	core/state/snapshot/generate.go
#	core/state/snapshot/journal.go
#	core/state/snapshot/snapshot.go
#	tests/state_test_util.go
2020-12-03 18:59:17 +01:00
Igor Mandrigin
5d98f8d40d post-rebase. fix imports 2020-12-03 17:37:38 +01:00
Martin Holst Swende
ca3d54a32c all: implement EIP-2929 (gas cost increases for state access opcodes) + yolo-v2 (#21509)
* core/vm, core/state: implement EIP-2929 + YOLOv2

* core/state, core/vm: fix some review concerns

* core/state, core/vm: address review concerns

* core/vm: address review concerns

* core/vm: better documentation

* core/vm: unify sload cost as fully dynamic

* core/vm: fix typo

* core/vm/runtime: fix compilation flaw

* core/vm/runtime: fix renaming-err leftovers

* core/vm: renaming

* params/config: use correct yolov2 chainid for config

* core, params: use a proper new genesis for yolov2

* core/state/tests: golinter nitpicks
# Conflicts:
#	cmd/evm/internal/t8ntool/execution.go
#	cmd/geth/chaincmd.go
#	cmd/geth/consolecmd.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	core/state/statedb.go
#	core/state/statedb_test.go
#	core/vm/evm.go
#	core/vm/interpreter.go
#	core/vm/logger.go
#	core/vm/runtime/runtime.go
#	eth/api_tracer.go
2020-12-03 17:37:38 +01:00
hwanjo
7929173dfd core: fix blockchain insert report time interval calculation (#21723) 2020-12-03 17:37:38 +01:00
Marius van der Wijden
f68ac9d1c3 core/vm: marshall returnData as hexstring in trace logs (#21715)
* core/vm: marshall returnData as hexstring in trace logs

* core/vm: marshall returnData as hexstring in trace logs
# Conflicts:
#	core/vm/gen_structlog.go
2020-12-03 17:37:38 +01:00
Igor Mandrigin
7026c4fefe update skip_analysis 2020-12-03 09:55:00 +01:00
ledgerwatch
8d2db43e2b
Fix for slow stage3 (#1377)
* Fix for slow stage3

* Cache sequence

* Small optimisation

* mutation sequence lock and sequence(0) use read transaction

* tx

* Fixes

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-11-30 11:49:52 +00:00
Alex Sharov
342c5fee89
[to discuss] More compact append implementation (#1372)
* more compact append implementation

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* do appned for dupsort buckets

* fix tests

* fix tests
2020-11-28 14:24:47 +00:00
Igor Mandrigin
b723f8915e update skip_analysis 2020-11-26 08:49:33 +01:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
Igor Mandrigin
449097d857 update skip_analysis 2020-11-19 11:39:32 +01:00
ledgerwatch
1a0e78a2e2
Remove changeset-based decorators from WalkAsOf (#1356)
* Remove decorators

* Current changes

* Fix and skip some tests for now

* Fix lint

* Specialise WalkAsOf

* Fix compile errors

* Fix lint

* Scan index files for key sizes

* More cleanup

* Fix to remote server

* Fix for walkasofstorage

* Fix/skip tests

* Fix tests

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-18 14:58:52 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
583fada5ad
history_last_but_one_chunk_dissapear (#1355) 2020-11-16 14:49:11 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
ledgerwatch
f55cb44001
Update skip analysis (#1351) 2020-11-13 08:35:14 +00:00
Igor Mandrigin
96a1648b83 update skip_analysis 2020-11-05 10:37:22 +01:00
Suhabe Bugrara
b18727911f
CFG analysis (#1327)
* First

* More on SA interpreter

* Fixup

* Add cfg action to hack binary that invokes the SaInterpreter. Added an operation handler for PUSH1

* refactor cfg tests into separate file

* Move cfg tests into separate file

* More refactoring into new file

* dataflow interpreter

* work on cfg0

* finish cfg0

* df works on base examples

* refactor into dataflow spec

* add bounded stack

* add harder example

* fix switch pass thru

* fix switch pass thru

* bug fix, and better printing

* manual merge

* restore call to test gencfg

* abstract interpretation based cfg analysis

* fix post signature

* use uint256 instead uint64, add post function

* preprocess stmts

* initial implementation of resolve

* fix resolve

* fix resolve

* print stmts for edges

* print stmts for edges

* print states

* print states

* bug fixes, debugging

* fix jumpi dest - first working impl

* reachability analysis to filter out dead edges

* add all transfer functions

* larger contract bytecodes from solc compiler

* simple solidity contract goes thru

* add deposit contract bytecode

* rename deposit contract test

* fix new contract arg

* Address non-determinism leading to imprecise results

* improve debugging output

* improve debugging output

* improve debugging output

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* add more test cases

* fix coverage bug

* debugging for non-termination

* fix bad fixpoint check

* fix data inference

* fix transfer function for halting stmts

* switch to deposit contract test, disable debugging

* add anly counter to viz, fix stmt.valid check

* show all preds, adjust anlycounter behavior

* dfs instead of bfs to fail earlier

* viz improvements

* add worklist size to viz

* add test case for private functions

* valueset analysis

* add more checks to fail earlier in the analysis to help debugging, improve debugging output, catch additional bad jumps

* delete old code

* delete old code

* delete old code

* fix up minor changes to jump table

* copy over comments from cgf-1 branch

* remove minor diffs

* add recompiled deposit contract

* graph viz

* cleanup/refactoring

* initial impl of viz

* script to run cfg anly and generate dot file

* div example

* accept bytecode from cmd line

* add minimal deposit contract example

* replace valueset analysis with stackset analysis

* get in sync with master

* sync with master

* fix linting

* fix linting

* fix linting

* reformatting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* harness for running over all contracts

* refactor anly, track coverage metrics

* breakdown unresolved into different types, fix bad opcode bug

* sort programs by frequency

* ingest used contracts from bigquery

* performance, concurrency, bug fixes

* more test cases, handle invalid jumps differently, remove duplicate edges, report analytics limit

* simplify concurrency

* correctly track short stack

* add new transfer function, fix stack len

* variable stack length, perf opts, inc anly count limit

* profiling

* test case for large state size

* use custom hash function for control

* timeouts

* cfg.sh

* increase to 5 min timeout

* track elpased time

* use ptr

* increase limits

* increase limits

* fix mem leak

* debug mem leak

* debug mem leak

* lower resource limits

* fix nil error

* add new lattice element

* re-enable

* cut down limits

* preliminary proof checker

* refactor batch mode to run cfg in subprocess,put memory limit

* remove hard wiring

* adjust limits

* update metrics tracking

* more succinct proof checker

* rewrite checker

* bug fixes on checker

* bug fix

* remove print stmts

* track proof size

* print proof size

* don't panic on process error

* compress proof

* go mody tidy

* code formatting

* fix capitalization

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* remove unnecessary files

* fix typo

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-30 12:24:14 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Horacio Mijail Antón Quiles
ad2ff23e6d
Add the missing CaptureStart and CaptureEnd (#1316)
* Add the missing CaptureStart and CaptureEnd

* gofmt
2020-10-29 12:44:11 +00:00
ledgerwatch
05c36c308a
Update skip_analysis (#1318) 2020-10-29 13:02:11 +01:00
Alex Sharov
60e9ea187a
pics: set IH comparator (#1300)
* set IH comparator

* set IH comparator

* Fix chain config

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-27 10:35:32 +00:00
Igor Mandrigin
0f1b2f3595 geth-1.9.23: post-rebase fixups 2020-10-26 17:16:00 +01:00
mr_franklin
4d59c11d20 all: replace RWMutex with Mutex in places where RLock is not used (#21622)
# Conflicts:
#	les/txrelay.go
#	miner/unconfirmed.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
f488346129 core: fix txpool off-by-one error (#21683) 2020-10-26 17:16:00 +01:00