Commit Graph

13130 Commits

Author SHA1 Message Date
Igor Mandrigin
e4596532d5 begin 2020.11.2 release cycle 2020-11-05 13:28:50 +01:00
Igor Mandrigin
96a1648b83 update skip_analysis 2020-11-05 10:37:22 +01:00
alex.sharov
5ec296a206 squash 2020-11-04 10:05:55 +07:00
Alex Sharov
1b4f721a61
fix_db_tools_build_dir (#1337) 2020-11-03 14:19:57 +07:00
ledgerwatch
8c376082ff
[WIP] Integrate header download with stage 1 (saving headers to the database) (#1330)
* move locking to the hd object itself, create ready channel

* Fix lint

* More integration

* Fix lint

* Connect to processing

* Set hd.files at recovery

* Serialise processSegment

* Fix lint

* Not prepend filesDir

* Write to the database

* Fix lint

* Set up genesis block

* Fix hard coded tips

* Fix chainbreak false alarm

* Recover from DB

* Skip non-header records

* Not initialise if recovered from DB

* Fix lint
2020-11-02 21:09:12 +00:00
Alex Sharov
6ede598ed6
less ops on autodupsort feature (#1335) 2020-11-02 21:07:58 +00:00
Alex Sharov
a72543bc16
Stop dial scheduler asap (#1334)
* stop dial scheduler asap

* stop dial scheduler asap
2020-11-02 21:07:26 +00:00
Giulio rebuffo
86f0411c53
added ETL documentation to Readthedocs (#1333) 2020-11-02 21:06:14 +00:00
Alex Sharov
73358fab1b
up master (#1332) 2020-11-01 08:28:11 +00:00
Giulio rebuffo
c351feffce
added documentation about snappy (#1331) 2020-11-01 08:27:38 +00:00
ledgerwatch
cceb8c2c3a
Sharded state experiment based on call_traces stage run via integration (#1329)
* Initial work on state shards

* Fix compile errors

* Intermediate

* Implement dispatcher

* Implemented sharded reader

* No cache

* Print comms

* No prepopulation

* Print dispatcher activity

* Print more

* Print errors

* Fix dispatcher

* Remove logging

* Dispatcher to enforce initial barrier

* Don't print that much

* Turn off cache pre-seeting for state sharding

* Artificial latency, fixed number of blocks to execute

* Fix
2020-10-31 11:51:56 +00:00
Giulio rebuffo
3d5f7ab892
added stage 3 documentation to rtd (#1328) 2020-10-30 22:16:10 +00:00
Horacio Mijail Antón Quiles
ed1819ec58
Opcode tracer (#1320)
* OpcodeTracer

* Cleanup, rename to opcode_tracer

* Bring back check_change_sets, remove repeated vars

* Cleanup, gofmt

* Linter, cleanup

* Rename "segments" to "basic blocks", "bblocks"

* gofmt

* Lintci madness
2020-10-30 19:44:00 +00:00
Alex Sharov
99af6c4592
mdbx_exclusive_open_for_migrations (#1326) 2020-10-30 12:25:30 +00:00
Suhabe Bugrara
b18727911f
CFG analysis (#1327)
* First

* More on SA interpreter

* Fixup

* Add cfg action to hack binary that invokes the SaInterpreter. Added an operation handler for PUSH1

* refactor cfg tests into separate file

* Move cfg tests into separate file

* More refactoring into new file

* dataflow interpreter

* work on cfg0

* finish cfg0

* df works on base examples

* refactor into dataflow spec

* add bounded stack

* add harder example

* fix switch pass thru

* fix switch pass thru

* bug fix, and better printing

* manual merge

* restore call to test gencfg

* abstract interpretation based cfg analysis

* fix post signature

* use uint256 instead uint64, add post function

* preprocess stmts

* initial implementation of resolve

* fix resolve

* fix resolve

* print stmts for edges

* print stmts for edges

* print states

* print states

* bug fixes, debugging

* fix jumpi dest - first working impl

* reachability analysis to filter out dead edges

* add all transfer functions

* larger contract bytecodes from solc compiler

* simple solidity contract goes thru

* add deposit contract bytecode

* rename deposit contract test

* fix new contract arg

* Address non-determinism leading to imprecise results

* improve debugging output

* improve debugging output

* improve debugging output

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* add more test cases

* fix coverage bug

* debugging for non-termination

* fix bad fixpoint check

* fix data inference

* fix transfer function for halting stmts

* switch to deposit contract test, disable debugging

* add anly counter to viz, fix stmt.valid check

* show all preds, adjust anlycounter behavior

* dfs instead of bfs to fail earlier

* viz improvements

* add worklist size to viz

* add test case for private functions

* valueset analysis

* add more checks to fail earlier in the analysis to help debugging, improve debugging output, catch additional bad jumps

* delete old code

* delete old code

* delete old code

* fix up minor changes to jump table

* copy over comments from cgf-1 branch

* remove minor diffs

* add recompiled deposit contract

* graph viz

* cleanup/refactoring

* initial impl of viz

* script to run cfg anly and generate dot file

* div example

* accept bytecode from cmd line

* add minimal deposit contract example

* replace valueset analysis with stackset analysis

* get in sync with master

* sync with master

* fix linting

* fix linting

* fix linting

* reformatting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* harness for running over all contracts

* refactor anly, track coverage metrics

* breakdown unresolved into different types, fix bad opcode bug

* sort programs by frequency

* ingest used contracts from bigquery

* performance, concurrency, bug fixes

* more test cases, handle invalid jumps differently, remove duplicate edges, report analytics limit

* simplify concurrency

* correctly track short stack

* add new transfer function, fix stack len

* variable stack length, perf opts, inc anly count limit

* profiling

* test case for large state size

* use custom hash function for control

* timeouts

* cfg.sh

* increase to 5 min timeout

* track elpased time

* use ptr

* increase limits

* increase limits

* fix mem leak

* debug mem leak

* debug mem leak

* lower resource limits

* fix nil error

* add new lattice element

* re-enable

* cut down limits

* preliminary proof checker

* refactor batch mode to run cfg in subprocess,put memory limit

* remove hard wiring

* adjust limits

* update metrics tracking

* more succinct proof checker

* rewrite checker

* bug fixes on checker

* bug fix

* remove print stmts

* track proof size

* print proof size

* don't panic on process error

* compress proof

* go mody tidy

* code formatting

* fix capitalization

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* remove unnecessary files

* fix typo

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-30 12:24:14 +00:00
Igor Mandrigin
1b2bf8bf61
fix git commit when using make docker (#1323)
* fix git commit when using `make docker`

* avoid warning about LATEST_COMMIT for linters in Docker
2020-10-30 08:43:24 +00:00
Alex Sharov
91871746b3
app code can specify TX flag to don't call FSync (#1322) 2020-10-30 08:43:11 +00:00
Horacio Mijail Antón Quiles
f8477e29ab
Move batch management into the writeReceipts case (#1321) 2020-10-30 08:41:35 +00:00
Alex Sharov
ed7dd7acd3
mdbx_fix_err_check (#1324) 2020-10-30 15:38:25 +07:00
alex.sharov
1cf5ee279d add incarnation definition 2020-10-30 12:09:55 +07:00
Igor Mandrigin
21f289ba74
dont send .git to Docker (#1319) 2020-10-29 16:39:05 +00:00
ledgerwatch
97a03a1ad7
[WIP] add interfaces subtree and switch headers POC to using shared proto files (#1315)
* Squashed 'interfaces/' content from commit c4e14d6dc

git-subtree-dir: interfaces
git-subtree-split: c4e14d6dc80a157c60acd42f0152d631d250601e

* Squashed 'interfaces/' changes from c4e14d6dc..84c5faa5f

84c5faa5f Update README.md
07db484a1 Update README.md
ea14fa4e3 Update README.md

git-subtree-dir: interfaces
git-subtree-split: 84c5faa5f956ea52119911f39ac5864b282dcec2

* Change proto paths

* Squashed 'interfaces/' changes from 84c5faa5f..eb9859d41

eb9859d41 Correct syntax
ae78fe955 Add go package

git-subtree-dir: interfaces
git-subtree-split: eb9859d4157351fac3b8bd2f477307a09978e6df

* Switch to shared proto files

* Fix generation

* Remove duplicate files

* Init hard-coded tips when not recovering

* Only check initiation when extending up
2020-10-29 16:38:46 +00:00
Alex Sharov
399db7ee3b
LMDB v0.9.27, and add all lmdb cli tools (#1303)
* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* clean

* clean

* clean

* clean

* clean

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:46 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Horacio Mijail Antón Quiles
ad2ff23e6d
Add the missing CaptureStart and CaptureEnd (#1316)
* Add the missing CaptureStart and CaptureEnd

* gofmt
2020-10-29 12:44:11 +00:00
Igor Mandrigin
feaac77390 begin 2020.11.01 release cycle 2020-10-29 13:10:51 +01:00
ledgerwatch
05c36c308a
Update skip_analysis (#1318) 2020-10-29 13:02:11 +01:00
ledgerwatch
ea060815ef
Remove z3 and semantics (#1314)
* Removed z3 submodule

* Remove semantics and z3 from build

* Fixes to CI
2020-10-28 17:52:00 +00:00
Igor Mandrigin
abf4fa5861
Don't require MDBX if you don't specify it (#1313)
* fix compilation

* add mdbx flag to the tg binary

* fix makefile

* fixups

* fix makefile

* fix readme

* fix linters

* fix tests
2020-10-28 12:17:18 +00:00
Alex Sharov
a139f31dcd
Exclusive lock manual (#1302)
* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* mdb tools

* resolve abs path

* rpcdaemon to check exclusive lock

* rpcdaemon to check exclusive lock

* exclusive lock for mdbx

* exclusive lock for mdbx
2020-10-28 09:52:15 +00:00
Igor Mandrigin
c17028e1b4
Update README.md 2020-10-28 09:35:51 +01:00
Igor Mandrigin
0aff30d2f9
Update README.md 2020-10-28 09:35:23 +01:00
ledgerwatch
722c4562f0
Some additions to the DB walkthrough (#1309)
* Fix tables

* Some corrections to DB walkthrough
2020-10-28 07:18:48 +00:00
Alex Sharov
28cb4a9b3d
Readme load dump tables (#1311) 2020-10-28 10:51:23 +07:00
Alex Sharov
7ae38f5fe3
check canonical receipt (#1310) 2020-10-28 10:24:17 +07:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
b00ris
d2d5da8b09
add snapshot support for rpcdaemon (#1304) 2020-10-27 22:31:47 +00:00
ledgerwatch
32278539ac
Use BTree instead of maps for Mutation (batch) (#1308)
* Try mutation based on B-tree

* Use pointers

* Fix NPE

* Cleanup
2020-10-27 22:30:18 +00:00
Igor Mandrigin
9ad6978259
Refactor flags + fix logs (#1306) 2020-10-27 16:53:49 +01:00
ledgerwatch
1e6c29a9ef
Revert "[WIP - Discussion - Perf Testing] Use BTree instead of maps for Mutation (batch) (#1294)" (#1305)
This reverts commit a9ae477d8f.
2020-10-27 14:35:25 +00:00
ledgerwatch
a9ae477d8f
[WIP - Discussion - Perf Testing] Use BTree instead of maps for Mutation (batch) (#1294)
* Try mutation based on B-tree

* Use pointers

* Fix NPE

* Cleanup
2020-10-27 13:37:37 +00:00
Alex Sharov
60e9ea187a
pics: set IH comparator (#1300)
* set IH comparator

* set IH comparator

* Fix chain config

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-27 10:35:32 +00:00
Igor Mandrigin
0f1b2f3595 geth-1.9.23: post-rebase fixups 2020-10-26 17:16:00 +01:00
Felix Lange
aa60deec2d params: go-ethereum v1.9.23 stable
# Conflicts:
#	params/version.go
2020-10-26 17:16:00 +01:00
Felix Lange
0c54d2e649 p2p/discover: implement v5.1 wire protocol (#21647)
This change implements the Discovery v5.1 wire protocol and
also adds an interactive test suite for this protocol.
# Conflicts:
#	cmd/devp2p/discv5cmd.go
#	p2p/discover/v5_encoding.go
#	p2p/discover/v5_encoding_test.go
#	p2p/discover/v5_udp.go
#	p2p/discover/v5_udp_test.go
#	p2p/discover/v5wire/session.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
26069ae7a4 miner: set etherbase even if mining isn't possible at the moment (#21707) 2020-10-26 17:16:00 +01:00
gary rong
ee6e0cbd5c params: update CHTs (#21706) 2020-10-26 17:16:00 +01:00
Marius van der Wijden
ac604670a3 accounts/keystore: fix flaky test (#21703)
* accounts/keystore: add timeout to test to prevent failure on travis

The TestWalletNotifications test sporadically fails on travis.
This is because we shutdown the event collection before all events are received.
Adding a small timeout (10 milliseconds) allows the collector to be scheduled
and to consume all pending events before we shut it down.

* accounts/keystore: added newlines back in

* accounts/keystore: properly fix the walletNotifications test
2020-10-26 17:16:00 +01:00
Martin Holst Swende
735c5b631c miner: don't interrupt mining after successful sync (#21701)
* miner: exit loop when downloader Done or Failed

Following the logic of the comment at the method,
this fixes a regression introduced at 7cf56d6f064869cb62b1673f9ee437020c595391
, which would allow external parties to DoS with
blocks, preventing mining progress.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: remove ineff assign (lint)

Signed-off-by: meows <b5c6@protonmail.com>

* miner: update test re downloader events

Signed-off-by: meows <b5c6@protonmail.com>

* Revert "miner: remove ineff assign (lint)"

This reverts commit eaefcd34ab4862ebc936fb8a07578aa2744bc058.

* Revert "miner: exit loop when downloader Done or Failed"

This reverts commit 23abd34265aa246c38fc390bb72572ad6ae9fe3b.

* miner: add test showing imprecise TestMiner

Signed-off-by: meows <b5c6@protonmail.com>

* miner: fix waitForMiningState precision

This helper function would return an affirmation
on the first positive match on a desired bool.

This was imprecise; it return false positives
by not waiting initially for an 'updated' value.

This fix causes TestMiner_2 to fail, which is
expected.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: remove TestMiner_2 demonstrating broken test

This test demonstrated the imprecision of the test
helper function waitForMiningState. This function
has been fixed with 6d365c2851, and this test test
may now be removed.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: fix test regarding downloader event/mining expectations

See comment for logic.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: add test describing expectations for downloader/mining events

We expect that once the downloader emits a DoneEvent,
signaling a successful sync, that subsequent StartEvents
are not longer permitted to stop the miner.

This prevents a security vulnerability where forced syncs via
fake high blocks would stall mining operation.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: use 'canStop' state to fix downloader event handling

- Break downloader event handling into event
separating Done and Failed events. We need to
treat these cases differently since a DoneEvent
should prevent the miner from being stopped on
subsequent downloader Start events.

- Use canStop state to handle the one-off
case when a downloader first succeeds.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: improve comment wording

Signed-off-by: meows <b5c6@protonmail.com>

* miner: start mining on downloader events iff not already mining

Signed-off-by: meows <b5c6@protonmail.com>

* miner: refactor miner update logic w/r/t downloader events

This makes mining pause/start logic regarding downloader
events more explicit. Instead of eternally handling downloader
events after the first done event, the subscription is closed
when downloader events are no longer actionable.

Signed-off-by: meows <b5c6@protonmail.com>

* miner: fix handling downloader events on subcription closed

Signed-off-by: meows <b5c6@protonmail.com>

* miner: (lint:gosimple) use range over chan instead of for/select

Signed-off-by: meows <b5c6@protonmail.com>

* miner: refactor update loop to remove race condition

The go routine handling the downloader events handling
vars in parallel with the parent routine, causing a
race condition.

This change, though ugly, remove the condition while
still allowing the downloader event subscription to be
closed when the miner has no further use for it (ie DoneEvent).

* miner: alternate fix for miner-flaw

Co-authored-by: meows <b5c6@protonmail.com>
# Conflicts:
#	miner/miner_test.go
2020-10-26 17:16:00 +01:00
Giuseppe Bertone
c690231219 cmd/geth, cmd/utils: fixed flags name (#21700)
# Conflicts:
#	cmd/geth/main.go
#	cmd/geth/usage.go
2020-10-26 17:16:00 +01:00