Commit Graph

12939 Commits

Author SHA1 Message Date
ledgerwatch
f55cb44001
Update skip analysis (#1351) 2020-11-13 08:35:14 +00:00
ledgerwatch
15fdf93d6c
Delete unnecessary pictures (#1350) 2020-11-12 19:41:30 +00:00
Igor Mandrigin
f1becbe0a3 begin 2020.11.3 release cycle 2020-11-12 13:03:43 +01:00
ledgerwatch
10f338b8f7
Add lmdb guide visualisations (#1348)
* Add lmdb guide visualisations

* updated names, remove double-tap visualisations
2020-11-11 17:18:34 +00:00
ledgerwatch
e71c3cae70
Lmdb visualisation + freelist stability fixes (#1345)
* Visualisations

* Use graphviz records

* Fix lint

* Fix lint

* Reduced leaf pages

* Fix

* Overflow pages

* Fix lint

* Fix lint

* Fix lint

* Add dupsort generate

* DupSort experiments

* Add dupsort nodes

* More interesting dupsort

* freelist

* Fragmented freelist

* Fix lint

* drop T2 too

* Handle overflow pages in FREE_DBI

* Skipping in the MAIN_DBI leaves

* Switch to double-tap LMDB

* Fix to double-tap

* With/without reader visualisations

* Flip flopping without reader

* Tidy mods

* Fix lint, switch to new lmdb-go release

* mod tidy
2020-11-11 09:35:58 +00:00
Igor Mandrigin
ed9672620b
Granular rpc control (Allow list for RPC daemon) (#1341) 2020-11-10 10:08:42 +01:00
Thomas Jay Rush
546b91f47e
Updating RPC tests in Postman (#1340) 2020-11-09 09:52:18 +01:00
Igor Mandrigin
c0f9b57b42
rpcdaemon: read chainConfig from the DB (remove hardcode) (#1339)
* read chainConfig from the db

* get dbtx properly
2020-11-08 06:46:53 +01:00
Igor Mandrigin
ab878ed59a
Ethstats (#1338) 2020-11-06 08:59:50 +01:00
Igor Mandrigin
e4596532d5 begin 2020.11.2 release cycle 2020-11-05 13:28:50 +01:00
Igor Mandrigin
96a1648b83 update skip_analysis 2020-11-05 10:37:22 +01:00
alex.sharov
5ec296a206 squash 2020-11-04 10:05:55 +07:00
Alex Sharov
1b4f721a61
fix_db_tools_build_dir (#1337) 2020-11-03 14:19:57 +07:00
ledgerwatch
8c376082ff
[WIP] Integrate header download with stage 1 (saving headers to the database) (#1330)
* move locking to the hd object itself, create ready channel

* Fix lint

* More integration

* Fix lint

* Connect to processing

* Set hd.files at recovery

* Serialise processSegment

* Fix lint

* Not prepend filesDir

* Write to the database

* Fix lint

* Set up genesis block

* Fix hard coded tips

* Fix chainbreak false alarm

* Recover from DB

* Skip non-header records

* Not initialise if recovered from DB

* Fix lint
2020-11-02 21:09:12 +00:00
Alex Sharov
6ede598ed6
less ops on autodupsort feature (#1335) 2020-11-02 21:07:58 +00:00
Alex Sharov
a72543bc16
Stop dial scheduler asap (#1334)
* stop dial scheduler asap

* stop dial scheduler asap
2020-11-02 21:07:26 +00:00
Giulio rebuffo
86f0411c53
added ETL documentation to Readthedocs (#1333) 2020-11-02 21:06:14 +00:00
Alex Sharov
73358fab1b
up master (#1332) 2020-11-01 08:28:11 +00:00
Giulio rebuffo
c351feffce
added documentation about snappy (#1331) 2020-11-01 08:27:38 +00:00
ledgerwatch
cceb8c2c3a
Sharded state experiment based on call_traces stage run via integration (#1329)
* Initial work on state shards

* Fix compile errors

* Intermediate

* Implement dispatcher

* Implemented sharded reader

* No cache

* Print comms

* No prepopulation

* Print dispatcher activity

* Print more

* Print errors

* Fix dispatcher

* Remove logging

* Dispatcher to enforce initial barrier

* Don't print that much

* Turn off cache pre-seeting for state sharding

* Artificial latency, fixed number of blocks to execute

* Fix
2020-10-31 11:51:56 +00:00
Giulio rebuffo
3d5f7ab892
added stage 3 documentation to rtd (#1328) 2020-10-30 22:16:10 +00:00
Horacio Mijail Antón Quiles
ed1819ec58
Opcode tracer (#1320)
* OpcodeTracer

* Cleanup, rename to opcode_tracer

* Bring back check_change_sets, remove repeated vars

* Cleanup, gofmt

* Linter, cleanup

* Rename "segments" to "basic blocks", "bblocks"

* gofmt

* Lintci madness
2020-10-30 19:44:00 +00:00
Alex Sharov
99af6c4592
mdbx_exclusive_open_for_migrations (#1326) 2020-10-30 12:25:30 +00:00
Suhabe Bugrara
b18727911f
CFG analysis (#1327)
* First

* More on SA interpreter

* Fixup

* Add cfg action to hack binary that invokes the SaInterpreter. Added an operation handler for PUSH1

* refactor cfg tests into separate file

* Move cfg tests into separate file

* More refactoring into new file

* dataflow interpreter

* work on cfg0

* finish cfg0

* df works on base examples

* refactor into dataflow spec

* add bounded stack

* add harder example

* fix switch pass thru

* fix switch pass thru

* bug fix, and better printing

* manual merge

* restore call to test gencfg

* abstract interpretation based cfg analysis

* fix post signature

* use uint256 instead uint64, add post function

* preprocess stmts

* initial implementation of resolve

* fix resolve

* fix resolve

* print stmts for edges

* print stmts for edges

* print states

* print states

* bug fixes, debugging

* fix jumpi dest - first working impl

* reachability analysis to filter out dead edges

* add all transfer functions

* larger contract bytecodes from solc compiler

* simple solidity contract goes thru

* add deposit contract bytecode

* rename deposit contract test

* fix new contract arg

* Address non-determinism leading to imprecise results

* improve debugging output

* improve debugging output

* improve debugging output

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* add more test cases

* fix coverage bug

* debugging for non-termination

* fix bad fixpoint check

* fix data inference

* fix transfer function for halting stmts

* switch to deposit contract test, disable debugging

* add anly counter to viz, fix stmt.valid check

* show all preds, adjust anlycounter behavior

* dfs instead of bfs to fail earlier

* viz improvements

* add worklist size to viz

* add test case for private functions

* valueset analysis

* add more checks to fail earlier in the analysis to help debugging, improve debugging output, catch additional bad jumps

* delete old code

* delete old code

* delete old code

* fix up minor changes to jump table

* copy over comments from cgf-1 branch

* remove minor diffs

* add recompiled deposit contract

* graph viz

* cleanup/refactoring

* initial impl of viz

* script to run cfg anly and generate dot file

* div example

* accept bytecode from cmd line

* add minimal deposit contract example

* replace valueset analysis with stackset analysis

* get in sync with master

* sync with master

* fix linting

* fix linting

* fix linting

* reformatting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* harness for running over all contracts

* refactor anly, track coverage metrics

* breakdown unresolved into different types, fix bad opcode bug

* sort programs by frequency

* ingest used contracts from bigquery

* performance, concurrency, bug fixes

* more test cases, handle invalid jumps differently, remove duplicate edges, report analytics limit

* simplify concurrency

* correctly track short stack

* add new transfer function, fix stack len

* variable stack length, perf opts, inc anly count limit

* profiling

* test case for large state size

* use custom hash function for control

* timeouts

* cfg.sh

* increase to 5 min timeout

* track elpased time

* use ptr

* increase limits

* increase limits

* fix mem leak

* debug mem leak

* debug mem leak

* lower resource limits

* fix nil error

* add new lattice element

* re-enable

* cut down limits

* preliminary proof checker

* refactor batch mode to run cfg in subprocess,put memory limit

* remove hard wiring

* adjust limits

* update metrics tracking

* more succinct proof checker

* rewrite checker

* bug fixes on checker

* bug fix

* remove print stmts

* track proof size

* print proof size

* don't panic on process error

* compress proof

* go mody tidy

* code formatting

* fix capitalization

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* remove unnecessary files

* fix typo

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-30 12:24:14 +00:00
Igor Mandrigin
1b2bf8bf61
fix git commit when using make docker (#1323)
* fix git commit when using `make docker`

* avoid warning about LATEST_COMMIT for linters in Docker
2020-10-30 08:43:24 +00:00
Alex Sharov
91871746b3
app code can specify TX flag to don't call FSync (#1322) 2020-10-30 08:43:11 +00:00
Horacio Mijail Antón Quiles
f8477e29ab
Move batch management into the writeReceipts case (#1321) 2020-10-30 08:41:35 +00:00
Alex Sharov
ed7dd7acd3
mdbx_fix_err_check (#1324) 2020-10-30 15:38:25 +07:00
alex.sharov
1cf5ee279d add incarnation definition 2020-10-30 12:09:55 +07:00
Igor Mandrigin
21f289ba74
dont send .git to Docker (#1319) 2020-10-29 16:39:05 +00:00
ledgerwatch
97a03a1ad7
[WIP] add interfaces subtree and switch headers POC to using shared proto files (#1315)
* Squashed 'interfaces/' content from commit c4e14d6dc

git-subtree-dir: interfaces
git-subtree-split: c4e14d6dc80a157c60acd42f0152d631d250601e

* Squashed 'interfaces/' changes from c4e14d6dc..84c5faa5f

84c5faa5f Update README.md
07db484a1 Update README.md
ea14fa4e3 Update README.md

git-subtree-dir: interfaces
git-subtree-split: 84c5faa5f956ea52119911f39ac5864b282dcec2

* Change proto paths

* Squashed 'interfaces/' changes from 84c5faa5f..eb9859d41

eb9859d41 Correct syntax
ae78fe955 Add go package

git-subtree-dir: interfaces
git-subtree-split: eb9859d4157351fac3b8bd2f477307a09978e6df

* Switch to shared proto files

* Fix generation

* Remove duplicate files

* Init hard-coded tips when not recovering

* Only check initiation when extending up
2020-10-29 16:38:46 +00:00
Alex Sharov
399db7ee3b
LMDB v0.9.27, and add all lmdb cli tools (#1303)
* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* clean

* clean

* clean

* clean

* clean

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:46 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Horacio Mijail Antón Quiles
ad2ff23e6d
Add the missing CaptureStart and CaptureEnd (#1316)
* Add the missing CaptureStart and CaptureEnd

* gofmt
2020-10-29 12:44:11 +00:00
Igor Mandrigin
feaac77390 begin 2020.11.01 release cycle 2020-10-29 13:10:51 +01:00
ledgerwatch
05c36c308a
Update skip_analysis (#1318) 2020-10-29 13:02:11 +01:00
ledgerwatch
ea060815ef
Remove z3 and semantics (#1314)
* Removed z3 submodule

* Remove semantics and z3 from build

* Fixes to CI
2020-10-28 17:52:00 +00:00
Igor Mandrigin
abf4fa5861
Don't require MDBX if you don't specify it (#1313)
* fix compilation

* add mdbx flag to the tg binary

* fix makefile

* fixups

* fix makefile

* fix readme

* fix linters

* fix tests
2020-10-28 12:17:18 +00:00
Alex Sharov
a139f31dcd
Exclusive lock manual (#1302)
* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* mdb tools

* resolve abs path

* rpcdaemon to check exclusive lock

* rpcdaemon to check exclusive lock

* exclusive lock for mdbx

* exclusive lock for mdbx
2020-10-28 09:52:15 +00:00
Igor Mandrigin
c17028e1b4
Update README.md 2020-10-28 09:35:51 +01:00
Igor Mandrigin
0aff30d2f9
Update README.md 2020-10-28 09:35:23 +01:00
ledgerwatch
722c4562f0
Some additions to the DB walkthrough (#1309)
* Fix tables

* Some corrections to DB walkthrough
2020-10-28 07:18:48 +00:00
Alex Sharov
28cb4a9b3d
Readme load dump tables (#1311) 2020-10-28 10:51:23 +07:00
Alex Sharov
7ae38f5fe3
check canonical receipt (#1310) 2020-10-28 10:24:17 +07:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
b00ris
d2d5da8b09
add snapshot support for rpcdaemon (#1304) 2020-10-27 22:31:47 +00:00
ledgerwatch
32278539ac
Use BTree instead of maps for Mutation (batch) (#1308)
* Try mutation based on B-tree

* Use pointers

* Fix NPE

* Cleanup
2020-10-27 22:30:18 +00:00
Igor Mandrigin
9ad6978259
Refactor flags + fix logs (#1306) 2020-10-27 16:53:49 +01:00
ledgerwatch
1e6c29a9ef
Revert "[WIP - Discussion - Perf Testing] Use BTree instead of maps for Mutation (batch) (#1294)" (#1305)
This reverts commit a9ae477d8f.
2020-10-27 14:35:25 +00:00
ledgerwatch
a9ae477d8f
[WIP - Discussion - Perf Testing] Use BTree instead of maps for Mutation (batch) (#1294)
* Try mutation based on B-tree

* Use pointers

* Fix NPE

* Cleanup
2020-10-27 13:37:37 +00:00