Commit Graph

12520 Commits

Author SHA1 Message Date
Alex Sharov
8c9a55bb21
[To test on HDD] Write changesets to tx instead of mutation (#978)
* use mutation over tx

* clear

* add .CommitAndBegin() method

* multip put 2

* right sorting

* write non-random things to tx

* write non-random things to tx

* disable multi put 2

* clean

* clean

* remove multiput2

* clean

* clean

* clean

* clean

* clean

* add receipts
2020-08-29 08:32:51 +01:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
Alex Sharov
bdb2e354ce
restapi_to_follog_geth_cli_variables (#993) 2020-08-28 11:27:56 +01:00
ledgerwatch
335aedc7e3
Cut down the stateGrowth generation code (#973)
* Cut down the stateGrowth generation

* Fix NPE

* Fix for lastTimestamp

* Adjust to the current state

* Potential fix

* Fix for shorter keys in the state

* Extract code

* Iterate over code
2020-08-28 10:31:37 +01:00
Alex Sharov
d30df505c8
Don't read cwd in blockchain validator, remove debug code (#992)
* print cwd

* don't read cwd, remove debug code

* oops, overdelete

* clean
2020-08-28 10:30:22 +01:00
Alex Sharov
d73b036cff
python_bublic_print_cli (#994) 2020-08-28 14:19:34 +07:00
Alex Sharov
de16812a94
move tx limit from 16 to 32 blocks (#988) 2020-08-28 07:28:08 +01:00
Alex Sharov
df05a581dd
Add tx pool stage to transaction (#981)
* add tx_pool stage to tx

* clean

* don't call commit in read tx

* don't call commit in read tx

* clean

* more logs

* fix tx pool race

* more logs

* clean
2020-08-28 07:27:48 +01:00
Suhabe Bugrara
8b31944611
Control Flow Analysis (#990)
* First

* More on SA interpreter

* Fixup

* Add cfg action to hack binary that invokes the SaInterpreter. Added an operation handler for PUSH1

* refactor cfg tests into separate file

* Move cfg tests into separate file

* More refactoring into new file

* dataflow interpreter

* work on cfg0

* finish cfg0

* df works on base examples

* refactor into dataflow spec

* add bounded stack

* add harder example

* fix switch pass thru

* fix switch pass thru

* bug fix, and better printing

* manual merge

* restore call to test gencfg

* abstract interpretation based cfg analysis

* fix post signature

* use uint256 instead uint64, add post function

* preprocess stmts

* initial implementation of resolve

* fix resolve

* fix resolve

* print stmts for edges

* print stmts for edges

* print states

* print states

* bug fixes, debugging

* fix jumpi dest - first working impl

* reachability analysis to filter out dead edges

* add all transfer functions

* larger contract bytecodes from solc compiler

* simple solidity contract goes thru

* add deposit contract bytecode

* rename deposit contract test

* fix new contract arg

* Address non-determinism leading to imprecise results

* improve debugging output

* improve debugging output

* improve debugging output

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* add more test cases

* fix coverage bug

* debugging for non-termination

* fix bad fixpoint check

* fix data inference

* fix transfer function for halting stmts

* switch to deposit contract test, disable debugging

* add anly counter to viz, fix stmt.valid check

* show all preds, adjust anlycounter behavior

* dfs instead of bfs to fail earlier

* viz improvements

* add worklist size to viz

* add test case for private functions

* valueset analysis

* add more checks to fail earlier in the analysis to help debugging, improve debugging output, catch additional bad jumps

* delete old code

* delete old code

* delete old code

* fix up minor changes to jump table

* copy over comments from cgf-1 branch

* remove minor diffs

* add recompiled deposit contract

* graph viz

* cleanup/refactoring

* initial impl of viz

* script to run cfg anly and generate dot file

* div example

* accept bytecode from cmd line

* add minimal deposit contract example

* replace valueset analysis with stackset analysis

* get in sync with master

* sync with master

* fix linting

* fix linting

* fix linting

* reformatting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-08-28 07:26:49 +01:00
Alex Sharov
af81bb0de6
[merge after release] Simplified loader (#979)
* v0

* more docs, less customization

* more docs, less customization

* more docs, less customization

* don't store db reference

* rely on fact that lmdb key/value valid until end of transaction

* clean

* clean
2020-08-27 19:21:50 +01:00
Chase Wright
9d3c641664
Update Supported JSON-RPC calls in README (#983)
* Update Supported JSON-RPC calls in README

* Fix Hyperlinks
* Update list

* Include net / net_version

* Rename net.go to net_api.go

* Fix hyperlink for renamed net_api.go file
2020-08-27 19:19:58 +01:00
Igor Mandrigin
03c1b48180 calver: start 2020.09.01 release 2020-08-27 10:55:52 +02:00
Igor Mandrigin
12aae871fd skip_analysis: update block number 2020-08-27 10:35:32 +02:00
Alex Sharov
f54812184b
make all in Dockerfile (#980) 2020-08-27 13:45:24 +07:00
Alex Sharov
584a5581c8
show "Done!" message only if stage took more than 30 sec (#977) 2020-08-27 12:43:37 +07:00
Alex Sharov
714e5041e2
Better logs3 (#976) 2020-08-26 19:06:51 +07:00
Alex Sharov
9091ce62bb
KV reduce features amount (#975)
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize

* remove bolt_db.go file

* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
Alex Sharov
522287ac18
Transactional cycle (#966)
* v0

* v1

* v3

* v4

* clean

* temporary fix of txpool

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* Add debug logs about tx start/commit

* add condition

* tx pool to not hold own db

* try to enable TxPool in integration

* exclude tx pool from tx

* exclude tx pool from integration

* reduce limit

* fix integration

* clean

* clean

* clean

* clean

* clean

* exclude tx pool unwind

* exclude tx pool unwind in integration

* fix integration tx pool

* fix commit

* fix current stage after unwind

* fix current stage after unwind

* fix linter

* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.

* move body and headers unwind out of tx

* fix unwind order after reboot

* add support external tx to exec stage

* clean

* clean

* clean

* clean

* clean

* add logs

* better id check

* better id check
2020-08-26 07:02:10 +01:00
Alex Sharov
325785aba4
add net namespace (#974) 2020-08-25 20:04:01 +07:00
Alex Sharov
c61f8a5bdb
After reboot unwind stack has wrong order (#972)
* fix unwind order after reboot

* Fix test

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-08-24 18:15:22 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Alex Sharov
43a4e34a6e
[to discuss] exec blocks without ObjectDb: Mutation over TxDb (#947)
* use mutation over tx

* clean

* clear

* add .CommitAndBegin() method

* clean

* increase timings for logging

* return ideal batch size
2020-08-24 12:07:59 +01:00
Giulio rebuffo
2cd825ab63
added message for --rpc (#968)
* added message for --rpc

* more messages for other unused flags
2020-08-24 07:32:14 +01:00
Igor Mandrigin
9b81829616
etl: create a subfolder in datadir for temp files. (#965) 2020-08-23 10:53:01 +01:00
Alex Sharov
2facf92533
Better logs2 (#964) 2020-08-23 06:55:17 +07:00
Giulio rebuffo
c2b3d137f7
implemented eth_getblockbyhash in rpcdaemon (#963) 2020-08-22 21:13:38 +01:00
Alex Sharov
a6be18b915
ticker-based logs (#954)
* timer-based logs

* timer-based logs

* delegate progress calculation to user

* delegate progress calculation to user

* delegate progress calculation to user

* clear

* add logs to senders recovery

* use default dir in integration

* more logs

* more logs
2020-08-22 12:12:33 +02:00
Igor Mandrigin
18df6cd182
Fix parallel recovery senders (#962)
* fix parallel context creation

* fix a typo
2020-08-22 12:08:47 +02:00
ledgerwatch
7c16160087
Add IntraBlockState copy function to RPC API (#960) 2020-08-21 21:53:23 +01:00
Igor Mandrigin
a119a3a35f
Simplify build system (#955) 2020-08-21 14:38:11 +02:00
Giulio rebuffo
0ffe32ae16
State growth restore (#951)
* new state growth

* lint
2020-08-21 07:32:11 +01:00
Alex Sharov
ebbaa55672
sub-transaction to not call fsync, sub-transactions to not call runtime.LockThread() (#945) 2020-08-21 07:31:43 +01:00
Igor Mandrigin
f46a72bdd9
Print approximate progress for ETL stages (#950) 2020-08-21 07:30:30 +01:00
Igor Mandrigin
fbcf48f29d
Store ethash datafiles in a separate directory. (#949) 2020-08-20 15:59:06 +01:00
ledgerwatch
01f211f3b8
Revert "IH: no retain list (#946)" (#948)
This reverts commit f86ed64288.
2020-08-20 13:37:34 +01:00
Alex Sharov
f86ed64288
IH: no retain list (#946)
* experiment of not using retain list

* better handle sub-tx

* save progress

* don't touch collector

* don't run collector on sub-tx because of `mdb_cursor_put: MDB_TXN_FULL: Transaction has too many dirty pages - transaction too big`

* clear

* clear
2020-08-20 11:42:43 +01:00
Igor Mandrigin
3f81472831
Never store Ethash caches to disk (#944)
* always keep ethash caches in ram

* fix tests
2020-08-20 11:12:51 +01:00
Alex Sharov
f827f86c11
Rpcdaemon as lib 2 (#943) 2020-08-20 10:52:27 +07:00
ledgerwatch
fd7e91a51c
Update skip_analysis (#942) 2020-08-19 20:34:45 +01:00
ledgerwatch
53f7efb4b0
Fix cmd/state flags (#941) 2020-08-19 14:52:03 +01:00
Alex Sharov
1dcc2b141a
Rpcdaemon as lib (#940)
* share config object

* create default config and logger

* move db connection to common func

* move server start to cli package

* clear

* clear

* rename cli to rpc

* use unified SetupLogger func

* make all root flag persistent

* use common flags in different packages

* use common flags in different packages

* move TraceTx method to eth package

* use native slice flags

* create package "turbo"

* disable geth api

* disable geth api

* move more data types to turbo/adapter package

* add support for customApiList

* run more

* run more

* run more

* dog-food

* move DoCall

* move DoCall

* fix tests

* fix test
2020-08-19 12:46:20 +01:00
b00ris
eb46dd68df
Account range testing (#938)
* save state

* save state

* save new timeouts

* compare passed

* save state

* cleanup
2020-08-19 07:33:49 +01:00
Evgeny Danilenko
3b40819444
grpc regenerate (#936)
* grpc regenerate

* use string type for id in net_version
2020-08-18 20:22:49 +03:00
ledgerwatch
09440dccd3
Print memory stats (#934) 2020-08-17 16:27:42 +01:00
Evgeny Danilenko
6976f58f6a
net api service (#933) 2020-08-17 16:27:29 +01:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction (#928)
* drop Bucket object from KV abstraction

* add some non-public methods

* rebase master

* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
7138d6740e
add logging to ih stage (#921)
* add logging to loader

* better logging
2020-08-17 08:24:48 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
820d6d0adf
[merge after release] enable IH sequence feature (#920)
* add logging to loader

* enable IH sequence feature

* clean
2020-08-17 07:46:22 +01:00
Alex Sharov
0e253e7336
lmdb transactions of unlimited size (#918)
* add logging to loader

* use pure tx in etl loading, logs in mutation commit

* clean

* bletter logging and more cleanup

* bletter logging and more cleanup

* increase batch size to 500M

* better batch commit logging

* async fsync

* sync fsync

* sync fsync

* unify logging

* fix corner-case when etl can use empty bucket name

* fix tests

* better logging

* better logging

* rebase master

* remove lmdb.NoMetaSync flag for now

* consistent walk and multi-walk

* clean

* sub tx

* add consistent multi-put

* implement dupsort support in one new cursor method

* clear
2020-08-17 07:45:52 +01:00