Commit Graph

2538 Commits

Author SHA1 Message Date
Artem Vorotnikov
aadb6e47cc
Remove hashed changesets (#1444) 2021-01-19 17:42:26 +07:00
ledgerwatch
7c8bae8591
Implementation of debug_traceCall, add stateDiff to trace_call (#1442)
* Implementation of debug_traceCall

* Fix compile error

* Implementing stateDiff in trace_call - WIP

* fixes to stateDiff, better parsing in rpctest

* Fix lint

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-01-18 11:13:19 +00:00
ledgerwatch
820216b77f
Block body download - more recent updates (#1441)
* Combined

* Fix lint

* Fix lint

* Initial bodydownload package

* Updates

* fix lint and process incoming body

* Fix compile errors

* Use deliveries

* Ready for initial testing

* Better handling of missing temp files dir

* Intermediate testing

* First working code

* Fix races, more efficient request scheduling

* Remove excessive logging

* Fix lint

* Interruptibility, fix lint

* Remove more logging

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-01-15 10:58:04 +00:00
Alex Sharov
7fd1f19435
txdb lazy cursors creation, grpc version up (#1440)
* txdb lazy, grpc version up

* tidy

* fix lint
2021-01-15 09:38:09 +00:00
Alex Sharov
e6f1e43e6f
lmdb: move fsync inside commit (#1433) 2021-01-04 08:17:18 +00:00
ledgerwatch
baafc90bc6
Fixup after merge (#1432)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-01-02 20:22:35 +00:00
b00ris
8db5790838
State snapshot sync (#1417)
* move experiments to new branch&reorganise kv_snapshot

* walk&modify tests

* added delete from snapshot tests

* fmt

* state snapshot debug

* snapshot validation passed. copy state snapshot

* debug

* snapshot cursor.Prev test

* Prev works correct. Added Current check

* add err check

* added walk forward and backward test

* before refactoring

* refactoring

* execution with snapshot debug

* fix

* remove useless test

* before dupcursor implimentation

* tests with prev and delete works

* execution based on state snapshot passed

* remove useless tests

* blocks to 1140000 passed

* clean verifier

* cleanup state generation

* clean verify && seeder

* remove debug code

* tests passed

* fix lint

* save state

* test passed

* fix lint

* add state hash

* fix lint
2021-01-02 19:28:37 +00:00
Alex Sharov
75cb938980
rpcdaemon performance improve - less reading blocks, less allocs on server (#1426)
* rpcdaemon performance improve - less reading blocks, less allocs on server

* rpcdaemon performance improve - less reading blocks, less allocs on server

* rpcdaemon performance improve - less reading blocks, less allocs on server

* rpcdaemon performance improve - less reading blocks, less allocs on server

* don't use global variables

* df

* avoid use non-transactional db instance to reduce amount of cursors
2021-01-02 19:28:22 +00:00
Alex Sharov
fa1d92bd15
Rm zstd (#1427) 2020-12-28 22:14:16 +07:00
Alex Sharov
d7d9ce0a8e
fix go vet (#1425) 2020-12-26 10:18:16 +07:00
Alex Sharov
311fb591c4
fix check index (#1424) 2020-12-26 09:01:00 +07:00
Artem Vorotnikov
8c0ecaeea9
Remove StageData (#1419) 2020-12-21 11:01:37 +00:00
Chase Wright
9db5debc6c
Fix order of print to match stage numbers (#1416)
IntermediateHashes and HashState are flipped
2020-12-17 21:11:54 +07:00
ledgerwatch
3287b44c2e
Bad unwind of recreation of CREATE2 contract - unit test and potentia… (#1412)
* Bad unwind of recreation of CREATE2 contract - unit test and potential fix

* Fix unit test

* Fix lint

* Fix lint

* Fixup hack

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-16 17:16:37 +00:00
Alex Sharov
7262fd503b
Mdbx devel 5 (#1413)
* tx

* fix tests

* fix tests

* fix tests

* sequence
2020-12-16 14:35:14 +00:00
ledgerwatch
71ef201c17
Extract bodies utility (#1409)
* Extract bodies

* Print headers too
2020-12-14 12:09:59 +00:00
Thomas Jay Rush
76e244f24f
Adding specific Postman testing for tracing (#1402)
* Adding specific Postman testing for tracing

* Updating tests
2020-12-14 11:28:03 +00:00
ledgerwatch
0953fd42cb
WIP trace_call implementation (#1399)
* trace_call initial

* Fix tests

* More tracing

* Add more fields to the action

* Completed first example query

* Add initial bench11 to compare trace_call with OpenEthereum

* Exclude precompile calls from call traces

* Add self-destruct, call types, more comparison in rpctest

* Support for execution errors

* Stack underflow error and delegatecall value

* Fix lint

* Fix suicide traceAddress, Bad instruction error

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-14 11:27:52 +00:00
ledgerwatch
2cc36dac87
further cleanup, unit tests for RPC daemon and tracing improvements (#1392)
* Cleanup and rpcdaemon unit tests

* Fix

* Fix

* Fix lint

* Test for debug_traceTransaction

* Add NoRefunds option

* Compile fix, test for no refunds

* Fix compile

* Add poly contract, fix compile errors

* No refunds now work

* Fix NPE in rpcdaemon

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-09 18:24:08 +00:00
ledgerwatch
5ea590c18e
State cache switching writes to reads during commit (#1368)
* State cache init

* More code

* Fix lint

* More tests

* More tests

* More tests

* Fix test

* Transformations

* remove writeQueue, before fixing the tests

* Fix tests

* Add more tests, incarnation to the code items

* Fix lint

* Fix lint

* Remove shards prototype, add incarnation to the state reader code

* Clean up and replace cache in call_traces stage

* fix flaky test

* Save changes

* Readers to use addrHash, writes - addresses

* Fix lint

* Fix lint

* More accurate tracking of size

* Optimise for smaller write batches

* Attempt to integrate state cache into Execution stage

* cacheSize to default flags

* Print correct cache sizes and batch sizes

* cacheSize in the integration

* Fix tests

* Fix lint

* Remove print

* Fix exec stage

* Fix test

* Refresh sequence on write

* No double increment

* heap.Remove

* Try to fix alignment

* Refactoring, adding hashItems

* More changes

* Fix compile errors

* Fix lint

* Wrapping cached reader

* Wrap writer into cached writer

* Turn state cache off by default

* Fix plain state writer

* Fix for code/storage mixup

* Fix tests

* Fix clique test

* Better fix for the tests

* Add test and fix some more

* Fix compile error|

* More functions

* Fixes

* Fix for the tests

* sepatate DeletedFlag and AbsentFlag

* Minor fixes

* Test refactoring

* More changes

* Fix some tests

* More test fixes

* More test fixes

* Fix lint

* Move blockchain_test to be able to use stagedsync

* More fixes

* Fixes and cleanup

* Fix tests in turbo/stages

* Fix lint

* Fix lint

* Intemediate

* Fix tests

* Intemediate

* More fixes

* Compilation fixes

* More fixes

* Fix compile errors

* More test fixes

* More fixes

* More test fixes

* Fix compile error

* Fixes

* Fix

* Fix

* More fixes

* Fixes

* More fixes and cleanup

* Further fix

* Check gas used and bloom with header

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-12-08 09:44:29 +00:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Alex Sharov
27a24746b7
remove GetLogsByHash (#1390) 2020-12-04 10:27:24 +00:00
Thomas Jay Rush
362e9a1773
Update docs testing (#1385)
* Remove unused files

* Updating testing definition

* Updating auto-generated docs

* Update go code to make agree with silkrpc testing
2020-12-04 10:24:49 +00:00
Igor Mandrigin
63ec2ec9f1 post-rebase fixes 2020-12-03 18:59:17 +01:00
rene
040dcebf57 cmd/devp2p/internal/ethtest: add correct chain files and improve test output (#21782)
This PR replaces the old test genesis.json and chain.rlp files in the testdata
directory for the eth protocol test suite, and also adds documentation for
running the eth test suite locally.

It also improves the test output text and adds more timeouts.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	cmd/devp2p/internal/ethtest/suite.go
2020-12-03 18:59:17 +01:00
Felix Lange
6dee5bbdb5 cmd/devp2p, internal/utesting: implement TAP output (#21760)
TAP is a text format for test results. Parsers for it are available in many languages,
making it easy to consume. I want TAP output from our protocol tests because the
Hive wrapper around them needs to know about the test names and their individual
results and logs. It would also be possible to just write this info as JSON, but I don't
want to invent a new format.

This also improves the normal console output for tests (when running without --tap).
It now prints -- RUN lines before any output from the test, and indents the log output
by one space.
# Conflicts:
#	cmd/devp2p/discv4cmd.go
#	cmd/devp2p/discv5cmd.go
#	cmd/devp2p/rlpxcmd.go
2020-12-03 18:59:17 +01:00
Martin Holst Swende
39c494f4f3 utils, params: add yolov2 bootnode 2020-12-03 18:59:17 +01:00
Felix Lange
eb955580c0 cmd/devp2p/internal/ethtest: update test chain (#21742)
The old one was wrong in two ways: the first block in chain.rlp was the
genesis block, and the genesis difficulty was below minimum difficulty.

This also contains some other fixes to the test.
2020-12-03 18:59:17 +01:00
Igor Mandrigin
5d98f8d40d post-rebase. fix imports 2020-12-03 17:37:38 +01:00
Martin Holst Swende
ca3d54a32c all: implement EIP-2929 (gas cost increases for state access opcodes) + yolo-v2 (#21509)
* core/vm, core/state: implement EIP-2929 + YOLOv2

* core/state, core/vm: fix some review concerns

* core/state, core/vm: address review concerns

* core/vm: address review concerns

* core/vm: better documentation

* core/vm: unify sload cost as fully dynamic

* core/vm: fix typo

* core/vm/runtime: fix compilation flaw

* core/vm/runtime: fix renaming-err leftovers

* core/vm: renaming

* params/config: use correct yolov2 chainid for config

* core, params: use a proper new genesis for yolov2

* core/state/tests: golinter nitpicks
# Conflicts:
#	cmd/evm/internal/t8ntool/execution.go
#	cmd/geth/chaincmd.go
#	cmd/geth/consolecmd.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	core/state/statedb.go
#	core/state/statedb_test.go
#	core/vm/evm.go
#	core/vm/interpreter.go
#	core/vm/logger.go
#	core/vm/runtime/runtime.go
#	eth/api_tracer.go
2020-12-03 17:37:38 +01:00
ledgerwatch
9a233da93d
Add test for getChainConfig (#1375)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-28 15:08:17 +00:00
Andrew Ashikhmin
8a877c227a
Integrate Silkworm execution (#1344)
* Expose C Handle of lmdbTx

* LoadExecutionFunctionPointer

* silkworm ExecuteBlocks

* linter

* CLI flag for SilkwormExecutionFunc

* linter

* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage

* Commit transaction after Silkworm execution

* Fix batch initialization

* Make batch_size uint64_t instead of size_t

* max_block in silkworm_execute_blocks

* More accurate logProgress

* Use CHandle exposed by lmdb-go

* Enable Silkworm on Linux only

* Update lmdb-go

* Error when attempting to use Silkworm not on Linux

* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go

* Integrate Silkworm into cmd/integration

* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution

* Refactor SpawnExecuteBlocksStage

* linters

* Small clean-ups

* Move ChangeSetHook inside executeBlockWithGo

* No need to tamper with batch size in executeBlocksWithSilkworm

* fix

* Fix

* Fixup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-28 15:08:02 +00:00
Alex Sharov
4565bd43c1
Mdbx v0.9.2 (#1373)
* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* up devel

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* squash

* switch to master

* mdbx 0.9.2

* mdbx 0.9.2

* %<<$!%%%%%%<o'{}k.###

* %<<$!%%%%%%<o'{}k.###

* merge

* merge

* merge

* merge

* ad

* ad

* ad

* ad
2020-11-28 14:26:28 +00:00
Alex Sharov
67de2a3a46
To mdbx dupfixed (#1371) 2020-11-23 11:18:21 +07:00
Alex Sharov
ccd1ad5832
Sequence in to mdbx (#1370) 2020-11-23 11:15:43 +07:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
Chase Wright
a8b7a59977
Update rpcdaemon README with eth_(un)subscribe (#1365)
* `eth_subscribe` is supported over WebSocket
* Only `newHeads` is supported at this time
* `eth_unsubscribe is supported
2020-11-19 15:39:13 +01:00
lightclient
377b979c98
Accept discover,netrestrict,staticpeer arguments for sentry (#1364)
* accept discover,netrestrict,staticpeer arguments for sentry

* appease go linter
2020-11-19 08:14:41 +00:00
ledgerwatch
1a0e78a2e2
Remove changeset-based decorators from WalkAsOf (#1356)
* Remove decorators

* Current changes

* Fix and skip some tests for now

* Fix lint

* Specialise WalkAsOf

* Fix compile errors

* Fix lint

* Scan index files for key sizes

* More cleanup

* Fix to remote server

* Fix for walkasofstorage

* Fix/skip tests

* Fix tests

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2020-11-18 14:58:52 +00:00
Igor Mandrigin
393c9965ae
rpcdaemon: subscriptions, newHeads (#1359)
* fix `make grpc` on new checkouts

* update proto files

* add some stub

* prototype with fake events

* notifying about events

* pass events

* events are being sent

* transfer headers to filters

* create the “filters” struct

* implement new heads

* PoC of New Heads subscription

* fix keep alive

* fixups for the client

* add “type” to the event

* support header event type on client

* better stage refactor

* fixup for the eth backend

* fixups

* fix tests

* fix tests

* fix linters

* address comments

* remove unused log
2020-11-17 19:13:41 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00
ledgerwatch
10f338b8f7
Add lmdb guide visualisations (#1348)
* Add lmdb guide visualisations

* updated names, remove double-tap visualisations
2020-11-11 17:18:34 +00:00
ledgerwatch
e71c3cae70
Lmdb visualisation + freelist stability fixes (#1345)
* Visualisations

* Use graphviz records

* Fix lint

* Fix lint

* Reduced leaf pages

* Fix

* Overflow pages

* Fix lint

* Fix lint

* Fix lint

* Add dupsort generate

* DupSort experiments

* Add dupsort nodes

* More interesting dupsort

* freelist

* Fragmented freelist

* Fix lint

* drop T2 too

* Handle overflow pages in FREE_DBI

* Skipping in the MAIN_DBI leaves

* Switch to double-tap LMDB

* Fix to double-tap

* With/without reader visualisations

* Flip flopping without reader

* Tidy mods

* Fix lint, switch to new lmdb-go release

* mod tidy
2020-11-11 09:35:58 +00:00
Igor Mandrigin
ed9672620b
Granular rpc control (Allow list for RPC daemon) (#1341) 2020-11-10 10:08:42 +01:00
Thomas Jay Rush
546b91f47e
Updating RPC tests in Postman (#1340) 2020-11-09 09:52:18 +01:00
Igor Mandrigin
c0f9b57b42
rpcdaemon: read chainConfig from the DB (remove hardcode) (#1339)
* read chainConfig from the db

* get dbtx properly
2020-11-08 06:46:53 +01:00
Igor Mandrigin
ab878ed59a
Ethstats (#1338) 2020-11-06 08:59:50 +01:00
alex.sharov
5ec296a206 squash 2020-11-04 10:05:55 +07:00
ledgerwatch
8c376082ff
[WIP] Integrate header download with stage 1 (saving headers to the database) (#1330)
* move locking to the hd object itself, create ready channel

* Fix lint

* More integration

* Fix lint

* Connect to processing

* Set hd.files at recovery

* Serialise processSegment

* Fix lint

* Not prepend filesDir

* Write to the database

* Fix lint

* Set up genesis block

* Fix hard coded tips

* Fix chainbreak false alarm

* Recover from DB

* Skip non-header records

* Not initialise if recovered from DB

* Fix lint
2020-11-02 21:09:12 +00:00
Alex Sharov
a72543bc16
Stop dial scheduler asap (#1334)
* stop dial scheduler asap

* stop dial scheduler asap
2020-11-02 21:07:26 +00:00
ledgerwatch
cceb8c2c3a
Sharded state experiment based on call_traces stage run via integration (#1329)
* Initial work on state shards

* Fix compile errors

* Intermediate

* Implement dispatcher

* Implemented sharded reader

* No cache

* Print comms

* No prepopulation

* Print dispatcher activity

* Print more

* Print errors

* Fix dispatcher

* Remove logging

* Dispatcher to enforce initial barrier

* Don't print that much

* Turn off cache pre-seeting for state sharding

* Artificial latency, fixed number of blocks to execute

* Fix
2020-10-31 11:51:56 +00:00
Horacio Mijail Antón Quiles
ed1819ec58
Opcode tracer (#1320)
* OpcodeTracer

* Cleanup, rename to opcode_tracer

* Bring back check_change_sets, remove repeated vars

* Cleanup, gofmt

* Linter, cleanup

* Rename "segments" to "basic blocks", "bblocks"

* gofmt

* Lintci madness
2020-10-30 19:44:00 +00:00
Suhabe Bugrara
b18727911f
CFG analysis (#1327)
* First

* More on SA interpreter

* Fixup

* Add cfg action to hack binary that invokes the SaInterpreter. Added an operation handler for PUSH1

* refactor cfg tests into separate file

* Move cfg tests into separate file

* More refactoring into new file

* dataflow interpreter

* work on cfg0

* finish cfg0

* df works on base examples

* refactor into dataflow spec

* add bounded stack

* add harder example

* fix switch pass thru

* fix switch pass thru

* bug fix, and better printing

* manual merge

* restore call to test gencfg

* abstract interpretation based cfg analysis

* fix post signature

* use uint256 instead uint64, add post function

* preprocess stmts

* initial implementation of resolve

* fix resolve

* fix resolve

* print stmts for edges

* print stmts for edges

* print states

* print states

* bug fixes, debugging

* fix jumpi dest - first working impl

* reachability analysis to filter out dead edges

* add all transfer functions

* larger contract bytecodes from solc compiler

* simple solidity contract goes thru

* add deposit contract bytecode

* rename deposit contract test

* fix new contract arg

* Address non-determinism leading to imprecise results

* improve debugging output

* improve debugging output

* improve debugging output

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* fix for bug causing incorrect analysis results

* add more test cases

* fix coverage bug

* debugging for non-termination

* fix bad fixpoint check

* fix data inference

* fix transfer function for halting stmts

* switch to deposit contract test, disable debugging

* add anly counter to viz, fix stmt.valid check

* show all preds, adjust anlycounter behavior

* dfs instead of bfs to fail earlier

* viz improvements

* add worklist size to viz

* add test case for private functions

* valueset analysis

* add more checks to fail earlier in the analysis to help debugging, improve debugging output, catch additional bad jumps

* delete old code

* delete old code

* delete old code

* fix up minor changes to jump table

* copy over comments from cgf-1 branch

* remove minor diffs

* add recompiled deposit contract

* graph viz

* cleanup/refactoring

* initial impl of viz

* script to run cfg anly and generate dot file

* div example

* accept bytecode from cmd line

* add minimal deposit contract example

* replace valueset analysis with stackset analysis

* get in sync with master

* sync with master

* fix linting

* fix linting

* fix linting

* reformatting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* harness for running over all contracts

* refactor anly, track coverage metrics

* breakdown unresolved into different types, fix bad opcode bug

* sort programs by frequency

* ingest used contracts from bigquery

* performance, concurrency, bug fixes

* more test cases, handle invalid jumps differently, remove duplicate edges, report analytics limit

* simplify concurrency

* correctly track short stack

* add new transfer function, fix stack len

* variable stack length, perf opts, inc anly count limit

* profiling

* test case for large state size

* use custom hash function for control

* timeouts

* cfg.sh

* increase to 5 min timeout

* track elpased time

* use ptr

* increase limits

* increase limits

* fix mem leak

* debug mem leak

* debug mem leak

* lower resource limits

* fix nil error

* add new lattice element

* re-enable

* cut down limits

* preliminary proof checker

* refactor batch mode to run cfg in subprocess,put memory limit

* remove hard wiring

* adjust limits

* update metrics tracking

* more succinct proof checker

* rewrite checker

* bug fixes on checker

* bug fix

* remove print stmts

* track proof size

* print proof size

* don't panic on process error

* compress proof

* go mody tidy

* code formatting

* fix capitalization

* fix linting

* fix linting

* fix linting

* fix linting

* fix linting

* remove unnecessary files

* fix typo

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-30 12:24:14 +00:00
Horacio Mijail Antón Quiles
f8477e29ab
Move batch management into the writeReceipts case (#1321) 2020-10-30 08:41:35 +00:00
ledgerwatch
97a03a1ad7
[WIP] add interfaces subtree and switch headers POC to using shared proto files (#1315)
* Squashed 'interfaces/' content from commit c4e14d6dc

git-subtree-dir: interfaces
git-subtree-split: c4e14d6dc80a157c60acd42f0152d631d250601e

* Squashed 'interfaces/' changes from c4e14d6dc..84c5faa5f

84c5faa5f Update README.md
07db484a1 Update README.md
ea14fa4e3 Update README.md

git-subtree-dir: interfaces
git-subtree-split: 84c5faa5f956ea52119911f39ac5864b282dcec2

* Change proto paths

* Squashed 'interfaces/' changes from 84c5faa5f..eb9859d41

eb9859d41 Correct syntax
ae78fe955 Add go package

git-subtree-dir: interfaces
git-subtree-split: eb9859d4157351fac3b8bd2f477307a09978e6df

* Switch to shared proto files

* Fix generation

* Remove duplicate files

* Init hard-coded tips when not recovering

* Only check initiation when extending up
2020-10-29 16:38:46 +00:00
Alex Sharov
399db7ee3b
LMDB v0.9.27, and add all lmdb cli tools (#1303)
* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* mdb tools

* clean

* clean

* clean

* clean

* clean

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:46 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
ledgerwatch
ea060815ef
Remove z3 and semantics (#1314)
* Removed z3 submodule

* Remove semantics and z3 from build

* Fixes to CI
2020-10-28 17:52:00 +00:00
Alex Sharov
a139f31dcd
Exclusive lock manual (#1302)
* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* exclusive lock for chaindata folder

* mdb tools

* resolve abs path

* rpcdaemon to check exclusive lock

* rpcdaemon to check exclusive lock

* exclusive lock for mdbx

* exclusive lock for mdbx
2020-10-28 09:52:15 +00:00
ledgerwatch
722c4562f0
Some additions to the DB walkthrough (#1309)
* Fix tables

* Some corrections to DB walkthrough
2020-10-28 07:18:48 +00:00
Alex Sharov
531d36efcb
mdbx support (#1235) 2020-10-28 10:18:10 +07:00
b00ris
d2d5da8b09
add snapshot support for rpcdaemon (#1304) 2020-10-27 22:31:47 +00:00
Igor Mandrigin
9ad6978259
Refactor flags + fix logs (#1306) 2020-10-27 16:53:49 +01:00
Alex Sharov
60e9ea187a
pics: set IH comparator (#1300)
* set IH comparator

* set IH comparator

* Fix chain config

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-27 10:35:32 +00:00
Igor Mandrigin
0f1b2f3595 geth-1.9.23: post-rebase fixups 2020-10-26 17:16:00 +01:00
Felix Lange
0c54d2e649 p2p/discover: implement v5.1 wire protocol (#21647)
This change implements the Discovery v5.1 wire protocol and
also adds an interactive test suite for this protocol.
# Conflicts:
#	cmd/devp2p/discv5cmd.go
#	p2p/discover/v5_encoding.go
#	p2p/discover/v5_encoding_test.go
#	p2p/discover/v5_udp.go
#	p2p/discover/v5_udp_test.go
#	p2p/discover/v5wire/session.go
2020-10-26 17:16:00 +01:00
Giuseppe Bertone
c690231219 cmd/geth, cmd/utils: fixed flags name (#21700)
# Conflicts:
#	cmd/geth/main.go
#	cmd/geth/usage.go
2020-10-26 17:16:00 +01:00
rene
d25a8f7282 cmd/devp2p/internal/ethtest: improve eth test suite (#21615)
This fixes issues with the protocol handshake and status exchange
and adds support for responding to GetBlockHeaders requests.
# Conflicts:
#	cmd/devp2p/internal/ethtest/suite.go
#	cmd/devp2p/internal/ethtest/types.go
2020-10-26 17:16:00 +01:00
Felix Lange
adb60a3a71 cmd/faucet: enable DNS discovery for known networks (#21636)
# Conflicts:
#	cmd/faucet/faucet.go
#	cmd/utils/flags.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
328becb968 accounts, signer: implement gnosis safe support (#21593)
* accounts, signer: implement gnosis safe support

* common/math: add type for marshalling big to dec

* accounts, signer: properly sign gnosis requests

* signer, clef: implement account_signGnosisTx

* signer: fix auditlog print, change rpc-name (signGnosisTx to signGnosisSafeTx)

* signer: pass validation-messages/warnings to the UI for gnonsis-safe txs

* signer/core: minor change to validationmessages of typed data
2020-10-26 17:16:00 +01:00
mr_franklin
bcf31729f7 cmd/bootnode,internal/debug: fix some comments (#21623) 2020-10-26 17:16:00 +01:00
ledgerwatch
7554428884
POC of header downloader - splitting into 2 processes: sentry + downloader (via gRPC) (#1291)
* Splitting sentry and downloader - the beginning

* A bit more

* More on sentry

* More gRPC

* Sentry and downloader separated

* Update binding for stable version of grpc

* Better bufferSize flag

* Fix lint

* Send pelanties

* Fix lint

* Remove hard-coded tips on connect

* Tidy the logs a bit

* Deal with hardTips on Recovery

* Print hard tips

* Hide empty anchors

* Request headers after receiving a message

* Better waking up

* Print hard-coded block numbers

* Print outgoing requests

* Debug logging

* In the middle protection

* Sentry not to lose peers when core disconnects
2020-10-25 22:05:37 +00:00
Evgeny Danilenko
b6687fc9aa
initial (#1296) 2020-10-25 21:34:00 +00:00
ledgerwatch
dd2c047cdf
[WIP] Defrag utility for LMDB database (#1268)
* Initial

* Read freelist pages

* Fix

* Fix lint

* Fix lint

* Fix lint
2020-10-25 10:10:55 +00:00
Alex Sharov
6f27e3043b
lmdb_readonly_dont_create_folder (#1293) 2020-10-25 08:39:09 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Ricardo Guilherme Schmidt
e9572ae981
Fix infinite rpc.gascap on eth_estimateGas (#1292)
* Fix infinite rpc.gascap on eth_estimateGas

* use `gasCap.Sign() > 0` instead of `gasCap.Uint64() != 0`
2020-10-25 08:19:59 +00:00
Thomas Jay Rush
1af3e09d75
1209 automate documentation (#1290)
* Automate documentation

* Updating

* Updating

* Fixing lint

* Updating readthedocs documentation

* Updating testing

* Updating testing
2020-10-24 18:03:52 +01:00
Alex Sharov
e85b388061
Protect from wrong class use: make ethdb.Tx incompatible with rawdb.DatabaseReader (#1288)
* protect_from_wrong_class_use

* more cases
2020-10-24 09:09:20 +01:00
Alex Sharov
77a4b3305d
rawdb to return errors: part 1 (#1286)
* ReadBlockByNumber and ReadChainConfig return error

* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:

ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
2020-10-24 07:57:09 +01:00
Alex Sharov
6584b43b12
Allow start read transactions from ethdb.Database interface (#1287)
* allow_read_transactions

* fix lint

* fix lint

* more cases
2020-10-24 07:55:43 +01:00
Alex Sharov
5e53c79f50
stable version of grpc codegen, drop kv_remote_v1 support (#1285) 2020-10-24 07:54:03 +01:00
alex.sharov
c043a7c876 prometheus docs for developers 2020-10-24 10:15:53 +07:00
Alex Sharov
76f1b05cb2
Move tmpdir definition to app-start, move migrations folder inside tmpdir (#1282)
* extract tmpdir to app-level-code

* extract tmpdir to app-level-code

* save progresss
2020-10-23 12:18:45 +01:00
Alex Sharov
dd5258d524
save progresss (#1283) 2020-10-23 11:52:18 +01:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274)
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
2020-10-21 18:01:40 +01:00
alex.sharov
ee0abb3095 improve prometheus docs 2020-10-21 11:24:22 +07:00
alex.sharov
761af4f372 rebase master 2020-10-21 11:23:22 +07:00
alex.sharov
67c295ef15 rebase master 2020-10-21 11:22:25 +07:00
Thomas Jay Rush
2aa6fb50ca
Cleans up some custom RPC commands (#1270)
* Moving a few files to be more consistent

* Collecting together custom code under tg_ namespace

* Small cleanups

* Updating test cases

* Uncomment eth_gasPrice but still unimplemented
2020-10-20 22:16:28 +01:00
Alex Sharov
5dbe1724be
Bitmap etl (#1227)
* save progress

* save progress

* save progress

* save progress

* etl bimaps

* etl bimaps

* etl bimaps

* etl bimaps

* a

* a

* a

* a

* a

* a

* a

* save progress

* save progress

* save progress

* a

* fix_race_condition_on_zstd_build

* clean

* clean

* clean

* clean

* clean

* go mod tidy

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* fix corner case

* commit every

* call traces etl

* call traces etl

* call traces etl

* return missed error

* go mod tidy

* go mod tidy

* go mod tidy

* rebase to master

* remove trash

* print full key if it has 8 leading zeroes
2020-10-19 14:11:01 +01:00
Evgeny Danilenko
3bec75cdc4
Extract nonstandard APIs and introduce tg namespace (#1248)
* initial

* update rpc deamon readme

* use read-only transactions
2020-10-18 20:44:28 +01:00
ledgerwatch
d2e5cc5cb5
Better migration - receipts_cbor_encode not to increase size of the database (#1250)
* Add print_migration command to integration

* Not run migrations for print commands

* Reverse migration

* Fix lint

* Add datadir to run_migrations

* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Fix

* receipt sizes

* fix print

* Switch to another version of lmdb

* Collector from existing files

* Not load

* Commit after clearing

* Add methods for clear/drop buckets by multiple transactions

* Fix to lmdb

* Not remove temp files

* Not load

* Re-enable loading

* Prints

* Try to fix refill

* Migration in 3 steps

* tidy mod

* Fix lint

* Fix tests

* Migrations for CBOR receipts in 3 txs

* Fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-17 14:00:12 +01:00
b00ris
e5f6412da2
Update snapshot hash (#1243)
* remove useless bucket

* update snapshot hashes

* remove log
2020-10-17 08:48:20 +01:00
Evgeny Danilenko
946d4fac5b
ForkID api (#1244)
* forkID api

* merge fork calls

* optional param
2020-10-16 15:15:10 +03:00
Thomas Jay Rush
c6c82b7569
Finish moving functions to consistent locations to ease testing/documentation of RPC (#1241)
* Cleaning up comments to start

* Renaming get_receipts.go to eth_receipts.go

* Stubbing out RPC mining routines

* Move block-related commands to eth_block.go

* Move transaction-related commands to eth_txs.go

* Move account-related commands to eth_accounts.go

* Move system-related commands to eth_system.go

* Final cleanup

* Updating testing
2020-10-14 16:59:42 +01:00
Alex Sharov
a8f38ba1c3
fix db opening (#1240) 2020-10-14 13:05:54 +01:00
Thomas Jay Rush
dcdcc1a30f
Fixes issue #1232 (#1233)
* Fixes issue #1232

* Adding/updating test cases

* Updating as per comments

* Use tx instead of db in GetModifiedAccounts and clean up a bit

* option to specify LMDB map size for cmd/integration commands (#1238)

* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Use mapSize for first opening

* Fix for cobra

* Extend lmdb.mapSize for all commands

* Fix lint

* Fixes issue #1232

* Adding/updating test cases

* Updating as per comments

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-10-14 12:01:44 +01:00
ledgerwatch
234e35851c
option to specify LMDB map size for cmd/integration commands (#1238)
* Add lmdb.mapSize to integration

* Make lmdb.mapSize effective

* Use mapSize for first opening

* Fix for cobra

* Extend lmdb.mapSize for all commands

* Fix lint
2020-10-13 23:42:04 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00
Igor Mandrigin
ef2353c0e0 turbo-api: custom state writers 2020-10-13 13:35:48 +02:00
Thomas Jay Rush
d6587ba250
Simple file renaming to make automated testing easier (#1230)
* Simple file renaming to make automated testing easier

* Fixing comment
2020-10-12 15:17:34 +01:00
Igor Mandrigin
f3ce1ce423
turbo-api: add a plug-in mechanism for possible Bloom and other filters (#1229)
* Add a convenience method to replace stages

* allow more parameters to the execution function

* add a plug-in point for stuff like bloom filters, etc

* add comments

* fixups
2020-10-12 14:58:45 +01:00
Thomas Jay Rush
1bd46089e9
Start of re-working of RPC code to align with testing better (#1225)
* Starting process of rearranging RPC code

* First version of automated testing

* Putting RPC docs in their own folder

* Fixing build

* Making varible names more consistent

* Adding deprecated RPC commands for db_

* Adding deprecated RPC commands for shh_

* Adding deprecated RPC commands for eth_
2020-10-12 09:39:33 +01:00
ledgerwatch
b747ab5324
[WIP] CallTraces index (#1157)
* Initial commit for CallTraces index

* Fix compilation

* fix lint, add comment

* Fix integration

* Add Close function to ethdb.Cursor, fix some compile errors

* Try to stop cursor leak in Get

* Fix compile errors in RPC daemon

* Fix compile errors

* fixing another way

* Some fixes

* More fixes

* More fixes

* More fixes

* Fixes to core/state

* Fix lint

* Fix lint

* Fixes

* Stage caching for call trace stage

* Add mem stats

* Try to stop the leak

* Turn off debug

* Chunks for 10k blocks

* Print

* Revert "Print"

This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.

* Revert "Chunks for 10k blocks"

This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.

* Trying to fix the leak

* Don't compute receipts in re-tracing

* Not compose block

* Print speed, fix receipts, bigger caches

* Fix lint

* Utilise changeset info

* Counters

* Use NoReceipts and ReadOnly

* ReadOnly is incompatible with caching

* Skip test leaking transactions

* Fix block test

* Change disable message for call-traces stage

* Use block option for call traces integration

* Fix retracing due to incarnation
2020-10-12 09:39:04 +01:00
Thomas Jay Rush
2f0a9d666c
Fixes 'too many other files opened' problem with Generate Storage History stage on Mac (#1217)
* Added back fdlimit to increase number of file descriptors

* Fixing a test file that fails on Mac with too few file descriptors available

* Trying to fix ci

* Trying to fix ci

* Trying to fix ci 3

* Fixing ci
2020-10-10 22:15:07 +01:00
Alex Sharov
fc3cd4d5c3
Rpcdaemon: private api V2 (#1214)
* save progress

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown json server

* GracefulShutdown json server

* fix lint

* clean

* hack hugeFreelist

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* save progress

* fix_race_condition_on_zstd_build

* fix_race_condition_on_zstd_build

* better close cursor

* save progress

* open read tx in all api methods

* clean

* clean
2020-10-10 13:24:56 +01:00
Alex Sharov
9cf10a8f85
Rpcdaemon: Gracefull shutdown, better grpc stream close (#1196)
* save progress

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown json server

* GracefulShutdown json server

* fix lint

* fix lint

* can

* clean

* test

* test

* test

* increase keepalive timeout, increase vegeta timeout, mark server-side errors

* increase keepalive timeout, increase vegeta timeout, mark server-side errors

* hack hugeFreelist

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* fix_race_condition_on_zstd_build
2020-10-10 07:06:54 +01:00
Alex Sharov
88a099702b
rawdb.ReadCanonicalHash to return err (#1204)
* read_canonical_hash_return_err

* hack hugeFreelist

* fix tests
2020-10-10 07:05:56 +01:00
Alex Sharov
53484d60f2
LMDB: don't search pages sequence in freelist if value is > X (#1213)
* lmdb_no_freepages_sequence_search_for_large_values

* go mod tidy
2020-10-10 07:05:18 +01:00
Alex Sharov
79815e4298
include_lmdb_stat_tool (#1206) 2020-10-08 08:11:36 +01:00
ledgerwatch
d912c9929e
No prefetch on remote cursors in RPC daemon (#1201) 2020-10-07 21:50:03 +01:00
ledgerwatch
1af6cbe37b
Revert "Download/seed headers and block bodies via BitTorrent (#1195)" (#1202)
This reverts commit 94360212cb.
2020-10-07 21:49:44 +01:00
ledgerwatch
94360212cb
Download/seed headers and block bodies via BitTorrent (#1195)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-07 13:15:49 +01:00
Alex Sharov
09d0020669
build docker on ci (#1198)
* build docker on ci

* setup docker

* build docker on ci
2020-10-07 11:12:10 +01:00
ledgerwatch
bdb0155856
Revert "Add sync by bittorrent snapshots (#1160)" (#1194)
This reverts commit 4ebade3583.
2020-10-06 22:31:57 +01:00
ledgerwatch
834eaedc00
pass_root_ctx_to_p2p_server (#1192)
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-06 20:25:01 +01:00
b00ris
4ebade3583
Add sync by bittorrent snapshots (#1160)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-06 20:24:48 +01:00
Igor Mandrigin
70f6ec3b49 post-rebase fixes 2020-10-06 14:12:09 +02:00
rene
8a12d561f2 cmd/devp2p: add eth protocol test suite (#21598)
This change adds a test framework for the "eth" protocol and some basic
tests. The tests can be run using the './devp2p rlpx eth-test' command.
# Conflicts:
#	core/forkid/forkid.go
#	eth/discovery.go
#	eth/handler.go
#	eth/helper_test.go
#	eth/protocol_test.go
2020-10-06 14:12:09 +02:00
Binacs
f4397a3485 cmd/clef, cmd/geth: use SplitAndTrim from cmd/utils (#21579)
# Conflicts:
#	cmd/geth/retesteth.go
#	cmd/utils/flags.go
2020-10-06 14:12:09 +02:00
rene
d57fb49ed3 p2p: move rlpx into separate package (#21464)
This change moves the RLPx protocol implementation into a separate package,
p2p/rlpx. The new package can be used to establish RLPx connections for
protocol testing purposes.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	p2p/rlpx/rlpx.go
#	p2p/rlpx/rlpx_test.go
#	p2p/server_test.go
2020-10-06 14:12:09 +02:00
Marius van der Wijden
1f0014a34e cmd/utils: use preconfigured testnet flags instead of networkid (#21561)
* cmd/utils: use preconfigured testnet flags instead of networkid

* cmd/utils: shorter description

Co-authored-by: Martin Holst Swende <martin@swende.se>

* Update flags.go

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	cmd/utils/flags.go
2020-10-06 14:12:09 +02:00
gary rong
cbe7a5c80a cmd, eth: offer maxprice flag for overwritting price cap (#21531)
* cmd, eth: offer maxprice flag for overwritting price cap

* eth: rename default price cap
# Conflicts:
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	eth/gasprice/gasprice.go
2020-10-06 14:12:09 +02:00
ledgerwatch
9418ebf860
Add some logging to mint (aka block body gap check) (#1189) 2020-10-06 07:36:47 +01:00
ledgerwatch
b652768241
Revert "pass_root_ctx_to_p2p_server (#1179)" (#1182)
This reverts commit b8c3880254.
2020-10-04 17:18:06 +01:00
Alex Sharov
b8c3880254
pass_root_ctx_to_p2p_server (#1179) 2020-10-03 11:15:01 +01:00
Alex Sharov
85219b651e
Revert "rpcdaemon to cache chainconfig (#1173)" (#1177)
This reverts commit cc4dd3695d.
2020-10-03 15:07:49 +07:00
Nikola
ef7c837f2c
Update rpcdaemon README (#1175) 2020-10-02 15:03:53 +02:00
Alex Sharov
cc4dd3695d
rpcdaemon to cache chainconfig (#1173)
* rpcdaemon to cache chainconfig

* topic normalization

* topic normalization
2020-10-02 13:52:02 +01:00
Alex Sharov
809d79b15f
Switch to cbor (#1172)
* switch receipts to cbor

* switch receipts to cbor

* rpcdaemon to cache chainconfig

* rpcdaemon to cache chainconfig

* rpcdaemon to cache chainconfig

* rpcdaemon to cache chainconfig
2020-10-02 13:51:20 +01:00
Alex Sharov
6ccf57dace
switch to go implementation of roaring bitmaps for alpine support (#1165) 2020-10-02 15:16:21 +07:00
Alex Sharov
61fbe34024
Better cbor support (#1163) 2020-10-02 10:56:13 +07:00
Alex Sharov
48cf5fb43e
History-style sharding for bitmap indices (#1159) 2020-10-02 10:54:11 +07:00
Thomas Jay Rush
5ebf997160
Fixes to a few tracing issues (#1161)
* Starting to look at performance

* Fixes issue #1142 - incorrect error strings

* Uses a recent fix in Geth callTracer code to return balance to self-destruct traces
2020-10-01 07:02:11 +01:00
ledgerwatch
7bb9cb4274
Remove BloomStatus from the API (#1155)
* Remove BloomStatus from the API

* Trying to find missing space

* Fix lint
2020-09-29 15:09:29 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124)
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
ledgerwatch
14288b8cd0
Headers poc 5 - Intermediate (#1145)
* Add headers persistence

* Print flushBuffer

* Fix indexing problem

* Not skip hard-coded headers if the files are empty

* Fix lint

* print anchor state after init

* Properly construct file names

* Add check sub-command

* Fix lint

* Fix lint

* Fix lint

* Print more info when recovering

* Fix recovering

* Fix recovery

* Add anchors also as tips

* 2-level priority queue for tips

* Initialise tipQueue in anchor

* update maxTipHeight

* fix type

* Add anchors to the anchorQueue and rebuild anchorQueue on deletion

* Fix NPE

* fix recovery

* User buffersize, add hard coded headers to buffer

* Reinit anchorQueue

* Schedule requests after recovery

* No fix

* Remove duplicates

* Report duplicate headers

* Log buffer additions

* Fix for duplicate headers

* Try to fix duplicate headers again

* remove TODO comment

* Use LLRB instead of heap for anchors

* Print reserveTip info

* Correctly replace anchors in the tree

* Remove excessive logging

* Print tips attached to the anchor

* Print tips better

* limitTips instead of reserveTip

* Print forked headers

* Use pointers in llrb

* Print tipStretch

* Print limitTips

* Mininise AnchorItem

* Put anchors into the tree

* Fix totalDiff calculation, but is it needed?

* Remove totalDifficulty from anchors

* CheckInitiation

* Fix tests

* Fix lint, print more at check initiation

* Better output for hard-coded anchors

* Print hard-coded anchors better

* Prioritise anchors with short stretches

* Prioritise by chainSize

* Use linked list instead of heap for requestQueue

* Fix problem of no requests

* Push front

* Fix lint

* Not verify PoW for too far in the past and future

* Fix Calculation of totalDifficulty when Connect

* Fix hard tips

* Another fix for tips
2020-09-27 21:32:05 +01:00
Horacio Mijail Antón Quiles
252863fedf
Use the passed VMConfig (#1141)
* Use the passed VMConfig

* Explicitly pass the vmConfig into runBlock
2020-09-27 21:31:49 +01:00
Thomas Jay Rush
784bc9dc85 Fixes issue #1139 - crash when running chaindata mode (#1140)
* Fixes issue #1139 - crash when running chaindata mode

* Cleaning up error messages as per feedback
2020-09-27 12:11:05 -04:00
Thomas Jay Rush
5b4445ecea Revert "Fixes issue #1139 - crash when running chaindata mode (#1140)"
This reverts commit 00415eee76.
2020-09-27 12:06:37 -04:00
Thomas Jay Rush
00415eee76
Fixes issue #1139 - crash when running chaindata mode (#1140)
* Fixes issue #1139 - crash when running chaindata mode

* Cleaning up error messages as per feedback
2020-09-26 22:01:11 +01:00
ledgerwatch
4be6174a15
Add comment about ContractAddress in receipts, followed by some simplification (#1137)
* Add comment about receipts

* Panic on error

* Ignore errors in the simulated backend

* Fix lint
2020-09-26 13:38:10 +01:00
Thomas Jay Rush
c0a03ad728
Responses to comments on previous tracing PR (#1138)
* Responses to comments on previous tracing PR

* Cleanup

* Cleanup
2020-09-26 07:41:34 +01:00
Thomas Jay Rush
b437fab006
[WIP] Tracing enhancements (#1133)
* Adding some testdata for RPC tracing

* Adding command line option to allow switching trace export format

* Moved types into a separate file for clarity

* Added endpoints and stub functions for other trace routines

* Added experimental end points for issuance, blockReward, and uncleReward

* Moved supporting functions for filtering to _filtering file

* Moved Filter function to _filtering file

* Reordering functions for clarity

* Cleaning up

* Implemented trace_get - 7 of 8 tests pass

* Implemented trace_transaction - 2 of 8 tests pass

* Updating test cases

* Turning off lint temporarily. Will be turned back on

* Extended trace_filter (parity traces, reward traces) - 10 of 13 tests pass

* Implemented trace_block - 4 of 6 tests passed

* Cleanup test cases

* Added refundAddress and selfDestructedAddress to self destruct traces

* Total hack fix to missing gasUsed in some tests. Fixes tests, needs to be replaced
2020-09-25 13:12:36 +01:00
ledgerwatch
28e2b45f5e
Return zero balance for non-existent accounts instead of crashing (#1135) 2020-09-23 20:18:23 +02:00
Igor Mandrigin
65d8a12bb1
turbo-api: Add docs to some public structs, methods and fields (#1127) 2020-09-21 16:10:25 +02:00
ledgerwatch
5aab794c9b
Headers poc 4 - intermediate changes (#1125)
* Process BlockHeadersMsg

* Print invalidations

* Print reason for parent-child mismatch

* Fix off by one error

* Fix ExtendDown

* Replace tipLimiter with an LRU cache

* Retrieve 256 headers at a time

* Hard coding of tips, process new block hashes

* extract headers

* Write hard-coded headers into a file

* Add total difficulty

* Insert hard coded headers

* Log i

* Initiate download from hard-coded anchors

* Print the state of anchors

* Add hard-coded headers as tips

* Spare the peers

* Refine LRU caching of tips

* Print length of anchor trees

* Limit requests to 192 headers

* range error fix

* Print better, retain at least one tip in the LRU cache

* Skip duplicate line ends

* Throttle requests, display peer name

* Back off, intermediate tip queue

* Print block header timestamp

* Better logging

* Log less

* More logging of evictions

* Fix queue push

* Fix queue pop

* Exclude duplicate tips

* Fix tips logging

* Fix moving

* Add port flag

* Check headers

* Print extra

* Add mixDigest to serialisation

* Fix difficulty in hard-coded headers

* Avoid duplicate header requests

* Update peer characteristics on NewBlockMsg

* Remove backoff when peer responds

* Fix tests

* Fix lint

* Fix lint
2020-09-20 10:47:24 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) (#1113)
* added tls auth

* added client side

* put --tls

* fixed flag

* Add key/cert generation instructions, turn off Common Name verification

* Add CLI arguments and Warning

* Lint

* Update the doc about Internal IP

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00