Commit Graph

2099 Commits

Author SHA1 Message Date
ledgerwatch
665b1d88b9
Speed up GenerateChain by using intermediate hashes (#736)
* GenerateChain to support intermediate hashes

* Fix formatting

* Changeset stats

* Fix formatting
2020-07-10 22:37:34 +01:00
Alex Sharov
8335700331
Integration: compare to reference_db after all cycles, better cancelation, add to makefile (#733)
* cmp to reference db if --reference_chaindata provided

* graceful shutdown

* add ./cmd/integration to makefile

* add interruption support for different stages

* don't compare errors directly
2020-07-10 07:03:18 +01:00
ledgerwatch
72dcf478d3
Fix debug_storageRangeAt in rpc daemon (#732)
* Fix storageRangeAt

* Fix formatting

* Fix lint

* Fix lint

* Prevent racing by closing db after the blockchain
2020-07-10 06:44:01 +01:00
Alex Sharov
d98382fdf8
Integration add checkChangeSet into state_stages command (#728)
Integration add checkChangeSet into state_stages command (#728)
2020-07-10 10:25:40 +07:00
Alex Sharov
686f186545
integration - to use same blockchain config as production (#727) 2020-07-09 13:54:10 +01:00
Evgeny Danilenko
5f77570556
Make transaction pool startable and stoppable. Staged sync. (#730)
* after concurrency testing

* fixes for linters

* add an extra parameter

* change init order for protocolManager

* green tests
2020-07-09 13:53:35 +01:00
ledgerwatch
887fc4f99c
GenerateChain without using TrieDbState (#726)
* Different chain generation

* More fixes

* Fix TestSideLogRebirth

* Fix the rest of core tests

* Fix accounts/abi/bind

* Fix compile errors for cmd/pics

* Fix consensus/clique tests

* More fixes

* Fix core/state tests

* Fix eth tests

* Fix eth/downloader tests

* Fix more tests

* Fix compile errors

* Some more fixes

* Save

* Exists

* Fix last tests

* Clean up

* Fix formatting

* Fix linter

* Fix formatting

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint
2020-07-09 07:15:28 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
2689b35da9
TxLookup unwind: Remove entries for blocks between unwindPoint+1 and stage.BlockNumber (#723) 2020-07-08 05:46:04 +01:00
Alex Sharov
495f95f688
Add commands "compare_states", add command "stage9", fix unwind of stage9 (it walked over whole db) (#722)
* ./cmd/integration compare_states, ./cmd/integration stage9

* TxLookup fix unwind - it walked over all db
2020-07-07 17:07:14 +07:00
Alex Sharov
fb933bc9e3
move lmdb-go under ledgerwatch org (#720) 2020-07-07 16:18:09 +07:00
Alex Sharov
f7cc86452c
Move stages from hack.go to ./cmd/integration (#719)
* move stages from hack.go to ./cmd/integration

* move stages from hack.go to ./cmd/integration
2020-07-07 08:11:10 +01:00
Alex Sharov
305f8aff02
Integration tests2 (#714)
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```
2020-07-07 11:00:25 +07:00
Alex Sharov
d3ede58ff0
stage3, fix: if header stage is ehead of body stage (#717)
* fix case: if header stage is ehead of body stage

* fix case: if header stage is ehead of body stage

* fix case: if header stage is ehead of body stage

* allow stage3 to sync to given block

* allow stage3 to sync to given block
2020-07-06 11:11:01 +01:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes (#716)
* Fixing history generation

* Try fixes

* Reset from 0

* Cleanup
2020-07-06 07:34:24 +01:00
Alex Sharov
98f8ccc561
Command for long and heavy integration tests (#712)
* cmd for integration tests

* cmd for integration tests
2020-07-05 07:18:21 +01:00
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature (#711) 2020-07-05 07:17:27 +01:00
Alex Sharov
c429e6c3a4
Experiments: how much will take state in DupSort feature and how fast can delete large DupSort key (#709) 2020-07-04 21:43:23 +07:00
ledgerwatch
cccc3ab460
Stage3 optimise (#708)
* Cherry-picked

* Reset3

* Fix formatting

* Print before commit

* Use ETL

* Fix linter

* Use different keys for ETL

* Fix formatting

* Update progress correctly

* Reset stage5

* Fix stage3 hack
2020-07-03 18:55:43 +01:00
ledgerwatch
49181a298b
Special bucket for transaction senders (#707)
* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test
2020-07-03 08:23:49 +01:00
Alex Sharov
ec70cd2a9c
Revert "Only useful IH cursor (#694)" (#703)
This reverts commit 63c998c91f.

# Conflicts:
#	cmd/hack/hack.go
#	eth/stagedsync/stage_interhashes.go
2020-07-02 17:55:35 +07:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00
Alex Sharov
e28e455b77
small fix of miss-type in ih stage (#697) 2020-06-30 05:44:41 +01:00
Alex Sharov
a7f5b1ca08
KV: defer friendly rollback (#696)
* AbstractKV: defer friendly rollback

* AbstractKV: defer friendly rollback
2020-06-30 04:48:46 +01:00
Alex Sharov
63c998c91f
Only useful IH cursor (#694)
* cursor which returns only useful IH

* cursor which returns only useful IH
2020-06-28 13:51:18 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692)
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
2020-06-28 07:10:27 +01:00
ledgerwatch
4c456a649f
Separating HashState and IntermediateHashes stages again, fixing errors (#688)
* TestWatchNoDir not to be parallel

* Print ks and inc

* Print addrHash

* Change the buffer

* Print loading

* Skip

* More logging

* Error out earlier

* Handle empty codes

* Handle empty codes

* Remove logging

* Compare states

* Not do stage5

* compareBucket

* Preimage

* Clearer errors

* No need to clean up contract code

* Restore stage6

* Printing

* Skip the skipping

* Print all

* Change buffer type

* Add limit to stage5

* Always fail

* Remove exessive logging

* Restore buffer type

* Revert

* Print when exception

* Reenable skipping

* Skip storage items for deleted contracts

* not shortcut

* Remove removeAccount

* Re-enable state hashing

* Default to plain state

* Disable hashing state

* Reenable reset5

* Print unfurl list

* Enable removingAccount

* No printing

* Reenable stage5 commit

* Swap order of stages

* Prevent backwards promotion, reset tx lookup

* reset finish

* Introduce storage item replacement

* See if unwind works

* Restore removingAccount

* Don't do removeAccount for unwinding

* Possible fix

* Proper(er) fix

* Don't exclude unwinding

* Remove unwinding flag

* Fix formatting

* Fix lint

* Not to ignore blocks if they cause reorg

* Fix test, separate stages again

* Fix TestUnwind

* Fix stage

* Swap unwinding

* Revert to unwinding flag

* Print unfurl list

* Print

* Print inside receive

* Print after

* No printing

* Cleanup

* Not use blockCache when doing GetBlock
2020-06-26 07:46:05 +01:00
Igor Mandrigin
69a20de5cc
Fix broken export file usage. (#680) 2020-06-20 21:07:39 +01:00
ledgerwatch
5812a649ce
Combine stage5 and stage6 (#679)
* Handle bucket error

* Replace with 0

* SetMaxDBs

* Set MaxDb before opening'

* Merge stage5 and stage6

* Fix lint

* Make downloader tests not parallel
2020-06-20 10:07:22 +01:00
ledgerwatch
fd98914c28
Stage6 - intermediate hashes (#677)
* First cut of the stage6

* Fix formatter

* Introduce state6

* Fix target number for stage6

* Fix linter

* Fix linter

* Reset in regenerate

* Correct block number

* Fix linter

* Fix linter

* Fix

* Upper case

* Skeleton to debug'

* Fix formatting

* Added codehash correction

* Fix TestUnwind

* fix test

* Fix linter

* Introduce unwind6

* Report adjustment error

* Code hashes included into stage5 incremental promotion and unwind

* Fix formatting

* Cleanup

* fix TestUnwind

* Cleanup

* Fix compile error

* Fix formatting

* unwind4

* Disable verifyRoot in stage5, fix hacks

* Remove verifyRoot function

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-06-18 22:27:11 +01:00
Alex Sharov
38f63f9af2
make lmdb default db (#675)
* make lmdb default db

* switch nodes db to lmdb

* log errors in nodedb

* fix miss typing

* fix default test target in makefile

* report error of .Get method

* add debug info

* run tests with debug

* run tests with debug

* remove default db from TEST_DB env

* cleanup

* wait for goroutines finish

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db
2020-06-18 19:13:58 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Igor Mandrigin
93f6c4c01a post-rebase fixups 2020-06-15 19:38:54 +03:00
Péter Szilágyi
1aef8b30fc cmd/geth: fix the fork orders for DAO tests 2020-06-15 19:38:13 +03:00
Martin Holst Swende
7347d036b0 cmd, core, params: inital support for yolo-v1 testnet (#21154)
* core,params,puppeth: inital support for yolo-v1 testnet

* cmd/geth, core: add yolov1 console flag

* cmd, core, params: YoloV1 bakein fixups

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/vm/interpreter.go
#	params/config.go
2020-06-15 19:38:13 +03:00
rene
ed8c1246bb metrics: replace gosigar with gopsutil (#21041)
* replace gosigar with gopsutil

* removed check for whether GOOS is openbsd

* removed accidental import of runtime

* potential fix for difference in units between gosig and gopsutil

* fixed lint error

* remove multiplication factor

* uses cpu.ClocksPerSec as the multiplication factor

* changed dependency from shirou to renaynay (#20)

* updated dep

* switching back from using renaynay fork to using upstream as PRs were merged on upstream

* removed empty line

* optimized imports

* tidied go mod
# Conflicts:
#	cmd/geth/main.go
#	go.mod
#	go.sum
2020-06-15 19:38:13 +03:00
Felix Lange
204d9dc985 cmd/utils: fix default DNS discovery configuration (#21124) 2020-06-15 19:38:13 +03:00
Martin Michlmayr
732a20d468 all: fix typos in comments (#21118)
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/puppeth/module_dashboard.go
#	core/blockchain_test.go
#	core/rawdb/chain_iterator.go
#	core/state/snapshot/difflayer.go
#	core/state/snapshot/iterator.go
#	core/state/snapshot/iterator_fast.go
#	core/state/snapshot/snapshot.go
#	core/state/snapshot/wipe.go
#	core/tx_pool.go
#	les/clientpool.go
#	les/peer.go
#	mobile/doc.go
#	trie/committer.go
#	trie/database.go
2020-06-15 19:38:13 +03:00
Felföldi Zsolt
1ca95e8352 les, les/lespay: implement new server pool (#20758)
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.

Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4

# Conflicts:
#	cmd/utils/flags.go
#	core/forkid/forkid.go
#	les/client.go
#	les/client_handler.go
#	les/commons.go
#	les/distributor.go
#	les/enr_entry.go
#	les/fetcher.go
#	les/lespay/client/valuetracker.go
#	les/metrics.go
#	les/peer.go
#	les/protocol.go
#	les/retrieve.go
#	les/server.go
#	les/serverpool.go
#	les/test_helper.go
#	les/utils/expiredvalue.go
#	les/utils/weighted_select.go
#	les/utils/weighted_select_test.go
#	params/bootnodes.go
2020-06-15 19:38:12 +03:00
Martin Holst Swende
fd4f6397ee cmd/clef, signer/core: password input fixes (#20960)
* cmd/clef, signer/core: use better terminal input for passwords, make it possible to avoid boot-up warning

* all: move commonly used prompter to isolated (small) package

* cmd/clef: Add new --acceptWarn to clef README

* cmd/clef: rename flag 'acceptWarn' to 'suppress-bootwarn'

Co-authored-by: ligi <ligi@ligi.de>
# Conflicts:
#	cmd/clef/main.go
#	cmd/devp2p/dnscmd.go
#	cmd/ethkey/utils.go
#	cmd/geth/accountcmd.go
#	cmd/geth/chaincmd.go
#	cmd/geth/main.go
#	cmd/wnode/main.go
#	console/bridge.go
#	console/console.go
#	console/console_test.go
#	signer/core/cliui.go
2020-06-15 19:38:12 +03:00
Satpal
82ee3c7711 cmd: fix log contexts (#21077) 2020-06-15 19:38:12 +03:00
Alex Sharov
b05fafc628
Optimize multi put (#657)
* remove unused slice from MultiPut

* mutation: reuse tuples slice and preallocate bucketPuts

* use bucketPool in kv_lmdb

* remove duplicated check of context status

* more benchmarks

* remove reusage of puts
2020-06-12 18:59:06 +01:00
Alex Sharov
15096f273d
Remove ctx from Open. Stop goroutines on Close. (#650)
* remove ctx from MustOpen

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)

* enable RawReads and add native implementation of Get/Has methods
2020-06-12 10:31:21 +01:00
Alex Sharov
9961fef39e
experiment with rewriting gas_limits to python+lmdb (just to see how it can be) (#654) 2020-06-12 09:25:40 +01:00
Alex Sharov
351c188b02
send bolt buckets stats to prometheus (#653) 2020-06-12 09:24:46 +01:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index (#648)
* Debug infos

* bucketStats lmdb by default

* Use bools instead of strings

* Fix linters

* Fix linter

* Finally fix

* Fix compile error

* Fix test

* Fix formatting

* Fix shadowing

* Cleanup

* cleanup
2020-06-11 07:19:15 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
Alex Sharov
fd61af7042
lmdb bucket stats (#645)
* lmdb bucket stats

* fix lint warnings

* add bucket.Size method

* add bucket.Size method
2020-06-10 07:42:26 +01:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
Igor Mandrigin
0ee2d83970
stagedsync: allow long stages/unwinds to be interrupted (#643)
use `s.UpdateWithStageData(db, <block number>, <key>)` to store the key

use `s.StageData` with `etl.NextKey` to restart ETL from where it was interrupted.
2020-06-09 15:49:49 +03:00