Commit Graph

2083 Commits

Author SHA1 Message Date
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature (#711) 2020-07-05 07:17:27 +01:00
Alex Sharov
c429e6c3a4
Experiments: how much will take state in DupSort feature and how fast can delete large DupSort key (#709) 2020-07-04 21:43:23 +07:00
ledgerwatch
cccc3ab460
Stage3 optimise (#708)
* Cherry-picked

* Reset3

* Fix formatting

* Print before commit

* Use ETL

* Fix linter

* Use different keys for ETL

* Fix formatting

* Update progress correctly

* Reset stage5

* Fix stage3 hack
2020-07-03 18:55:43 +01:00
ledgerwatch
49181a298b
Special bucket for transaction senders (#707)
* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test
2020-07-03 08:23:49 +01:00
Alex Sharov
ec70cd2a9c
Revert "Only useful IH cursor (#694)" (#703)
This reverts commit 63c998c91f.

# Conflicts:
#	cmd/hack/hack.go
#	eth/stagedsync/stage_interhashes.go
2020-07-02 17:55:35 +07:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00
Alex Sharov
e28e455b77
small fix of miss-type in ih stage (#697) 2020-06-30 05:44:41 +01:00
Alex Sharov
a7f5b1ca08
KV: defer friendly rollback (#696)
* AbstractKV: defer friendly rollback

* AbstractKV: defer friendly rollback
2020-06-30 04:48:46 +01:00
Alex Sharov
63c998c91f
Only useful IH cursor (#694)
* cursor which returns only useful IH

* cursor which returns only useful IH
2020-06-28 13:51:18 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692)
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
2020-06-28 07:10:27 +01:00
ledgerwatch
4c456a649f
Separating HashState and IntermediateHashes stages again, fixing errors (#688)
* TestWatchNoDir not to be parallel

* Print ks and inc

* Print addrHash

* Change the buffer

* Print loading

* Skip

* More logging

* Error out earlier

* Handle empty codes

* Handle empty codes

* Remove logging

* Compare states

* Not do stage5

* compareBucket

* Preimage

* Clearer errors

* No need to clean up contract code

* Restore stage6

* Printing

* Skip the skipping

* Print all

* Change buffer type

* Add limit to stage5

* Always fail

* Remove exessive logging

* Restore buffer type

* Revert

* Print when exception

* Reenable skipping

* Skip storage items for deleted contracts

* not shortcut

* Remove removeAccount

* Re-enable state hashing

* Default to plain state

* Disable hashing state

* Reenable reset5

* Print unfurl list

* Enable removingAccount

* No printing

* Reenable stage5 commit

* Swap order of stages

* Prevent backwards promotion, reset tx lookup

* reset finish

* Introduce storage item replacement

* See if unwind works

* Restore removingAccount

* Don't do removeAccount for unwinding

* Possible fix

* Proper(er) fix

* Don't exclude unwinding

* Remove unwinding flag

* Fix formatting

* Fix lint

* Not to ignore blocks if they cause reorg

* Fix test, separate stages again

* Fix TestUnwind

* Fix stage

* Swap unwinding

* Revert to unwinding flag

* Print unfurl list

* Print

* Print inside receive

* Print after

* No printing

* Cleanup

* Not use blockCache when doing GetBlock
2020-06-26 07:46:05 +01:00
Igor Mandrigin
69a20de5cc
Fix broken export file usage. (#680) 2020-06-20 21:07:39 +01:00
ledgerwatch
5812a649ce
Combine stage5 and stage6 (#679)
* Handle bucket error

* Replace with 0

* SetMaxDBs

* Set MaxDb before opening'

* Merge stage5 and stage6

* Fix lint

* Make downloader tests not parallel
2020-06-20 10:07:22 +01:00
ledgerwatch
fd98914c28
Stage6 - intermediate hashes (#677)
* First cut of the stage6

* Fix formatter

* Introduce state6

* Fix target number for stage6

* Fix linter

* Fix linter

* Reset in regenerate

* Correct block number

* Fix linter

* Fix linter

* Fix

* Upper case

* Skeleton to debug'

* Fix formatting

* Added codehash correction

* Fix TestUnwind

* fix test

* Fix linter

* Introduce unwind6

* Report adjustment error

* Code hashes included into stage5 incremental promotion and unwind

* Fix formatting

* Cleanup

* fix TestUnwind

* Cleanup

* Fix compile error

* Fix formatting

* unwind4

* Disable verifyRoot in stage5, fix hacks

* Remove verifyRoot function

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-06-18 22:27:11 +01:00
Alex Sharov
38f63f9af2
make lmdb default db (#675)
* make lmdb default db

* switch nodes db to lmdb

* log errors in nodedb

* fix miss typing

* fix default test target in makefile

* report error of .Get method

* add debug info

* run tests with debug

* run tests with debug

* remove default db from TEST_DB env

* cleanup

* wait for goroutines finish

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db
2020-06-18 19:13:58 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Igor Mandrigin
93f6c4c01a post-rebase fixups 2020-06-15 19:38:54 +03:00
Péter Szilágyi
1aef8b30fc cmd/geth: fix the fork orders for DAO tests 2020-06-15 19:38:13 +03:00
Martin Holst Swende
7347d036b0 cmd, core, params: inital support for yolo-v1 testnet (#21154)
* core,params,puppeth: inital support for yolo-v1 testnet

* cmd/geth, core: add yolov1 console flag

* cmd, core, params: YoloV1 bakein fixups

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/vm/interpreter.go
#	params/config.go
2020-06-15 19:38:13 +03:00
rene
ed8c1246bb metrics: replace gosigar with gopsutil (#21041)
* replace gosigar with gopsutil

* removed check for whether GOOS is openbsd

* removed accidental import of runtime

* potential fix for difference in units between gosig and gopsutil

* fixed lint error

* remove multiplication factor

* uses cpu.ClocksPerSec as the multiplication factor

* changed dependency from shirou to renaynay (#20)

* updated dep

* switching back from using renaynay fork to using upstream as PRs were merged on upstream

* removed empty line

* optimized imports

* tidied go mod
# Conflicts:
#	cmd/geth/main.go
#	go.mod
#	go.sum
2020-06-15 19:38:13 +03:00
Felix Lange
204d9dc985 cmd/utils: fix default DNS discovery configuration (#21124) 2020-06-15 19:38:13 +03:00
Martin Michlmayr
732a20d468 all: fix typos in comments (#21118)
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/puppeth/module_dashboard.go
#	core/blockchain_test.go
#	core/rawdb/chain_iterator.go
#	core/state/snapshot/difflayer.go
#	core/state/snapshot/iterator.go
#	core/state/snapshot/iterator_fast.go
#	core/state/snapshot/snapshot.go
#	core/state/snapshot/wipe.go
#	core/tx_pool.go
#	les/clientpool.go
#	les/peer.go
#	mobile/doc.go
#	trie/committer.go
#	trie/database.go
2020-06-15 19:38:13 +03:00
Felföldi Zsolt
1ca95e8352 les, les/lespay: implement new server pool (#20758)
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.

Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4

# Conflicts:
#	cmd/utils/flags.go
#	core/forkid/forkid.go
#	les/client.go
#	les/client_handler.go
#	les/commons.go
#	les/distributor.go
#	les/enr_entry.go
#	les/fetcher.go
#	les/lespay/client/valuetracker.go
#	les/metrics.go
#	les/peer.go
#	les/protocol.go
#	les/retrieve.go
#	les/server.go
#	les/serverpool.go
#	les/test_helper.go
#	les/utils/expiredvalue.go
#	les/utils/weighted_select.go
#	les/utils/weighted_select_test.go
#	params/bootnodes.go
2020-06-15 19:38:12 +03:00
Martin Holst Swende
fd4f6397ee cmd/clef, signer/core: password input fixes (#20960)
* cmd/clef, signer/core: use better terminal input for passwords, make it possible to avoid boot-up warning

* all: move commonly used prompter to isolated (small) package

* cmd/clef: Add new --acceptWarn to clef README

* cmd/clef: rename flag 'acceptWarn' to 'suppress-bootwarn'

Co-authored-by: ligi <ligi@ligi.de>
# Conflicts:
#	cmd/clef/main.go
#	cmd/devp2p/dnscmd.go
#	cmd/ethkey/utils.go
#	cmd/geth/accountcmd.go
#	cmd/geth/chaincmd.go
#	cmd/geth/main.go
#	cmd/wnode/main.go
#	console/bridge.go
#	console/console.go
#	console/console_test.go
#	signer/core/cliui.go
2020-06-15 19:38:12 +03:00
Satpal
82ee3c7711 cmd: fix log contexts (#21077) 2020-06-15 19:38:12 +03:00
Alex Sharov
b05fafc628
Optimize multi put (#657)
* remove unused slice from MultiPut

* mutation: reuse tuples slice and preallocate bucketPuts

* use bucketPool in kv_lmdb

* remove duplicated check of context status

* more benchmarks

* remove reusage of puts
2020-06-12 18:59:06 +01:00
Alex Sharov
15096f273d
Remove ctx from Open. Stop goroutines on Close. (#650)
* remove ctx from MustOpen

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)

* enable RawReads and add native implementation of Get/Has methods
2020-06-12 10:31:21 +01:00
Alex Sharov
9961fef39e
experiment with rewriting gas_limits to python+lmdb (just to see how it can be) (#654) 2020-06-12 09:25:40 +01:00
Alex Sharov
351c188b02
send bolt buckets stats to prometheus (#653) 2020-06-12 09:24:46 +01:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index (#648)
* Debug infos

* bucketStats lmdb by default

* Use bools instead of strings

* Fix linters

* Fix linter

* Finally fix

* Fix compile error

* Fix test

* Fix formatting

* Fix shadowing

* Cleanup

* cleanup
2020-06-11 07:19:15 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
Alex Sharov
fd61af7042
lmdb bucket stats (#645)
* lmdb bucket stats

* fix lint warnings

* add bucket.Size method

* add bucket.Size method
2020-06-10 07:42:26 +01:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
Igor Mandrigin
0ee2d83970
stagedsync: allow long stages/unwinds to be interrupted (#643)
use `s.UpdateWithStageData(db, <block number>, <key>)` to store the key

use `s.StageData` with `etl.NextKey` to restart ETL from where it was interrupted.
2020-06-09 15:49:49 +03:00
Alex Sharov
9f11dabede
Enable golint back but with conditions (#641)
* enable_golint_with_conditions

* enable_golint_with_conditions
2020-06-09 12:00:58 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Alex Sharov
f00f4b5dff
KV: Tx objects pool (#637)
* Lmdb internal objects reuse (built-in feature of lmdb):
- lmdb read transactions pool
- lmdb also support cursors reuse, but not implemented in this PR

And kv abstraction objects reuse:
- lmdbKV pool of all tx objects
- boltKV pool of all tx objects
- badgerKV pool of all tx objects

* switch makefile back to bolt

* switch makefile back to bolt

* lmdb test run

* switch makefile back to bolt

* cursors pool

* run lmdb tests

* make kv objects pool global

* switch makefile back to bolt

* remove badgers GOMAXPROC setup, because our app tunned for sequential read/writes, not for random throughput

* simplify code
2020-06-08 11:53:03 +01:00
ledgerwatch
0fb37c807e
Introduce intermediate hashes stage (#634)
* Query progress

* Run stage4 offline

* More thorough resetState

* Correct BlockNumber

* Fix formatting

* State loop

* do every 200k blocks

* Shift to 6.6m

* Close dbs in tests

* Stage2 with option of no reset

* every 100k blocks

* Reset state before stage5

* Introduce another stage

* Check compile errors

* Fix linter

* Fix linter

* Disable unreliable test

* Fix test

* Remove unreachable code
2020-06-07 09:56:46 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
Evgeny Danilenko
8d7e603f07
stack pool (#627) 2020-06-05 17:45:56 +03:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger (#624)
* lmdb tests

* trigger ci

* fix tests

* disable parallelism

* disable parallelism

* cleanup resources

* cleanup resources

* reduce concurency

* try run tests on bolt

* try run tests on bolt

* fix downloader test

* run bolt tests

* rely on interface instead of exact instance

* Rename AbstractKV to KV

* don't use separator for badger

* don't initialize badger cursor - because it not used here

* fix linter

* try reduce badger compactors

* compat with master

* try lmdb

* try lmdb

* try lmdb

* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option

* allow to close closed db

* release

* release

* ideal batch size for badger

* ideal batch size for badger
2020-06-05 10:25:33 +01:00
Alex Sharov
7aa050ca79
[Breaking change in db!] UI for Buckets stat (#607)
* UI for buckets size
2020-06-04 16:35:42 +07:00
Andrew Ashikhmin
33c28f7fac
[GC] uint256 rather than big.Int in Transaction (#614)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray
2020-06-04 08:43:08 +01:00
Igor Mandrigin
0533eed812
Refactor Stagedsync part 1 (#610) 2020-06-03 17:25:44 +03:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments (#608)
* First commit

* Fix the one-off error

* Fix formatting

* Ability to execute stage5 separately

* Clean up intermediate hashes and stage5 progress

* Fix linter

* Print original keys when extracting

* channel

* More logging

* More logging

* Remove excess logging

* fix stage2

* Revert

* Fix stage2

* Add provider exhausted

* Sort sortable buffer

* Fix test

* Another cleanup

* Remove exhaust log
2020-06-03 13:03:40 +01:00
Igor Mandrigin
1e20ed255d
move staged sync to its own package (#605) 2020-06-02 17:52:50 +03:00
ledgerwatch
4ce69916dc
Fix rebuilding the Intermediate Hash bucket (#602)
* Changes

* First attempt to fix

* Sync works

* Fix compile

* Fix linter

* Fix test

* Fix test

* Debug

* Propagate tracing

* Trace default receiver

* More tracing

* Correct unfurl

* Remove tracing

* Try to retain trie for debugging
2020-06-02 08:46:21 +01:00
Igor Mandrigin
c16d3da1b4
etl: startkey for Load + OnLoadCommit callback (#601) 2020-06-01 17:14:40 +03:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes (#597)
* Introduce hashCollector

* Add HashCollector to SubTrieLoader

* Fix linter

* Reset hashed state

* Not to regenerate the hashed state

* Not to delete state

* Fix linter

* Print expected hash in the beginning

* Simplify

* Remove initialTrie

* Use etl to buffer intermediate hashes

* Copy values, not insert empty key

* Compress instead of decompress

* Enhance file buffer logging, fix linter

* Fix compile errors

* Fix log

* Fix logging

* Exclude zero key again

* Add rewind

* Restrict timestamps

* Fix

* Fix formatting

* Incorporate separation

* Extract identityLoadFunction

* Fix formatting
2020-05-31 13:23:34 +01:00
b00ris
266f9a5208
Plain state index (#595)
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
2020-05-31 07:57:47 +01:00