Commit Graph

419 Commits

Author SHA1 Message Date
Igor Mandrigin
c60aff8058
Promote hashed state initially (#577) 2020-05-26 16:37:25 +03:00
ledgerwatch
810cf13bf8
Explicit incarnation map in the database for deleted contracts (#578)
* Explicit map of incarnations

* Fix linter
2020-05-26 13:27:21 +01:00
andrew
8d6ff0b769 Post-merge fix 2020-05-25 13:36:19 +02:00
Igor Mandrigin
db746bba7a
"Unwind" for the execution phase when plain state is selected (#571)
* "Unwind" for the execution phase when plain state is selected

* test stub (fails)

* tests (one with incarnations fails)

* test fixups

* fix tests: cleanup contract code bucket
2020-05-25 12:11:56 +01:00
ledgerwatch
154b9a9ca2
Stop blockchain when terminating downloader (#574)
* Print

* Stop blockchain on Terminate

* Remove prints
2020-05-25 08:19:08 +01:00
ledgerwatch
f6be0b601e
Fix history generation and optimise via pre-sorting in the files (#572)
* Rewrite

* Fix linter, make counter

* Skip hash stage

* Fix linter

* Add common

* Fix wrong bucket

* Use temp files to generate index

* Fix hack, defer

* Add logging when creating files

* Properly pass datadir

* Bigger buffers, less logging

* Log current key, enforce batching

* Limit to 4m, print more

* Use ReadFull

* Optimised version of storage generation

* Don't print entire key

* Commit at the end

* Fix linter

* Remove a bit of copy-paste

* More copy-paste reduction

* Fixes

* Fix key length

* Not delete files too early

* Fix linter

* Fix logging

* Fix starting block

* Skip test

* Remove limit on the stage 4

* Fix test while disabling the hash check

* Fix tests

* Fix unreachable
2020-05-24 15:48:15 +01:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices (#569)
* save state

* add current index feature

* fix test

* remove logs

* Only execute 1000 blocks

* Reset history index

* Correct action

* Increase batch size

* Increase chunk size, print memory stats

* Fix linter

* Remove unused from

* Split into 2 staged

* Use storage history gen

* remove log

* Not to run tx_cacher in staged mode

* Not to recover during stage 2

* Not to recover during stage 2

* Remove counter

Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
ledgerwatch
6d883b5350
Fix staged sync issue introduced by LRU caches (#568)
* readPlainAccount

* readPlainAccount

* Try to disable accountCache

* Try to not separate changeBatch

* The actual fix
?

* Move to fastcache, include incarnation

* Fix linter

* Fix linter

* Disable caches

* Fix linter

* Reenable all except storage cache

* Cosmetics

* Disable account cache

* Disable code size cache

* Fix code caching

* Apply the same fix to db_state_writer
2020-05-22 16:11:00 +01:00
ledgerwatch
8b97c5c620
Remove memory leak caused by accountCache, count batch size in bytes (#565)
* Lower cache sizes

* Add memory profiling over http

* No code cache

* Calculate db batch size in bytes

* Fixes

* Fixes

* Increase batch

* Fix linter

* Restore account caching, with copying

* Reintroduce code cache

* Add fixed overhead per key

* Print batch size

* Fix batch size

* Reduce batch size

* 50 Mb

* Fix linter
2020-05-21 21:55:39 +01:00
ledgerwatch
e2848ac666
Staged exec: RLU cache between StateReader and StateWriter, separate batches for state and changesets (#542)
* Add cpu profiling

* Splt state and change batches

* Log commits

* Remove

* Log memory

* Add lru caches for staged execution

* Fix

* Fix for nil account

* Fix linter

* Fix linter

* Increase state batch size

* Add memory profiling over http

* Reduce cache sizes

* Fix storage cache

* Fix

* Fix

* Fix

* Fix

* Cleanup

* Fix linter

* Reduce logging

* Reduce logging
2020-05-21 13:27:52 +01:00
ledgerwatch
5904332035
hack --action resetState to create both hashed and plain genesis, clean all buckets (#563)
* Reset plain genesis

* Change to optimised freelist implementation

* Fix linter

* Revert "Change to optimised freelist implementation"

This reverts commit 65e28f1737801178e95585381a789785504ec318.

* Noerrcheck
2020-05-21 06:18:25 +01:00
Guillaume Ballet
61d8a49ece eth/downloader: minor typo fixes in comments (#21035)
# Conflicts:
#	trie/sync.go
2020-05-20 15:26:22 +03:00
Péter Szilágyi
2f13f9835a eth: fix shutdown regression to abort downloads, not just cancel 2020-05-20 15:26:22 +03:00
Alex Sharov
02dd405e93
MGR Schedule v0 (#550)
* MGR Make Schedule v0
2020-05-15 15:58:36 +07:00
ledgerwatch
f9d83c655b
Cap number of goroutines to 3 (#549) 2020-05-15 08:52:58 +01:00
Igor Mandrigin
fd77eaf86a
Staged Sync: Execution phase should use "plain state" (#548)
* introduce PlainStateReader with fallbacks

* no 10.000 changes in tests

* even less iterations

* remove even more iterations

* add `go run ./cmd/geth --syncmode staged --plainstate` flag

* fix serialization calls

* make a more sensible file default

doesn’t affect anything, because this flag is always overriden when parsing CLI. but still.
2020-05-15 08:52:45 +01:00
ledgerwatch
9f3f768ab4
Rename Resolver => SubTrieLoader, ResolveSet => RetainList (#544)
* Rename Resolver => SubTrieLoader, ResolveSet => RetainList

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 15:24:43 +01:00
ledgerwatch
82a67f4d48
eth_getProof step2 - Decouple Resolver from Trie, remove ResolveRequest (#543)
* Removing PrepareResolveParams and NeedResolution

* Remove ResolveRequest

* Decouple resolver from trie

* Remove hooks from resolver

* Fix util compilation

* More test fixes

* Compiled tests in trie

* Fix trie tests

* Fix linter

* Fix linter

* Fix linter

* Fix linter
2020-05-12 08:22:45 +01:00
ledgerwatch
41b5fce078
resolver: merge HashBuilder-s and ResolveSet-s, decouple from ResolveRequest-s (#540)
* merge hbStorage and hb in resolver

* Fix prepare

* Remove tracing by default

* cleanup

* Fix linter

* Not to include incarnation into the ResolveRequest

* Fix linter

* Fix linter

* Fix test

* Fix linter

* Revert

* Fix for infinite loop

* Fix linter

* Collapse rss into rs

* remove reliance of requests in resolver

* Remove currentReq

* Remove fields from ResolveRequest

* Fix linter
2020-05-11 05:46:07 +01:00
ledgerwatch
b4d2e57428
Introduce incarnationMap in both DbStateReader and DbStateWriter (#534)
* Add incarnationMap into db_state_reader

* Fix test

* Add incarnationMap back to the DbStateWriter
2020-05-08 05:52:55 +01:00
ledgerwatch
2fd8f1704c
Restore EVMC integration for staged sync (accidentally broken), resetState (#529)
* Add resetState

* resetState not to disturb header chain

* Fix linter

* Fix linter

* Restore evmc integration
2020-05-07 06:59:00 +01:00
ledgerwatch
f47d661c34
Proper unwinding in the staged sync (#521)
* introduce fork

* Fix unwind test

* Unwind now really works in the test

* Fix linter

* Fix linter

* Fix linter

* Not to handle block and block hash announced in staged sync
2020-05-06 18:42:10 +01:00
Alex Sharov
2686bf2f3b
Remove topLevels param of resolver (#515)
* remove topLevels param of resolver

* revert to master version of ih
2020-05-05 21:08:00 +01:00
Igor Mandrigin
cea453be74
Support EVMC (EVMone) (#509)
* add stuff

* vendor evmc and patch import paths

* some fixups

* compiles!

* fixups for evm initialization

* less chatty logs

* fix wrong import in test

* better (time based) logging

* remove unnecessary printf

* evmc as a module

* glue evmc code

* use a proper fork for evmc

* post-rebase fixups

* fix linterz
2020-05-04 17:13:51 +01:00
ledgerwatch
5d6fb16a97
[WIP] Rewinding for the staged sync, test for the staged sync (#508)
* Initial rework

* Fix lint

* boilerplate for the test

* Fix linter

* More to the test

* fix linter

* Save progress

* Test works

* Fix linter

* More on test

* Fix test, linter

* Fix lint

* Fix test
2020-05-03 13:39:50 +01:00
Alex Sharov
af4a99b977
Merge account and storage resolvers (#504)
* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* Debug

This reverts commit 9c5eb69465f25607d546b03359b2cbcb1bd46689.

* Fix linters

* add_incarnation_to_acc_root_in_ih

* merge cached resolver into stateful resolver

* - move account root set to "post iteration" of resolver
- rename "cache" to IntermediateHash

* remove blockNR and bucket params from walker

* fix out of range panic

* calc acc.Root on the fly

* remove fieldSet field from resolver, make logic of root - lazy

* remove 2 parameters

* working version of forward-only walk over Acc and Storage

* improve test

* rebase master

* save progress - more tests for PrepareResolveParams, add dedicated ResolveSet for storage.
Problem: See duplicates in ResolveSet hexes. Next test failing: oracle_test.go

* skip old incarnations

* don't rebuild when 0 requests

* fix tests

* start from account key when need resolve storage

* Error: stateless prototype faced hashNode when extracting witness

* Statless works: copy touches

* Remove getAccRoot function

* Remove "isAccount" parameter from resolver signature

* Fix: use correct storageResolveSet in finaliseStorageRoot

* Fix: when startKey changed - reset storage buffers also

* Fix: if account incarnation=0 - set EmptyRoot

* Fix: remove account roots by default from IntermediateHash bucket

* Fix: skip abandoned storage - which appeared just after startKey

* Fix: did reset acc key incorrectly

* Fix: clean previous key if receive IH

* Fix: IH observer - subscribe only to branch nodes (was subscribed to value nodes also)

* Add DISABLE_IH and STORE_ACCOUNT_ROOT env variables for tests

* Remove accNode from IH cycle

* Fix flags

* Fix: reset succStorage also

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: skip IH if it has wrong incarnation

* Fix: use rssStorage to HashOnly check

* Fix: remove termination symbol from resolveRequest

* cleanup

* Fix: skip abandoned storage after IH

* remove inc

* remove inc from rss

* tr.succStorage.Reset()

* remove inc from rss

* Remove hard-coding

* succ.Reset

* Enable CalcTrieRoots

* Proper dumping of the trie

* Debug

* Fix for CalcTrieRoot

* Fix another inteference bug

* Temp

* Fix test

* Cleanup

* remove STORE_ACCOUNT_ROOT=true flag

* Fix linter

* Fix linter

* Disable getnodedata by default

* Fix test

* Fix test

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-05-02 19:00:57 +01:00
Andrew Ashikhmin
7b40cbb6fa
Incarnation should be read by StateReader, not StateWriter (#506)
* GetCurrentAccountIncarnation

* Incarnation should be read by StateReader, not StateWriter

* Use GetHistoricalAccountIncarnation in DbState

* RemoteReader ReadAccountIncarnation

* Handle the case where a contract has self-destructed, then Eth sent to it, then it got recreated again
2020-05-02 19:00:42 +01:00
Igor Mandrigin
30d41380ee
Parallel signature verification (#499)
* create context

* parallel recovery

* recovery should use the parallel contexts

* linters

* linters

* remove a leftover

* fix refactoring leftover
2020-04-29 14:17:28 +03:00
Igor Mandrigin
f130d00b8c
Parallel sender recovery in staged sync (#496)
* fix stuff

* recover senders in a separate goroutine

* better way of saving state

* dont fail hash checks because of block 0

* debug stuff

* Revert "debug stuff"

This reverts commit 6d4be2255b29944f6e52516b37656a8eafdb415e.

* fix linter
2020-04-28 13:29:09 +01:00
ledgerwatch
7211b0f261
Add senders recovery stage to staged sync (#494)
* Maintenance commit

* Fixes

* Fixes

* Fix linters

* Fix linter

* Fix linter
2020-04-28 06:36:51 +01:00
ledgerwatch
a09547e822
Truncate history index after reorgs (#492)
* Refactor RewindData

* Add cpu profile

* Index truncation

* Fix compile

* Fix linter

* Fix linter

* Fix tests

* Create test

* Remove Remove

* Clarifying comment
2020-04-28 06:36:33 +01:00
Igor Mandrigin
c516110121
Staged Sync: check trie root hash stage (#490) 2020-04-27 16:27:37 +03:00
ledgerwatch
e1cda82eb8
Fix canonical hashes in staged bodies sync (#487)
* Fix canonical hashes

Remove changes in hack.go

* incarnations and db_state_reader

* Fix compilation

* Fix linter

* Test fixes
2020-04-26 22:58:26 +01:00
Igor Mandrigin
468bd65109
Add "execute blocks" stage to the staged sync. (#486)
* extract functions to different files

* post-rebase fixups

* stage v0

* add a log

* better log entries

* fixes in log messages

* fix some stuff

* review fixes

* fix linters

* pw as a variable

* fix a test

* add Byzantium check

* batch save progress too
2020-04-26 17:02:38 +01:00
ledgerwatch
971024aab1
Stage reorgs (#485)
* Reorg for staged sync

* Fix test

* Fix linter
2020-04-26 12:22:11 +01:00
ledgerwatch
c0b87340a2
Introduce staged sync (#482)
* Introducing staged sync

* Fix linter

* Fix linter

* Sync progress in the DB

* Fix linter

* Fix linter

* Not to wake body and receipt downloaders in the header stage

* Next stage

* Fix tests

* Fix linter

* More on block bodies downloading

* More on downloading bodes

* Remove debug

* Fix test compilation

* Fix tests

* Fix linter

* Fix linter

* Turn off the miner during staged sync

* More fixes
2020-04-24 17:57:43 +01:00
Igor Mandrigin
8c3d19fd4c
geth 1.9.13 (#469)
* core: initial version of state snapshots

* core/state: lazy sorting, snapshot invalidation

* core/state/snapshot: extract and split cap method, cover corners

* snapshot: iteration and buffering optimizations

* core/state/snapshot: unlink snapshots from blocks, quad->linear cleanup

* 123

* core/rawdb, core/state/snapshot: runtime snapshot generation

* core/state/snapshot: fix difflayer origin-initalization after flatten

* add "to merge"

* core/state/snapshot: implement snapshot layer iteration

* core/state/snapshot: node behavioural difference on bloom content

* core: journal the snapshot inside leveldb, not a flat file

* core/state/snapshot: bloom, metrics and prefetcher fixes

* core/state/snapshot: move iterator out into its own files

* core/state/snapshot: implement iterator priority for fast direct data lookup

* core/state/snapshot: full featured account iteration

* core/state/snapshot: faster account iteration, CLI integration

* core: fix broken tests due to API changes + linter

* core/state: fix an account resurrection issue

* core/tests: test for destroy+recreate contract with storage

* squashme

* core/state/snapshot, tests: sync snap gen + snaps in consensus tests

* core/state: extend snapshotter to handle account resurrections

* core/state: fix account root hash update point

* core/state: fix resurrection state clearing and access

* core/state/snapshot: handle deleted accounts in fast iterator

* core: more blockchain tests

* core/state/snapshot: fix various iteration issues due to destruct set

* core: fix two snapshot iterator flaws, decollide snap storage prefix

* core/state/snapshot/iterator: fix two disk iterator flaws

* core/rawdb: change SnapshotStoragePrefix to avoid prefix collision with preimagePrefix

* params: begin v1.9.13 release cycle

* cmd/checkpoint-admin: add some documentation (#20697)

* go.mod: update duktape to fix sprintf warnings (#20777)

This revision of go-duktype fixes the following warning

```
duk_logging.c: In function ‘duk__logger_prototype_log_shared’:
duk_logging.c:184:64: warning: ‘Z’ directive writing 1 byte into a region of size between 0 and 9 [-Wformat-overflow=]
  184 |  sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
      |                                                                ^
In file included from /usr/include/stdio.h:867,
                 from duk_logging.c:5:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:36:10: note: ‘__builtin___sprintf_chk’ output between 25 and 85 bytes into a destination of size 32
   36 |   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   37 |       __bos (__s), __fmt, __va_arg_pack ());
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

* core/rawdb: fix freezer table test error check

Fixes: Condition is always 'false' because 'err' is always 'nil'

* core/rawdb: improve table database (#20703)

This PR fixes issues in TableDatabase.

TableDatabase is a wrapper of underlying ethdb.Database with an additional prefix.
The prefix is applied to all entries it maintains. However when we try to retrieve entries
from it we don't handle the key properly. In theory the prefix should be truncated and
only user key is returned. But we don't do it in some cases, e.g. the iterator and batch
replayer created from it. So this PR is the fix to these issues.

* eth: when triggering a sync, check the head header TD, not block

* internal/web3ext: fix clique console apis to work on missing arguments

* rpc: dont log an error if user configures --rpcapi=rpc... (#20776)

This just prevents a false negative ERROR warning when, for some unknown
reason, a user attempts to turn on the module rpc even though it's already going
to be on.

* node, cmd/clef: report actual port used for http rpc (#20789)

* internal/ethapi: don't set sender-balance to maxuint, fixes #16999 (#20783)

Prior to this change, eth_call changed the balance of the sender account in the
EVM environment to 2^256 wei to cover the gas cost of the call execution.
We've had this behavior for a long time even though it's super confusing.

This commit sets the default call gasprice to zero instead of updating the balance,
which is better because it makes eth_call semantics less surprising. Removing
the built-in balance assignment also makes balance overrides work as expected.

* metrics: disable CPU stats (gosigar) on iOS

* cmd/devp2p: tweak DNS TTLs (#20801)

* cmd/devp2p: tweak DNS TTLs

* cmd/devp2p: bump treeNodeTTL to four weeks

* cmd/devp2p: lower route53 change limit again (#20819)

* cmd/devp2p: be very correct about route53 change splitting (#20820)

Turns out the way RDATA limits work is documented after all,
I just didn't search right. The trick to make it work is to
count UPSERTs twice.

This also adds an additional check to ensure TTL changes are
applied on existing records.

* graphql, node, rpc: fix typos in comments (#20824)

* eth: improve shutdown synchronization (#20695)

* eth: improve shutdown synchronization

Most goroutines started by eth.Ethereum didn't have any shutdown sync at
all, which lead to weird error messages when quitting the client.

This change improves the clean shutdown path by stopping all internal
components in dependency order and waiting for them to actually be
stopped before shutdown is considered done. In particular, we now stop
everything related to peers before stopping 'resident' parts such as
core.BlockChain.

* eth: rewrite sync controller

* eth: remove sync start debug message

* eth: notify chainSyncer about new peers after handshake

* eth: move downloader.Cancel call into chainSyncer

* eth: make post-sync block broadcast synchronous

* eth: add comments

* core: change blockchain stop message

* eth: change closeBloomHandler channel type

* eth/filters: fix typo on unindexedLogs function's comment (#20827)

* core: bump txpool tx max size to 128KB

* snapshotter/tests: verify snapdb post-state against trie (#20812)

* core/state/snapshot: basic trie-to-hash implementation

* tests: validate snapshot after test

* core/state/snapshot: fix review concerns

* cmd, consensus: add option to disable mmap for DAG caches/datasets (#20484)

* cmd, consensus: add option to disable mmap for DAG caches/datasets

* consensus: add benchmarks for mmap with/with lock

* cmd/clef: add newaccount command (#20782)

* cmd/clef: add newaccount command

* cmd/clef: document clef_New, update API versioning

* Update cmd/clef/intapi_changelog.md

Co-Authored-By: ligi <ligi@ligi.de>

* Update signer/core/uiapi.go

Co-Authored-By: ligi <ligi@ligi.de>

Co-authored-by: ligi <ligi@ligi.de>

* eth: add debug_accountRange API (#19645)

This new API allows reading accounts and their content by address range.

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>

* travis: allow cocoapods deploy to fail (#20833)

* metrics: improve TestTimerFunc (#20818)

The test failed due to what appears to be fluctuations in time.Sleep, which is
not the actual method under test. This change modifies it so we compare the
metered Max to the actual time instead of the desired time.

* README: update private network genesis spec with istanbul (#20841)

* add istanbul and muirGlacier to genesis states in README

* remove muirGlacier, relocate istanbul

* cmd/evm: Rework execution stats (#20792)

- Dump stats also for --bench flag.
- From memory stats only show number and size of allocations. This is what `test -bench` shows. I doubt others like number of GC runs are any useful, but can be added if requested.
- Now the mem stats are for single execution in case of --bench.

* cmd/devp2p, cmd/wnode, whisper: add missing calls to Timer.Stop (#20843)

* p2p/server: add UDP port mapping goroutine to wait group (#20846)

* accounts/abi faster unpacking of int256 (#20850)

* p2p/discv5: add missing Timer.Stop calls (#20853)

* miner/worker: add missing timer.Stop call (#20857)

* cmd/geth: fix bad genesis test (#20860)

* eth/filters: add missing Ticker.Stop call (#20862)

* eth/fetcher: add missing timer.Stop calls (#20861)

* event: add missing timer.Stop call in TestFeed (#20868)

* metrics: add missing calls to Ticker.Stop in tests (#20866)

* ethstats: add missing Ticker.Stop call (#20867)

* p2p/discv5, p2p/testing: add missing Timer.Stop calls in tests (#20869)

* core: add missing Timer.Stop call in TestLogReorgs (#20870)

* rpc: add missing timer.Stop calls in websocket tests (#20863)

* crypto/ecies: improve concatKDF (#20836)

This removes a bunch of weird code around the counter overflow check in
concatKDF and makes it actually work for different hash output sizes.

The overflow check worked as follows: concatKDF applies the hash function N
times, where N is roundup(kdLen, hashsize) / hashsize. N should not
overflow 32 bits because that would lead to a repetition in the KDF output.

A couple issues with the overflow check:

- It used the hash.BlockSize, which is wrong because the
  block size is about the input of the hash function. Luckily, all standard
  hash functions have a block size that's greater than the output size, so
  concatKDF didn't crash, it just generated too much key material.
- The check used big.Int to compare against 2^32-1.
- The calculation could still overflow before reaching the check.

The new code in concatKDF doesn't check for overflow. Instead, there is a
new check on ECIESParams which ensures that params.KeyLen is < 512. This
removes any possibility of overflow.

There are a couple of miscellaneous improvements bundled in with this
change:

- The key buffer is pre-allocated instead of appending the hash output
  to an initially empty slice.
- The code that uses concatKDF to derive keys is now shared between Encrypt
  and Decrypt.
- There was a redundant invocation of IsOnCurve in Decrypt. This is now removed
  because elliptic.Unmarshal already checks whether the input is a valid curve
  point since Go 1.5.

Co-authored-by: Felix Lange <fjl@twurst.com>

* rpc: metrics for JSON-RPC method calls (#20847)

This adds a couple of metrics for tracking the timing
and frequency of method calls:

- rpc/requests gauge counts all requests
- rpc/success gauge counts requests which return err == nil
- rpc/failure gauge counts requests which return err != nil
- rpc/duration/all timer tracks timing of all requests
- rpc/duration/<method>/<success/failure> tracks per-method timing

* mobile: use bind.NewKeyedTransactor instead of duplicating (#20888)

It's better to reuse the existing code to create a keyed transactor
than to rewrite the logic again.

* internal/ethapi: add CallArgs.ToMessage method (#20854)

ToMessage is used to convert between ethapi.CallArgs and types.Message.
It reduces the length of the DoCall method by about half by abstracting out
the conversion between the CallArgs and the Message. This should improve the
code's maintainability and reusability.

* eth, les: fix flaky tests (#20897)

* les: fix flaky test

* eth: fix flaky test

* cmd/geth: enable metrics for geth import command (#20738)

* cmd/geth: enable metrics for geth import command

* cmd/geth: enable metrics-flags for import command

* core/vm: use a callcontext struct (#20761)

* core/vm: use a callcontext struct

* core/vm: fix tests

* core/vm/runtime: benchmark

* core/vm: make intpool push inlineable, unexpose callcontext

* docs/audits: add discv5 protocol audits from LA and C53 (#20898)

* .github: change gitter reference to discord link in issue template (#20896)

* couple of fixes to docs in clef (#20900)

* p2p/discover: add initial discovery v5 implementation (#20750)This adds an implementation of the current discovery v5 spec.There is full integration with cmd/devp2p and enode.Iterator in thisversion. In theory we could enable the new protocol as a replacement ofdiscovery v4 at any time. In practice, there will likely be a few morechanges to the spec and implementation before this can happen.

* build: upgrade to golangci-lint 1.24.0 (#20901)

* accounts/scwallet: remove unnecessary uses of fmt.Sprintf

* cmd/puppeth: remove unnecessary uses of fmt.Sprintf

* p2p/discv5: remove unnecessary use of fmt.Sprintf

* whisper/mailserver: remove unnecessary uses of fmt.Sprintf

* core: goimports -w tx_pool_test.go

* eth/downloader: goimports -w downloader_test.go

* build: upgrade to golangci-lint 1.24.0

* accounts/abi/bind: Refactored topics  (#20851)

* accounts/abi/bind: refactored topics

* accounts/abi/bind: use store function to remove code duplication

* accounts/abi/bind: removed unused type defs

* accounts/abi/bind: error on tuples in topics

* Cosmetic changes to restart travis build

Co-authored-by: Guillaume Ballet <gballet@gmail.com>

* node: allow websocket and HTTP on the same port (#20810)

This change makes it possible to run geth with JSON-RPC over HTTP and
WebSocket on the same TCP port. The default port for WebSocket
is still 8546.

    geth --rpc --rpcport 8545 --ws --wsport 8545

This also removes a lot of deprecated API surface from package rpc.
The rpc package is now purely about serving JSON-RPC and no longer
provides a way to start an HTTP server.

* crypto: improve error messages in LoadECDSA (#20718)

This improves error messages when the file is too short or too long.
Also rewrite the test for SaveECDSA because LoadECDSA has its own
test now.

Co-authored-by: Felix Lange <fjl@twurst.com>

* changed date of rpcstack.go since new file (#20904)

* accounts/abi/bind: fixed erroneous filtering of negative ints (#20865)

* accounts/abi/bind: fixed erroneous packing of negative ints

* accounts/abi/bind: added test cases for negative ints in topics

* accounts/abi/bind: fixed genIntType for go 1.12

* accounts/abi: minor  nitpick

* cmd: deprecate --testnet, use named networks instead (#20852)

* cmd/utils: make goerli the default testnet

* cmd/geth: explicitly rename testnet to ropsten

* core: explicitly rename testnet to ropsten

* params: explicitly rename testnet to ropsten

* cmd: explicitly rename testnet to ropsten

* miner: explicitly rename testnet to ropsten

* mobile: allow for returning the goerli spec

* tests: explicitly rename testnet to ropsten

* docs: update readme to reflect changes to the default testnet

* mobile: allow for configuring goerli and rinkeby nodes

* cmd/geth: revert --testnet back to ropsten and mark as legacy

* cmd/util: mark --testnet flag as deprecated

* docs: update readme to properly reflect the 3 testnets

* cmd/utils: add an explicit deprecation warning on startup

* cmd/utils: swap goerli and ropsten in usage

* cmd/geth: swap goerli and ropsten in usage

* cmd/geth: if running a known preset, log it for convenience

* docs: improve readme on usage of ropsten's testnet datadir

* cmd/utils: check if legacy `testnet` datadir exists for ropsten

* cmd/geth: check for legacy testnet path in console command

* cmd/geth: use switch statement for complex conditions in main

* cmd/geth: move known preset log statement to the very top

* cmd/utils: create new ropsten configurations in the ropsten datadir

* cmd/utils: makedatadir should check for existing testnet dir

* cmd/geth: add legacy testnet flag to the copy db command

* cmd/geth: add legacy testnet flag to the inspect command

* les, les/lespay/client: add service value statistics and API (#20837)

This PR adds service value measurement statistics to the light client. It
also adds a private API that makes these statistics accessible. A follow-up
PR will add the new server pool which uses these statistics to select
servers with good performance.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* README: update min go version to 1.13 (#20911)

* travis, appveyor, build, Dockerfile: bump Go to 1.14.2 (#20913)

* travis, appveyor, build, Dockerfile: bump Go to 1.14.2

* travis, appveyor: force GO111MODULE=on for every build

* core/rawdb: fix data race between Retrieve and Close (#20919)

* core/rawdb: fixed data race between retrieve and close

closes https://github.com/ethereum/go-ethereum/issues/20420

* core/rawdb: use non-atomic load while holding mutex

* all: simplify and fix database iteration with prefix/start (#20808)

* core/state/snapshot: start fixing disk iterator seek

* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start

* les, core/state/snapshot: iterator fixes

* all: remove two iterator methods

* all: rename Iteratee.NewIteratorWith -> NewIterator

* ethdb: fix review concerns

* params: update CHTs for the 1.9.13 release

* params: release Geth v1.9.13

* added some missing files

* post-rebase fixups

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: gary rong <garyrong0905@gmail.com>
Co-authored-by: Alex Willmer <alex@moreati.org.uk>
Co-authored-by: meowsbits <45600330+meowsbits@users.noreply.github.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rene <41963722+renaynay@users.noreply.github.com>
Co-authored-by: Ha ĐANG <dvietha@gmail.com>
Co-authored-by: Hanjiang Yu <42531996+de1acr0ix@users.noreply.github.com>
Co-authored-by: ligi <ligi@ligi.de>
Co-authored-by: Wenbiao Zheng <delweng@gmail.com>
Co-authored-by: Adam Schmideg <adamschmideg@users.noreply.github.com>
Co-authored-by: Jeff Wentworth <jeff@curvegrid.com>
Co-authored-by: Paweł Bylica <chfast@gmail.com>
Co-authored-by: ucwong <ucwong@126.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Luke Champine <luke.champine@gmail.com>
Co-authored-by: Boqin Qin <Bobbqqin@gmail.com>
Co-authored-by: William Morriss <wjmelements@gmail.com>
Co-authored-by: Guillaume Ballet <gballet@gmail.com>
Co-authored-by: Raw Pong Ghmoa <58883403+q9f@users.noreply.github.com>
Co-authored-by: Felföldi Zsolt <zsfelfoldi@gmail.com>
2020-04-19 18:31:47 +01:00
Boqin Qin
92f415e6aa eth/downloader: fix possible data race by inconsistent field protection (#20690) 2020-03-16 16:40:19 +03:00
rjl493456442
34c43f7040 core, eth: implement eth/65 transaction fetcher 2020-02-27 17:21:20 +03:00
Evgeny Danilenko
17a4a56634
Refactor mining. Remove a few mining goroutines (#338)
* add context

* extract chain events

* run commit in goroutines

* mine only on canonical

* typo

* linters

* fmt

* mark unused methods

* restore stress test

* test single miner

* remove unsafe Trie storage

* remove locks from miner

* restore interrupt

* remove result goroutine

* remove unconfirmedBlocks

* cherry-pick 04a1d475ff1a36ad8f92fec80385df18c52bdc1f

* extract uncles

* one miner succeeded

* restore context cancel

* cleanup

* skip an unstable test

* remove pending state

* use context instead of interrupt func

* calculate sealHash only once

* comment out unstable test

* after merge

* fix after merge

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-02-03 15:02:26 +03:00
alex.sharov
76dc061d8b FakePeer - reduce interface size 2019-12-13 09:56:30 +06:00
Igor Mandrigin
ec6350fa15 apply post-rebase fixups 2019-12-03 16:27:57 +01:00
Martin Holst Swende
7df0db66d6 eth/downloader: fix data race in downloader 2019-12-03 13:06:36 +01:00
Alexey Akhunov
fe01bccbb8 Apply Turbo-Geth modifications to go-ethereum codebase 2019-11-01 21:52:03 +01:00
Christian Muehlhaeuser
a32a2b933a cmd, contracts, eth, p2p, signer, whisper: fixed ineffectual assignments (#19869)
Fixed assigning values to variables we don't end up using.
2019-07-22 13:34:41 +03:00
大彬
4ac04ae0fe all: replace fmt.Print* calls with t.Log* in tests (#19670) 2019-07-17 13:20:24 +02:00
Péter Szilágyi
b02958b9c5
core, ethdb, metrics, p2p: expose various counter metrics for grafana 2019-06-11 09:49:13 +03:00
gary rong
7641bbe535 eth/downloader: make syncing error more obvious (#19413) 2019-06-05 14:00:46 +02:00
Péter Szilágyi
f35975ea21
core/rawdb, eth/downloader: align 64bit atomic fields 2019-05-17 01:45:05 +03:00
Péter Szilágyi
536b3b416c
cosensus, core, eth, params, trie: fixes + clique history cap 2019-05-16 10:39:35 +03:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong
80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
Péter Szilágyi
9effd64290
core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
Péter Szilágyi
85726fdb02
eth/downloader: fix header delays during chain dedup 2019-05-07 16:09:11 +03:00
gary rong
749ccab9a4 eth/downloader: enable unsync-protection for light client (#19496)
* eth/downloader: enable unsync-protection for light client

* eth/downloader: fix tests
2019-04-25 11:31:23 +03:00
Péter Szilágyi
38f6b85638
eth, les, light: enforce CHT checkpoints on fast-sync too 2019-04-17 13:16:15 +03:00
Péter Szilágyi
7221cb1434
core, eth, les, light: scope receipt functionality a bit cleaner 2019-04-15 13:42:26 +03:00
Péter Szilágyi
91eec1251c
cmd, core, eth, trie: get rid of trie cache generations (#19262)
* cmd, core, eth, trie: get rid of trie cache generations

* core, trie: get rid of remainder of cache gen boilerplate
2019-03-14 15:25:12 +02:00
gary rong
6c312a24b6 eth/downloader: fix ancestor searching for light syncing (#19136) 2019-03-14 12:19:03 +02:00
Péter Szilágyi
1612267a4b
eth/downloader: fix nil droppeer in state sync 2019-03-07 12:37:03 +02:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
Matthew Halpern
3c62f965e7 eth: remove redundant parentheses (#19108) 2019-02-18 17:42:22 +02:00
lhendre
d88441025f cmd,eth: 16400 Add an option to stop geth once in sync. WIP for light mode (#17321)
* cmd, eth: Added in the flag to step geth once sync based on input

* cmd, eth: 16400 Add an option to stop geth once in sync.

* cmd: 16400 Add an option to stop geth once in sync. WIP

* cmd/geth/main, les/fletcher: added in light mode support

* cmd/geth/main, les/fletcher: Cleaned Comments and code for light mode

* cmd: 16400 Fixed formatting issue and cleaned code

* cmd, eth, les: 16400 Fixed formatting issues

* cmd, eth, les: Performed gofmt to update formatting

* cmd, eth, les: Fixed bugs resulting formatting

* cmd/geth, eth/, les: switched to downloader event

* eth: Fixed styling and gen_config

* eth/: Fix nil error in config file

* cmd/geth: Updated countdown log

* les/fetcher.go: Removed depcreated channel

* eth/downloader.go: Removed deprecated select

* cmd/geth, cmd/utils: Fixed minor issues

* eth: Reverted config files to proper format

* eth: Fixed typo in config file

* cmd/geth, eth/down: Updated code to use header time stamp

* eth/downloader: Changed the time threshold to 10 minutes

* cmd/geth, eth/downloader: Updated downloading event to pass latest header

* cmd/geth: Updated main to use right timer object

* cmd/geth: Removed unused failed event

* cmd/geth: added in correct time field with type assertion

* cmd/geth, cmd/utils: Updated flag to use boolean

* cmd/geth, cmd/utils, eth/downloader: Cleaned up code based on recommendations

* cmd/geth: Removed unneeded import

* cmd/geth, eth/downloader: fixed event field and suggested changes

* cmd/geth, cmd/utils: Updated flag and linting issue
2019-01-30 08:40:36 +01:00
Dave McGregor
33d233d3e1
vendor, crypto, swarm: switch over to upstream sha3 package 2019-01-04 09:26:07 +02:00
Wenbiao Zheng
81c3dc728f eth/downloader: progress in stateSync not used anymore (#17998) 2018-12-21 23:36:14 +01:00
Martin Holst Swende
5f251a6448
downloader: fix edgecase where returned index is OOB for downloader (#18335)
* downloader: fix edgecase where returned index is OOB for downloader

* downloader: documentation

Co-Authored-By: holiman <martin@swende.se>
2018-12-20 10:46:08 +01:00
Péter Szilágyi
174083c3ae
eth/downloader: fix light client cht binary search issue 2018-11-28 13:46:13 +02:00
Péter Szilágyi
accc0fab4f
core, eth/downloader: fix ancestor lookup for fast sync 2018-11-16 13:21:20 +02:00
Martin Holst Swende
51b2f1620c
downloader: different sync strategy 2018-11-16 11:54:36 +02:00
mr_franklin
b91766fe6d eth: fix comment typo (#18114)
* consensus/clique: fix comment typo

* eth,eth/downloader: fix comment typo
2018-11-15 16:31:24 +02:00
Felix Lange
0bcff8f525
eth/downloader: speed up tests by generating chain only once (#17916)
* core: speed up GenerateChain

Use a mock implementation of ChainReader instead of creating
and destroying a BlockChain object for each generated block.

* eth/downloader: speed up tests by generating chain only once

This change reworks the downloader tests so they share a common test
blockchain instead of generating a chain in every test. The tests are
roughly twice as fast now.
2018-11-07 15:07:43 +01:00
Wenbiao Zheng
b35165555d eth/downloader: remove the expired id directly (#17963) 2018-11-07 15:30:19 +02:00
Martin Holst Swende
eea3ae42a3 core, eth/downloader: fix validation flaw, fix downloader printout flaw (#17974) 2018-11-07 14:47:11 +02:00
Martin Holst Swende
dc6648bb58 downloader: measure successfull deliveries, not failed (#17983)
* downloader: measure successfull deliveries, not failed

* downloader: fix typos
2018-11-07 14:18:07 +02:00
Wenbiao Zheng
6810933640 eth/downloader: SetBlocksIdle is not used (#17962)
__
 <(o )___
  ( ._> /
   `---'
2018-10-24 01:27:49 +02:00
Wenbiao Zheng
3088c122d8 eth/downloader: fix comment typos (#17956) 2018-10-23 13:21:16 +02:00
Péter Szilágyi
6ee3b26f44
eth/downloader: fix invalid hash chain error due to head mini reorg 2018-10-05 10:45:02 +03:00
Wenbiao Zheng
024b22c30e eth/downloader: use intermediate variable for better readability (#17510) 2018-09-29 22:13:39 +02:00
Péter Szilágyi
faa69bea1c
core, eth: fix goimports for Go 1.11 2018-09-19 11:47:09 +03:00
Wenbiao Zheng
6a33954731 core, eth, trie: use common/prque (#17508) 2018-09-03 17:33:21 +02:00
Wenbiao Zheng
d1aa605f1e all: remove the duplicate 'the' in annotations (#17509) 2018-08-27 11:49:29 +03:00
williambannas
9402f96597 eth: conform better to the golint standards (#16783)
* eth: made changes to conform better to the golint standards

* eth: fix comment nit
2018-06-14 13:14:52 +03:00
Wenbiao Zheng
38c7eb0f26 trie: rename TrieSync to Sync and improve hexToKeybytes (#16804)
This removes a golint warning: type name will be used as trie.TrieSync by
other packages, and that stutters; consider calling this Sync.

In hexToKeybytes len(hex) is even and (even+1)/2 == even/2, remove the +1.
2018-05-29 17:48:43 +02:00
kiel barry
84f8c0cc1f common: improve documentation comments (#16701)
This commit adds many comments and removes unused code.
It also removes the EmptyHash function, which had some uses
but was silly.
2018-05-29 12:42:21 +02:00
gary rong
7beccb29be all: get rid of error when creating memory database (#16716)
* all: get rid of error when create mdb

* core: clean up variables definition

* all: inline mdb definition
2018-05-09 15:24:25 +03:00
Péter Szilágyi
6cf0ab38bd
core/rawdb: separate raw database access to own package (#16666) 2018-05-07 14:35:06 +03:00
Domino Valdano
7cf83cee52 eth/downloader: fix for Issue #16539 (#16546) 2018-04-23 10:01:21 +03:00
Wuxiang
8f8774cf6d all: fix various typos (#16533)
* fix typo

* fix typo

* fix typo
2018-04-19 16:32:02 +03:00
gary rong
de2a7bb764 eth/downloader: wait for all fetcher goroutines to exit before terminating (#16509) 2018-04-16 11:37:48 +03:00
gary rong
1100e8ba63 eth/downloader: flush state sync data before exit (#16280) 2018-04-09 14:46:27 +02:00
Yusup
7aad81f881 eth: fix typos (#16414) 2018-04-04 12:25:02 +02:00
gary rong
77da203547 eth: update higest block we know during the sync if a higher was found (#16283)
* eth: update higest block we know during the sync if a higher was found

* eth: avoid useless sync in fast sync
2018-03-09 11:51:30 +02:00
Péter Szilágyi
ca64a122d3 eth/downloader: save and load trie sync progress (#16224) 2018-03-03 00:52:39 +01:00
Anton Evangelatov
ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Péter Szilágyi
96dad6b6f6
eth/downloader: don't require state for ancestor lookups 2018-02-11 14:43:56 +02:00
Péter Szilágyi
eb07dbb079
eth, light: minor light client startup cleanups 2018-02-08 07:49:23 +02:00
Péter Szilágyi
55599ee95d core, trie: intermediate mempool between trie and database (#15857)
This commit reduces database I/O by not writing every state trie to disk.
2018-02-05 17:40:32 +01:00
Martin Holst Swende
bc0666fb27 eth/downloader: fix #15858 by checking if downloader dropPeer function is set (#15992) 2018-02-05 15:38:06 +02:00
Péter Szilágyi
566d5c0777
core, eth, les, light: get rid of redundant methods 2018-01-30 18:42:00 +02:00
Péter Szilágyi
6f69cdd109
all: switch gas limits from big.Int to uint64 2018-01-03 14:45:35 +02:00