Commit Graph

597 Commits

Author SHA1 Message Date
Thomas Jay Rush
2f0a9d666c
Fixes 'too many other files opened' problem with Generate Storage History stage on Mac (#1217)
* Added back fdlimit to increase number of file descriptors

* Fixing a test file that fails on Mac with too few file descriptors available

* Trying to fix ci

* Trying to fix ci

* Trying to fix ci 3

* Fixing ci
2020-10-10 22:15:07 +01:00
Alex Sharov
9cf10a8f85
Rpcdaemon: Gracefull shutdown, better grpc stream close (#1196)
* save progress

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown grpc server, send to server close signal before canceling context

* GracefulShutdown json server

* GracefulShutdown json server

* fix lint

* fix lint

* can

* clean

* test

* test

* test

* increase keepalive timeout, increase vegeta timeout, mark server-side errors

* increase keepalive timeout, increase vegeta timeout, mark server-side errors

* hack hugeFreelist

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* up streams limit

* fix_race_condition_on_zstd_build
2020-10-10 07:06:54 +01:00
Alex Sharov
88a099702b
rawdb.ReadCanonicalHash to return err (#1204)
* read_canonical_hash_return_err

* hack hugeFreelist

* fix tests
2020-10-10 07:05:56 +01:00
Alex Sharov
53484d60f2
LMDB: don't search pages sequence in freelist if value is > X (#1213)
* lmdb_no_freepages_sequence_search_for_large_values

* go mod tidy
2020-10-10 07:05:18 +01:00
ledgerwatch
1af6cbe37b
Revert "Download/seed headers and block bodies via BitTorrent (#1195)" (#1202)
This reverts commit 94360212cb.
2020-10-07 21:49:44 +01:00
ledgerwatch
94360212cb
Download/seed headers and block bodies via BitTorrent (#1195)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-07 13:15:49 +01:00
ledgerwatch
bdb0155856
Revert "Add sync by bittorrent snapshots (#1160)" (#1194)
This reverts commit 4ebade3583.
2020-10-06 22:31:57 +01:00
b00ris
4ebade3583
Add sync by bittorrent snapshots (#1160)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-06 20:24:48 +01:00
Igor Mandrigin
70f6ec3b49 post-rebase fixes 2020-10-06 14:12:09 +02:00
Binacs
f4397a3485 cmd/clef, cmd/geth: use SplitAndTrim from cmd/utils (#21579)
# Conflicts:
#	cmd/geth/retesteth.go
#	cmd/utils/flags.go
2020-10-06 14:12:09 +02:00
Marius van der Wijden
1f0014a34e cmd/utils: use preconfigured testnet flags instead of networkid (#21561)
* cmd/utils: use preconfigured testnet flags instead of networkid

* cmd/utils: shorter description

Co-authored-by: Martin Holst Swende <martin@swende.se>

* Update flags.go

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	cmd/utils/flags.go
2020-10-06 14:12:09 +02:00
gary rong
cbe7a5c80a cmd, eth: offer maxprice flag for overwritting price cap (#21531)
* cmd, eth: offer maxprice flag for overwritting price cap

* eth: rename default price cap
# Conflicts:
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	eth/gasprice/gasprice.go
2020-10-06 14:12:09 +02:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) (#1113)
* added tls auth

* added client side

* put --tls

* fixed flag

* Add key/cert generation instructions, turn off Common Name verification

* Add CLI arguments and Warning

* Lint

* Update the doc about Internal IP

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon (#1089)
* added eth_getStorageAt

* used uint32

* now its 256

* incarnation

* added TLS handshake

* code minified

* lint

* minified client

* Update flags.go

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt (#1085)
* kill bolt

* kill bolt

* fix test

* clean
2020-09-09 11:09:55 +01:00
ledgerwatch
c6975414e8
[WIP] Warmup the state bucket for HDD sync (#1032)
* Warmup

* Log the warmup

* Make warmup interruptible

* Make warmup a command line option for HDD

* Activate hdd flag
2020-09-02 12:24:27 +01:00
ledgerwatch
961b1d57c4
Remove fdlimit - ramping up open file handles limit (#1011) 2020-08-30 09:01:39 +01:00
Alex Sharov
07ca5c643a
eth_syncing (#991)
* eth_syncing

* linters

* fix readme

* cleanup cli params

* go mod tidy

* remove memsize ui dep

* remove bloomfilter

* implement net_peerCount

* remove legacy flags
2020-08-29 08:24:50 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync (#970)
* Log in GetAsOf

* Default plain state

* trace

* More test fixes

* Fix lint

* Fix

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix for events

* Repair events

* Fix lint

* Fix lint

* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Giulio rebuffo
2cd825ab63
added message for --rpc (#968)
* added message for --rpc

* more messages for other unused flags
2020-08-24 07:32:14 +01:00
Igor Mandrigin
3f81472831
Never store Ethash caches to disk (#944)
* always keep ethash caches in ram

* fix tests
2020-08-20 11:12:51 +01:00
Alex Sharov
1dcc2b141a
Rpcdaemon as lib (#940)
* share config object

* create default config and logger

* move db connection to common func

* move server start to cli package

* clear

* clear

* rename cli to rpc

* use unified SetupLogger func

* make all root flag persistent

* use common flags in different packages

* use common flags in different packages

* move TraceTx method to eth package

* use native slice flags

* create package "turbo"

* disable geth api

* disable geth api

* move more data types to turbo/adapter package

* add support for customApiList

* run more

* run more

* run more

* dog-food

* move DoCall

* move DoCall

* fix tests

* fix test
2020-08-19 12:46:20 +01:00
Igor Mandrigin
ae784e4b0a fixups1 2020-08-15 22:08:57 +02:00
Igor Mandrigin
d848233275 Merge branch 'master' into geth-1.9.19
# Conflicts:
#	cmd/geth/consolecmd.go
#	cmd/rpcdaemon/service/service.go
#	cmd/utils/flags.go
#	eth/backend.go
2020-08-15 21:55:09 +02:00
Igor Mandrigin
d6fe697ee1 fixups. make geth successful 2020-08-15 19:32:05 +02:00
Alex Sharov
90fa69a9e5
Swap IH and HashState stages (#922)
* move ih stage v0

* save progress

* save working version

* baby step

* baby step

* baby step

* non-baby step

* cleanup
2020-08-15 14:49:36 +01:00
Evgeny Danilenko
beb123f784
fix (#924) 2020-08-15 12:51:06 +01:00
Péter Szilágyi
abb1eccdf8 cmd/utils: grant snapshot cache to trie if disabled (#21416)
* cmd/utils: grant snapshot cache to trie if disabled

* eth: fix up default non-mainnet cache distribution
# Conflicts:
#	cmd/utils/flags.go
#	eth/config.go
2020-08-14 16:43:48 +02:00
rene
fe1bb62bad node: refactor package node (#21105)
This PR significantly changes the APIs for instantiating Ethereum nodes in
a Go program. The new APIs are not backwards-compatible, but we feel that
this is made up for by the much simpler way of registering services on
node.Node. You can find more information and rationale in the design
document: https://gist.github.com/renaynay/5bec2de19fde66f4d04c535fd24f0775.

There is also a new feature in Node's Go API: it is now possible to
register arbitrary handlers on the user-facing HTTP server. In geth, this
facility is used to enable GraphQL.

There is a single minor change relevant for geth users in this PR: The
GraphQL API is no longer available separately from the JSON-RPC HTTP
server. If you want GraphQL, you need to enable it using the
./geth --http --graphql flag combination.

The --graphql.port and --graphql.addr flags are no longer available.

# Conflicts:
#	cmd/faucet/faucet.go
#	cmd/geth/chaincmd.go
#	cmd/geth/config.go
#	cmd/geth/consolecmd.go
#	cmd/geth/main.go
#	cmd/utils/flags.go
#	cmd/wnode/main.go
#	core/rawdb/freezer.go
#	eth/api_backend.go
#	eth/backend.go
#	ethclient/ethclient_test.go
#	ethstats/ethstats.go
#	graphql/service.go
#	internal/ethapi/backend.go
#	les/api_backend.go
#	les/api_test.go
#	les/checkpointoracle/oracle.go
#	les/client.go
#	les/commons.go
#	les/server.go
#	miner/stresstest/stress_clique.go
#	miner/stresstest/stress_ethash.go
#	mobile/geth.go
#	node/api.go
#	node/node.go
#	node/node_example_test.go
#	node/node_test.go
#	node/rpcstack.go
#	node/rpcstack_test.go
#	node/service.go
#	node/service_test.go
#	node/utils_test.go
#	p2p/simulations/examples/ping-pong.go
#	p2p/testing/peerpool.go
#	p2p/testing/protocolsession.go
#	p2p/testing/protocoltester.go
#	whisper/mailserver/server_test.go
#	whisper/whisperv6/api_test.go
#	whisper/whisperv6/filter_test.go
#	whisper/whisperv6/whisper.go
#	whisper/whisperv6/whisper_test.go
2020-08-14 16:33:59 +02:00
gary rong
1bc2bef061 cmd, core, eth, light, trie: dump clean cache periodically (#20391)
* cmd, core, eth, light, trie: dump clean cache periodically

* eth: update config

* trie: minor fix

* core, trie: address comments

* eth: remove useless

* trie: print clean cache dump start too

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/blockchain.go
#	core/genesis.go
#	core/state/database.go
#	eth/api_tracer.go
#	eth/backend.go
#	eth/config.go
#	light/postprocess.go
#	trie/database.go
2020-08-12 12:01:13 +02:00
Alex Sharov
760fe0f0b6
DupSort of hash state (#896) 2020-08-12 09:57:55 +07:00
Evgeny Danilenko
3980fa7d45
Grps eth_sendTransaction (#882)
* implementation

* tidy gomod

* linters

* fix cmd test

* fix

* fix lint
2020-08-11 22:09:30 +01:00
Igor Mandrigin
8600616c3d geth 1.9.18 post-rebase fixups 2020-08-08 17:33:35 +02:00
meowsbits
dd3e143b55 cmd/utils: implement configurable developer (--dev) account options (#21301)
* geth,utils: implement configurable developer account options

Prior to this change --dev (developer) mode
generated one account with an empty password,
irrespective of existing --password and --miner.etherbase
options.

This change makes --dev mode compatible with these
existing flags.

--dev mode may now be used in conjunction with
--password and --miner.etherbase flags to configure
the developer faucet using an existing keystore or
in creating a new account.

Signed-off-by: meows <b5c6@protonmail.com>

* main: remove key/pass flags from usage developer section

These flags are included already in other sections,
and it is not desired to duplicate them.

They were originally included in this section
along with added support for these flags in the
developer mode.

Signed-off-by: meows <b5c6@protonmail.com>
2020-08-07 13:51:15 +02:00
Péter Szilágyi
a61131dfa7 cmd/utils: reuse existing genesis in persistent dev mode
# Conflicts:
#	cmd/utils/flags.go
2020-08-07 13:51:04 +02:00
rene
3f4770275d cmd/clef: change --rpcport to --http.port and update flags in docs (#21318)
# Conflicts:
#	cmd/abigen/main.go
#	cmd/checkpoint-admin/main.go
#	cmd/clef/main.go
#	cmd/ethkey/main.go
#	cmd/evm/main.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
2020-08-07 13:23:29 +02:00
gary rong
478da39c6d cmd, node: dump empty value config (#21296) 2020-08-07 12:47:04 +02:00
meowsbits
396937a291 cmd/geth: allow configuring metrics HTTP server on separate endpoint (#21290)
Exposing /debug/metrics and /debug/metrics/prometheus was dependent
on --pprof, which also exposes other HTTP APIs. This change makes it possible
to run the metrics server on an independent endpoint without enabling pprof.
# Conflicts:
#	cmd/utils/flags.go
#	internal/debug/flags.go
#	metrics/exp/exp.go
#	mobile/geth.go
2020-08-07 12:33:14 +02:00
gary rong
1a99c54719 eth/gasprice: lighter gas price oracle for light client (#20409)
This PR reduces the bandwidth used by the light client to compute the
recommended gas price. The current mechanism for suggesting the price is:

- retrieve recent 20 blocks
- get the lowest gas price of these blocks
- sort the price array and return the middle(60%) one

This works for full nodes, which have all blocks available locally.
However, this is very expensive for the light client because the light
client needs to retrieve block bodies from the network.

The PR changes the default options for light client. With the new config,
the light client only retrieves the two latest blocks, but in order to
collect more sample transactions, the 3 lowest prices are collected from
each block.

This PR also changes the behavior for empty blocks. If the block is empty,
the lastest price is reused for sampling.

# Conflicts:
#	eth/gasprice/gasprice.go
2020-08-07 12:29:19 +02:00
Martin Holst Swende
5d0b4ca3ff rpc, internal/ethapi: default rpc gascap at 25M + better error message (#21229)
* rpc, internal/ethapi: default rpc gascap at 50M + better error message

* eth,internal: make globalgascap uint64

* core/tests: fix compilation failure

* eth/config: gascap at 25M + minor review concerns
# Conflicts:
#	core/tx_pool_test.go
#	internal/ethapi/api.go
#	les/api_backend.go
2020-08-07 12:27:17 +02:00
Martin Holst Swende
14e3fda786 cmd/evm: add state transition tool for testing (#20958)
This PR implements the EVM state transition tool, which is intended
to be the replacement for our retesteth client implementation.
Documentation is present in the cmd/evm/README.md file.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	cmd/evm/main.go
#	core/state/dump.go
#	core/state/state_test.go
#	tests/state_test_util.go
2020-08-07 11:38:07 +02:00
Binacs
5f3516ccf7 cmd: abstract getPassPhrase functions into one (#21219)
* [cmd] Abstract `getPassPhrase` functions into one.

* cmd/ethkey: fix compilation failure

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
# Conflicts:
#	cmd/clef/main.go
#	cmd/ethkey/utils.go
#	cmd/geth/accountcmd.go
2020-08-07 11:25:25 +02:00
meowsbits
f106dd529e utils: fix ineffectual miner config flags (#21271)
Without use of global, these flags didn't actually modify
miner configuration, since we weren't grabbing from the
proper context scope, which should be global (vs. subcommand).

Signed-off-by: meows <b5c6@protonmail.com>
2020-08-07 11:23:24 +02:00
gary rong
2740fc90d9 cmd, eth, internal, les: add txfee cap (#21212)
* cmd, eth, internal, les: add gasprice cap

* cmd/utils, eth: add default value for gasprice cap

* all: use txfee cap

* cmd, eth: add fix

* cmd, internal: address comments
# Conflicts:
#	les/api_backend.go
2020-08-07 11:18:39 +02:00
Alex Sharov
f41959f030
drop badger support (#869)
* drop badger support

* rebase master
2020-08-05 16:33:45 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
Igor Mandrigin
e732ccd438 add lmdb.mapSize param 2020-08-05 10:17:50 +02:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation (#851)
* Jumpdest skipping optimisation

* Fix formatting

* Move skipAnalysis into vmConfig, introduce tracing ability

* Improve detection logging

* Added release instructions

* Fix lint
2020-08-01 17:56:57 +01:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb (#788) 2020-07-27 19:15:48 +07:00
Igor Mandrigin
d05cfa22f6
Make plain state + staged sync the default (#782)
* make plain state + staged sync the default

* remove sync mode flag completely

* one thing

* fix the console

* simplify code

* fix and skip tests

* fixup for console tests
2020-07-25 18:18:18 +01:00