Commit Graph

579 Commits

Author SHA1 Message Date
Alex Sharov
59d05dc5fe
hide file exists err (#2218) 2021-06-22 11:09:45 +01:00
Alex Sharov
bc271f92ce
To break dependency to db implementation - move all db classes to "ethdb/kv" and leave in "ethdb" only interfaces (#2195) 2021-06-19 15:21:53 +07:00
ledgerwatch
535ce8fb9c
Try to solve sentry hanging problem + p2p impovement from geth (#2191)
* Try to solve sentry hanging problem + p2p impovement from geth

* Fix import

* Finer stream lock

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-06-18 12:47:46 +01:00
Alex Sharov
e9f510c2b7
erigon as lib (#2186) 2021-06-18 10:35:11 +07:00
Artem Vorotnikov
af836a6200
Nuke LMDB (#2167) 2021-06-16 13:57:58 +03:00
Artem Vorotnikov
9a8c0f9669
Always enable eth65 (#2166)
* Always enable eth65

* Add ListenAddr65 to defaults
2021-06-15 17:57:26 +01:00
BitBaseBit
7ed337cdcc
Implemented panic handling, graceful shutdown and reporting for all goroutines that don't explicitly handle them. (#2153)
* implemented crash reporting for all goroutine panics that aren't handled explicitly

* implemented crash reporting for all goroutine panics that aren't handled explicitly

* changed node defaults back to originals after testing

* implemented panic handling for all goroutines that don't explicitly handle them, outputting the stack trace to a file in crashreports

* handling panics on all goroutines gracefully

* updated missing call

* error assignment

* implemented suggestions

* path.Join added

* implemented Evgeny's suggestions

* changed path.Join to filepath.Join for cross-platform

* added err check

* updated RecoverStackTrace to LogPanic

* updated closures

* removed call of common.Go to some goroutines

* updated scope capture

* removed testing files

* reverted back to original method, I feel like its less intrusive

* update filename for clarity
2021-06-13 17:41:39 +01:00
Alex Sharov
eee871aedf
Add db label (later will add txpool db) (#2081)
* add db label - later will add txpool db

* move commit marker to kv_mdbx behind Chain label

* Update migrations.go

* clean

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2021-06-04 15:56:49 +01:00
Alex Sharov
759af6a15d
cli param to enable eth65 and counfigure it's port (#2075)
* cli param to enable eth65

* save

* save
2021-06-02 08:43:24 +01:00
Alex Sharov
3016b96e88
1 Sentry per protocol (#2046) 2021-05-30 09:53:30 +07:00
ledgerwatch
87d24be960
Drop download.v2 flag (#2039)
* Drop download.v2 flag

* Lint

* Skip tests

* Skip test

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-28 16:32:34 +01:00
Alex Sharov
3a9b38756e
pass cli flag to enode (#2033) 2021-05-28 08:56:05 +07:00
Alex Sharov
ffb5fceb6e
switch node db to mdbx (#2022) 2021-05-27 10:08:33 +07:00
Alex Sharov
14c15cba43
Check version of remote services (#1989)
* save

* save

* Squashed 'interfaces/' content from commit 08c32a09e

git-subtree-dir: interfaces
git-subtree-split: 08c32a09e40b1e6fcb5922e723191c9477545356

* Revert "Squashed 'interfaces/' content from commit 08c32a09e"

This reverts commit 8393d9fd

* save

* seve

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* ensure versions compatibility of all remote services

* Revert "Squashed 'interfaces/' content from commit dd6a42724"

This reverts commit 2a764bf9

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* Revert "Squashed 'interfaces/' content from commit dd6a42724"

This reverts commit 52621846

* Squashed 'interfaces/' content from commit dd6a42724

git-subtree-dir: interfaces
git-subtree-split: dd6a42724401f34c21662ca1aa1718effb92320d

* a

* a

* a

* a

* a

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-22 11:00:13 +01:00
Alex Sharov
0be3044b7e
rename (#1978)
* rename

* rename "make grpc"

* rename "abi bindings templates"

* rename "abi bindings templates"
2021-05-20 19:25:53 +01:00
Alex Sharov
215623da80
Windows support (#1944) 2021-05-20 23:46:12 +07:00
Alex Sharov
9d66aa0fe3
better support for mdbx in snapshots (#1971) 2021-05-20 18:12:07 +07:00
Alex Sharov
781a67a4ee
Auto-Close db in tests by using t.Cleanup (#1956) 2021-05-19 10:47:28 +07:00
Alex Sharov
cceaf75b16
use t.TempDir() (#1955) 2021-05-18 19:13:16 +07:00
ledgerwatch
ebf20b1adc
Peer logging in sentry + fresh hashes for mainnet and ropsten (#1924)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2021-05-13 01:04:07 +01:00
Alex Sharov
249b3098cb
add --sentry.api.addr flag (#1850)
* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr

* add --sentry.api.addr
2021-04-30 16:09:03 +01:00
Alex Sharov
ef77eb3053
tx.RwCursor() to return err (first step of removing lazy cursors) (#1656) 2021-04-02 13:36:49 +07:00
Alex Sharov
b4a2d823c4
old loader remove v1 (#1641) 2021-03-31 14:43:12 +07:00
Artem Vorotnikov
d4c10c9a47
Port rpcdaemon to KV interface (#1627) 2021-03-30 12:53:54 +03:00
Artem Vorotnikov
9b8cdc0f22
Fix lints and remove more unused code (#1621) 2021-03-29 10:58:45 +07:00
Artem Vorotnikov
afe551ba1c
Remove puppeth & account management (#1610) 2021-03-26 21:05:42 +03:00
Artem Vorotnikov
6a6f5469a1
KV: Split RO and RW transactions (#1577) 2021-03-21 20:15:25 +07:00
Igor Mandrigin
a122920fa8 and more lint fixes 2021-03-18 11:53:34 +01:00
Alexey Sharp
519b19f8aa Compilation fixes 2021-03-14 08:00:37 +00:00
Felföldi Zsolt
e3f0782057 les: UDP pre-negotiation of available server capacity (#22183)
This PR implements the first one of the "lespay" UDP queries which
is already useful in itself: the capacity query. The server pool is making
use of this query by doing a cheap UDP query to determine whether it is
worth starting the more expensive TCP connection process.
# Conflicts:
#	les/client.go
#	les/clientpool.go
#	les/clientpool_test.go
#	les/enr_entry.go
#	les/server.go
#	les/vflux/client/serverpool.go
#	les/vflux/client/serverpool_test.go
#	les/vflux/server/balance.go
#	les/vflux/server/balance_test.go
#	les/vflux/server/prioritypool.go
#	les/vflux/server/prioritypool_test.go
#	p2p/nodestate/nodestate.go
2021-03-12 15:53:04 +01:00
Felix Lange
b958c6447a p2p/dnsdisc: fix hot-spin when all trees are empty (#22313)
In the random sync algorithm used by the DNS node iterator, we first pick a random
tree and then perform one sync action on that tree. This happens in a loop until any
node is found. If no trees contain any nodes, the iterator will enter a hot loop spinning
at 100% CPU.

The fix is complicated. The iterator now checks if a meaningful sync action can
be performed on any tree. If there is nothing to do, it waits for the next root record
recheck time to arrive and then tries again.

Fixes #22306
2021-03-10 14:31:53 +01:00
Martin Holst Swende
d1b207ff7e eth: implement eth66 (#22241)
* eth/protocols/eth: split up the eth protocol handlers

* eth/protocols/eth: define eth-66 protocol messages

* eth/protocols/eth: poc implement getblockheaders on eth/66

* eth/protocols/eth: implement remaining eth-66 handlers

* eth/protocols: define handler map for eth 66

* eth/downloader: use protocol constants from eth package

* eth/protocols/eth: add ETH66 capability

* eth/downloader: tests for eth66

* eth/downloader: fix error in tests

* eth/protocols/eth: use eth66 for outgoing requests

* eth/protocols/eth: remove unused error type

* eth/protocols/eth: define protocol length

* eth/protocols/eth: fix pooled tx over eth66

* protocols/eth/handlers: revert behavioural change which caused tests to fail

* eth/downloader: fix failing test

* eth/protocols/eth: add testcases + fix flaw with header requests

* eth/protocols: change comments

* eth/protocols/eth: review fixes + fixed flaw in RequestOneHeader

* eth/protocols: documentation

* eth/protocols/eth: review concerns about types
# Conflicts:
#	eth/downloader/downloader_test.go
#	eth/downloader/peer.go
2021-03-10 11:33:45 +01:00
Martin Holst Swende
aaeb4a40a3 eth: don't wait for snap registration if we're not running snap (#22272)
Prevents a situation where we (not running snap) connects with a peer running snap, and get stalled waiting for snap registration to succeed (which will never happen), which cause a waitgroup wait to halt shutdown
2021-03-10 10:26:58 +01:00
Péter Szilágyi
08ad6aaec7 eth: check snap satelliteness, delegate drop to eth (#22235)
* eth: check snap satelliteness, delegate drop to eth

* eth: better handle eth/snap satellite relation, merge reg/unreg paths
# Conflicts:
#	eth/handler.go
#	eth/peer.go
2021-03-09 13:55:09 +01:00
Felföldi Zsolt
0d8a69b8c6 les: switch to new discv5 (#21940)
This PR enables running the new discv5 protocol in both LES client
and server mode. In client mode it mixes discv5 and dnsdisc iterators
(if both are enabled) and filters incoming ENRs for "les" tag and fork ID.
The old p2p/discv5 package and all references to it are removed.

Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
#	cmd/bootnode/main.go
#	cmd/faucet/faucet.go
#	cmd/utils/flags.go
#	les/client.go
#	les/commons.go
#	les/enr_entry.go
#	les/server.go
#	les/serverpool.go
#	les/serverpool_test.go
#	mobile/discover.go
#	mobile/params.go
#	p2p/discv5/database.go
#	p2p/discv5/metrics.go
#	p2p/discv5/net.go
#	p2p/discv5/net_test.go
#	p2p/discv5/node.go
#	p2p/discv5/node_test.go
#	p2p/discv5/sim_test.go
#	p2p/discv5/table.go
#	p2p/discv5/table_test.go
#	p2p/discv5/ticket.go
#	p2p/discv5/topic.go
#	p2p/discv5/topic_test.go
#	p2p/discv5/udp.go
#	p2p/server.go
2021-03-09 12:54:06 +01:00
Martin Holst Swende
ff2b58887f eth, p2p: reserve half peer slots for snap peers during snap sync (#22171)
* eth, p2p: reserve half peer slots for snap peers during snap sync

* eth: less logging

* eth: rework the eth/snap peer reservation logic

* eth: rework the eth/snap peer reservation logic (again)
2021-03-09 12:46:30 +01:00
Guillaume Ballet
e235d3c43a cmd/utils: don't enumerate USB unless --usb is set (#22130)
USB enumeration still occured. Make sure it will only occur if --usb is set.
This also deprecates the 'NoUSB' config file option in favor of a new option 'USB'.
# Conflicts:
#	cmd/geth/accountcmd_test.go
#	cmd/geth/consolecmd_test.go
#	cmd/geth/dao_test.go
#	cmd/geth/genesis_test.go
#	cmd/geth/les_test.go
2021-03-09 10:36:28 +01:00
Felix Lange
82f1d29cf1 p2p/enode: avoid crashing for invalid IP (#21981)
The database panicked for invalid IPs. This is usually no problem
because all code paths leading to node DB access verify the IP, but it's
dangerous because improper validation can turn this panic into a DoS
vulnerability. The quick fix here is to just turn database accesses
using invalid IP into a noop. This isn't great, but I'm planning to
remove the node DB for discv5 long-term, so it should be fine to have
this quick fix for half a year.

Fixes #21849
2021-03-08 14:25:11 +01:00
Nishant Das
419d7c62aa p2p/discover: fix deadlock in discv5 message dispatch (#21858)
This fixes a deadlock that could occur when a response packet arrived
after a call had already received enough responses and was about to
signal completion to the dispatch loop.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-03-08 12:43:11 +01:00
Alex Prut
f032b6d18e all: remove redundant conversions and import names (#21903)
# Conflicts:
#	accounts/accounts.go
#	accounts/keystore/wallet.go
#	accounts/scwallet/wallet.go
#	accounts/usbwallet/wallet.go
#	cmd/clef/main.go
#	cmd/devp2p/dnscmd.go
#	cmd/evm/compiler.go
#	cmd/evm/disasm.go
#	cmd/evm/runner.go
#	cmd/evm/staterunner.go
#	cmd/geth/config.go
#	cmd/geth/main.go
#	cmd/geth/usage.go
#	cmd/utils/flags.go
#	consensus/ethash/ethash.go
#	core/state/statedb.go
#	core/tx_pool_test.go
#	core/vm/evm.go
#	core/vm/gas_table.go
#	core/vm/instructions.go
#	core/vm/operations_acl.go
#	eth/downloader/api.go
#	eth/filters/api.go
#	eth/filters/filter_system.go
#	eth/filters/filter_system_test.go
#	eth/tracers/tracer.go
#	internal/debug/flags.go
#	internal/ethapi/api.go
#	internal/flags/helpers.go
#	miner/unconfirmed_test.go
#	mobile/ethereum.go
#	trie/iterator.go
#	turbo/trie/database.go
2021-03-08 12:43:02 +01:00
Alex Prut
86dd1f91dd all: simplify nested complexity and if blocks ending with a return statement (#21854)
Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
# Conflicts:
#	cmd/utils/flags.go
#	console/bridge.go
#	crypto/bls12381/g2.go
#	les/benchmark.go
#	les/lespay/server/balance.go
#	les/lespay/server/balance_tracker.go
#	les/lespay/server/prioritypool.go
#	les/odr_requests.go
#	les/serverpool.go
#	les/serverpool_test.go
#	p2p/nodestate/nodestate_test.go
#	trie/committer.go
2021-03-08 12:34:16 +01:00
Martin Holst Swende
e4e21a8834 p2p: avoid spinning loop on out-of-handles (#21878)
* p2p: avoid busy-loop on temporary errors

* p2p: address review concerns
2021-03-08 12:29:12 +01:00
gary rong
a8cbb48d1e p2p/simulations/adapters/exec: fix some issues (#21801)
- Remove the ws:// prefix from the status endpoint since
  the ws:// is already included in the stack.WSEndpoint().
- Don't register the services again in the node start.
  Registration is already done in the initialization stage.
- Expose admin namespace via websocket.
  This namespace is necessary for connecting the peers via websocket.
- Offer logging relevant options for exec adapter.
  It's really painful to mix all log output in the single console. So
  this PR offers two additional options for exec adapter in this case
  testers can config the log output(e.g. file output) and log level
  for each p2p node.
2020-12-03 18:59:17 +01:00
gary rong
0a48344dfe les, p2p/simulations/adapters: fix issues found while simulating les (#21761)
This adds a few tiny fixes for les and the p2p simulation framework:

LES Parts

- Keep the LES-SERVER connection even it's non-synced

  We had this idea to reject the connections in LES protocol if the les-server itself is
  not synced. However, in LES protocol we will also receive the connection from another
  les-server. In this case even the local node is not synced yet, we should keep the tcp
  connection for other protocols(e.g. eth protocol).

- Don't count "invalid message" for non-existing GetBlockHeadersMsg request

  In the eth syncing mechanism (full sync, fast sync, light sync), it will try to fetch
  some non-existent blocks or headers(to ensure we indeed download all the missing chain).
  In this case, it's possible that the les-server will receive the request for
  non-existent headers. So don't count it as the "invalid message" for scheduling
  dropping.

- Copy the announce object in the closure

  Before the les-server pushes the latest headers to all connected clients, it will create
  a closure and queue it in the underlying request scheduler. In some scenarios it's
  problematic. E.g, in private networks, the block can be mined very fast. So before the
  first closure is executed, we may already update the latest_announce object. So actually
  the "announce" object we want to send is replaced.

  The downsize is the client will receive two announces with the same td and then drop the
  server.

P2P Simulation Framework

- Don't double register the protocol services in p2p-simulation "Start".

  The protocols upon the devp2p are registered in the "New node stage". So don't reigster
  them again when starting a node in the p2p simulation framework

- Add one more new config field "ExternalSigner", in order to use clef service in the
  framework.

# Conflicts:
#	les/server_handler.go
2020-12-03 18:59:17 +01:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
Alex Sharov
a72543bc16
Stop dial scheduler asap (#1334)
* stop dial scheduler asap

* stop dial scheduler asap
2020-11-02 21:07:26 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297)
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-29 13:19:31 +00:00
Igor Mandrigin
0f1b2f3595 geth-1.9.23: post-rebase fixups 2020-10-26 17:16:00 +01:00
Felix Lange
0c54d2e649 p2p/discover: implement v5.1 wire protocol (#21647)
This change implements the Discovery v5.1 wire protocol and
also adds an interactive test suite for this protocol.
# Conflicts:
#	cmd/devp2p/discv5cmd.go
#	p2p/discover/v5_encoding.go
#	p2p/discover/v5_encoding_test.go
#	p2p/discover/v5_udp.go
#	p2p/discover/v5_udp_test.go
#	p2p/discover/v5wire/session.go
2020-10-26 17:16:00 +01:00
Martin Holst Swende
926cc78870 eth, p2p: use truncated names (#21698)
* peer: return localAddr instead of name to prevent spam

We currently use the name (which can be freely set by the peer) in several log messages.
This enables malicious actors to write spam into your geth log.
This commit returns the localAddr instead of the freely settable name.

* p2p: reduce usage of peer.Name in warn messages

* eth, p2p: use truncated names

* Update peer.go

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
2020-10-26 17:16:00 +01:00