Commit Graph

4814 Commits

Author SHA1 Message Date
Mark Holt
5935b11b24
Devnet macos startup and windows committed memory startup fixes (#7832)
The fixes here fix a couple of issues related to devnet start-up

1. macos threading and syscall error return where causing multi node
start to both not wait and fail
2. On windows creating DB's with the default 2 TB mapsize causes the os
to reserve about 4GB of committed memory per DB. This may not be used -
but is reserved by the OS - so a default bor node reserves around 10GB
of storage. Starting many nodes causes the OS page file to become
exhausted.

To fix this the consensus DB's now use the node's OpenDatabase function
rather than their own, which means that the consensus DB's take notice
of the config.MdbxDBSizeLimit.

This fix leaves one 4GB committed memory allocation in the TX pool which
needs its own MapSize setting.

---------

Co-authored-by: Alex Sharp <akhounov@gmail.com>
2023-07-02 22:37:23 +01:00
sonicWhale
e18db89ced
RPCTxFeeCap flag for rpc daemon (#7824) 2023-06-30 10:02:27 +07:00
Andrew Ashikhmin
1f7de0e5cf
Rename StageLoopStep to StageLoopIteration (#7820)
Rename for clarity to highlight that not only one stage is executed, but
the entire loop.
2023-06-29 17:53:23 +02:00
ledgerwatch
0a1229bd76
Fixed for the devnet runner (#7808)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
2023-06-28 18:21:27 +01:00
Mark Holt
bcc2a4a2f8
Cleaned up error handling in network and node start-up (#7811)
The check in catches errors in the node start-up code and makes sure
that the network is stopped if any node fails to start cleanly, and
that5 it returns an error - so that any calling code can take
appropriate action.
2023-06-28 18:21:15 +01:00
Manav Darji
6f0a02ac42
[bor] Implement bor_getSnapshotProposerSequence RPC method (#7770)
Implements `bor_getSnapshotProposerSequence` RPC method which returns an
ordered set of validators (selected to mine) for a given block number.

```
> curl http://localhost:8545 -X POST --data '{"jsonrpc":"2.0","method":"bor_getSnapshotProposerSequence","params":["0x235A310"],"id":1}' -H "Content-Type: application/json"

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "Signers": [
      {
        "Signer": "0xcfef2a3dc244ef7d0fb93c45e762d671445c4569",
        "Difficulty": 5
      },
      {
        "Signer": "0x3a22c8bc68e98b0faf40f349dd2b2890fae01484",
        "Difficulty": 4
      },
      {
        "Signer": "0xbe188d6641e8b680743a4815dfa0f6208038960f",
        "Difficulty": 3
      },
      {
        "Signer": "0xc26880a0af2ea0c7e8130e6ec47af756465452e8",
        "Difficulty": 2
      },
      {
        "Signer": "0xc275dc8be39f50d12f66b6a63629c39da5bae5bd",
        "Difficulty": 1
      }
    ],
    "Diff": 5,
    "Author": "0xcfef2a3dc244ef7d0fb93c45e762d671445c4569"
  }
}
```
2023-06-27 22:15:45 +07:00
Alex Sharov
1a1ca49ffa
change --db.pagesize default to 8KB (#7798)
reasons:
- mainnet: even nodes with small FreeList - still have millions of pages
there `GC: 46446830 5.8%`. Probability of getting into state where space
re-use will be slower than free-list grow is > 0% (we now using db
version which limiting freelist-overhead, but increasing such
probability)
- polygon: size is > 8Tb
- hardware slowly moving towards bigger pageSizes (because for
OS/Hardware) maintenance of pages metadata is also not free (metadata,
lists, LRU, etc...). Macbook's default pagesize now is 16Kb. Network
disks in cloud are also likely working with 16Kb pages.

pros:
- less db fragmentation (better FS-level compression)
- less overflow pages in DB (which also reducing free-list overhead)
- smaller free-list 
- bigger key-size-limit 
- no 8Tb db size limit
- can setup FS - to also use bigger pagesize - it will reduce FS
overhead also
- reducing amount of page-faults during batch-reads (if FS pagesize
match)
- less write syscalls during commit (when WriteMap disabled)

cons:
- ~10% more IO: because of more RAM waste and just because need
read/write bigger pages (not all updates are co-located).
2023-06-26 19:44:25 +07:00
Andrew Ashikhmin
a24eae8d6c
EIP-4844: Handle data gas in txpool (#7779)
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/1029
2023-06-23 11:10:23 +02:00
Alex Sharov
ded166d73d
e3: release chain maker changes (#7787) 2023-06-23 10:07:42 +07:00
Alex Sharov
7bad6d3fd9
erigon_info metric (#7769) 2023-06-21 11:13:50 +07:00
Mark Holt
051cad0fdd
Devnet diagnostics (#7762)
Added support tunnel to the devnet cmd. In order to get this to run I
made the following changes:

* Create a public function 
* Added non root logging

I have also added commentary to the readme to explain the additional
command line arguments needed to integrate with diagnostics. In summary,
if you set the --diagnostics.url the devenet will wait for diagnostic
requests rather than exiting

---------

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2023-06-20 10:11:55 +01:00
Somnath Banerjee
71231140a5
Block Gas limit contract integration for Chiado (#7727)
Changes summary:
- Continue with the gasLimit check skip in ``verifyHeader`` of
``merge.go`` for unless pre-merge block and blockGasLimitContract
present
- Refactor ``aura.go`` a bit
- Have ``sysCall`` method customized to be able to call state (contract)
at a parent (or any other) header state
2023-06-19 22:06:51 +05:30
Alex Sharov
eae2d9a79c
Disable ci tests on another servers: step 2 (#7752) 2023-06-17 10:27:58 +07:00
Alex Sharov
d3c3be9c91
e2: optimize tests speed (#7738) 2023-06-15 16:09:11 +07:00
Andrew Ashikhmin
252e57b76f
Base BlobTx on DynamicFeeTransaction (#7736)
Continuation of PR #7715
2023-06-15 10:57:27 +02:00
Alex Sharov
e5023775aa
Enforce blockReader interface (#7737)
- breaks dependency from staged_sync to package with block_reader
implementation
- breaks dependency from snap_sync to package with block_reader
implementation
- breaks dependency from mining to txpool implementation
2023-06-15 13:11:51 +07:00
Mark Holt
f110102023
Devnet scenarios (#7723)
This is an update to the devnet code which introduces the concept of
configurable scenarios. This replaces the previous hard coded execution
function.

The intention is that now both the network and the operations to run on
the network can be described in a data structure which is configurable
and composable.

The operating model is to create a network and then ask it to run
scenarios:

```go
network.Run(
		runCtx,
		scenarios.Scenario{
			Name: "all",
			Steps: []*scenarios.Step{
				&scenarios.Step{Text: "InitSubscriptions", Args: []any{[]requests.SubMethod{requests.Methods.ETHNewHeads}}},
				&scenarios.Step{Text: "PingErigonRpc"},
				&scenarios.Step{Text: "CheckTxPoolContent", Args: []any{0, 0, 0}},
				&scenarios.Step{Text: "SendTxWithDynamicFee", Args: []any{recipientAddress, services.DevAddress, sendValue}},
				&scenarios.Step{Text: "AwaitBlocks", Args: []any{2 * time.Second}},
			},
		})
```
The steps here refer to step handlers which can be defined as follows:

```go
func init() {
	scenarios.MustRegisterStepHandlers(
		scenarios.StepHandler(GetBalance),
	)
}

func GetBalance(ctx context.Context, addr string, blockNum requests.BlockNumber, checkBal uint64) {
...
```
This commit is an initial implementation of the scenario running - which
is working, but will need to be enhanced to make it more usable &
developable.

The current version of the code is working and has been tested with the
dev network, and bor withoutheimdall. There is a multi miner bor
heimdall configuration but this is yet to be tested.

Note that by default the scenario runner picks nodes at random on the
network to send transactions to. this causes the dev network to run very
slowly as it seems to take a long time to include transactions where the
nonce is incremented across nodes. It seems to take a long time for the
nonce to catch up in the transaction pool processing. This is yet to be
investigated.
2023-06-14 12:35:22 +01:00
Giulio rebuffo
4f812efbc8
Better version of libp2p where losing peers bug fixed (#7726)
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-06-14 13:14:52 +02:00
Andrew Ashikhmin
862d7b6bc6
Move ChainID from CommonTx to DynamicFeeTransaction (#7732)
For legacy transactions ChainID is optional (missing in
pre-[EIP155](https://eips.ethereum.org/EIPS/eip-155) transactions) and
is derived from `V` anyway.

Also, cherry pick https://github.com/ethereum/go-ethereum/pull/27452.
2023-06-14 11:44:05 +02:00
Alex Sharov
1b14785d79
ws unfreeze (#7724)
attempt to address next issue: 

> when I'm having a lot of websocket connections the node is freezing
and then it needs like 10 mins to sync. Then if I keep pushing requests
it falls out of sync all the time
2023-06-14 16:25:05 +07:00
racytech
8b312d54e8
eip-4844: desszing blobtx: initial commit (#7715) 2023-06-14 11:01:03 +02:00
Alex Sharov
f4d8bd84b3
Crypto lib up (#7730) 2023-06-14 10:43:29 +07:00
Alex Sharov
5ff8246e2a
e4: release some tests (#7729) 2023-06-14 10:01:00 +07:00
Alex Sharov
ae1b193d6f
use 1 constant in tests to configure initialCycle variable (#7720) 2023-06-13 10:55:59 +07:00
omahs
c21d77aa20
Fix: typos (#7710)
Fix: typos
2023-06-12 14:39:52 +07:00
Giulio rebuffo
63006611ec
Added concurrent bls (#7709) 2023-06-12 00:40:03 +02:00
Enrique Jose Avila Asapche
e380ee4bdd
[Caplin] added configurations for beacon api (#7699)
Now you can configure:

Read timeout (default: 5s)
Write timeout  (default: 5s)
On/off (default: ON)
Port (default:5555)
Address (default:localhost)
Protocol (default: tcp)

#7684
2023-06-11 19:34:33 +02:00
Alex Sharov
afd8ea94d0
e3: move domains to tables.go (#7707) 2023-06-11 21:12:05 +07:00
Mark Holt
f9acbff96c
eth_sendRawTransaction: remove logging, because we have --txpool.trace.senders which enabling more verbose logging for given senders list (#7686)
When testing with `Bor` consensus turned on I discovered that
`SendRawTransaction` returns a 0x000... hash when transactions are
submitted during block transitions. This turns out to be spurious in the
sense that the transaction insertion is successful.

The cause is that `ReadCurrentBlockNumber` returns a nil block number.

This in turn is caused by the following: In `accessors_chain.go` there
are two methods: `WriteHeader` and `WriteHeadHeaderHash` when the first
is called the block number is written for the header. The second writes
the header has, but there is no guarantee when it does that the head
header will have been written yet. In fact it seems to happen sometime
later.

The problem for `SendRawTransation` is that it begins a transaction
after inserting into the txpool. And depending on timing this
transaction may see only the `WriteHeadHeaderHash` insertion, and hence
can't read the block number.

I have mitigated this by opening the db transaction before calling the
tx pool insertion, meaning that it is more likely to have a clean view
of the DB.

I have also moved the chain id check earlier in the code - as I think
that if this is invalid the method should not try to insert transactions
in the first place.

The `ReadCurrentBlockNumber` is only used to produce a log message - so
I've changed this to not fail the whole function but to just log an
unknown sender. Which means that the hash is still returned to the
sender after a successful txpool insertion
2023-06-11 13:18:04 +07:00
Alex Sharov
3b89ed81d6
e3: erigon_getBalanceChanges (#7704) 2023-06-11 11:49:53 +07:00
Giulio rebuffo
88410559c6
Forgot to configure flags (#7701) 2023-06-10 08:53:24 +02:00
Mark Holt
1e575ea172
rpc service and stageloop logger updates (#7696)
This is another update to logging to replace the root logger with a
contextual logger
2023-06-10 07:39:39 +01:00
Mark Holt
62b2375de9
7667 cli params vhost (#7669)
This change adds 'any' as an alternate wildcard to '*'.

I have updated all doc references in the main erigon repo - let me know
if there is anywhere else that needs changing.
2023-06-09 14:12:11 +01:00
Giulio rebuffo
614769f7ac
Prepare regression testings for kube (#7697) 2023-06-09 14:38:28 +02:00
Alex Sharov
1546a79771
lru lib version up (#7695) 2023-06-09 13:46:58 +07:00
Alex Sharov
afa4f53ae2
e3: use deserializeV3 for history (#7691) 2023-06-09 10:53:04 +07:00
Enrique Jose Avila Asapche
1cb1c303d4
added a listener for beacon api and added get genesis (#7659)
Added two new flags beacon.api.port and beacon.api.addr

Now we can listen for beacon api and get beacon genesis

---------

Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
2023-06-08 09:43:27 +02:00
Manav Darji
421ee911d2
Indore hardfork for polygon mumbai and fixes in bor consensus (#7681)
This PR does the following things:
- Updates the hardfork number of the upcoming Indore hardfork schedule
at block 36877056.
- Refactoring to `CommitStates` method of bor consensus
- Fixes a bug in triggering mining
2023-06-08 09:20:40 +07:00
Giulio rebuffo
07e2100510
Added Regeression testing for Caplin (#7679) 2023-06-07 20:01:32 +02:00
Andrew Ashikhmin
fa0cfcbec7
Remove the Rinkeby testnet (#7677)
See https://github.com/ethereum/go-ethereum/pull/27406
2023-06-07 07:42:36 +02:00
Mark Holt
37a8450d5f
Added non root logger to bor.ValidatorSet (#7668)
I've added a non root logger to bor.ValidatorSet validator set. This
creates a signature change on a number of calling functions to propagate
the logger. This is mostly constrained to the bor package but impacts a
number of tests and utilities which call the validators set.
2023-06-06 10:55:59 +01:00
Alex Sharov
250ecfc4d1
Fix integration tests run3 (#7674) 2023-06-06 16:50:04 +07:00
Alex Sharov
bf9f5067f3
fix integration tests run (#7672) 2023-06-06 13:49:01 +07:00
Giulio rebuffo
3a8d5c4d0a
improved freezer for Caplin (#7670) 2023-06-06 03:24:59 +02:00
Alex Sharov
e14664d53b
Store Canonical/NonCanonical/Bad blocks/senders/txs in same tables. Make re-org cheaper (#7648)
- allow store non-canonical blocks/senders
- optimize re-org: don't update/delete most of data
- allow mark chain as `Bad` - will be not visible by eth_getBlockByHash,
but can read if have hash+num
2023-06-05 12:36:24 +07:00
Alex Sharov
f62b6fd722
e3: release switch (#7663) 2023-06-05 12:36:02 +07:00
Giulio rebuffo
e45ed7d139
Added record mode for Caplin (#7660) 2023-06-05 01:52:55 +02:00
Mark Holt
415cf86250
refactor to allow switchable consensus and multiple communicating nodes (#7646)
This branch is intended to allow the devnet to be used for testing
multiple consents types beyond the default clique. It is initially being
used to test Bor consensus for polygon.

It also has the following refactoring:

### 1.  Network configuration

The two node arg building functions miningNodeArgs and nonMiningNodeArgs
have been replaced with a configuration struct which is used to
configure:

```go
network := &node.Network{
		DataDir: dataDir,
		Chain:   networkname.DevChainName,
		//Chain:              networkname.BorDevnetChainName,
		Logger:             logger,
		BasePrivateApiAddr: "localhost:9090",
		BaseRPCAddr:        "localhost:8545",
		Nodes: []node.NetworkNode{
			&node.Miner{},
			&node.NonMiner{},
		},
	}
```
and start multiple nodes

```go
network.Start()
```
Network start will create a network of nodes ensuring that all nodes are
configured with non clashing network ports set via command line
arguments on start-up.

### 2. Request Routing

The `RequestRouter` has been updated to take a 'target' rather than
using a static dispatcher which routes to a single node on the network.
Each node in the network has its own request generator so command and
services have more flexibility in request routing and
`ExecuteAllMethods` currently takes the `node.Network` as an argument
and can pick which node (node 0 for the moment) to send requests to.
2023-06-04 20:53:05 +01:00
Alex Sharov
feea6b954c
simplify chain config read by rpcdaemon (#7656) 2023-06-04 11:49:10 +07:00
Alex Sharov
c8e717c957
rawdb methods to rw db schema version (#7653) 2023-06-04 09:20:22 +07:00