This is to fix an issue with resource usage if the db.size.limit is
increased from its default setting of 2TB. This is applied to the chain
DB, but should not be used on the consensus DB which has smaller data
requirements. Expanding both DBs results in excessive RAM being reserved
by the underlying OS.
An update to the devnet to introduce a local heimdall to facilitate
multiple validators without the need for an external process, and hence
validator registration/staking etc.
In this initial release only span generation is supported.
It has the following changes:
* Introduction of a local grpc heimdall interface
* Allocation of accounts via a devnet account generator ()
* Introduction on 'Services' for the network config
"--chain bor-devnet --bor.localheimdall" will run a 2 validator network
with a local service
"--chain bor-devnet --bor.withoutheimdall" will sun a single validator
with no heimdall service as before
---------
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
The fixes here fix a couple of issues related to devnet start-up
1. macos threading and syscall error return where causing multi node
start to both not wait and fail
2. On windows creating DB's with the default 2 TB mapsize causes the os
to reserve about 4GB of committed memory per DB. This may not be used -
but is reserved by the OS - so a default bor node reserves around 10GB
of storage. Starting many nodes causes the OS page file to become
exhausted.
To fix this the consensus DB's now use the node's OpenDatabase function
rather than their own, which means that the consensus DB's take notice
of the config.MdbxDBSizeLimit.
This fix leaves one 4GB committed memory allocation in the TX pool which
needs its own MapSize setting.
---------
Co-authored-by: Alex Sharp <akhounov@gmail.com>
The check in catches errors in the node start-up code and makes sure
that the network is stopped if any node fails to start cleanly, and
that5 it returns an error - so that any calling code can take
appropriate action.
This change adds 'any' as an alternate wildcard to '*'.
I have updated all doc references in the main erigon repo - let me know
if there is anywhere else that needs changing.
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established
---------
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
That's an initial PR mostly for code review, not ready for production
use
Got basic GraphQL working when querying a single block
---------
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
this pr adds CLI flag to allow the rpcdaemon to bind to a TCP port.
this is very useful if one wants to maintain a remote connection with
the rpcdaemon without using websocket. This is useful because a lot of
issues come with the websocket protocol (compression, max size, etc).
TCP socket gets around these things (it is just raw json over tcp
stream)
the rpc package already supports this, it was just a matter of adding
the bind.
try `echo
'{"jsonrpc":"2.0","method":"eth_blockNumber","id":"1","params":[""]}' |
nc localhost 8548` as a basic test
to test. Subscriptions are also working (idk how to send keepalives with
netcat)
the default rpc.(*Client).Dial method does not support TCP. I have not
included that in this PR. The code for such is as follow
```
// DialTCP create a new TCP client that connects to the given endpoint.
//
// The context is used for the initial connection establishment. It does not
// affect subsequent interactions with the client.
func DialTCP(ctx context.Context, endpoint string) (*Client, error) {
parsed, err := url.Parse(endpoint)
if err != nil {
return nil, err
}
ans := make(chan *Client)
errc := make(chan error)
go func() {
client, err := newClient(ctx, func(ctx context.Context) (ServerCodec, error) {
conn, err := net.Dial("tcp", parsed.Host)
if err != nil {
return nil, err
}
return NewCodec(conn), nil
})
if err != nil {
errc <- err
return
}
ans <- client
}()
select {
case err := <-errc:
return nil, err
case a := <-ans:
return a, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
// DialContext creates a new RPC client, just like Dial.
//
// The context is used to cancel or time out the initial connection establishment. It does
// not affect subsequent interactions with the client.
func DialContext(ctx context.Context, rawurl string) (*Client, error) {
u, err := url.Parse(rawurl)
if err != nil {
return nil, err
}
switch u.Scheme {
case "http", "https":
return DialHTTP(rawurl)
case "ws", "wss":
return DialWebsocket(ctx, rawurl, "")
case "tcp":
return DialTCP(ctx, rawurl)
case "stdio":
return DialStdIO(ctx)
case "":
return DialIPC(ctx, rawurl)
default:
return nil, fmt.Errorf("no known transport for URL scheme %q", u.Scheme)
}
}
```
let me know if you would like me to add this to the PR as well. the TCP
connection can then be established with `rpc.Dial("tcp://host:port")`
Previously "in-memory" MDBX instances for fork validation and mining
were created inside `os.TempDir()`. We should create them inside
Erigon's datadir so that the file permissions and the disk are the same
as for the main database.
Prerequisite: https://github.com/ledgerwatch/erigon-lib/pull/676.
