This is an update of:
https://github.com/ledgerwatch/erigon/pull/7846
which uses a local fork of victoria metrics to include the changes that
https://github.com/anshalshukla added to the original for we where
using.
It also includes code to address the duplicate metrics issue identified
here:
https://github.com/ledgerwatch/erigon/issues/8053
It has one more associated fix which is to correctly add a metadata
label to counters, these where previously labelled as gauges.
e.g.
```
# TYPE p2p_peers counter
p2p_peers 0
```
rather than
```
# TYPE p2p_peers gauge
p2p_peers 0
```
---------
Co-authored-by: Anshal Shukla <53994948+anshalshukla@users.noreply.github.com>
Co-authored-by: Anshal Shukla <shukla.anshal85@gmail.com>
This request implements the insertion of Bor ephemeral transactions into
snapshot indexes.
I does this by taking the block hash from the header index and passing
it to the transaction indexer to add an additional index entry per block
into the transaction hash -> block index.
The passed entries are currently contained in an in memory array which
is (32 * number of blocks / sprint size) bytes.
In addition to the functional code there is also an update to the
`dump_test.go` so that it runs `DumpBlocks` to exercise the indexing
code. To facilitate this the `InsertChain` method in `mock_sentry` has
been modified so that it can process >128 blocks.
The code in this request also includes additional bor/consensus code
with the following functions:
`CalculateSprint`
`CalculateSprintCount`
The first function is a modification of the code in erigon-lib so that
the sprints are numerically rather than lexically ordered. This code
should be migrated to erigon-lib and should have its sprint set
calculated once from its underlying map rather than this process being
repeated every calculation.
---------
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Co-authored-by: ledgerwatch <akhounov@gmail.com>
Co-authored-by: Enrique Jose Avila Asapche <eavilaasapche@gmail.com>
Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
The fixes here fix a couple of issues related to devnet start-up
1. macos threading and syscall error return where causing multi node
start to both not wait and fail
2. On windows creating DB's with the default 2 TB mapsize causes the os
to reserve about 4GB of committed memory per DB. This may not be used -
but is reserved by the OS - so a default bor node reserves around 10GB
of storage. Starting many nodes causes the OS page file to become
exhausted.
To fix this the consensus DB's now use the node's OpenDatabase function
rather than their own, which means that the consensus DB's take notice
of the config.MdbxDBSizeLimit.
This fix leaves one 4GB committed memory allocation in the TX pool which
needs its own MapSize setting.
---------
Co-authored-by: Alex Sharp <akhounov@gmail.com>
- breaks dependency from staged_sync to package with block_reader
implementation
- breaks dependency from snap_sync to package with block_reader
implementation
- breaks dependency from mining to txpool implementation
I've added a non root logger to bor.ValidatorSet validator set. This
creates a signature change on a number of calling functions to propagate
the logger. This is mostly constrained to the bor package but impacts a
number of tests and utilities which call the validators set.
- stage_senders: don't re-calc existing senders
- stage_tx_lookup: prune less blocks per iteration - because
random-deletes are expensive. pruning must not slow-down sync.
- prune data even if --snap.stop is set
- "prune as-much-as-possible at startup" is not very good idea: at
initialCycle machine can be cold and prune will cause big downtime, no
reason to produce much freelist in 1 tx. People may also restart erigon
- because of some bug - and it will cause unexpected downtime (usually
Erigon startup very fast). So, I just remove all `initialSync`-related
logic in pruning.
- fix lost metrics about disk write byte/sec
- always RLock all snapshots - to guarantee consistency
- introduce class View (analog of RoTx and MakeContext)
- move read methods to View object
- View object will be managed by temporal_tx
---------
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
This is the beginning of the series of changes to make it possible to
run multiple instances of erigon inside a single process (as devnet tool
does), with the logging from these processes going to respective log
files correctly.
This is the first part where the initial infrastructure is being
established
---------
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
`excessDataGas` has been partially made eip-4844 ready, so instead of
passing nils to functions, now it actually assigned to some value (it is
expected to be nil until cancun update).
added subcommand `state_domains` of `bin/integration` to start
processing of existing chaindata with use of state domains and
commitment.
It creates two directory in `datadir`: `state` and `statedb` for files
and mdbx respectively.
This version runs blocks one after another and produces merged files.
Want to add some metrics to export later.
This branch implements support of btree indexes instead of .kvi files.
Kvi also created, but not used during search/lookup, only for size
comparation.
Btree allows to make seek calls, while recsplit indexes does not support
that query.
During initial tests, btree indexes are ~1-3% smaller than recsplit but
avg lookup time bigger up to 1.5us (which compensates with ability to
fetch next key).
BtAlloc
```
337000 searches, last took 5.417µs avg=4.385µs next_access_last[of 10000 keys] 105ns
338000 searches, last took 3.792µs avg=4.385µs next_access_last[of 10000 keys] 103ns
339000 searches, last took 5.125µs avg=4.386µs next_access_last[of 10000 keys] 103ns
340000 searches, last took 4.75µs avg=4.386µs next_access_last[of 10000 keys] 103ns
341000 searches, last took 3.875µs avg=4.386µs next_access_last[of 10000 keys] 104ns
342000 searches, last took 6.042µs avg=4.387µs next_access_last[of 10000 keys] 107ns
343000 searches, last took 3.209µs avg=4.388µs next_access_last[of 10000 keys] 108ns
344000 searches, last took 2.292µs avg=4.393µs next_access_last[of 10000 keys] 113ns
345000 searches, last took 2.417µs avg=4.394µs next_access_last[of 10000 keys] 106ns
346000 searches, last took 2.625µs avg=4.394µs next_access_last[of 10000 keys] 103ns
347000 searches, last took 4.417µs avg=4.397µs next_access_last[of 10000 keys] 104ns
348000 searches, last took 5.458µs avg=4.398µs next_access_last[of 10000 keys] 103ns
349000 searches, last took 4.875µs avg=4.4µs next_access_last[of 10000 keys] 102ns
350000 searches, last took 4.25µs avg=4.4µs next_access_last[of 10000 keys] 104ns
351000 searches, last took 6.25µs avg=4.403µs next_access_last[of 10000 keys] 106ns
352000 searches, last took 3.125µs avg=4.404µs next_access_last[of 10000 keys] 106ns
```
Recsplit
```
354000 searches, last took 1.209µs avg=3.081µs
355000 searches, last took 1.333µs avg=3.076µs
356000 searches, last took 1.541µs avg=3.071µs
357000 searches, last took 1.333µs avg=3.066µs
358000 searches, last took 1.333µs avg=3.062µs
359000 searches, last took 1.167µs avg=3.057µs
360000 searches, last took 1.791µs avg=3.052µs
361000 searches, last took 1.209µs avg=3.048µs
362000 searches, last took 1.375µs avg=3.043µs
363000 searches, last took 1.166µs avg=3.038µs
364000 searches, last took 1.625µs avg=3.033µs
365000 searches, last took 1.125µs avg=3.029µs
366000 searches, last took 1.208µs avg=3.024µs
367000 searches, last took 1.333µs avg=3.019µs
368000 searches, last took 1.791µs avg=3.015µs
369000 searches, last took 1.333µs avg=3.01µs
370000 searches, last took 1.166µs avg=3.006µs
371000 searches, last took 1.125µs avg=3.001µs
372000 searches, last took 1.333µs avg=2.997µs
```
https://github.com/ledgerwatch/erigon-lib/pull/891/files
