Commit Graph

20669 Commits

Author SHA1 Message Date
ledgerwatch
2064edc5e6
Add arguments (no-op) (#8653) 2023-11-04 17:44:34 +00:00
ledgerwatch
138dceb639
Make some functions in bor exportable (no-op) (#8650) 2023-11-04 12:54:31 +00:00
ledgerwatch
6b09679774
Fix typo in make output (#8649) 2023-11-04 11:11:15 +00:00
ledgerwatch
a77e33e7c4
Introduce extra functions for BorSpans (no-op) (#8648) 2023-11-04 10:59:07 +00:00
ledgerwatch
1ffa3fcf94
Properly remove borspans snapshots after merges (#8647) 2023-11-04 09:58:56 +00:00
a
2aab8f496c
rlp2 (#8643)
rlp2 is a package that aims to replace the existing erigon-lib/rlp
package and the erigon/common/rlp

it is called rlp2 for now because it requires breaking changes to
erigon-lib/rlp and i do not have the time right now to test all current
uses of such functions

however, the encoder/decoder characteristics of rlp2 might be desirable
for caplin, and also for execution layer parsing blob txns, so im
putting it in a folder called rlp2 (note that it exports package rlp for
easier switching later)

importantly, rlp2 is designed for single-pass decoding with the ability
to skip elements one does not care about. it also is zero alloc.
2023-11-04 09:22:11 +07:00
Giulio rebuffo
36595994e8
Bumps down Bodies Timeout from 30 secs to 2 secs (#8619)
Now our timings are 8.6 MB/sec
2023-11-03 21:44:35 +01:00
battlmonstr
9db2d56aac
devnet: scenarios as tests (#8640) 2023-11-03 15:25:27 +01:00
Andrew Ashikhmin
8288dcb5c2
erigon-lib: remove unused constants from protocol.go (#8644) 2023-11-03 10:20:36 +01:00
Alex Sharov
714fa0fdc7
mainnet 18.3M snaps (#8639) 2023-11-02 14:14:39 +07:00
battlmonstr
d92898a508
p2p: silkworm sentry (#8527) 2023-11-02 08:35:13 +07:00
Alex Sharov
329d18ef6f
snapshots: reduce merge limit of blocks to 100K (#8614)
Reason: 
- produce and seed snapshots earlier on chain tip. reduce depnedency on
"good peers with history" at p2p-network.
Some networks have no much archive peers, also ConsensusLayer clients
are not-good(not-incentivised) at serving history.
- avoiding having too much files:
more files(shards) - means "more metadata", "more lookups for
non-indexed queries", "more dictionaries", "more bittorrent
connections", ...
less files - means small files will be removed after merge (no peers for
this files).


ToDo:
[x] Recent 500K - merge up to 100K 
[x] Older than 500K - merge up to 500K 
[x] Start seeding 100k files
[x] Stop seeding 100k files after merge (right before delete)

In next PR: 
[] Old version of Erigon must be able download recent hashes. To achieve
it - at first start erigon will download preverified hashes .toml from
s3 - if it's newer that what we have (build-in) - use it.
2023-11-01 23:22:35 +07:00
battlmonstr
35696afca1
tests: test that erigon can be used as a library (#8494) 2023-11-01 11:44:01 +01:00
battlmonstr
3698e7f476
devnet: configuration fixes (#8592)
* fix "genesis hash does not match" when dev nodes connect  
The "dev" nodes need to have the same --miner.etherbase in order to
generate the same genesis ExtraData by DeveloperGenesisBlock(). Override
DevnetEtherbase global var that's used if --miner.etherbase is not
passed. (for NonBlockProducer case)

* fix missing private key for the hardcoded DevnetEtherbase  
Fixes panic if SigKey is not found. Bor non-producers will use a default
`DevnetEtherbase` while Dev nodes modify it. Save hardcoded
DevnetEtherbase/DevnetSignPrivateKey into accounts so that SigKey can
recover it.

* refactor devnet.node to contain Node config  
This avoids interface{} type casts and fixes an error with
Heimdall.validatorSet == nil

* add connection retries to rpcCall and Subscribe of requestGenerator  
Fixes "connection refused" errors due to node not ready to handle early
RPC requests.

* fix deadlock in Heimdall.NodeStarted

* fix GetBlockByNumber
Fixes "cannot unmarshal string into Go struct field body.transactions of
type jsonrpc.RPCTransaction"

* demote "no of blocks on childchain is less than confirmations
required" to Info (#8626)

* demote "mismatched pending subpool size" to Debug (#8615)

* revert wiggle testing code
2023-11-01 11:08:47 +01:00
Alex Sharov
8e54a965ca
webseed: don't download .torrent files (#8629) 2023-11-01 15:50:35 +07:00
Alex Sharov
b5ea87d8c5
mdbx_empty_kv_dup (#8635) 2023-11-01 15:50:27 +07:00
Alex Sharov
d4b19b3bb6
pool: limit broadcast tx size by 4kb (#8634)
similar to geth
2023-11-01 09:56:02 +07:00
yyjia
22332f72eb
fix: batch close confict with flush (#8613)
both batch.Flush() and batch.close() close channel, I think flush is not
necessary to close channel
2023-11-01 09:08:47 +07:00
battlmonstr
eb65ffbaa7
config: avoid OOM in docker using cgroups v2 limit (#6646) (#8632)
Tested on Debian 11 by:

    $ git revert HEAD
    $ make docker
    $ docker run --memory=8.8G <sha>

before the fix outputs:

    15.6 GB

after the fix outputs:

    8.8 GB
2023-11-01 09:02:34 +07:00
Mark Holt
c20af7d985
Add code to handle close/cancel semantics (#8633) 2023-10-31 18:43:23 +00:00
Andrew Ashikhmin
cabd2b21dd
Mumbai: fix fork ID for Agra (#8630)
otherwise we start losing maticnetwork/bor peers on devp2p
2023-10-31 10:35:53 +01:00
a
d8d7d8d5df
Enable h2c for http handler. add https handler for http2 (#8610)
new flag examples.

--https.enabled
--https.addr="0.0.0.0"
--https.port=443
--https.url="unix:///file.wow"
--https.cert="keyfile.cert" 
--https.key="certfile.cert"

also adds support for h2c to the http handler - http2 protocol without tls.
2023-10-31 04:14:20 -05:00
Alex Sharov
c90bff7e22
mdbx: use release tag (#8628) 2023-10-31 10:11:40 +07:00
Anshal Shukla
83bbe0dd37
Fix Agra hf scheduling on Mumbai (#8620)
Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
2023-10-30 15:18:47 +01:00
Giulio rebuffo
513fd50fa5
Compress snapshots for Caplin with lz4 level=1 (#8609) 2023-10-30 13:48:14 +01:00
Alex Sharov
c23e5a1abf
downloader: preparations for reducing blocks merge limit (#8612) 2023-10-30 13:46:35 +07:00
Alex Sharov
b311da959f
downloader: webseed better error messages (#8611) 2023-10-30 12:13:45 +07:00
Andrew Ashikhmin
8f67a8c556
Txpool: db unlimited grow fix (#8596) (#8608)
read transaction was opened before stream.Recv(), but stream.Recv() is
blocking infinity loop. so, this read transaction never rollback -
causing unlimited db grow.

---------

Merge PR #8596 into `devel`

---------

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-10-29 11:13:06 +01:00
Andrew Ashikhmin
8231cdaede
downloader: less webseed logs (#8586) (#8607)
Merge PR #8586 into `devel`

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
2023-10-29 10:37:50 +01:00
a
60b3f1e7f0
bind-ipc (#8590) 2023-10-29 12:34:03 +07:00
yyjia
216f20d5f8
batch flush can without read tx (#8597) 2023-10-29 12:33:47 +07:00
Alex Sharov
5bb91bb77c
"erigon snapshots retire": prune, then retire, then prune (#8606) 2023-10-29 12:33:31 +07:00
Giulio rebuffo
0e5af0a69c
Added beacon snapshots download (#8601) 2023-10-28 17:41:50 +02:00
Dmytro
9adf31b8eb
bytes transfet separated by capability and category (#8568)
Co-authored-by: Mark Holt <mark@distributed.vision>
2023-10-27 22:30:28 +03:00
Alex Sharov
f36d090f39
make db-tools: suspend enum-int linter error (#8598)
```
 error: conflicting types for ‘mdbx_get_datacmp’ due to enum/integer mismatch; have ‘int (*(unsigned int))(const MDBX_val *, const MDBX_val *)’ {aka ‘int (*(unsigned int))(const struct iovec *, const struct iovec *)’} [-Werror=enum-int-mismatch]
26609 | __cold MDBX_cmp_func *mdbx_get_datacmp(unsigned flags) {
      |                       ^~~~~~~~~~~~~~~~
mdbx.h:4932:1: note: previous declaration of ‘mdbx_get_datacmp’ with type ‘int (*(MDBX_db_flags_t))(const MDBX_val *, const MDBX_val *)’ {aka ‘int (*(MDBX_db_flags_t))(const struct iovec *, const struct iovec *)’}
```
can't upgrade mdbx version for now
2023-10-27 11:01:11 +07:00
battlmonstr
f1c81dc14e
devnet: fix node startup on macOS (#8569)
* call getEnode before NodeStarted to make sure it is ready for RPC
calls
* fix connection error detection on macOS
* use a non-default p2p port to avoid conflicts
* disable bor milestones on local heimdall
* generate node keys for static peers config
2023-10-26 12:58:01 +07:00
Somnath
043ccef4ca
Fix null ptr in debug_traceTx (#8585)
Newly introduced `t.logGaps` was being set to `nil` and still accessed
within `clearFailedLogs`. This PR changes the ordering, moving the nil
setting to `CaptureTxEnd`.
2023-10-26 12:56:27 +07:00
Andrew Ashikhmin
e1c71bda2e
params: begin 2.54 release cycle (#8582) 2023-10-25 15:51:58 +02:00
Alex Sharov
0e1fa8dc3e
blocksReadAheadFunc: to calc engine.Author in background (#8499)
Bor consensus: this calc is heavy and has cache
2023-10-25 20:09:09 +07:00
Alex Sharov
7aa79a5bbb
TestState: share db (#8576) 2023-10-25 20:08:50 +07:00
Andrew Ashikhmin
656c3d239e
params: remove dev from v2.53.0 (#8581) 2023-10-25 15:05:05 +02:00
Andrew Ashikhmin
38e91809f9
Revert "Move validator set snapshot computation to bor_heimdall stage… (#8580)
PR #8202 might cause Issue #8550, so reverting it until Alexey's return.

This reverts commit 2ce98f8337.
2023-10-25 14:02:31 +02:00
Anshal Shukla
95424b5ccc
fix nil error in bor devnet (#8578) 2023-10-25 10:16:43 +01:00
Alex Sharov
52c4489b04
sapshots: remove modtime protection (#8573)
Historically we had several times when:
- erigon downloaded new version of .seg file
- or didn't finish download and start indexing

this was a "quick-fix protection" against this cases
but now we have other protections for this cases
let's try to remove this one - because it's not compatible with "copy
datadir" and "restore datadir from backup" scenarios
2023-10-25 11:35:45 +07:00
Alex Sharov
3d3c0bec18
downloader: log.Trace non-important log messages instead of skipping it (#8574) 2023-10-25 11:35:22 +07:00
Giulio rebuffo
d7448fdb3f
Added functional beacon snapshots reader and generator to Caplin (#8570)
This PR adds beacon blocks snapshots and beacon blocks snapshot
generator to Caplin, plus a snapshot verifier CLI
2023-10-24 21:32:29 +02:00
Dmytro
b9a8c2d6e6
sentry: error checking fix (#8566) 2023-10-24 10:49:48 +02:00
Somnath
c1697caea3
Update Readme - gnosis size (#8538) 2023-10-24 09:57:13 +02:00
lupin012
e42642e353
rpctest: Fix error mgt from bench1 to bench9 (#8548) 2023-10-24 12:55:33 +07:00
Alex Sharov
dd8920008f
readme: faq on public rpc api (#8541) 2023-10-24 12:55:22 +07:00