Commit Graph

16948 Commits

Author SHA1 Message Date
Alex Sharov
8afeee56c8
Downloader extract, step2 (#6076) 2022-11-20 10:41:30 +07:00
Philippe Schommers
806117fbab
feat: add chiado config (#6058)
Hey guys, I'm trying to add the Chiado network ([Gnosis'
testnet](https://docs.gnosischain.com/about/networks/chiado)) now that
Gnosis can be synced fully with Erigon, so we can test it on the testnet
as well.

This is mostly inspired from
cd5ef32f37.

Probably missing:
- [ ] The right consensus config (currently only a copy of Gnosis)
- [ ] Fixes to the chainspec?
- [ ] Presumably something in `cl/clparams/config.go`

Current state:
```
$ ./build/bin/erigon --chain=chiado --log.console.verbosity=debug
WARN[11-16|11:52:28.188] no log dir set, console logging only 
WARN[11-16|11:52:28.193] no log dir set, console logging only 
INFO[11-16|11:52:28.193] Build info                               git_branch=feat/chiado git_tag=v2021.10.03-2291-g17fae73f8 git_commit=17fae73f8af5348ba7c04684f2a2978daf81b67e
INFO[11-16|11:52:28.193] Starting Erigon on                       devnet=chiado
INFO[11-16|11:52:28.194] Maximum peer count                       ETH=100 total=100
INFO[11-16|11:52:28.194] starting HTTP APIs                       APIs=eth,erigon,engine
INFO[11-16|11:52:28.194] torrent verbosity                        level=WRN
INFO[11-16|11:52:30.300] Set global gas cap                       cap=50000000
INFO[11-16|11:52:30.302] Opening Database                         label=chaindata path=/home/filoozom/.local/share/erigon/chiado/chaindata
INFO[11-16|11:52:30.310] Initialised chain configuration          config="{ChainID: 10200, Homestead: 0, DAO: <nil>, DAO Support: false, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 0, Muir Glacier: <nil>, Berlin: 0, London: 0, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Engine: aura}" genesis=0xf463abeb7ee27fa62be3ac36a264e8174ee3458da451e6403df47618fd2cf415
WARN[11-16|11:52:30.311] Incorrect snapshot enablement            got=true change_to=false
INFO[11-16|11:52:30.311] Effective                                prune_flags= snapshot_flags= history.v3=false
INFO[11-16|11:52:30.312] Initialising Ethereum protocol           network=10200
INFO[11-16|11:52:30.329] Starting private RPC server              on=127.0.0.1:9090
INFO[11-16|11:52:30.329] new subscription to logs established 
INFO[11-16|11:52:30.329] rpc filters: subscribing to Erigon events 
DBUG[11-16|11:52:30.330] Establishing event subscription channel with the RPC daemon ... 
INFO[11-16|11:52:30.330] New txs subscriber joined 
INFO[11-16|11:52:30.330] new subscription to newHeaders established 
INFO[11-16|11:52:30.330] Reading JWT secret                       path=/home/filoozom/.local/share/erigon/chiado/jwt.hex
INFO[11-16|11:52:30.331] HTTP endpoint opened for Engine API      url=localhost:8551 ws=true ws.compression=true
INFO[11-16|11:52:30.332] HTTP endpoint opened                     url=localhost:8545 ws=false ws.compression=true grpc=false
DBUG[11-16|11:52:30.336] Couldn't add port mapping                proto=tcp extport=30304 intport=30304 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
DBUG[11-16|11:52:30.336] Couldn't add port mapping                proto=udp extport=30304 intport=30304 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
DBUG[11-16|11:52:30.336] QuerySeeds read nodes from the node DB   count=0
DBUG[11-16|11:52:30.341] [1/16 Snapshots] DONE                    in=134.7µs
INFO[11-16|11:52:30.341] [txpool] Started 
INFO[11-16|11:52:30.341] [2/16 Headers] Waiting for headers...    from=0
DBUG[11-16|11:52:30.341] [Downloader] Request skeleton            anchors=0 top seen height=0 highestInDb=0
DBUG[11-16|11:52:30.343] Couldn't add port mapping                proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
DBUG[11-16|11:52:30.343] Couldn't add port mapping                proto=udp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
DBUG[11-16|11:52:30.344] QuerySeeds read nodes from the node DB   count=0
DBUG[11-16|11:52:30.346] QuerySeeds read nodes from the node DB   count=0
INFO[11-16|11:52:30.347] Started P2P networking                   version=67 self=enode://47d2e31d90fe140bfd967f147c1d4e8a4834b4c6b895a4bb7082100be60aa5e9a73e98e996da9b0257f02f9ad39b683755abe6ca3e9aefe0170a478a8559dcfb@127.0.0.1:30304 name=erigon/v2.30.0-dev-17fae73f/linux-amd64/go1.18.1
DBUG[11-16|11:52:30.351] QuerySeeds read nodes from the node DB   count=0
INFO[11-16|11:52:30.351] Started P2P networking                   version=66 self=enode://47d2e31d90fe140bfd967f147c1d4e8a4834b4c6b895a4bb7082100be60aa5e9a73e98e996da9b0257f02f9ad39b683755abe6ca3e9aefe0170a478a8559dcfb@127.0.0.1:30303 name=erigon/v2.30.0-dev-17fae73f/linux-amd64/go1.18.1
DBUG[11-16|11:52:31.342] [Downloader] Request skeleton            anchors=0 top seen height=0 highestInDb=0
[...]
DBUG[11-16|11:52:32.342] [Downloader] Request skeleton            anchors=0 top seen height=0 highestInDb=0
INFO[11-16|11:37:00.062] [p2p] GoodPeers                          eth66=0 eth67=0
INFO[11-16|11:37:00.077] [txpool] stat                            block=0 pending=0 baseFee=0 queued=0 alloc=42.7MB sys=79.5MB
INFO[11-16|11:37:00.089] [2/16 Headers] No block headers to write in this log period block number=0
INFO[11-16|11:37:00.089] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0
DBUG[11-16|11:37:00.089] [Downloader] Queue sizes                 anchors=0 links=0 persisted=1
DBUG[11-16|11:37:00.089] [Downloader] Request skeleton            anchors=0 top seen height=0 highestInDb=0
[...]
DBUG[11-16|11:37:06.095] [Downloader] Request skeleton            anchors=0 top seen height=0 highestInDb=0
```

I guess it comes down to:
```
--- FAIL: TestDefaultBSCGenesisBlock (0.34s)
    genesis_test.go:27: 
                Error Trace:    /home/filoozom/projects/erigon/core/genesis_test.go:27
                                                        /home/filoozom/projects/erigon/core/genesis_test.go:30
                Error:          Not equal: 
                                expected: []byte{0xe8, 0x72, 0x46, 0x2c, 0x6b, 0xd5, 0xf4, 0x2, 0xec, 0x81, 0xde, 0x7c, 0x5b, 0xd2, 0x82, 0x3e, 0x13, 0x7c, 0x66, 0x6b, 0x78, 0xe8, 0x2b, 0x7e, 0xb0, 0xbe, 0x95, 0xaf, 0x5e, 0xce, 0xa1, 0x8d}
                                actual  : []byte{0xad, 0xa4, 0x4f, 0xd8, 0xd2, 0xec, 0xab, 0x8b, 0x8, 0xf2, 0x56, 0xaf, 0x7, 0xad, 0x3e, 0x77, 0x7f, 0x17, 0xfb, 0x43, 0x4f, 0x8f, 0x8e, 0x67, 0x8b, 0x31, 0x2f, 0x57, 0x62, 0x12, 0xba, 0x9a}
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,4 +1,4 @@
                                 ([]uint8) (len=32) {
                                - 00000000  e8 72 46 2c 6b d5 f4 02  ec 81 de 7c 5b d2 82 3e  |.rF,k......|[..>|
                                - 00000010  13 7c 66 6b 78 e8 2b 7e  b0 be 95 af 5e ce a1 8d  |.|fkx.+~....^...|
                                + 00000000  ad a4 4f d8 d2 ec ab 8b  08 f2 56 af 07 ad 3e 77  |..O.......V...>w|
                                + 00000010  7f 17 fb 43 4f 8f 8e 67  8b 31 2f 57 62 12 ba 9a  |...CO..g.1/Wb...|
                                 }
                Test:           TestDefaultBSCGenesisBlock
                Messages:       chiado
FAIL
FAIL    github.com/ledgerwatch/erigon/core      0.823s
```

----------

Turns out that the `code` in Erigon's chainspec is not the same as
`constructor` in Nethermind for example.
Comparison for Sokol:
- Erigon:
https://github.com/ledgerwatch/erigon/blob/devel/core/allocs/sokol.json#L20-L34
- Nethermind:
https://github.com/NethermindEth/nethermind/blob/master/src/Nethermind/Chains/sokol.json#L248-L252
2022-11-18 12:54:18 +01:00
Alex Sharov
4ba8a6a381
Update README.md 2022-11-18 12:48:04 +03:00
alex.sharov
faa92f9bbc save 2022-11-18 04:31:11 +03:00
lupin012
a442293be1
ChainId conditional in Legacy transaction (#5856)
impacted API:
     eth_getTransactionByHash()
     eth_getTransactionByNumber()
     eth_getTransactionByBlockHashAndIndex()
     eth_getTransactionByBlockNumberAndIndex()
     eth_getBlockByHash()
     eth_getBlockByNumber()

1) In case of legacy transitions the chainId field should be inserted
only if V is not 27/28.
This seems also the Geth/v1.10.23 behaviour via infura

2) In case of dynamicFee/AccessList transaction the access_list should
be inserted in the response also if empty
This is done correctly in cmd/rpcdaemon/commands/eth_api.go and NOT in
internal/ethapi/api.go
This seems also the Geth/v1.10.23 behaviour via infura
2022-11-18 03:15:32 +03:00
lupin012
8ceb1638a0
add totalDifficulty to GetBlockByTimestamp (#5827)
Added totalDifficulty to the returned response to be similar to the data
returned by eth_getBlockByNumber() or eth_getBlockByHash()
2022-11-18 03:15:00 +03:00
lupin012
21d24a20a0
fix on number of samples for blocks (#5826)
eth_gasPrice(): A maximum of 3 samples should be taken per block (see
sample const)
The getBlockPrices() function takes 3 samples on the first block and one
on the others.
In the check s.Len() >= limit s.Len() after first block will contain 3
samples and so the loop will be executed only one time for each block
NOT three times
2022-11-18 03:14:49 +03:00
Alex Sharov
4509f496c6
bsc: new blocks snapshots (#6075) 2022-11-18 03:09:07 +03:00
Nathan (Blaise) Bruer
ca89620e56
SendMessageToRandomPeers() now is truely pseudo-random (#6050)
Fixes an issue where if the peer count didn't change much, we'd send the
message to the same peers nearly every time.

We now pick new random peers on every call to
SendMessageToRandomPeers().
2022-11-18 02:44:43 +03:00
Enrique Jose Avila Asapche
a4b9da9bd4
fixed nil ptr err (#6035)
The from field would sometimes be nil causing an error when trying to
access it, instead changed it to be an empty address #5882
2022-11-18 02:43:22 +03:00
Manav Darji
1da4d3abbf
eth, consensus/bor: fixes and improvements related to mining (#6051)
In context of https://github.com/ledgerwatch/erigon/issues/5694, this PR
adds some fixes and improvement in the mining flow. Also, a relevant
change in txpool (present in erigon-lib) is made here:
https://github.com/ledgerwatch/erigon-lib/pull/737

#### Changes in triggering mining in `startMining()`
The mining module didn't honour the block time as a simple 3 second
timer and a notifier from txpool was used to trigger mining. This would
cause inconsistencies, at least with the bor consensus. Hence, a geth
like approach is used instead for simplicity. A new head channel
subscription is added in the `startMining()` loop which would notify the
addition of new block. Hence, this would make sure that the block time
is being honoured. Moreover, the fixed 3 second timer is replaced by the
`miner.recommit` value set using flags.

####  Changes in the arrangement of calls made post mining
When all the mining stages are completed, erigon writes all the data in
a cache. It then processes the block through all the stages as it would
process a block received from P2P. In this case, some of the stages
aren't really required. Like the block header and body download stage is
not required as the block was mined locally. Even execution stage is not
required as it already went through it in the mining stages.

Now, we encountered an issue where the chain was halted and kept mining
the same block again and again (liveness issue). The root cause is
because of an error in a stage of it's parent block. This stage turns
out to be the 4th stage which is "Block body download" stage. This stage
tries to download the block body from peers using the headers. As, we
mined this block locally we don't really need to download anything (or
process anything again). Hence, it reaches out to the cache which we
store for the block body.

Interestingly that cache turned out to be empty for some blocks. This
was because post mining, before adding block header and body to a cache,
we call the broadcast method which starts the staged sync. So,
technically it’s a bit uncertain at any stage if the block header and
body has been written or not.(see
[this](https://github.com/ledgerwatch/erigon/blob/devel/eth/backend.go#L553-L572)).
To achieve complete certainty, we rearranged the calls with the write to
cache being called first and broadcast next. This pretty much solves the
issue as now we’re sure that we’d always have a block body in the cache
when we reach the body download stage.

#### Misc changes
This PR also adds some logs in bor consensus.
2022-11-18 02:39:16 +03:00
Mark Shields
0d854fea6e
feat(k8s): base with google-kubernetes-engine overlay example (#6041)
re:
https://discord.com/channels/687972960811745322/738982866670714901/1041381487754354689

Introduces a more modular base with a google-kubernetes-engine overlay
example.
2022-11-18 02:37:53 +03:00
Alex Sharov
874d96fb47
e3: Non parallel exec (with exec in RO goroutine) (#6073) 2022-11-17 20:52:12 +03:00
Alex Sharov
c50d4e6c0b
e3: panic logging (#6072) 2022-11-17 16:17:02 +03:00
Alex Sharov
d5c23c4df4
e3: enable test (#6071) 2022-11-17 16:06:57 +03:00
Alex Sharov
04293ef67a
Revert "e3: more mainnet snapshots" (#6069)
Reverts ledgerwatch/erigon#6068
2022-11-17 06:20:58 +03:00
Alex Sharov
17541b26da
e3: more mainnet snapshots (#6068) 2022-11-17 06:15:53 +03:00
Alex Sharov
66e1dc2dfd
better format p2p error (#6066) 2022-11-16 19:46:25 +03:00
Alex Sharov
591c2c5ff4
e3: disalbe ci test (#6064) 2022-11-16 19:12:02 +03:00
Giulio rebuffo
7ea8de4a3f
added checkpoint saving (#6060) 2022-11-16 16:19:40 +01:00
Alex Sharov
e7a3c0789a
Disable ci3 (#6063) 2022-11-16 15:53:46 +03:00
Alex Sharov
30bf54ebf2
disable ci3 (#6062) 2022-11-16 15:49:32 +03:00
Alex Sharov
93926646f8
e3: prepare downloader for extraction (#6061) 2022-11-16 15:48:23 +03:00
Alex Sharov
7d034d0c79
e3: fix stuck exec after recon (#6059) 2022-11-16 14:22:49 +03:00
Alex Sharov
7957e69773
mdbx: configure merge threshold (#6057) 2022-11-16 09:44:34 +03:00
alex.sharov
bfda9bc731 save 2022-11-16 09:43:56 +03:00
Nebojsa Urosevic
44f909767d
Feat: add eth_exitst endpoint (#5992)
Feat: add `eth_exists` endpoint.

By introducing this endpoint, Erigon will fully enable remote execution
for pre-EIP158 blocks.
2022-11-16 07:12:52 +03:00
Andrew Ashikhmin
d9f342bb43
Restore MigrateFlags (#6055)
Another postfix to PR #6047. This fixes `--datadir` flag passed to
`erigon init`. See also
https://github.com/ethereum/go-ethereum/pull/24751.
2022-11-15 21:38:31 +01:00
Giulio rebuffo
3a2d3be080
Added persistence in LightClient (#6056) 2022-11-15 20:39:18 +01:00
Alex Sharov
242b5a76df
e3: fix doneCount variable (#6053) 2022-11-15 18:03:10 +03:00
Andrew Ashikhmin
ec9db8662c
Fix command line help (#6052)
Postfix after PR #6047.
2022-11-15 12:03:06 +01:00
Andrew Ashikhmin
14c0643476
Enable both eth/66 and eth/67 by default (#6048) 2022-11-15 10:41:56 +01:00
Andrew Ashikhmin
32629bdce4
Upgrade urfave/cli to v2 (#6047)
See also
https://github.com/urfave/cli/blob/main/docs/migrate-v1-to-v2.md
2022-11-14 17:33:57 +01:00
Andrew Ashikhmin
8152d5c7e4
txpool: sync.Mutex instead of RWMutex (#6045)
Pick up https://github.com/ledgerwatch/erigon-lib/pull/735
2022-11-14 12:24:14 +01:00
Giulio rebuffo
ae01026ca9
Customizable networking for Erigon Lightclient (#6032) 2022-11-11 23:22:47 +01:00
Andrew Ashikhmin
5b3f2b8c5f
Update interfaces (#6031)
Pick up
813d816908
& https://github.com/ledgerwatch/interfaces/pull/131.
2022-11-11 17:16:39 +01:00
ledgerwatch
5fc2ec4b52
Enable snapshot creation for Gnosis Chain (#6030)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-11-11 15:38:59 +00:00
Max Revitt
74308f2500
fix(test): eliminate race conditions (#6019) 2022-11-11 15:22:09 +00:00
Andrew Ashikhmin
09c5e4699b
Fix panic("durations cannot be empty") (#6029) 2022-11-11 15:24:58 +01:00
Andrew Ashikhmin
04e3a18463
Block difficulty can be really big on Gnosis Chain (#6028)
For example, see https://gnosisscan.io/block/24938312.

This fixes the following errors:
```
t=2022-11-11T12:39:01+0000 lvl=dbug msg="Handling incoming message" stream=RecvMessage err="newBlock66: too large block difficulty: bitlen 128"
t=2022-11-11T12:39:01+0000 lvl=dbug msg="Handling incoming message" stream=RecvMessage err="newBlock66: too large block difficulty: bitlen 128"
t=2022-11-11T12:39:01+0000 lvl=dbug msg="Handling incoming message" stream=RecvMessage err="newBlock66: too large block difficulty: bitlen 128"
```
2022-11-11 15:24:46 +01:00
Alex Sharov
c21a266236
Allow bigger jumps in 1 RwTx - for consistency (#6026) 2022-11-11 16:30:28 +07:00
Alex Sharov
90952c4110
e3: wal ram configure (#6023) 2022-11-11 11:28:22 +07:00
ledgerwatch
de26ccf6f6
Try non-parallel tests (#6021)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-11-10 23:47:35 +00:00
ledgerwatch
14906fec17
Bump version (#6020) 2022-11-10 22:18:01 +00:00
Giulio rebuffo
f8916e9226
Made Lightclient default (#5813)
lightclient cl is default, for external cl, run `--externalcl`
2022-11-10 17:06:04 +00:00
hexoscott
ea1edf4a11
trace json error fix (#6014)
write trace transaction errors to json and continue
2022-11-10 15:37:16 +00:00
hexoscott
475fa08fdf
eth_gasPrice performance tweaks (#6016)
Code news up an oracle for every call so existing cache checks always
came back as 0. Moved cache up a layer and pass in via the new
`gasprice.Cache` interface. Looked at putting the oracle instance onto
the ethApi itself to re-use it that way, but the backend transaction
made it a little hard work as we can't re-use that. This seemed cleaner
but happy to take feedback.

Locally takes me from ±2.5k rps to ±43k rps so quite a difference there.
(k6 with 1000 virtual users)
2022-11-10 15:36:04 +00:00
Mike Neuder
bf34bed004
Adding GetStatus to the req/resp domain (#6010)
Part of https://github.com/ledgerwatch/erigon/issues/5884 defined in the
spec
https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#status.

We need this functionality to continue
https://github.com/ledgerwatch/erigon/issues/5965 because we need to
determine how many blocks to request from the checkpoint block.
2022-11-10 14:14:26 +01:00
Andrew Ashikhmin
424f38a9b6
AuRa: iterate over durations in order (#6013)
Iteration order over go maps is random.
2022-11-10 13:47:57 +01:00
ledgerwatch
7f12b809ec
Improvements to the downloader (#6012)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-11-10 10:33:10 +00:00