According to EIP-1186 the `proof` parts of the response to eth_getProof
should be returned "starting with the stateRoot-Node, following the path
of the SHA3 (address) as key." Currently, the proof is returned in
traversal order, rather than from the root.
Although all of the proof elements are there and correct, this is
contrary to the EIP and will cause problems for some clients. The
existing rpc test uses a map to lookup proof elements by hash, rather
than by index, so this bug was not initially caught.
This commit fixes the behavior, updates the existing test, and adds
additional checks to the rpc test.
Co-authored-by: Jason Yellick <jason@enya.ai>
`excessDataGas` has been partially made eip-4844 ready, so instead of
passing nils to functions, now it actually assigned to some value (it is
expected to be nil until cancun update).
when assigning default bor parameters, need to make sure config exists
first. since not sending the config will result in it being nil, not the
empty struct.
it would appear that an extra `e` was added somewhere along the way to
the cli flags for txpool. If this was intentional I was curious as to
why, otherwise it seems like fixing the flag would remedy some
confusion. Thank you
Upon further investigation it would appear
https://github.com/ledgerwatch/erigon/blob/devel/cmd/utils/flags.go#L173
shows corrected spelling.
Separate:
- new tasks coming by channel with limited capacity (touching the limit
will block - like sync.Cond.Wait() does, but composable). as a result -
don't need track queue size (it will not increase conflict-rate).
- re-exec tasks are going to priority-queue (which is also higher
priority than 1-st `chan`).
This PR completes the implementation of `eth_getProof` by adding support
for storage proofs.
Because storage proofs are potentially overlapping, the existing
strategy of simply aggregating all proofs together into a single result
was challenging. Instead, this commit rewires things to introduce a
ProofRetainer, which aggregates proofs and their corresponding nibble
encoded paths in the trie. Once all of the proofs have been aggregated,
the caller requests the proof result, which then iterates over the
aggregated proofs, placing each into the relevant proof array into the
result.
Although there are tests for `eth_getProof` as an RPC and for the new
`ProofRetainer` code, the code coverage for the proof generation over
complex tries is lacking. But, since this is not a new problem I'll plan
to follow up this PR with an additional one adding more coverage into
`turbo/trie`.
---------
Co-authored-by: Jason Yellick <jason@enya.ai>
## Changelog
Fixed the following Grafana panels:
* DB Size
* GC and State
* DB Pages Ops/sec
* Commit Counters
* Getrusage
* Latest Block
Also partially fixes https://github.com/ledgerwatch/erigon/issues/7226
@shyba hi, seems this lib doesn't work with
ghcr.io/goreleaser/goreleaser-cross (which producing release binaries)
removing it for now, feel free to add it in future - if can make it work
with goreleaser-cross
see: https://github.com/ledgerwatch/erigon/issues/7210
- Fix tx numbers expected
- Check for baseFee subpool size too
- Fix tx counting
- Wait for blocks in baseFee subpool to be promoted and mined
- Add BlockNumber request
Small change in core.NewEVMBlockContext and now it expects
excessDataGas. This will be used in state transition to compute data fee
for eip-4844 data blobs. The logic that computes it will be added in the
next PRs.
This PR contains very small EIP-4844 additions. GasPool is modified and
now it is a struct with 2 fields "gas" and "dataGas" (blobs are priced
in dataGas). ExcessDataGas block header field added. ExcessDataGas
needed to compute the data gas price. EIP-4844 helper functions are
added as well.
This PR starts with a few small commits of code cleanup. Reviewed
separately they should hopefully obviously be functionally no-ops. I'm
happy to strip these out and submit them separately if desired.
The final commit is to add support for older blocks as a parameter to
eth_getProof. In order to compute proofs, the function leverages the
staged sync unwinding code to bring the hashed state table back to its
historic state, as well as to build a list of trie nodes which need to
be invalidated/re-computed in the trie computation. Because these
operations could be expensive for very old blocks, it limits the
distance proofs are allowed from the head. It also adds some additional
checks for correctness, as well as tests which verify the
implementation.
This was discussed a bit on Discord in the db-format topic.
---------
Co-authored-by: Jason Yellick <jason@enya.ai>
Hi,
I'm syncing Gnosis on a Celeron N5100 to get familiar with the codebase.
In the process I managed to optimize some things from profiling.
Since I'm not yet on the project Discord, I decided to open this PR as a
suggestion. This pass all tests here and gave me a nice boost for that
platform, although I didn't have time to benchmark it yet.
* reuse VM Memory objects with sync.Pool. It starts with 4k as `evmone`
[code
suggested](0897edb001/lib/evmone/execution_state.hpp (L49))
as a good value.
* set32 simplification: mostly cosmetic
* sha256-simd: Celeron has SHA instructions. We should probably do the
same for torrent later, but this already helped as it is very CPU bound
on such a low end processor. Maybe that helps ARM as well.
This PR adds missing tests for eth_getProof and does some mild
refactoring with switching from strings to more strict types. It's
likely best/most easily reviewed commit by commit.
Note, the tests include quite a number of helper types and functions for
doing the proof validation. This is largely because unlike Geth,
Erigon's approach to trie computations only requires serializing the
trie nodes, not deserializing them. Consequently, it wasn't obvious how
to leverage the existing trie code for doing deserialization and proof
checks. I checked on Discord, but, there were no suggestions. Of course,
any feedback is welcome and I'd be happy to remove this code if it can
be avoided.
Additionally, I've opted to change the interface type for `GetProof` to
use a `common.Hash` for the storage keys instead of a `string`. I
_think_ this should be fairly safe, as until very recently it was
unimplemented. That being said, since it's an interface, it has the
potential to break other consumers, anyone who was generating mocks
against it etc. There's additionally a `GetStorageAt` that follows the
same parameter. I'd be happy to submit a PR modifying this one as well
if desired.
Also, as a small note, there is test code for checking storage proofs,
but, storage proofs aren't currently supported by the implementation. My
hope is to add storage proofs and historic proofs in a followup PR.
---------
Co-authored-by: Jason Yellick <jason@enya.ai>
Gateway reported an issue with a trace returning an odd result, similar
to the recent problem we'd seen with gnosis. I found that
debug_traceTransaction worked fine so found where the differences were.
trace_transaction wasn't checking for service transactions so the trace
failed around fees when it shouldn't.
A number of code paths use callManyTransactions so they should all now
check for service messages where needed.
added subcommand `state_domains` of `bin/integration` to start
processing of existing chaindata with use of state domains and
commitment.
It creates two directory in `datadir`: `state` and `statedb` for files
and mdbx respectively.
This version runs blocks one after another and produces merged files.
Want to add some metrics to export later.
This is a partial implementation of eth_getProof (see issue #1349),
supporting only a request for the latest block and an empty list of
storage keys (i.e. Account proof only). I don't know if there's a better
way of implementing this, but this was what I could come up with.
Posting it here in case it's useful.
Example output:
```
> eth.getProof("0x67b1d87101671b127f5f8714789C7192f7ad340e", [], 'latest')
{
accountProof: ["0xf90131a0252c9d4ed347b4cf3fdccaea3ccef0a507e6bd4dbe4dcd98609b7195347c4062a0ab8cdb808c8303bb61fb48e276217be9770fa83ecf3f90f2234d558885f5abf18080a01a697e814758281972fcd13bc9707dbcd2f195986b05463d7b78426508445a04a0b5d7a91be5ee273cce27e2ad9a160d2faadd5a6ba518d384019b68728a4f62f4a0c2c799b60a0cd6acd42c1015512872e86c186bcf196e85061e76842f3b7cf86080a0e73919d9f472eec11f6da95518503f5527a98b9428f7a02c4f55bf51854214e480a06301b39b2ea8a44df8b0356120db64b788e71f52e1d7a6309d0d2e5b86fee7cb8080a01b7779e149cadf24d4ffb77ca7e11314b8db7097e4d70b2a173493153ca2e5a0a066a7662811491b3d352e969506b420d269e8b51a224f574b3b38b3463f43f0098080", "0xf8518080808080a0a00135c9ec2655cb6a47ab7ad27d6fc150d9cba8b3d4a702e879179116a68a60808080808080a02fb46956347985b9870156b5747712899d213b1636ad4fe553c63e33521d567a80808080", "0xf873a02056274a27dd7524955417c11ecd917251cc7c4c8310f4c7e4bd3c304d3d9a79b850f84e808a021e19e0c9bab2400000a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470"],
address: "0x67b1d87101671b127f5f8714789c7192f7ad340e",
balance: "0x21e19e0c9bab2400000",
codeHash: "0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470",
nonce: "0x0",
storageHash: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
storageProof: []
}
> eth.getBlock('latest').stateRoot
"0x6a0673c691edfa4c4528323986bb43c579316f436ff6f8b4ac70854bbd95340b"
```
Passing consensus-specs tests for Capella. Processing of withdrawals and
ExecutionChanges. efficient non-validation implemented. Refactored:
ExecutionPayload/ExecutionPayloadHeader.
This branch implements support of btree indexes instead of .kvi files.
Kvi also created, but not used during search/lookup, only for size
comparation.
Btree allows to make seek calls, while recsplit indexes does not support
that query.
During initial tests, btree indexes are ~1-3% smaller than recsplit but
avg lookup time bigger up to 1.5us (which compensates with ability to
fetch next key).
BtAlloc
```
337000 searches, last took 5.417µs avg=4.385µs next_access_last[of 10000 keys] 105ns
338000 searches, last took 3.792µs avg=4.385µs next_access_last[of 10000 keys] 103ns
339000 searches, last took 5.125µs avg=4.386µs next_access_last[of 10000 keys] 103ns
340000 searches, last took 4.75µs avg=4.386µs next_access_last[of 10000 keys] 103ns
341000 searches, last took 3.875µs avg=4.386µs next_access_last[of 10000 keys] 104ns
342000 searches, last took 6.042µs avg=4.387µs next_access_last[of 10000 keys] 107ns
343000 searches, last took 3.209µs avg=4.388µs next_access_last[of 10000 keys] 108ns
344000 searches, last took 2.292µs avg=4.393µs next_access_last[of 10000 keys] 113ns
345000 searches, last took 2.417µs avg=4.394µs next_access_last[of 10000 keys] 106ns
346000 searches, last took 2.625µs avg=4.394µs next_access_last[of 10000 keys] 103ns
347000 searches, last took 4.417µs avg=4.397µs next_access_last[of 10000 keys] 104ns
348000 searches, last took 5.458µs avg=4.398µs next_access_last[of 10000 keys] 103ns
349000 searches, last took 4.875µs avg=4.4µs next_access_last[of 10000 keys] 102ns
350000 searches, last took 4.25µs avg=4.4µs next_access_last[of 10000 keys] 104ns
351000 searches, last took 6.25µs avg=4.403µs next_access_last[of 10000 keys] 106ns
352000 searches, last took 3.125µs avg=4.404µs next_access_last[of 10000 keys] 106ns
```
Recsplit
```
354000 searches, last took 1.209µs avg=3.081µs
355000 searches, last took 1.333µs avg=3.076µs
356000 searches, last took 1.541µs avg=3.071µs
357000 searches, last took 1.333µs avg=3.066µs
358000 searches, last took 1.333µs avg=3.062µs
359000 searches, last took 1.167µs avg=3.057µs
360000 searches, last took 1.791µs avg=3.052µs
361000 searches, last took 1.209µs avg=3.048µs
362000 searches, last took 1.375µs avg=3.043µs
363000 searches, last took 1.166µs avg=3.038µs
364000 searches, last took 1.625µs avg=3.033µs
365000 searches, last took 1.125µs avg=3.029µs
366000 searches, last took 1.208µs avg=3.024µs
367000 searches, last took 1.333µs avg=3.019µs
368000 searches, last took 1.791µs avg=3.015µs
369000 searches, last took 1.333µs avg=3.01µs
370000 searches, last took 1.166µs avg=3.006µs
371000 searches, last took 1.125µs avg=3.001µs
372000 searches, last took 1.333µs avg=2.997µs
```
https://github.com/ledgerwatch/erigon-lib/pull/891/files
- [x] setup infra
- [x] provide readme
- [x] setup secrets (keys) for infra
- [x] resolve todos
- [x] remove unnecessary test actions triggers
- [x] update go path (should be installed on runner properly - unlike my
self hosted quick setup)
- [x] provide docs on using checksum and signature to verify the binary
**Current Problem**
When X transactions are created, with M transactions out of them having
`gasFeeCap` greater than the current `baseFeePerGas` for the block, and
N transactions having `gasFeeCap` lower, all X transactions get added to
the yellow pool (baseFee) in the txpool and they never get mined.
However when X transactions all having `gasFeeCap` higher than the
`baseFeePerGas` are created, they all get added to the green pool
(pending) in the txpool and thus get mined.
This phenomenon can be inspected in the
`signEIP1559TxsLowerAndHigherThanBaseFee2` function where the number of
transactions that should have `gasFeeCap` higher than and lower than the
current `baseFeePerGas` is specified in its parameter - `func
signEIP1559TxsLowerAndHigherThanBaseFee2(amountLower, amountHigher
int...`.
When `amountLower` is set as `0`, all transactions created from
`amountHigher` will be mined because they have `gasFeeCap` >
`baseFeePerGas`. When `amountLower` has a value > `0`, all the
transactions created (including transactions created with
`amountHigher`) will go to the yellow pool (baseFee) and thus do not get
mined.
The CLI logging currently emits the configured address when noting that
the HTTP or Engine RPC services are started. This is _usually_ the same
as the listening address, but not always. In particular, when the bind
directive specifies an ambiguous parameter, like port 0 for an ephemeral
port, the actually bound address will differ.
This change will help users to operate Erigon while binding to ephemeral
ports. This is especially helpful for developers trying to run multiple
instances of Erigon concurrently in tests which may or may not be
executing in parallel.
Co-authored-by: Jason Yellick <jason@enya.ai>
That's an initial PR mostly for code review, not ready for production
use
Got basic GraphQL working when querying a single block
---------
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Fees going to the gas fee recipient should be based on the actual gas
used (available in the receipt) rather than the gas limit in a
transaction. This fixes Hive test "GetPayloadV2 Block Value".
Also `engine_getPayloadBodiesByRangeV1` params should be encoded as hex
strings.
So... for blocks that don't exist, the eth_getBlockTransactionCount RPCs
should return `null`. Erigon does this correctly right now and this is
covered by the unit test in
[rpcdaemon/commands/corner_cases_support_test.go](https://github.com/ledgerwatch/erigon/blob/devel/cmd/rpcdaemon/commands/corner_cases_support_test.go).
But, for blocks that don't have any transactions, like
[703696](https://etherscan.io/block/703696), the RPC also returns
`null`.
I've changed this to correctly return `0` and have added 2 new tests to
handle it. (I had to add another block to the test chain in
rpcdaemon/rpcdaemontest/test_util.go and had to fix a few other tests
that broke as a result.)
Resolves#6775
this pr does the following:
1. adds new function to ApiImpl `GetFilterLogs` which should implement
`eth_getFilterLogs` (eth_getFilterChanges except with only logs filters)
2. changes the ID generator of rpchelper.Filters to use crypto/rand.
3. switched logs subscriptions to use the secure ID instead of number
4. changes subcription ids from an 8 byte hex string to a 16 byte
It turns out that "standard" BSC nodes based on Geth, do not propagate
new block hashes and blocks, at least towards Erigon nodes. This is a
workaround
---------
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
In v2.36.0 we correctly trace this transaction, however due to a code
ordering issue in previous releases, there was an 'out of gas' response,
because we fell through to the code here where
ErrContractAddressCollision will be returned as such. This change isn't
the fix, but ensures the actual error is returned.
