Refactors filters.go such that map+locks are now in their own class
Move logic for safely canceling & draining channel to its own class
changed subscriptions to ask for size of buffer and construct its own
channel
marked as draft b/c need to do live testing.
Moved some of the tracers to `eth/tracers/logger` to make it more
similar to go-ethereum.
Removed Erigon-specific `Capture-` functions
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
In `debug_traceCallMany` and `eth_callMany` the `IntraBlockState` is not
being finalized after each replayed transaction and after each call.
This can cause several problems:
- False gas consumption calculation. Specifically in `SSTORE` dynamic
gas calculation being affected by "warm" and "cold" storage slots, and
by "original" value comparison.
- Ability to call contracts that were `SELFDESTRUCT`ed in the replayed
transactions or during previous calls, as the self-destruction has not
been finalized.
see https://github.com/ledgerwatch/erigon/issues/6373 as an example.
This pull request adds a call to `FinalizeTx` after each `ApplyMessage`
and `TraceTx`
This fixes Otterscan tx tracer, which broke on the following
refactoring:
60cb4e2bbb
It used to receive value == -1 for delegatecall, now it is receiving
nil, so we safeguard against it now.
Remove `callType` argument from `CaptureStart`, and change `callType
vm.CallType` to `typ vm.OpCode` in `CaptureEnter` and move to the first
position (as it is in geth)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
Splitting function `CaptureStart` into `CaptureStart` (when depth == 0)
and `CaptureEnter` (when depth > 0), while removing argument `depth`.
Same with splitting `CaptureEnd` into `CaptureEnd` and `CaptureExit`
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
In this first step, the new functions `CaptureTxStart` and
`CaptureTxEnd` are introduced to all tracers
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro-2.local>
FIrst of all, thanks in advance for your advice.
I've found that `trace_replayTransaction` is having trouble tracing some
Polygon State Sync transactions such as
[`0xd5f4f8c3cd85cf65e8df23a2c1ae02aefda1e6293db0c3a9ddcc08cee8ca1131`](https://polygonscan.com/tx/0xd5f4f8c3cd85cf65e8df23a2c1ae02aefda1e6293db0c3a9ddcc08cee8ca1131),
just like [the case of the previous PR][p].
[p]: https://github.com/ledgerwatch/erigon/pull/6286
```shell
$ curl -XPOST 'http://localhost:8545' \
-H 'Content-Type: application/json' \
--data '{"method":"trace_replayTransaction","params":["0xd5f4f8c3cd85cf65e8df23a2c1ae02aefda1e6293db0c3a9ddcc08cee8ca1131",["trace","stateDiff"]],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":null}
```
This is because RPCDaemon doesn't query for blocks by Bor Hash, even
though `api.txnLookup` has failed. Same as #6286.
Same as #6286 and #6288
This patch closes#6276
```shell
$ curl -XPOST 'http://localhost:8545' \
-H 'Content-Type: application/json' \
--data '{"method":"eth_getTransactionReceipt","params":["0x31ce15ce9a1ff347f4204a1ed3625861165c53ae08743c1f36a32865c62744c6"],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"block has less receipts than expected: 0 \u003c= 0, block: 36635776"}}
```
cc. @0xKrishna
FIrst of all, thanks in advance for your advice.
I've found that `eth_getTransactionByHash` is having trouble retrieving
some Polygon State Sync transactions such as
[`0xd5f4f8c3cd85cf65e8df23a2c1ae02aefda1e6293db0c3a9ddcc08cee8ca1131`](https://polygonscan.com/tx/0xd5f4f8c3cd85cf65e8df23a2c1ae02aefda1e6293db0c3a9ddcc08cee8ca1131),
while `eth_getBlockByNumber` can retrieve them without problem.
```shell
$ curl -XPOST 'http://localhost:8545' \
-H 'Content-Type: application/json' \
--data '{"method":"eth_getTransactionByHash","params":["0xd5f4f8c3cd85cf65e8df23a2c1ae02aefda1e6293db0c3a9ddcc08cee8ca1131"],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":null}
```
This is because RPCDaemon doesn't query for blocks by Bor Hash, even
though `api.txnLookup` has failed.
Works around a flaw in the upgrade logic of the system contracts. Since
they are updated directly, without first being self-destructed and then
re-created, the usual incarnation logic does not get activated, and all
historical records of the code of these contracts are retrieved as the
most recent version. This problem will not exist in erigon3, but until
then, a workaround will be used to access code of such contracts through
a special structure, `SystemContractCodeLookup`
Fixes https://github.com/ledgerwatch/erigon/issues/5865
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
So there is an issue with tracing certain blocks/transactions on
Polygon, for example:
```
> '{"method": "trace_transaction","params":["0xb198d93f640343a98f90d93aa2b74b4fc5c64f3a649f1608d2bfd1004f9dee0e"],"id":1,"jsonrpc":"2.0"}'
```
gives the error `first run for txIndex 1 error: insufficient funds for
gas * price + value: address 0x10AD27A96CDBffC90ab3b83bF695911426A69f5E
have 16927727762862809 want 17594166808296934`
The reason is that this transaction is from the author of the block,
which doesn't have enough ETH to pay for the gas fee + tx value if he's
not the block author receiving transactions fees.
The issue is that currently the APIs are using `ethash.NewFaker()`
Engine for running traces, etc. which doesn't know how to get the author
for a specific block (which is consensus dependant); as it was noting in
several TODO comments.
The fix is to pass the Engine to the BaseAPI, which can then be used to
create the right Block Context. I chose to split the current Engine
interface in 2, with Reader and Writer, so that the BaseAPI only
receives the Reader one, which might be safer (even though it's only
used for getting the block Author).
this pr adds CLI flag to allow the rpcdaemon to bind to a TCP port.
this is very useful if one wants to maintain a remote connection with
the rpcdaemon without using websocket. This is useful because a lot of
issues come with the websocket protocol (compression, max size, etc).
TCP socket gets around these things (it is just raw json over tcp
stream)
the rpc package already supports this, it was just a matter of adding
the bind.
try `echo
'{"jsonrpc":"2.0","method":"eth_blockNumber","id":"1","params":[""]}' |
nc localhost 8548` as a basic test
to test. Subscriptions are also working (idk how to send keepalives with
netcat)
the default rpc.(*Client).Dial method does not support TCP. I have not
included that in this PR. The code for such is as follow
```
// DialTCP create a new TCP client that connects to the given endpoint.
//
// The context is used for the initial connection establishment. It does not
// affect subsequent interactions with the client.
func DialTCP(ctx context.Context, endpoint string) (*Client, error) {
parsed, err := url.Parse(endpoint)
if err != nil {
return nil, err
}
ans := make(chan *Client)
errc := make(chan error)
go func() {
client, err := newClient(ctx, func(ctx context.Context) (ServerCodec, error) {
conn, err := net.Dial("tcp", parsed.Host)
if err != nil {
return nil, err
}
return NewCodec(conn), nil
})
if err != nil {
errc <- err
return
}
ans <- client
}()
select {
case err := <-errc:
return nil, err
case a := <-ans:
return a, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
// DialContext creates a new RPC client, just like Dial.
//
// The context is used to cancel or time out the initial connection establishment. It does
// not affect subsequent interactions with the client.
func DialContext(ctx context.Context, rawurl string) (*Client, error) {
u, err := url.Parse(rawurl)
if err != nil {
return nil, err
}
switch u.Scheme {
case "http", "https":
return DialHTTP(rawurl)
case "ws", "wss":
return DialWebsocket(ctx, rawurl, "")
case "tcp":
return DialTCP(ctx, rawurl)
case "stdio":
return DialStdIO(ctx)
case "":
return DialIPC(ctx, rawurl)
default:
return nil, fmt.Errorf("no known transport for URL scheme %q", u.Scheme)
}
}
```
let me know if you would like me to add this to the PR as well. the TCP
connection can then be established with `rpc.Dial("tcp://host:port")`
feat:
1. `erigon_getLatestLogs` doesn't have to match the exact position of
the topics. It will match logs that contain the topics regardless of the
topics' position with original bloom filter. And it accepts `blockCount`
& `crit.ToBlock` params for better pagination.
This PR includes changes required for delhi hard fork schedule at block
`29638656` on mumbai testnet. It changes few major parameters.
1. Sprint length - the number of bor blocks post which a new validator
mines has been reduced from 64 to 16.
2. Block time - the block time which was increased earlier for some
experiments to 5 seconds has been reduced to 2 seconds (along with
backup multiplier and producer delay).
3. Base fee denominator - this fields has been increased from 8 to 16 to
smoothen the effect of EIP 1559.
One simple change to send the header in rather than the body allowing
re-use in a loop which saves the copy call when looping.
The other one for a reusable evm seems potentially dangerous so feedback
more than welcome on that one, local testing shows it gains me around
6k±rps so if it's safe is a good win, but I feel it will need more work.
I could only validate against goerli chain which seems to always return
the same value for eth_estimateGas so if anybody could validate it
against another chain that would be awesome.
impacted API:
eth_getTransactionByHash()
eth_getTransactionByNumber()
eth_getTransactionByBlockHashAndIndex()
eth_getTransactionByBlockNumberAndIndex()
eth_getBlockByHash()
eth_getBlockByNumber()
1) In case of legacy transitions the chainId field should be inserted
only if V is not 27/28.
This seems also the Geth/v1.10.23 behaviour via infura
2) In case of dynamicFee/AccessList transaction the access_list should
be inserted in the response also if empty
This is done correctly in cmd/rpcdaemon/commands/eth_api.go and NOT in
internal/ethapi/api.go
This seems also the Geth/v1.10.23 behaviour via infura
Code news up an oracle for every call so existing cache checks always
came back as 0. Moved cache up a layer and pass in via the new
`gasprice.Cache` interface. Looked at putting the oracle instance onto
the ethApi itself to re-use it that way, but the backend transaction
made it a little hard work as we can't re-use that. This seemed cleaner
but happy to take feedback.
Locally takes me from ±2.5k rps to ±43k rps so quite a difference there.
(k6 with 1000 virtual users)
Will gain more perf improvements when the change for kvcache goes in
from erigon-lib.
Hopefully can be validated with hive? My main concern is the re-use of
the state reader, I couldn't find any manipulation of it so it looks
safe to re-use.
feat: add `erigon_getLatestLogs` as a new feature API.
1. `erigon_getLatestLogs` returns latest logs that match the filter with
`logCount` length. Implementation is similar to `erigon_getLogs` but it
uses `ReverseIterator` which makes it more efficient to fetch the latest
logs.
Following our previous discussion on erigon's discord, this PR requests
to upstream all Otterscan modifications to erigon's repo.
That decision comes after getting feedback from lots of users at events
this year, and although it may introduce some friction for development,
it will make integrators life easier by having all our modifications
available out of box, e.g., dappnode users will get our RPCs since their
official packages are built from erigon repo.
I'm submitting the source-code as-is, please let me know if you think
there is a better code organization.
The current set of modifications comprises only new RPCs. There are some
proposals for extra-stages that would add new tables, but they are still
WIP and will be submitted separately in future after more testing.
this pr has two things in it
1. changed filter logs to use a map for the topics. this will speed up
queries with many topics in them. I still don't have a use case for this
though. i put is as a method of Logs, since that made sense to me, happy
to move it back out though.
2. allows json-rpc over http get request. since firefox is a great json
viewer (can search through, collapse large results) i often use it to
debug. it is also useful for sharing data with those who are less
familiar with command line / programming.
example get request:
http://rpcdaemon/?method=eth_getLogs¶ms=[{"fromBlock":"0xf2316b","toBlock":"0xf2316b"}]
it is based on the old jsonrpc http specification
https://www.jsonrpc.org/historical/json-rpc-over-http.html#encoded-parameters
except we also accept not base64 encoded params. since every eth rpc
request has a [], it will immediately fail validation for base64 and
attempt to use the parameters as a they are. otherwise it will attempt
to parse the rest of the payload as base64 and use that.
Co-authored-by: a <a@a.a>
Co-authored-by: gfx <86091021+gfxlabs@users.noreply.github.com>
- lives in internal/logging
- all log flags moved to internal/logging/flags
- allows continued use of root logger via log.Info etc.
- update logger to take change allowing string to lvl for 'trace'
Verbosity flag is overridden by log.console.verbosity. Logs will be
colocated if all run as one process, only split where progs are run as
separate processes, in a future update this will be addressed so for
example rpcdeamon will always log to it's own file
this pr changes filterLogs to use a pre computed hashset of addresses,
instead of iterating across the list of addresses once per log.
this greatly increases the speed of filter queries that use many
addresses and also return a large number of logs. In our case, we are
performing a query for all the trades performed in a uniswap v3 pool in
a 250 block range.
my benchmarks were performed with the data & code below:
address list gist is here
[addrs](2c30b0df43/gistfile1.txt)
```
c := NewRpcClient()
addrs := []common.Address{AddressListGist}
logs, err := c.FilterLogs(context.TODO(), ethereum.FilterQuery{
FromBlock:big.NewInt(15640000),
ToBlock: big.NewInt(15640250),
Addresses: addrs,
Topics: [][]common.Hash{
{
common.HexToHash("c42079f94a6350d7e6235f29174924f928cc2ac818eb64fed8004e115fbcca67"),
},
},
```
the query contains 8442 addresses, while the response contains 1277 logs
On average, current devel averages a 15.57 second response time on my machine after 10 runs, while the new filterLogs averages 1.05 seconds.
for CURRENT DEVEL, the profile is here: https://pprof.aaaaa.news/cd8dkv0tidul37sctmi0/flamegraph
for the filterLogs branch, the profile is here: https://pprof.aaaaa.news/cd8dlmgtidul37sctmig/flamegraph
while the tests pass with this branch, I am not really sure why filterLogs was originally programmed the way it was. Is there some sort of edge case / compatibility thing that I am missing with this change?
Co-authored-by: a <a@a.a>
the root cause is that when `inMemoryExecution` lambda gets created in
the `eth/backend.go`, it captures the reference of
`backend.notifications`, and so the execution of side-forks actually
adds notifications to there, and it all gets sent out to tx pool (and
RPC daemon) at the end of the stage loop (regardless of whether there
was forkchoice update or not)
so we can create a separate notification, but then somehow flush it to
the "main" nofitications when the in-memory exec state is flushed
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
* cleaned up storage and retrieval of state sync receipts
* added ctx to the error and got rid of unused var
* missing one
* small change
* log error
* ops
* read header number
* checking if status is succesful
* nil ptr err
* dont check for block reward or uncles if we are tracing POS
* comparing chainConfig to header difficulty
* fixed test
* difficulty == 0 then we are POS
* Fixes for trace_block
* Print
* Print
* extra bodies
* extra bodies
* Add canonical check
* First fix for bodies mismatch
* More cleanup
* Advance progress
* Reset snapshots in integration
* Cleanup
* Way to reset snapshots stage
* Not reset sequence if not needed
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* added a way to get latest executed block post POS
* added erigon_ExecutedBlockNumber into readme
* optional rpc.BlockNumber
* better message
* updated readme
* printing
* logging
* spam off
* ops
* clean up
* deleted all bor related stuff in snapshots
* adding bor tx to BorTxLookUp
* added readBorTxLookUp
* getting bor tx from bor lookup
* comments
* deleted bor related stuff
* deleting bor tx
* writing borTxLookUp in stage_txlookup
* using bytes
* little fixes
* updated erigon-lib
* using biEndian
* clean up
* setting bytes for big int
* log
* logging
* more logs
* delete logs
