Following our previous discussion on erigon's discord, this PR requests
to upstream all Otterscan modifications to erigon's repo.
That decision comes after getting feedback from lots of users at events
this year, and although it may introduce some friction for development,
it will make integrators life easier by having all our modifications
available out of box, e.g., dappnode users will get our RPCs since their
official packages are built from erigon repo.
I'm submitting the source-code as-is, please let me know if you think
there is a better code organization.
The current set of modifications comprises only new RPCs. There are some
proposals for extra-stages that would add new tables, but they are still
WIP and will be submitted separately in future after more testing.
this pr has two things in it
1. changed filter logs to use a map for the topics. this will speed up
queries with many topics in them. I still don't have a use case for this
though. i put is as a method of Logs, since that made sense to me, happy
to move it back out though.
2. allows json-rpc over http get request. since firefox is a great json
viewer (can search through, collapse large results) i often use it to
debug. it is also useful for sharing data with those who are less
familiar with command line / programming.
example get request:
http://rpcdaemon/?method=eth_getLogs¶ms=[{"fromBlock":"0xf2316b","toBlock":"0xf2316b"}]
it is based on the old jsonrpc http specification
https://www.jsonrpc.org/historical/json-rpc-over-http.html#encoded-parameters
except we also accept not base64 encoded params. since every eth rpc
request has a [], it will immediately fail validation for base64 and
attempt to use the parameters as a they are. otherwise it will attempt
to parse the rest of the payload as base64 and use that.
Co-authored-by: a <a@a.a>
Co-authored-by: gfx <86091021+gfxlabs@users.noreply.github.com>
- lives in internal/logging
- all log flags moved to internal/logging/flags
- allows continued use of root logger via log.Info etc.
- update logger to take change allowing string to lvl for 'trace'
Verbosity flag is overridden by log.console.verbosity. Logs will be
colocated if all run as one process, only split where progs are run as
separate processes, in a future update this will be addressed so for
example rpcdeamon will always log to it's own file
this pr changes filterLogs to use a pre computed hashset of addresses,
instead of iterating across the list of addresses once per log.
this greatly increases the speed of filter queries that use many
addresses and also return a large number of logs. In our case, we are
performing a query for all the trades performed in a uniswap v3 pool in
a 250 block range.
my benchmarks were performed with the data & code below:
address list gist is here
[addrs](2c30b0df43/gistfile1.txt)
```
c := NewRpcClient()
addrs := []common.Address{AddressListGist}
logs, err := c.FilterLogs(context.TODO(), ethereum.FilterQuery{
FromBlock:big.NewInt(15640000),
ToBlock: big.NewInt(15640250),
Addresses: addrs,
Topics: [][]common.Hash{
{
common.HexToHash("c42079f94a6350d7e6235f29174924f928cc2ac818eb64fed8004e115fbcca67"),
},
},
```
the query contains 8442 addresses, while the response contains 1277 logs
On average, current devel averages a 15.57 second response time on my machine after 10 runs, while the new filterLogs averages 1.05 seconds.
for CURRENT DEVEL, the profile is here: https://pprof.aaaaa.news/cd8dkv0tidul37sctmi0/flamegraph
for the filterLogs branch, the profile is here: https://pprof.aaaaa.news/cd8dlmgtidul37sctmig/flamegraph
while the tests pass with this branch, I am not really sure why filterLogs was originally programmed the way it was. Is there some sort of edge case / compatibility thing that I am missing with this change?
Co-authored-by: a <a@a.a>
the root cause is that when `inMemoryExecution` lambda gets created in
the `eth/backend.go`, it captures the reference of
`backend.notifications`, and so the execution of side-forks actually
adds notifications to there, and it all gets sent out to tx pool (and
RPC daemon) at the end of the stage loop (regardless of whether there
was forkchoice update or not)
so we can create a separate notification, but then somehow flush it to
the "main" nofitications when the in-memory exec state is flushed
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
* cleaned up storage and retrieval of state sync receipts
* added ctx to the error and got rid of unused var
* missing one
* small change
* log error
* ops
* read header number
* checking if status is succesful
* nil ptr err
