erigon-pulse/cmd/integration
Alex Sharov e02d6acc7d
bitmap indices for logs (#1124)
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
..
commands bitmap indices for logs (#1124) 2020-09-28 18:18:36 +01:00
main.go Rpcdaemon as lib 2 (#943) 2020-08-20 10:52:27 +07:00
Readme.md integration_write_receipts (#1098) 2020-09-11 17:33:46 +07:00

Integration - tool to run TurboGeth stages in custom way: run single stage, or run all stages but reorg every X blocks, etc...

By examples. All commands require parameter --chaindata=/path/to/chaindata - I will skip it for readability.

integration --help
integration print_stages --block=1_000_000

# run all stages (which do not need internet). 
integration state_stages --unwind=1 --unwind_every=10  # process 10 blocks, then reorg 1 block, then process 10 blocks, ... 
integration state_stages --unwind=10 --unwind_every=1  # process 1 block, then reorg 10 blocks, then process 1 blocks, ...

# Run single stage 
integration stage_senders 
integration stage_exec  
integration stage_exec --block=1_000_000 # stop at 1M block
integration stage_hash_state 
integration stage_ih 
integration stage_history
integration stage_tx_lookup

# Drop data of single stage 
integration stage_exec --reset     
integration stage_history --reset
... 

# reset all data after stage_senders
integration reset_state

# hack which allows to force clear unwind stack of all stages
clear_unwind_stack

The way I usually run it:

go run -trimpath ./cmd/integration state_stages --chaindata=/path/to/chaindata --unwind=10 --unwind_every=20 --pprof 

Pre-requirements of state_stages command:

  • Headers/Bodies must be downloaded
  • TxSenders stage must be executed