* rpcdaemon performance improve - less reading blocks, less allocs on server
* rpcdaemon performance improve - less reading blocks, less allocs on server
* rpcdaemon performance improve - less reading blocks, less allocs on server
* rpcdaemon performance improve - less reading blocks, less allocs on server
* don't use global variables
* df
* avoid use non-transactional db instance to reduce amount of cursors
* trace_call initial
* Fix tests
* More tracing
* Add more fields to the action
* Completed first example query
* Add initial bench11 to compare trace_call with OpenEthereum
* Exclude precompile calls from call traces
* Add self-destruct, call types, more comparison in rpctest
* Support for execution errors
* Stack underflow error and delegatecall value
* Fix lint
* Fix suicide traceAddress, Bad instruction error
* Fix lint
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Expose C Handle of lmdbTx
* LoadExecutionFunctionPointer
* silkworm ExecuteBlocks
* linter
* CLI flag for SilkwormExecutionFunc
* linter
* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage
* Commit transaction after Silkworm execution
* Fix batch initialization
* Make batch_size uint64_t instead of size_t
* max_block in silkworm_execute_blocks
* More accurate logProgress
* Use CHandle exposed by lmdb-go
* Enable Silkworm on Linux only
* Update lmdb-go
* Error when attempting to use Silkworm not on Linux
* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go
* Integrate Silkworm into cmd/integration
* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution
* Refactor SpawnExecuteBlocksStage
* linters
* Small clean-ups
* Move ChangeSetHook inside executeBlockWithGo
* No need to tamper with batch size in executeBlocksWithSilkworm
* fix
* Fix
* Fixup
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* more compact append implementation
* do appned for dupsort buckets
* do appned for dupsort buckets
* do appned for dupsort buckets
* do appned for dupsort buckets
* do appned for dupsort buckets
* fix tests
* fix tests
* fix `make grpc` on new checkouts
* update proto files
* add some stub
* prototype with fake events
* notifying about events
* pass events
* events are being sent
* transfer headers to filters
* create the “filters” struct
* implement new heads
* PoC of New Heads subscription
* fix keep alive
* fixups for the client
* add “type” to the event
* support header event type on client
* better stage refactor
* fixup for the eth backend
* fixups
* fix tests
* fix tests
* fix linters
* address comments
* remove unused log
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* change_set_dup
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* working version
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* aa
* squash
* squash
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* history_early_stop
* history_early_stop
* vmConfig with ReadOnly false
* auto_increment
* auto_increment
* rebase master
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
* save progress
* save progress
* save progress
* save progress
* etl bimaps
* etl bimaps
* etl bimaps
* etl bimaps
* a
* a
* a
* a
* a
* a
* a
* save progress
* save progress
* save progress
* a
* fix_race_condition_on_zstd_build
* clean
* clean
* clean
* clean
* clean
* go mod tidy
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* fix corner case
* commit every
* call traces etl
* call traces etl
* call traces etl
* return missed error
* go mod tidy
* go mod tidy
* go mod tidy
* rebase to master
* remove trash
* print full key if it has 8 leading zeroes
* Add print_migration command to integration
* Not run migrations for print commands
* Reverse migration
* Fix lint
* Add datadir to run_migrations
* Add lmdb.mapSize to integration
* Make lmdb.mapSize effective
* Fix
* receipt sizes
* fix print
* Switch to another version of lmdb
* Collector from existing files
* Not load
* Commit after clearing
* Add methods for clear/drop buckets by multiple transactions
* Fix to lmdb
* Not remove temp files
* Not load
* Re-enable loading
* Prints
* Try to fix refill
* Migration in 3 steps
* tidy mod
* Fix lint
* Fix tests
* Migrations for CBOR receipts in 3 txs
* Fix lint
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* Fixes issue #1232
* Adding/updating test cases
* Updating as per comments
* Use tx instead of db in GetModifiedAccounts and clean up a bit
* option to specify LMDB map size for cmd/integration commands (#1238)
* Add lmdb.mapSize to integration
* Make lmdb.mapSize effective
* Use mapSize for first opening
* Fix for cobra
* Extend lmdb.mapSize for all commands
* Fix lint
* Fixes issue #1232
* Adding/updating test cases
* Updating as per comments
Co-authored-by: ledgerwatch <akhounov@gmail.com>
* save state
* torrent experiments
* torrent experiment passed
* fixes after merge
* snapshot headers processing passed
* save state
* save state
* download headers works after snapshot processing
* save state
* save state
* save state
* save state
* add lazy load tx to snapshots, increase number of trackers
* save state
* speedup getting info
* change logging
* move to turbo package
* save state
* save state
* save state
* cleanup
* save state
* add test test
* save state
* lmdb debugging
* fix readonly mode
* save state
* fix build
* sync works
* save state
* save state
* save state
* allow cmd stages stageSenders use snapshots
* debugging failed hashing
* remove experimental tests
* remove torrent experimental tests
* fix lint
* extract snapshot wrapper
* metainfo checker
* add remote seeder
* add logs
* update gomod
* remove useless code
* fix lint&remove useless code
* extract verify snapshot to separated command
* skip debug test
* fix test
* change type of seedSnapshot flag
* add eth logger to torrent lib
* skip debug test
* add Close method
* review fixes
* fix lint
* tidy mods
* Fix compile
* Fix lint
* Fix rpcdaemon running in the docker
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* Initial commit for CallTraces index
* Fix compilation
* fix lint, add comment
* Fix integration
* Add Close function to ethdb.Cursor, fix some compile errors
* Try to stop cursor leak in Get
* Fix compile errors in RPC daemon
* Fix compile errors
* fixing another way
* Some fixes
* More fixes
* More fixes
* More fixes
* Fixes to core/state
* Fix lint
* Fix lint
* Fixes
* Stage caching for call trace stage
* Add mem stats
* Try to stop the leak
* Turn off debug
* Chunks for 10k blocks
* Print
* Revert "Print"
This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.
* Revert "Chunks for 10k blocks"
This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.
* Trying to fix the leak
* Don't compute receipts in re-tracing
* Not compose block
* Print speed, fix receipts, bigger caches
* Fix lint
* Utilise changeset info
* Counters
* Use NoReceipts and ReadOnly
* ReadOnly is incompatible with caching
* Skip test leaking transactions
* Fix block test
* Change disable message for call-traces stage
* Use block option for call traces integration
* Fix retracing due to incarnation
* save progress
* GracefulShutdown grpc server, send to server close signal before canceling context
* GracefulShutdown json server
* GracefulShutdown json server
* fix lint
* clean
* hack hugeFreelist
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* save progress
* fix_race_condition_on_zstd_build
* fix_race_condition_on_zstd_build
* better close cursor
* save progress
* open read tx in all api methods
* clean
* clean
* save progress
* GracefulShutdown grpc server, send to server close signal before canceling context
* GracefulShutdown grpc server, send to server close signal before canceling context
* GracefulShutdown json server
* GracefulShutdown json server
* fix lint
* fix lint
* can
* clean
* test
* test
* test
* increase keepalive timeout, increase vegeta timeout, mark server-side errors
* increase keepalive timeout, increase vegeta timeout, mark server-side errors
* hack hugeFreelist
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* up streams limit
* fix_race_condition_on_zstd_build
* save state
* torrent experiments
* torrent experiment passed
* fixes after merge
* snapshot headers processing passed
* save state
* save state
* download headers works after snapshot processing
* save state
* save state
* save state
* save state
* add lazy load tx to snapshots, increase number of trackers
* save state
* speedup getting info
* change logging
* move to turbo package
* save state
* save state
* save state
* cleanup
* save state
* add test test
* save state
* lmdb debugging
* fix readonly mode
* save state
* fix build
* sync works
* save state
* save state
* save state
* allow cmd stages stageSenders use snapshots
* debugging failed hashing
* remove experimental tests
* remove torrent experimental tests
* fix lint
* extract snapshot wrapper
* metainfo checker
* add remote seeder
* add logs
* update gomod
* remove useless code
* fix lint&remove useless code
* extract verify snapshot to separated command
* skip debug test
* fix test
* change type of seedSnapshot flag
* add eth logger to torrent lib
* skip debug test
* add Close method
* review fixes
* fix lint
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* save state
* torrent experiments
* torrent experiment passed
* fixes after merge
* snapshot headers processing passed
* save state
* save state
* download headers works after snapshot processing
* save state
* save state
* save state
* save state
* add lazy load tx to snapshots, increase number of trackers
* save state
* speedup getting info
* change logging
* move to turbo package
* save state
* save state
* save state
* cleanup
* save state
* add test test
* save state
* lmdb debugging
* fix readonly mode
* save state
* fix build
* sync works
* save state
* save state
* save state
* allow cmd stages stageSenders use snapshots
* debugging failed hashing
* remove experimental tests
* remove torrent experimental tests
* fix lint
* extract snapshot wrapper
* metainfo checker
* add remote seeder
* add logs
* update gomod
* remove useless code
* fix lint&remove useless code
* extract verify snapshot to separated command
* skip debug test
* fix test
* change type of seedSnapshot flag
* add eth logger to torrent lib
* skip debug test
* add Close method
* review fixes
* fix lint
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
* added tls auth
* added client side
* put --tls
* fixed flag
* Add key/cert generation instructions, turn off Common Name verification
* Add CLI arguments and Warning
* Lint
* Update the doc about Internal IP
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
* it compiles
* after recent master
* fix linters warnings
* grpcV7
* go mod tidy
* unmarshall adresses or adress
* fix linters
* after cr
* after cr
* after cr
* after cr
* fix tests
* remove dev version
* it compiles
* mod tidy
* fix bin deps
* use stable version of grpc
* switch back to master constructor
* switch back to master constructor
* add a bit docs
* add a bit docs
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* Use Append for changesets and receipts
* Fix lint
* Delete all receipts on unwind
* Fix lint
* Fix unwind keys for changeset
* Fix test
* Use writeReceipts flag
* Revert to Append
* Delete receipts too when resetting
* don't use global variables
* up deps
* clean
* use joined flags
* use single non-global buckets config
* clean
* copy map
* fix tests
* fix tests
* clean
* clean
* v0
* v1
* v3
* v4
* clean
* temporary fix of txpool
* Add debug logs about tx start/commit
* Add debug logs about tx start/commit
* Add debug logs about tx start/commit
* add condition
* tx pool to not hold own db
* try to enable TxPool in integration
* exclude tx pool from tx
* exclude tx pool from integration
* reduce limit
* fix integration
* clean
* clean
* clean
* clean
* clean
* exclude tx pool unwind
* exclude tx pool unwind in integration
* fix integration tx pool
* fix commit
* fix current stage after unwind
* fix current stage after unwind
* fix linter
* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.
* move body and headers unwind out of tx
* fix unwind order after reboot
* add support external tx to exec stage
* clean
* clean
* clean
* clean
* clean
* add logs
* better id check
* better id check
* timer-based logs
* timer-based logs
* delegate progress calculation to user
* delegate progress calculation to user
* delegate progress calculation to user
* clear
* add logs to senders recovery
* use default dir in integration
* more logs
* more logs
* experiment of not using retain list
* better handle sub-tx
* save progress
* don't touch collector
* don't run collector on sub-tx because of `mdb_cursor_put: MDB_TXN_FULL: Transaction has too many dirty pages - transaction too big`
* clear
* clear
* add logging to loader
* use pure tx in etl loading, logs in mutation commit
* clean
* bletter logging and more cleanup
* bletter logging and more cleanup
* increase batch size to 500M
* better batch commit logging
* async fsync
* sync fsync
* sync fsync
* unify logging
* fix corner-case when etl can use empty bucket name
* fix tests
* better logging
* better logging
* rebase master
* remove lmdb.NoMetaSync flag for now
* consistent walk and multi-walk
* clean
* sub tx
* add consistent multi-put
* implement dupsort support in one new cursor method
* clear
* return error from rawdb
* squash
* v14
* improve performance of put
* clean
* clean
* hide feature behind ENV variable
* cleanup
* cleanup
* disable ipc and make Readme less confusing (people thought points are depend on each-other)
* fix test
* cleanup
* cleanup
* limit incremental step size of stages 5 and 6
* remove limit, make incremental
* remove_load_start_key_from_etl
* switch to "struct of arrays" from "array of structs"
* switch to "struct of arrays" from "array of structs"
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
* save state
* walkAsOf by cs test passed
* cs search for plain state
* save state
* fix accounts tests
* refactor walkAsOf account tests
* fix storage test
* refactor walkAsOf storage tests
* fix lint
* fix test
* save state
* save state
* add test with fixed bits
* fmt
* add stages check
* fix lint
* fix lint
* remove obsoleted methods
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out // to file
--pprof --pprof.port=6060 // launch pprof server
--metrics // sends to prometheus
```
* make lmdb default db
* switch nodes db to lmdb
* log errors in nodedb
* fix miss typing
* fix default test target in makefile
* report error of .Get method
* add debug info
* run tests with debug
* run tests with debug
* remove default db from TEST_DB env
* cleanup
* wait for goroutines finish
* wait for goroutines finish before close db
* wait for goroutines finish before close db
* wait for goroutines finish before close db
* wait for goroutines finish before close db
* avoid 0 length key/values and seek arguments
* less abstractions in .get method, and copy memory from C space without unsafe conversion into go memory
* fix test
* don't call initCursor on happy path
* don't call initCursor on happy path
* don't run stale reads goroutine for inMem mode
* don't call initCursor on happy path
* remove buffers from cursor object - they are useful only in Badger implementation
* commit kv benchmark
* remove buffers from cursor object - they are useful only in Badger implementation
* remove buffers from cursor object - they are useful only in Badger implementation
* cancel server before return pipe to pool
* try to fix test
* set field db in managed tx
* remove unused slice from MultiPut
* mutation: reuse tuples slice and preallocate bucketPuts
* use bucketPool in kv_lmdb
* remove duplicated check of context status
* more benchmarks
* remove reusage of puts
* remove ctx from MustOpen
* remove ctx from Open. Stop goroutines on Close.
* remove ctx from Open. Stop goroutines on Close.
* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)
* enable RawReads and add native implementation of Get/Has methods
* Euphemerally -> Ephemerally
* Move StorageMode to ethdb and pass it to PrepareStagedSync
* linter
* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go
* Optionally write receipts in the execute stage
* memory profiler
* linter
* proper linter fix
* linter
* typo
* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
* Start from 9m7
* Regenerate IH + receipts
* Only stats for iH bucket
* Persist receipts
* Go all in
* Start from block 10m
* Convert DbState to use plain state
* Fix findHistory
* Hard-code export
* More fixes
* Fix test
* Fix formatting
* Introduce PlainDbState
* Actually return PlainDbState
* Fix formatting
* Fix name style lint
* Fix linters
* Fix history_test
* Fix blockchain_test
* Fix compile error
* Bucket stats from all buckets
* Lmdb internal objects reuse (built-in feature of lmdb):
- lmdb read transactions pool
- lmdb also support cursors reuse, but not implemented in this PR
And kv abstraction objects reuse:
- lmdbKV pool of all tx objects
- boltKV pool of all tx objects
- badgerKV pool of all tx objects
* switch makefile back to bolt
* switch makefile back to bolt
* lmdb test run
* switch makefile back to bolt
* cursors pool
* run lmdb tests
* make kv objects pool global
* switch makefile back to bolt
* remove badgers GOMAXPROC setup, because our app tunned for sequential read/writes, not for random throughput
* simplify code
* resetIH from scratch if needed
* lmdb
* add AbstractKV to loader, added new Object accessor around AbstractKV
* add lmdb cli flag
* add requirement of k!=nil on error in docs
* add Size method for compatibility
* read after put tests
* fix multiput nils
* simplify loops
* increase mmap size
* better error messages
* better error messages
* fix tests
* better error messages
* cleanup
* avoid bolt usage in test
* move hardcoded bucket name to dbutils
* register more buckets
* register more buckets
* fix test
* db based version of PrefixByCumulativeWitnessSize
* db based version of PrefixByCumulativeWitnessSize
* retain all in Trie by default
* fix WitnessLen logic in calcTrie roots
* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket
* handle corner cases in WL
* Use correct incarnation for IH bucket
* use name WitnessSize
* save progress towards db-only witness estimation
* results from trie and from db are still different
* less recursion
* correct incarnation in CumulativeSearch
* reuse results from previous Tick, separate concepts of parent and startKey
* experiment: if not including trie structure to WitnessSize will reduce cumulative error
* tool to generate all IH and tool to calculate assessment of cumulative error
* tool to generate all IH
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* better stats
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* calculate ticks size distribution
* estimate cumulative error
* fix linter
* resetIH from scratch if needed
* cleanup
* fix test
* fix test
* Not hash, keep the files
* Calculate savings
* Fix
* Fix
* Fix
* Fix
* RestAPI to support local boltdb
* Not error on read-only db
* Changes so far
* Continue
* More
* Roll back a bit
* Restore newline
* something compiles
* Fix restapi
* Fix block number
* Fix reads
* Use plain writer
* Maps for storage reads and writes
* Clean up coersions
* Fix accounts/abi/bind
* Fix tests
* More fixes
* more fixes
* More fixes
* Fixes
* Fixed core/state
* Fixed eth tests
* Move code, fix linter
* Fix test
* Fix linter
* Fix linter
* Fix linter, badger_db to support AbstractKV
* Increase IdealBatchSize for badger
* Fix linter
* Fix linter
* "Unwind" for the execution phase when plain state is selected
* test stub (fails)
* tests (one with incarnations fails)
* test fixups
* fix tests: cleanup contract code bucket
* save state
* add current index feature
* fix test
* remove logs
* Only execute 1000 blocks
* Reset history index
* Correct action
* Increase batch size
* Increase chunk size, print memory stats
* Fix linter
* Remove unused from
* Split into 2 staged
* Use storage history gen
* remove log
* Not to run tx_cacher in staged mode
* Not to recover during stage 2
* Not to recover during stage 2
* Remove counter
Co-authored-by: b00ris <b00ris@mail.ru>
