* Only insert hard-coded tips if both DB and files recovery failed
* Prevent deadlock
* Report some efficiency numbers
* Count properly unrequested bodies
* Initialise allRequests
* Increase timeout
* Fixes to scheduling
* Small fix
* Simplified scheduling
* Remove separate bodyLoop goroutine
* Update from DB at the beginnig of block bodies forward
* Timeout for repeating the request cycle
* Fix timeout
* Fix
* Increase timeout
* Increase timeout
* Try to make flow if possible
* Fix flow
* Lower timeout
* timeout for each blockNum
* Adjustable timeout
* Better log timing
* Track peers
* copy peerID
* fix scheduling
* Too much logging
* Print delivery speed
* Print committed blocks
* Fix race
* Sentry to only reset back-off timer when response to a request is received
* Print bytes/sec and wasted traffic
* Fix bandwidth accounting
* Less logging
* not to wake up on deliveries
* Spam every second
* Print peer map
* Fix npe, print requests
* Timestamps
* Improved logging
* Penalty for peers
* Log penalties only when disconnecting
* Try with smaller window
* window parameter
* Dealing with partially delivered requests
* Init bodyReq
* Fix array index
* More fix for NPE
* More NPE checks
* Print out body progress and header progress
* Fix ending condition
* Bring back waking up and penalties
* Fix duplicate tip
* Duplicate segment
* Fix lint
* Fix lint
* fix lint
* Fix lint
* Hard coded headers in the source files
* Fix lint
* Replace hix-sized header serialisation with rlp (to support clique)
* Remove anchor records from the files
* Fixeds for DB recovery
* Fix compilation
* Fix compile errors
* Fix formatting
* Fix lint
* Fix comments
* Remove headerLoop
* Properly terminate body download
* Support for StatusData p2p proto
* Fix forkid test
* Fix test
* Fix lint
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* move experiments to new branch&reorganise kv_snapshot
* walk&modify tests
* added delete from snapshot tests
* fmt
* state snapshot debug
* snapshot validation passed. copy state snapshot
* debug
* snapshot cursor.Prev test
* Prev works correct. Added Current check
* add err check
* added walk forward and backward test
* before refactoring
* refactoring
* execution with snapshot debug
* fix
* remove useless test
* before dupcursor implimentation
* tests with prev and delete works
* execution based on state snapshot passed
* remove useless tests
* blocks to 1140000 passed
* clean verifier
* cleanup state generation
* clean verify && seeder
* remove debug code
* tests passed
* fix lint
* save state
* test passed
* fix lint
* add state hash
* fix lint
* rpcdaemon performance improve - less reading blocks, less allocs on server
* rpcdaemon performance improve - less reading blocks, less allocs on server
* rpcdaemon performance improve - less reading blocks, less allocs on server
* rpcdaemon performance improve - less reading blocks, less allocs on server
* don't use global variables
* df
* avoid use non-transactional db instance to reduce amount of cursors
* trace_call initial
* Fix tests
* More tracing
* Add more fields to the action
* Completed first example query
* Add initial bench11 to compare trace_call with OpenEthereum
* Exclude precompile calls from call traces
* Add self-destruct, call types, more comparison in rpctest
* Support for execution errors
* Stack underflow error and delegatecall value
* Fix lint
* Fix suicide traceAddress, Bad instruction error
* Fix lint
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Cleanup and rpcdaemon unit tests
* Fix
* Fix
* Fix lint
* Test for debug_traceTransaction
* Add NoRefunds option
* Compile fix, test for no refunds
* Fix compile
* Add poly contract, fix compile errors
* No refunds now work
* Fix NPE in rpcdaemon
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* State cache init
* More code
* Fix lint
* More tests
* More tests
* More tests
* Fix test
* Transformations
* remove writeQueue, before fixing the tests
* Fix tests
* Add more tests, incarnation to the code items
* Fix lint
* Fix lint
* Remove shards prototype, add incarnation to the state reader code
* Clean up and replace cache in call_traces stage
* fix flaky test
* Save changes
* Readers to use addrHash, writes - addresses
* Fix lint
* Fix lint
* More accurate tracking of size
* Optimise for smaller write batches
* Attempt to integrate state cache into Execution stage
* cacheSize to default flags
* Print correct cache sizes and batch sizes
* cacheSize in the integration
* Fix tests
* Fix lint
* Remove print
* Fix exec stage
* Fix test
* Refresh sequence on write
* No double increment
* heap.Remove
* Try to fix alignment
* Refactoring, adding hashItems
* More changes
* Fix compile errors
* Fix lint
* Wrapping cached reader
* Wrap writer into cached writer
* Turn state cache off by default
* Fix plain state writer
* Fix for code/storage mixup
* Fix tests
* Fix clique test
* Better fix for the tests
* Add test and fix some more
* Fix compile error|
* More functions
* Fixes
* Fix for the tests
* sepatate DeletedFlag and AbsentFlag
* Minor fixes
* Test refactoring
* More changes
* Fix some tests
* More test fixes
* More test fixes
* Fix lint
* Move blockchain_test to be able to use stagedsync
* More fixes
* Fixes and cleanup
* Fix tests in turbo/stages
* Fix lint
* Fix lint
* Intemediate
* Fix tests
* Intemediate
* More fixes
* Compilation fixes
* More fixes
* Fix compile errors
* More test fixes
* More fixes
* More test fixes
* Fix compile error
* Fixes
* Fix
* Fix
* More fixes
* Fixes
* More fixes and cleanup
* Further fix
* Check gas used and bloom with header
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* Expose C Handle of lmdbTx
* LoadExecutionFunctionPointer
* silkworm ExecuteBlocks
* linter
* CLI flag for SilkwormExecutionFunc
* linter
* Call SilkwormExecutionFunc in SpawnExecuteBlocksStage
* Commit transaction after Silkworm execution
* Fix batch initialization
* Make batch_size uint64_t instead of size_t
* max_block in silkworm_execute_blocks
* More accurate logProgress
* Use CHandle exposed by lmdb-go
* Enable Silkworm on Linux only
* Update lmdb-go
* Error when attempting to use Silkworm not on Linux
* Move SilkwormFlag from cmd/utils/flags.go to turbo/cli/flags.go
* Integrate Silkworm into cmd/integration
* Check against ReaderBuilder/WriterBuilder/ChangeSetHook in Silkworm execution
* Refactor SpawnExecuteBlocksStage
* linters
* Small clean-ups
* Move ChangeSetHook inside executeBlockWithGo
* No need to tamper with batch size in executeBlocksWithSilkworm
* fix
* Fix
* Fixup
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* fix `make grpc` on new checkouts
* update proto files
* add some stub
* prototype with fake events
* notifying about events
* pass events
* events are being sent
* transfer headers to filters
* create the “filters” struct
* implement new heads
* PoC of New Heads subscription
* fix keep alive
* fixups for the client
* add “type” to the event
* support header event type on client
* better stage refactor
* fixup for the eth backend
* fixups
* fix tests
* fix tests
* fix linters
* address comments
* remove unused log
* save state
* save state
* save state
* refactoring
* fix
* save state
* save state
* fmt
* fix lint
* restore torrents for external downloader
* fix lint
* download
* skip debug test
* debug
* remote debug
* small cli fixes
* skip debug test
* external snapshot predownloader
* get rid of remote downloader
* fix lint
* clean makefile
* fix lint
* fix lint
* cleanup
* fix ci
* fmt
* remove proto from interfaces
* Squashed 'interfaces/' content from commit acd02bb94
git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
* move locking to the hd object itself, create ready channel
* Fix lint
* More integration
* Fix lint
* Connect to processing
* Set hd.files at recovery
* Serialise processSegment
* Fix lint
* Not prepend filesDir
* Write to the database
* Fix lint
* Set up genesis block
* Fix hard coded tips
* Fix chainbreak false alarm
* Recover from DB
* Skip non-header records
* Not initialise if recovered from DB
* Fix lint
* Initial work on state shards
* Fix compile errors
* Intermediate
* Implement dispatcher
* Implemented sharded reader
* No cache
* Print comms
* No prepopulation
* Print dispatcher activity
* Print more
* Print errors
* Fix dispatcher
* Remove logging
* Dispatcher to enforce initial barrier
* Don't print that much
* Turn off cache pre-seeting for state sharding
* Artificial latency, fixed number of blocks to execute
* Fix
* Splitting sentry and downloader - the beginning
* A bit more
* More on sentry
* More gRPC
* Sentry and downloader separated
* Update binding for stable version of grpc
* Better bufferSize flag
* Fix lint
* Send pelanties
* Fix lint
* Remove hard-coded tips on connect
* Tidy the logs a bit
* Deal with hardTips on Recovery
* Print hard tips
* Hide empty anchors
* Request headers after receiving a message
* Better waking up
* Print hard-coded block numbers
* Print outgoing requests
* Debug logging
* In the middle protection
* Sentry not to lose peers when core disconnects
* ReadBlockByNumber and ReadChainConfig return error
* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:
ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
* save state
* torrent experiments
* torrent experiment passed
* fixes after merge
* snapshot headers processing passed
* save state
* save state
* download headers works after snapshot processing
* save state
* save state
* save state
* save state
* add lazy load tx to snapshots, increase number of trackers
* save state
* speedup getting info
* change logging
* move to turbo package
* save state
* save state
* save state
* cleanup
* save state
* add test test
* save state
* lmdb debugging
* fix readonly mode
* save state
* fix build
* sync works
* save state
* save state
* save state
* allow cmd stages stageSenders use snapshots
* debugging failed hashing
* remove experimental tests
* remove torrent experimental tests
* fix lint
* extract snapshot wrapper
* metainfo checker
* add remote seeder
* add logs
* update gomod
* remove useless code
* fix lint&remove useless code
* extract verify snapshot to separated command
* skip debug test
* fix test
* change type of seedSnapshot flag
* add eth logger to torrent lib
* skip debug test
* add Close method
* review fixes
* fix lint
* tidy mods
* Fix compile
* Fix lint
* Fix rpcdaemon running in the docker
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* Initial commit for CallTraces index
* Fix compilation
* fix lint, add comment
* Fix integration
* Add Close function to ethdb.Cursor, fix some compile errors
* Try to stop cursor leak in Get
* Fix compile errors in RPC daemon
* Fix compile errors
* fixing another way
* Some fixes
* More fixes
* More fixes
* More fixes
* Fixes to core/state
* Fix lint
* Fix lint
* Fixes
* Stage caching for call trace stage
* Add mem stats
* Try to stop the leak
* Turn off debug
* Chunks for 10k blocks
* Print
* Revert "Print"
This reverts commit 5ffada4828d61e00e5dad1ca12c98258dfbbad00.
* Revert "Chunks for 10k blocks"
This reverts commit cfb9d498e782e5583d41c30abf0e2137da27383e.
* Trying to fix the leak
* Don't compute receipts in re-tracing
* Not compose block
* Print speed, fix receipts, bigger caches
* Fix lint
* Utilise changeset info
* Counters
* Use NoReceipts and ReadOnly
* ReadOnly is incompatible with caching
* Skip test leaking transactions
* Fix block test
* Change disable message for call-traces stage
* Use block option for call traces integration
* Fix retracing due to incarnation
* save state
* torrent experiments
* torrent experiment passed
* fixes after merge
* snapshot headers processing passed
* save state
* save state
* download headers works after snapshot processing
* save state
* save state
* save state
* save state
* add lazy load tx to snapshots, increase number of trackers
* save state
* speedup getting info
* change logging
* move to turbo package
* save state
* save state
* save state
* cleanup
* save state
* add test test
* save state
* lmdb debugging
* fix readonly mode
* save state
* fix build
* sync works
* save state
* save state
* save state
* allow cmd stages stageSenders use snapshots
* debugging failed hashing
* remove experimental tests
* remove torrent experimental tests
* fix lint
* extract snapshot wrapper
* metainfo checker
* add remote seeder
* add logs
* update gomod
* remove useless code
* fix lint&remove useless code
* extract verify snapshot to separated command
* skip debug test
* fix test
* change type of seedSnapshot flag
* add eth logger to torrent lib
* skip debug test
* add Close method
* review fixes
* fix lint
Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* save state
* torrent experiments
* torrent experiment passed
* fixes after merge
* snapshot headers processing passed
* save state
* save state
* download headers works after snapshot processing
* save state
* save state
* save state
* save state
* add lazy load tx to snapshots, increase number of trackers
* save state
* speedup getting info
* change logging
* move to turbo package
* save state
* save state
* save state
* cleanup
* save state
* add test test
* save state
* lmdb debugging
* fix readonly mode
* save state
* fix build
* sync works
* save state
* save state
* save state
* allow cmd stages stageSenders use snapshots
* debugging failed hashing
* remove experimental tests
* remove torrent experimental tests
* fix lint
* extract snapshot wrapper
* metainfo checker
* add remote seeder
* add logs
* update gomod
* remove useless code
* fix lint&remove useless code
* extract verify snapshot to separated command
* skip debug test
* fix test
* change type of seedSnapshot flag
* add eth logger to torrent lib
* skip debug test
* add Close method
* review fixes
* fix lint
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
* Add headers persistence
* Print flushBuffer
* Fix indexing problem
* Not skip hard-coded headers if the files are empty
* Fix lint
* print anchor state after init
* Properly construct file names
* Add check sub-command
* Fix lint
* Fix lint
* Fix lint
* Print more info when recovering
* Fix recovering
* Fix recovery
* Add anchors also as tips
* 2-level priority queue for tips
* Initialise tipQueue in anchor
* update maxTipHeight
* fix type
* Add anchors to the anchorQueue and rebuild anchorQueue on deletion
* Fix NPE
* fix recovery
* User buffersize, add hard coded headers to buffer
* Reinit anchorQueue
* Schedule requests after recovery
* No fix
* Remove duplicates
* Report duplicate headers
* Log buffer additions
* Fix for duplicate headers
* Try to fix duplicate headers again
* remove TODO comment
* Use LLRB instead of heap for anchors
* Print reserveTip info
* Correctly replace anchors in the tree
* Remove excessive logging
* Print tips attached to the anchor
* Print tips better
* limitTips instead of reserveTip
* Print forked headers
* Use pointers in llrb
* Print tipStretch
* Print limitTips
* Mininise AnchorItem
* Put anchors into the tree
* Fix totalDiff calculation, but is it needed?
* Remove totalDifficulty from anchors
* CheckInitiation
* Fix tests
* Fix lint, print more at check initiation
* Better output for hard-coded anchors
* Print hard-coded anchors better
* Prioritise anchors with short stretches
* Prioritise by chainSize
* Use linked list instead of heap for requestQueue
* Fix problem of no requests
* Push front
* Fix lint
* Not verify PoW for too far in the past and future
* Fix Calculation of totalDifficulty when Connect
* Fix hard tips
* Another fix for tips
* Process BlockHeadersMsg
* Print invalidations
* Print reason for parent-child mismatch
* Fix off by one error
* Fix ExtendDown
* Replace tipLimiter with an LRU cache
* Retrieve 256 headers at a time
* Hard coding of tips, process new block hashes
* extract headers
* Write hard-coded headers into a file
* Add total difficulty
* Insert hard coded headers
* Log i
* Initiate download from hard-coded anchors
* Print the state of anchors
* Add hard-coded headers as tips
* Spare the peers
* Refine LRU caching of tips
* Print length of anchor trees
* Limit requests to 192 headers
* range error fix
* Print better, retain at least one tip in the LRU cache
* Skip duplicate line ends
* Throttle requests, display peer name
* Back off, intermediate tip queue
* Print block header timestamp
* Better logging
* Log less
* More logging of evictions
* Fix queue push
* Fix queue pop
* Exclude duplicate tips
* Fix tips logging
* Fix moving
* Add port flag
* Check headers
* Print extra
* Add mixDigest to serialisation
* Fix difficulty in hard-coded headers
* Avoid duplicate header requests
* Update peer characteristics on NewBlockMsg
* Remove backoff when peer responds
* Fix tests
* Fix lint
* Fix lint
* added tls auth
* added client side
* put --tls
* fixed flag
* Add key/cert generation instructions, turn off Common Name verification
* Add CLI arguments and Warning
* Lint
* Update the doc about Internal IP
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
* First algorithm
* Update doc
* Second algorithm implemented
* Prepend (unfinished)
* More on prepend
* More fixes
* Change CalcDifficulty interface
* fix make test
* Fix formatting
* Another test
* More on Preprend
* Another test for Prepend
* More tests for Prepend
* More tests for Prepend
* Move files to turbo
* Add another test for Prepend
* Started adding Append algorithm
* More work on Append
* Start changing the terminology
* Split trees into segments
* More fixes
* Fix compilation
* Connect
* FindAnchors, FindTip
* Tip has anchor and not anchorParent field
* Remove Prepend and Append
* Add Connect
* NewAnchor algorithm
* More to recovery from files
* Add more to AddAnchor
* Request more headers
* Header fix-sizes serialisation
* Upgrade CI to 1.15
* Fix lint
* Fix lint
* replace RLP with fixed-sized serialisation
* Fix lint
* More algos
* Fix lint
* share config object
* create default config and logger
* move db connection to common func
* move server start to cli package
* clear
* clear
* rename cli to rpc
* use unified SetupLogger func
* make all root flag persistent
* use common flags in different packages
* use common flags in different packages
* move TraceTx method to eth package
* use native slice flags
* create package "turbo"
* disable geth api
* disable geth api
* move more data types to turbo/adapter package
* add support for customApiList
* run more
* run more
* run more
* dog-food
* move DoCall
* move DoCall
* fix tests
* fix test
