This adds a new API method on core.BlockChain to allow interrupting
running data inserts, and calls the method before shutting down the
downloader.
The BlockChain interrupt checks are now done through a method instead
of inlining the atomic load everywhere. There is no loss of efficiency from
this and it makes the interrupt protocol a lot clearer because the check is
defined next to the method that sets the flag.
# Conflicts:
# core/blockchain.go
# light/lightchain.go
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.
Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.
This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4
# Conflicts:
# cmd/utils/flags.go
# core/forkid/forkid.go
# les/client.go
# les/client_handler.go
# les/commons.go
# les/distributor.go
# les/enr_entry.go
# les/fetcher.go
# les/lespay/client/valuetracker.go
# les/metrics.go
# les/peer.go
# les/protocol.go
# les/retrieve.go
# les/server.go
# les/serverpool.go
# les/test_helper.go
# les/utils/expiredvalue.go
# les/utils/weighted_select.go
# les/utils/weighted_select_test.go
# params/bootnodes.go
* Fix body fetch
* Reduce spurious reorgs
* Exit the sync cycle after unwinds
* Fix out of range
* No stalling check for staged sync
* Disable failing tests
* Remove duplicate log message
* Fix UnwindTest and add assertions
* Fix formatting
* Cleanup
* Fix off by one error with bodies
* Remove rollback
* Profile all stages
* Try to recover senders with 8 goroutines
* fix CPU profiling for stage_bodies
* fix out-of-index
* Try full DAG for verfication of header seals
* Try to unroll fnvHash for performance
* SSE2 assembly for fnvHash16
* fnvHash16AVX2
* Revert changes to state.go
* check we're on 64-bit in useAVX2
* Shave a move off fnvHash16AVX2
* asmdecl doesn't know about VMOVD
* disable linter in the right place
* save state
* txlookup full results
* save state
* save state
* remove experiments
* some fix&lint
* add end key to txLookup and index generation
* change log message
* change log
* fix lint
* lint
* fix test
* Euphemerally -> Ephemerally
* Move StorageMode to ethdb and pass it to PrepareStagedSync
* linter
* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go
* Optionally write receipts in the execute stage
* memory profiler
* linter
* proper linter fix
* linter
* typo
* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
use `s.UpdateWithStageData(db, <block number>, <key>)` to store the key
use `s.StageData` with `etl.NextKey` to restart ETL from where it was interrupted.
* Start from 9m7
* Regenerate IH + receipts
* Only stats for iH bucket
* Persist receipts
* Go all in
* Start from block 10m
* Convert DbState to use plain state
* Fix findHistory
* Hard-code export
* More fixes
* Fix test
* Fix formatting
* Introduce PlainDbState
* Actually return PlainDbState
* Fix formatting
* Fix name style lint
* Fix linters
* Fix history_test
* Fix blockchain_test
* Fix compile error
* Bucket stats from all buckets
* Query progress
* Run stage4 offline
* More thorough resetState
* Correct BlockNumber
* Fix formatting
* State loop
* do every 200k blocks
* Shift to 6.6m
* Close dbs in tests
* Stage2 with option of no reset
* every 100k blocks
* Reset state before stage5
* Introduce another stage
* Check compile errors
* Fix linter
* Fix linter
* Disable unreliable test
* Fix test
* Remove unreachable code
* uint256 in rlp
* uint256 rather than big.Int in Transation
* linters
* more linters
* still linters
* Reduce garbage in writeUint256
* Experiment with GC in writeByteArray
* db based version of PrefixByCumulativeWitnessSize
* db based version of PrefixByCumulativeWitnessSize
* retain all in Trie by default
* fix WitnessLen logic in calcTrie roots
* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket
* handle corner cases in WL
* Use correct incarnation for IH bucket
* use name WitnessSize
* save progress towards db-only witness estimation
* results from trie and from db are still different
* less recursion
* correct incarnation in CumulativeSearch
* reuse results from previous Tick, separate concepts of parent and startKey
* experiment: if not including trie structure to WitnessSize will reduce cumulative error
* tool to generate all IH and tool to calculate assessment of cumulative error
* tool to generate all IH
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* better stats
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle
* calculate ticks size distribution
* estimate cumulative error
* fix linter
* resetIH from scratch if needed
* cleanup
* fix test
* fix test
* Not hash, keep the files
* Calculate savings
* Fix
* Fix
* Fix
* Fix
* RestAPI to support local boltdb
* Not error on read-only db
* Changes so far
* Continue
* More
* Roll back a bit
* Restore newline
* something compiles
* Fix restapi
* Fix block number
* Fix reads
* Use plain writer
* Maps for storage reads and writes
* Clean up coersions
* Fix accounts/abi/bind
* Fix tests
* More fixes
* more fixes
* More fixes
* Fixes
* Fixed core/state
* Fixed eth tests
* Move code, fix linter
* Fix test
* Fix linter
* Fix linter
* Fix linter, badger_db to support AbstractKV
* Increase IdealBatchSize for badger
* Fix linter
* Fix linter
* Move promotion before the shortcut
* Correct the logs
* Trying without the encoder
* Revert "Trying without the encoder"
This reverts commit 24c526bc2791fb6806b1c0d4f52a43b87c3959b9.
* Commit to db, not to the mutation
* Fix log more
* No need to log every deleted file
* Produce less garbage in GetState
* Still playing with mem allocation in GetCommittedState
* Pass key by pointer in GetState as well
* linter
* Avoid a memory allocation in opSload
* Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore
* linter
* linters
* small clean up
* "Unwind" for the execution phase when plain state is selected
* test stub (fails)
* tests (one with incarnations fails)
* test fixups
* fix tests: cleanup contract code bucket
* Produce less garbage in GetState
* Still playing with mem allocation in GetCommittedState
* Pass key by pointer in GetState as well
* linter
* Avoid a memory allocation in opSload
* Rewrite
* Fix linter, make counter
* Skip hash stage
* Fix linter
* Add common
* Fix wrong bucket
* Use temp files to generate index
* Fix hack, defer
* Add logging when creating files
* Properly pass datadir
* Bigger buffers, less logging
* Log current key, enforce batching
* Limit to 4m, print more
* Use ReadFull
* Optimised version of storage generation
* Don't print entire key
* Commit at the end
* Fix linter
* Remove a bit of copy-paste
* More copy-paste reduction
* Fixes
* Fix key length
* Not delete files too early
* Fix linter
* Fix logging
* Fix starting block
* Skip test
* Remove limit on the stage 4
* Fix test while disabling the hash check
* Fix tests
* Fix unreachable
* save state
* add current index feature
* fix test
* remove logs
* Only execute 1000 blocks
* Reset history index
* Correct action
* Increase batch size
* Increase chunk size, print memory stats
* Fix linter
* Remove unused from
* Split into 2 staged
* Use storage history gen
* remove log
* Not to run tx_cacher in staged mode
* Not to recover during stage 2
* Not to recover during stage 2
* Remove counter
Co-authored-by: b00ris <b00ris@mail.ru>
