This PR implements the following modifications
- Don't shortcut check if block is present, thus avoid disk lookup
- Don't check hash ancestry in early-check (it's still done in parallel checker)
- Don't check time.Now for every single header
Charts and background info can be found here: https://github.com/holiman/headerimport/blob/main/README.md
With these changes, writing 1M headers goes down to from 80s to 62s.
# Conflicts:
# consensus/ethash/consensus.go
# core/rawdb/freezer_table.go
This PR adds re-written difficulty calculators, which are based on uint256. It also adds a fuzzer + oss-fuzz integration for the new fuzzer. It does differential fuzzing between the new and old calculators.
Note: this PR does not actually enable the new calculators.
# Conflicts:
# consensus/ethash/consensus_test.go
* consensus/ethash: only use *reflect.SliceHeader, not reflect.SliceHeader. See comment here: https://github.com/golang/go/issues/40397\#issuecomment-663748689
* consensus/ethash: pr feedback from @mdempsky, makes a copy of dest such that is not mutated
* consensus/ethash: remove noop assign
* consensus/ethash: apply same fix to another location
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
* State cache init
* More code
* Fix lint
* More tests
* More tests
* More tests
* Fix test
* Transformations
* remove writeQueue, before fixing the tests
* Fix tests
* Add more tests, incarnation to the code items
* Fix lint
* Fix lint
* Remove shards prototype, add incarnation to the state reader code
* Clean up and replace cache in call_traces stage
* fix flaky test
* Save changes
* Readers to use addrHash, writes - addresses
* Fix lint
* Fix lint
* More accurate tracking of size
* Optimise for smaller write batches
* Attempt to integrate state cache into Execution stage
* cacheSize to default flags
* Print correct cache sizes and batch sizes
* cacheSize in the integration
* Fix tests
* Fix lint
* Remove print
* Fix exec stage
* Fix test
* Refresh sequence on write
* No double increment
* heap.Remove
* Try to fix alignment
* Refactoring, adding hashItems
* More changes
* Fix compile errors
* Fix lint
* Wrapping cached reader
* Wrap writer into cached writer
* Turn state cache off by default
* Fix plain state writer
* Fix for code/storage mixup
* Fix tests
* Fix clique test
* Better fix for the tests
* Add test and fix some more
* Fix compile error|
* More functions
* Fixes
* Fix for the tests
* sepatate DeletedFlag and AbsentFlag
* Minor fixes
* Test refactoring
* More changes
* Fix some tests
* More test fixes
* More test fixes
* Fix lint
* Move blockchain_test to be able to use stagedsync
* More fixes
* Fixes and cleanup
* Fix tests in turbo/stages
* Fix lint
* Fix lint
* Intemediate
* Fix tests
* Intemediate
* More fixes
* Compilation fixes
* More fixes
* Fix compile errors
* More test fixes
* More fixes
* More test fixes
* Fix compile error
* Fixes
* Fix
* Fix
* More fixes
* Fixes
* More fixes and cleanup
* Further fix
* Check gas used and bloom with header
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
* First algorithm
* Update doc
* Second algorithm implemented
* Prepend (unfinished)
* More on prepend
* More fixes
* Change CalcDifficulty interface
* fix make test
* Fix formatting
* Another test
* More on Preprend
* Another test for Prepend
* More tests for Prepend
* More tests for Prepend
* Move files to turbo
* Add another test for Prepend
* Started adding Append algorithm
* More work on Append
* Start changing the terminology
* Split trees into segments
* More fixes
* Fix compilation
* Connect
* FindAnchors, FindTip
* Tip has anchor and not anchorParent field
* Remove Prepend and Append
* Add Connect
* NewAnchor algorithm
* More to recovery from files
* Add more to AddAnchor
* Request more headers
* Header fix-sizes serialisation
* Upgrade CI to 1.15
* Fix lint
* Fix lint
* replace RLP with fixed-sized serialisation
* Fix lint
* More algos
* Fix lint
* close goruotines
* fix using defer
* close pm managers a bit later
* fmt
* finalizer for in-mem db
* after cr
* revert finalizer
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
* Profile all stages
* Try to recover senders with 8 goroutines
* fix CPU profiling for stage_bodies
* fix out-of-index
* Try full DAG for verfication of header seals
* Try to unroll fnvHash for performance
* SSE2 assembly for fnvHash16
* fnvHash16AVX2
* Revert changes to state.go
* check we're on 64-bit in useAVX2
* Shave a move off fnvHash16AVX2
* asmdecl doesn't know about VMOVD
* disable linter in the right place
* Query progress
* Run stage4 offline
* More thorough resetState
* Correct BlockNumber
* Fix formatting
* State loop
* do every 200k blocks
* Shift to 6.6m
* Close dbs in tests
* Stage2 with option of no reset
* every 100k blocks
* Reset state before stage5
* Introduce another stage
* Check compile errors
* Fix linter
* Fix linter
* Disable unreliable test
* Fix test
* Remove unreachable code
