Commit Graph

23 Commits

Author SHA1 Message Date
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Alex Sharov
8d04689041
close db in tests (#669) 2020-06-16 12:05:40 +07:00
ledgerwatch
dec30b698d
Fix execution for block 10094566 (#665)
* First sketch of the fix

* Re-enable stage4

* Fix formatting

* Park the changes

* Save updates

* Save update

* Cleanup

* Cleanup

* Cleanup

* Fix lint

* Cleanup

* Update

* Tx tracing

* Update

* Cleanup

* Switch tests back to bolt

* Test cleanup

* Update

* Polymorthic CREATE2 test

* Add assertions

* Fix formatting

* Fix test

* Cleanup

* Cleanup

* Cleanup
2020-06-15 22:24:08 +01:00
ledgerwatch
a8c9910f04
Various fixes to make sync work (#663)
* Fix body fetch

* Reduce spurious reorgs

* Exit the sync cycle after unwinds

* Fix out of range

* No stalling check for staged sync

* Disable failing tests

* Remove duplicate log message

* Fix UnwindTest and add assertions

* Fix formatting

* Cleanup

* Fix off by one error with bodies

* Remove rollback
2020-06-13 22:39:55 +01:00
Igor Mandrigin
96750b3711
Support ETL interruptions while promoting plain state (#662)
* lil etl changez

* fix test compile

* fixups to the unwinds

* add new buffer mode

* support unfinished transitions

* fixups

* fix tests

* linters

* linters
2020-06-13 16:03:38 +01:00
ledgerwatch
99b9c0fa99
Re-enable TestUnwind test and fix issues in the stage sync (#659)
* Fixes to stages

* Fix formatting

* Fixes to stages

* Fix formatting

* Fix linter

* Fix lint

* Updates

* Next update

* Update

* Test works

* Cleanup

* Fix formatting

* Fix formatting

* Cleanup

* Fix core test

* Reinstate randomness for reorgs
2020-06-13 13:39:04 +01:00
Andrew Ashikhmin
ebdac3a192
AVX2 fnvHash. Bump numOfGoroutines to 8 in stage_senders.go (#656)
* Profile all stages

* Try to recover senders with 8 goroutines

* fix CPU profiling for stage_bodies

* fix out-of-index

* Try full DAG for verfication of header seals

* Try to unroll fnvHash for performance

* SSE2 assembly for fnvHash16

* fnvHash16AVX2

* Revert changes to state.go

* check we're on 64-bit in useAVX2

* Shave a move off fnvHash16AVX2

* asmdecl doesn't know about VMOVD

* disable linter in the right place
2020-06-12 12:48:59 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
Igor Mandrigin
0ee2d83970
stagedsync: allow long stages/unwinds to be interrupted (#643)
use `s.UpdateWithStageData(db, <block number>, <key>)` to store the key

use `s.StageData` with `etl.NextKey` to restart ETL from where it was interrupted.
2020-06-09 15:49:49 +03:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Igor Mandrigin
1d740fc78e
refactor stagedsync unwind, add tests(#636)
* some stubs

* post rebase

* fixups

* linter

* linter 2

* Update stages.go
2020-06-08 07:01:26 +03:00
ledgerwatch
0fb37c807e
Introduce intermediate hashes stage (#634)
* Query progress

* Run stage4 offline

* More thorough resetState

* Correct BlockNumber

* Fix formatting

* State loop

* do every 200k blocks

* Shift to 6.6m

* Close dbs in tests

* Stage2 with option of no reset

* every 100k blocks

* Reset state before stage5

* Introduce another stage

* Check compile errors

* Fix linter

* Fix linter

* Disable unreliable test

* Fix test

* Remove unreachable code
2020-06-07 09:56:46 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
Evgeny Danilenko
3134066bba
Fix stage 3 shutdown (#629)
* check stop on every block inside jobs

* stop spawnRecoverSendersStage properly

* fmt

* close recoverSenders goroutines

* fmt
2020-06-06 16:09:57 +01:00
ledgerwatch
cd9813b82c
Stage5 unwind (#631)
* Unwinding for stage5 - start

* Fix formatting

* Integrated unwind, but no tests yet

* Fix formatting

* Fix lint

* Fix lint

* Test for unwind stage5

* Fix formatting
2020-06-06 12:03:29 +01:00
ledgerwatch
a8b438d21d
Incremental promotion for stage 5 (#628)
* Incremental promotion

* Fix compile errors

* Fix linter

* Transform state keys before loading

* Re-enable and fix the tests

* Fix formatting
2020-06-06 06:36:29 +01:00
Evgeny Danilenko
199ede36a2
Prevent slice make and copy in Pushes (#625)
* prevent slice make and copy in Pushes

* do not pad if the same length

* after CR

* linters

* linters

* linters
2020-06-05 19:46:34 +03:00
ledgerwatch
b20cdbe399
Enable staged sync (#618) 2020-06-04 12:34:22 +01:00
Andrew Ashikhmin
33c28f7fac
[GC] uint256 rather than big.Int in Transaction (#614)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray
2020-06-04 08:43:08 +01:00
Igor Mandrigin
0533eed812
Refactor Stagedsync part 1 (#610) 2020-06-03 17:25:44 +03:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments (#608)
* First commit

* Fix the one-off error

* Fix formatting

* Ability to execute stage5 separately

* Clean up intermediate hashes and stage5 progress

* Fix linter

* Print original keys when extracting

* channel

* More logging

* More logging

* Remove excess logging

* fix stage2

* Revert

* Fix stage2

* Add provider exhausted

* Sort sortable buffer

* Fix test

* Another cleanup

* Remove exhaust log
2020-06-03 13:03:40 +01:00
Igor Mandrigin
1e20ed255d
move staged sync to its own package (#605) 2020-06-02 17:52:50 +03:00