Igor Mandrigin
adf52465e3
move ./trie to ./turbo/trie ( #1114 )
...
Useful for minimizing merge conflicts when rebasing new geth
2020-09-14 11:33:39 +01:00
ledgerwatch
a98469c6cd
Remove changeset processing in FindHistory ( #1105 )
2020-09-12 08:24:13 +01:00
Igor Mandrigin
2a6478d799
turbo-api: create a second executable file with a custom stage example ( #1055 )
2020-09-05 18:07:27 +02:00
ledgerwatch
c06dbdadf9
Use Append for changesets and receipts ( #1018 )
...
* Use Append for changesets and receipts
* Fix lint
* Delete all receipts on unwind
* Fix lint
* Fix unwind keys for changeset
* Fix test
* Use writeReceipts flag
* Revert to Append
* Delete receipts too when resetting
2020-09-01 07:48:25 +01:00
Igor Mandrigin
637493e8f1
Merge branch 'master' into geth-1.9.20
2020-08-29 15:28:03 +02:00
Alex Sharov
8c9a55bb21
[To test on HDD] Write changesets to tx instead of mutation ( #978 )
...
* use mutation over tx
* clear
* add .CommitAndBegin() method
* multip put 2
* right sorting
* write non-random things to tx
* write non-random things to tx
* disable multi put 2
* clean
* clean
* remove multiput2
* clean
* clean
* clean
* clean
* clean
* add receipts
2020-08-29 08:32:51 +01:00
Giuseppe Bertone
5388193c2c
core/state: fixed some comments ( #21450 )
...
# Conflicts:
# core/state/state_object.go
# core/state/statedb.go
# core/state/statedb_test.go
2020-08-27 17:38:12 +02:00
Alex Sharov
9091ce62bb
KV reduce features amount ( #975 )
...
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize
* remove bolt_db.go file
* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync ( #970 )
...
* Log in GetAsOf
* Default plain state
* trace
* More test fixes
* Fix lint
* Fix
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix for events
* Repair events
* Fix lint
* Fix lint
* Change back to InsertChain
2020-08-24 18:00:40 +01:00
ledgerwatch
7c16160087
Add IntraBlockState copy function to RPC API ( #960 )
2020-08-21 21:53:23 +01:00
b00ris
eb46dd68df
Account range testing ( #938 )
...
* save state
* save state
* save new timeouts
* compare passed
* save state
* cleanup
2020-08-19 07:33:49 +01:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction ( #928 )
...
* drop Bucket object from KV abstraction
* add some non-public methods
* rebase master
* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
0e253e7336
lmdb transactions of unlimited size ( #918 )
...
* add logging to loader
* use pure tx in etl loading, logs in mutation commit
* clean
* bletter logging and more cleanup
* bletter logging and more cleanup
* increase batch size to 500M
* better batch commit logging
* async fsync
* sync fsync
* sync fsync
* unify logging
* fix corner-case when etl can use empty bucket name
* fix tests
* better logging
* better logging
* rebase master
* remove lmdb.NoMetaSync flag for now
* consistent walk and multi-walk
* clean
* sub tx
* add consistent multi-put
* implement dupsort support in one new cursor method
* clear
2020-08-17 07:45:52 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object ( #914 )
2020-08-14 13:41:18 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string ( #894 )
2020-08-11 06:55:32 +07:00
Igor Mandrigin
8600616c3d
geth 1.9.18 post-rebase fixups
2020-08-08 17:33:35 +02:00
Martin Holst Swende
14e3fda786
cmd/evm: add state transition tool for testing ( #20958 )
...
This PR implements the EVM state transition tool, which is intended
to be the replacement for our retesteth client implementation.
Documentation is present in the cmd/evm/README.md file.
Co-authored-by: Felix Lange <fjl@twurst.com>
# Conflicts:
# cmd/evm/main.go
# core/state/dump.go
# core/state/state_test.go
# tests/state_test_util.go
2020-08-07 11:38:07 +02:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key ( #868 )
2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key ( #858 )" ( #867 )
...
This reverts commit 7290bf519e
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key ( #858 )
2020-08-04 16:25:28 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) ( #854 )
...
* return error from rawdb
* squash
* v14
* improve performance of put
* clean
* clean
* hide feature behind ENV variable
* cleanup
* cleanup
* disable ipc and make Readme less confusing (people thought points are depend on each-other)
* fix test
* cleanup
* cleanup
2020-08-02 12:53:08 +01:00
ledgerwatch
cf799157cc
Jumpdest skip optimisation ( #851 )
...
* Jumpdest skipping optimisation
* Fix formatting
* Move skipAnalysis into vmConfig, introduce tracing ability
* Improve detection logging
* Added release instructions
* Fix lint
2020-08-01 17:56:57 +01:00
b00ris
9fc51a3efd
Account range ( #781 )
...
* save state
* cleanup
* add walkasof test for storage with chunks
* fix lint
* fix lint
* remove useless comments
* fix dump test
* move accountRange test to plain state
* get last block fixes
2020-07-27 07:09:50 +01:00
b00ris
12aa55bcbb
Remove incarnation inversion ( #755 )
...
* remove incarnation inversion
* fix lint
* remove incarnation inversion
* remove comment
* remove inversion from new code
* revert for unwind
2020-07-17 15:18:31 +01:00
Evgeny Danilenko
5ead20e288
close tx_cache and peer.handle goruotines properly ( #748 )
...
* close goruotines
* fix using defer
* close pm managers a bit later
* fmt
* finalizer for in-mem db
* after cr
* revert finalizer
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-07-16 14:27:24 +01:00
ledgerwatch
18e67cd07d
cmd/tester to initiate correct staged sync (but not reorg yet) ( #747 )
...
* Disable caches
* Fix cmd/integration?
* state root hash matches
* Cleanup
* Skip TestForkIDSplit
* Try to fix the test
2020-07-15 13:26:59 +01:00
Alex Sharov
6043c09ddf
Replace global buff pool by local, because of buffer leaks to larger pools ( #739 )
2020-07-14 08:56:47 +07:00
ledgerwatch
665b1d88b9
Speed up GenerateChain by using intermediate hashes ( #736 )
...
* GenerateChain to support intermediate hashes
* Fix formatting
* Changeset stats
* Fix formatting
2020-07-10 22:37:34 +01:00
Alex Sharov
8335700331
Integration: compare to reference_db after all cycles, better cancelation, add to makefile ( #733 )
...
* cmp to reference db if --reference_chaindata provided
* graceful shutdown
* add ./cmd/integration to makefile
* add interruption support for different stages
* don't compare errors directly
2020-07-10 07:03:18 +01:00
ledgerwatch
72dcf478d3
Fix debug_storageRangeAt in rpc daemon ( #732 )
...
* Fix storageRangeAt
* Fix formatting
* Fix lint
* Fix lint
* Prevent racing by closing db after the blockchain
2020-07-10 06:44:01 +01:00
Alex Sharov
d98382fdf8
Integration add checkChangeSet into state_stages command ( #728 )
...
Integration add checkChangeSet into state_stages command (#728 )
2020-07-10 10:25:40 +07:00
ledgerwatch
887fc4f99c
GenerateChain without using TrieDbState ( #726 )
...
* Different chain generation
* More fixes
* Fix TestSideLogRebirth
* Fix the rest of core tests
* Fix accounts/abi/bind
* Fix compile errors for cmd/pics
* Fix consensus/clique tests
* More fixes
* Fix core/state tests
* Fix eth tests
* Fix eth/downloader tests
* Fix more tests
* Fix compile errors
* Some more fixes
* Save
* Exists
* Fix last tests
* Clean up
* Fix formatting
* Fix linter
* Fix formatting
* Fix linter
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
2020-07-09 07:15:28 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf ( #724 )
...
* save state
* walkAsOf by cs test passed
* cs search for plain state
* save state
* fix accounts tests
* refactor walkAsOf account tests
* fix storage test
* refactor walkAsOf storage tests
* fix lint
* fix test
* save state
* save state
* add test with fixed bits
* fmt
* add stages check
* fix lint
* fix lint
* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Igor Mandrigin
9881a2ccb3
Fix broken witness generation. ( #721 )
2020-07-07 12:24:55 +01:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes ( #716 )
...
* Fixing history generation
* Try fixes
* Reset from 0
* Cleanup
2020-07-06 07:34:24 +01:00
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature ( #711 )
2020-07-05 07:17:27 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets ( #698 )
...
* Debugging changesets
* Kinda works
* Fix compile error
* Fix formatting
* Fix lint
* Duplicate entries kludge
* Fix compile error
* Cleanup
2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon ( #692 )
...
* Fixing history index
* Remove chunk generation, fix formatting
* Fix compile error, clean up hack.go
* Fix output tests
* Fix index generator test
* Fixed checkChangeSets
* Fix linter
2020-06-28 07:10:27 +01:00
b00ris
e8b617bbf2
Add sort.Search to account changeset ( #671 )
...
* fast search
* fix
* remove copy bytes
2020-06-16 15:03:35 +01:00
Alex Sharov
8d04689041
close db in tests ( #669 )
2020-06-16 12:05:40 +07:00
ledgerwatch
dec30b698d
Fix execution for block 10094566 ( #665 )
...
* First sketch of the fix
* Re-enable stage4
* Fix formatting
* Park the changes
* Save updates
* Save update
* Cleanup
* Cleanup
* Cleanup
* Fix lint
* Cleanup
* Update
* Tx tracing
* Update
* Cleanup
* Switch tests back to bolt
* Test cleanup
* Update
* Polymorthic CREATE2 test
* Add assertions
* Fix formatting
* Fix test
* Cleanup
* Cleanup
* Cleanup
2020-06-15 22:24:08 +01:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index ( #648 )
...
* Debug infos
* bucketStats lmdb by default
* Use bools instead of strings
* Fix linters
* Fix linter
* Finally fix
* Fix compile error
* Fix test
* Fix formatting
* Fix shadowing
* Cleanup
* cleanup
2020-06-11 07:19:15 +01:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts ( #644 )
...
* Euphemerally -> Ephemerally
* Move StorageMode to ethdb and pass it to PrepareStagedSync
* linter
* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go
* Optionally write receipts in the execute stage
* memory profiler
* linter
* proper linter fix
* linter
* typo
* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts ( #639 )
...
* Start from 9m7
* Regenerate IH + receipts
* Only stats for iH bucket
* Persist receipts
* Go all in
* Start from block 10m
* Convert DbState to use plain state
* Fix findHistory
* Hard-code export
* More fixes
* Fix test
* Fix formatting
* Introduce PlainDbState
* Actually return PlainDbState
* Fix formatting
* Fix name style lint
* Fix linters
* Fix history_test
* Fix blockchain_test
* Fix compile error
* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap ( #626 )
...
* optimize codeBitmap
* add lru
* remove custom lru
* jumpDests as explocit parameter
* lint
* lint
* linters
* default jumpDestsCache
* remove common.rand
2020-06-06 21:49:06 +01:00
Evgeny Danilenko
199ede36a2
Prevent slice make and copy in Pushes ( #625 )
...
* prevent slice make and copy in Pushes
* do not pad if the same length
* after CR
* linters
* linters
* linters
2020-06-05 19:46:34 +03:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger ( #624 )
...
* lmdb tests
* trigger ci
* fix tests
* disable parallelism
* disable parallelism
* cleanup resources
* cleanup resources
* reduce concurency
* try run tests on bolt
* try run tests on bolt
* fix downloader test
* run bolt tests
* rely on interface instead of exact instance
* Rename AbstractKV to KV
* don't use separator for badger
* don't initialize badger cursor - because it not used here
* fix linter
* try reduce badger compactors
* compat with master
* try lmdb
* try lmdb
* try lmdb
* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option
* allow to close closed db
* release
* release
* ideal batch size for badger
* ideal batch size for badger
2020-06-05 10:25:33 +01:00
Andrew Ashikhmin
33c28f7fac
[GC] uint256 rather than big.Int in Transaction ( #614 )
...
* uint256 in rlp
* uint256 rather than big.Int in Transation
* linters
* more linters
* still linters
* Reduce garbage in writeUint256
* Experiment with GC in writeByteArray
2020-06-04 08:43:08 +01:00
ledgerwatch
4ce69916dc
Fix rebuilding the Intermediate Hash bucket ( #602 )
...
* Changes
* First attempt to fix
* Sync works
* Fix compile
* Fix linter
* Fix test
* Fix test
* Debug
* Propagate tracing
* Trace default receiver
* More tracing
* Correct unfurl
* Remove tracing
* Try to retain trie for debugging
2020-06-02 08:46:21 +01:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes ( #597 )
...
* Introduce hashCollector
* Add HashCollector to SubTrieLoader
* Fix linter
* Reset hashed state
* Not to regenerate the hashed state
* Not to delete state
* Fix linter
* Print expected hash in the beginning
* Simplify
* Remove initialTrie
* Use etl to buffer intermediate hashes
* Copy values, not insert empty key
* Compress instead of decompress
* Enhance file buffer logging, fix linter
* Fix compile errors
* Fix log
* Fix logging
* Exclude zero key again
* Add rewind
* Restrict timestamps
* Fix
* Fix formatting
* Incorporate separation
* Extract identityLoadFunction
* Fix formatting
2020-05-31 13:23:34 +01:00
