Commit Graph

410 Commits

Author SHA1 Message Date
Alex Sharov
bf596c26f3
[merge after release] dupsort of plain state (#913)
* dupsort of plain state

* rebase master
2020-08-15 08:11:40 +01:00
Alex Sharov
3aed624521
[do not merge] Trie Loader: use only "retain=false" IH (#891) 2020-08-15 07:52:36 +07:00
Alex Sharov
16f09be0a8
add method .Last() (#909) 2020-08-12 10:49:52 +07:00
Alex Sharov
760fe0f0b6
DupSort of hash state (#896) 2020-08-12 09:57:55 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Igor Mandrigin
34fd8b0c17
Merge pull request #888 from ledgerwatch/geth-1.9.18
Geth 1.9.18
2020-08-08 18:05:56 +02:00
Felix Lange
8396d95411 common/math: use math/bits intrinsics for Safe* (#21316)
This is a resubmit of ledgerwatch/turbo-geth#556. The performance
benefit of this change is negligible, but it does remove a TODO.
2020-08-07 12:52:46 +02:00
Felix Lange
e38c354c27 Revert "core, txpool: less allocations when handling transactions (#21232)"
Reverting because this change started handling account balances as
uint64 in the transaction pool, which is incorrect.

This reverts commit af5c97aebe1d37486635521ef553cb8bd4bada13.

# Conflicts:
#	core/types/transaction.go
2020-08-07 12:52:34 +02:00
Marius van der Wijden
84f3ed1b88 core, txpool: less allocations when handling transactions (#21232)
* core: use uint64 for total tx costs instead of big.Int

* core: added local tx pool test case

* core, crypto: various allocation savings regarding tx handling

* Update core/tx_list.go

* core: added tx.GasPriceIntCmp for comparison without allocation

adds a method to remove unneeded allocation in comparison to tx.gasPrice

* core: handle pools full of locals better

* core/tests: benchmark for tx_list

* core/txlist, txpool: save a reheap operation, avoid some bigint allocs

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	common/math/integer.go
#	core/tx_list.go
#	core/tx_pool.go
#	core/types/transaction.go
2020-08-07 12:25:38 +02:00
Guillaume Ballet
d04e4b25c8 common/fdlimit: build on DragonflyBSD (#21241)
* common/fdlimit: build on DragonflyBSD

* review feedback
# Conflicts:
#	go.mod
#	go.sum
2020-08-07 11:21:19 +02:00
Igor Mandrigin
bbb918ea6c etl: update docs with the correct sort method 2020-08-06 18:22:58 +02:00
Igor Mandrigin
ba9f9ee7dd
Add docs for common/etl (#878)
* ETL readme

* add a link to the explainer

* fixups

* a note on marshaling
2020-08-06 14:02:41 +02:00
Igor Mandrigin
1495979983
etl: only print file sizes for actual files, not RAM (#877) 2020-08-06 09:33:09 +02:00
Igor Mandrigin
fa89cdefd5
log ETL temp files datasize (#873) 2020-08-05 16:33:58 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key (#868) 2020-08-05 17:13:35 +07:00
ledgerwatch
0514501937
Revert "Use stage name as db key (#858)" (#867)
This reverts commit 7290bf519e.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key (#858) 2020-08-04 16:25:28 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) (#854)
* return error from rawdb

* squash

* v14

* improve performance of put

* clean

* clean

* hide feature behind ENV variable

* cleanup

* cleanup

* disable ipc and make Readme less confusing (people thought points are depend on each-other)

* fix test

* cleanup

* cleanup
2020-08-02 12:53:08 +01:00
ledgerwatch
7bc45fab80
Remove preimage storage mode, write receipts by default (#809) 2020-07-29 14:00:39 +01:00
ledgerwatch
6372b6ef5a
refuse to append to index is value is not greater than the last element (#807)
* Make stage7 and stage8 idempotent

* Fix formatting

* fix

* Fix formatting
2020-07-29 11:22:48 +01:00
Alex Sharov
7f60d1b535
Remove load start key from etl (#757)
* limit incremental step size of stages 5 and 6

* remove limit, make incremental

* remove_load_start_key_from_etl

* switch to "struct of arrays" from "array of structs"

* switch to "struct of arrays" from "array of structs"

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-18 10:17:04 +01:00
b00ris
12aa55bcbb
Remove incarnation inversion (#755)
* remove incarnation inversion

* fix lint

* remove incarnation inversion

* remove comment

* remove inversion from new code

* revert for unwind
2020-07-17 15:18:31 +01:00
Andrew Ashikhmin
165ae21ae3
Support multiple incarnations of the same contract in a single StorageChangeSet (#749)
* Support multiple incarntions of the same contract in a single StorageChangeSet

* linter

* Remove common.ToHex
2020-07-16 10:53:34 +01:00
Alex Sharov
6043c09ddf
Replace global buff pool by local, because of buffer leaks to larger pools (#739) 2020-07-14 08:56:47 +07:00
ledgerwatch
665b1d88b9
Speed up GenerateChain by using intermediate hashes (#736)
* GenerateChain to support intermediate hashes

* Fix formatting

* Changeset stats

* Fix formatting
2020-07-10 22:37:34 +01:00
Alex Sharov
df2241f13e
remove debug (#734) 2020-07-10 10:29:30 +07:00
Alex Sharov
de5dd0f55e
use non-empty buffer to encode changest (#729) 2020-07-10 10:26:48 +07:00
Alex Sharov
d98382fdf8
Integration add checkChangeSet into state_stages command (#728)
Integration add checkChangeSet into state_stages command (#728)
2020-07-10 10:25:40 +07:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724)
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
305f8aff02
Integration tests2 (#714)
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```
2020-07-07 11:00:25 +07:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes (#716)
* Fixing history generation

* Try fixes

* Reset from 0

* Cleanup
2020-07-06 07:34:24 +01:00
Alex Sharov
98f8ccc561
Command for long and heavy integration tests (#712)
* cmd for integration tests

* cmd for integration tests
2020-07-05 07:18:21 +01:00
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature (#711) 2020-07-05 07:17:27 +01:00
ledgerwatch
49181a298b
Special bucket for transaction senders (#707)
* Senders in the separate bucket

* Pre-create txSenders bucket

* Fix compile error

* Fix compile error

* Fix compile error, remove block transformations

* Fix formatting

* Fix test
2020-07-03 08:23:49 +01:00
ledgerwatch
b498419dcb
Fix loader crash (kludge) (#706)
* Kludge to remove duplicates

* Fix

* Fix formatting

* Fix in Len function
2020-07-03 08:08:35 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692)
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
2020-06-28 07:10:27 +01:00
Alex Sharov
b26b00f72b
increase IO buffers size (#686)
* increase IO buffers size

* found SSD threshold
2020-06-25 08:34:40 +01:00
Alex Sharov
daba929bd8
seek only when need, calculate next subtrie correctly (#685) 2020-06-24 07:54:34 +01:00
b00ris
7ac009a7da
Fix last chunk order (#683)
* fix commit order

* fix lint
2020-06-21 15:13:17 +01:00
ledgerwatch
fd98914c28
Stage6 - intermediate hashes (#677)
* First cut of the stage6

* Fix formatter

* Introduce state6

* Fix target number for stage6

* Fix linter

* Fix linter

* Reset in regenerate

* Correct block number

* Fix linter

* Fix linter

* Fix

* Upper case

* Skeleton to debug'

* Fix formatting

* Added codehash correction

* Fix TestUnwind

* fix test

* Fix linter

* Introduce unwind6

* Report adjustment error

* Code hashes included into stage5 incremental promotion and unwind

* Fix formatting

* Cleanup

* fix TestUnwind

* Cleanup

* Fix compile error

* Fix formatting

* unwind4

* Disable verifyRoot in stage5, fix hacks

* Remove verifyRoot function

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-06-18 22:27:11 +01:00
b00ris
7df7880a5a
[etl] Split collect&commit to separate threads (#676)
* fast search

* fix

* remove copy bytes

* speedup collector

* move logs to commit gr

* fix lint

* fix test
2020-06-18 19:14:10 +01:00
Alex Sharov
38f63f9af2
make lmdb default db (#675)
* make lmdb default db

* switch nodes db to lmdb

* log errors in nodedb

* fix miss typing

* fix default test target in makefile

* report error of .Get method

* add debug info

* run tests with debug

* run tests with debug

* remove default db from TEST_DB env

* cleanup

* wait for goroutines finish

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db

* wait for goroutines finish before close db
2020-06-18 19:13:58 +01:00
b00ris
e8b617bbf2
Add sort.Search to account changeset (#671)
* fast search

* fix

* remove copy bytes
2020-06-16 15:03:35 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Igor Mandrigin
96750b3711
Support ETL interruptions while promoting plain state (#662)
* lil etl changez

* fix test compile

* fixups to the unwinds

* add new buffer mode

* support unfinished transitions

* fixups

* fix tests

* linters

* linters
2020-06-13 16:03:38 +01:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index (#648)
* Debug infos

* bucketStats lmdb by default

* Use bools instead of strings

* Fix linters

* Fix linter

* Finally fix

* Fix compile error

* Fix test

* Fix formatting

* Fix shadowing

* Cleanup

* cleanup
2020-06-11 07:19:15 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00