Commit Graph

12046 Commits

Author SHA1 Message Date
Alex Sharov
9f11dabede
Enable golint back but with conditions (#641)
* enable_golint_with_conditions

* enable_golint_with_conditions
2020-06-09 12:00:58 +01:00
Andrew Ashikhmin
24690499f3
Upgrade uint256 to v1.1.0 (#642)
* Upgrade uint256 to v1.1.0

* go mod tidy
2020-06-09 12:00:47 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Alex Sharov
9459a58918
fix lmdb mem leak (#640) 2020-06-09 09:43:33 +07:00
Andrew Ashikhmin
d1509f21b8
No-need to right-pad in makePush (#638) 2020-06-08 16:55:19 +01:00
Alex Sharov
f00f4b5dff
KV: Tx objects pool (#637)
* Lmdb internal objects reuse (built-in feature of lmdb):
- lmdb read transactions pool
- lmdb also support cursors reuse, but not implemented in this PR

And kv abstraction objects reuse:
- lmdbKV pool of all tx objects
- boltKV pool of all tx objects
- badgerKV pool of all tx objects

* switch makefile back to bolt

* switch makefile back to bolt

* lmdb test run

* switch makefile back to bolt

* cursors pool

* run lmdb tests

* make kv objects pool global

* switch makefile back to bolt

* remove badgers GOMAXPROC setup, because our app tunned for sequential read/writes, not for random throughput

* simplify code
2020-06-08 11:53:03 +01:00
Igor Mandrigin
1d740fc78e
refactor stagedsync unwind, add tests(#636)
* some stubs

* post rebase

* fixups

* linter

* linter 2

* Update stages.go
2020-06-08 07:01:26 +03:00
ledgerwatch
0fb37c807e
Introduce intermediate hashes stage (#634)
* Query progress

* Run stage4 offline

* More thorough resetState

* Correct BlockNumber

* Fix formatting

* State loop

* do every 200k blocks

* Shift to 6.6m

* Close dbs in tests

* Stage2 with option of no reset

* every 100k blocks

* Reset state before stage5

* Introduce another stage

* Check compile errors

* Fix linter

* Fix linter

* Disable unreliable test

* Fix test

* Remove unreachable code
2020-06-07 09:56:46 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
Evgeny Danilenko
3134066bba
Fix stage 3 shutdown (#629)
* check stop on every block inside jobs

* stop spawnRecoverSendersStage properly

* fmt

* close recoverSenders goroutines

* fmt
2020-06-06 16:09:57 +01:00
Alex Sharov
ec3a5f6d38
lmdb version up to 2020 (#632) 2020-06-06 13:47:34 +01:00
ledgerwatch
cd9813b82c
Stage5 unwind (#631)
* Unwinding for stage5 - start

* Fix formatting

* Integrated unwind, but no tests yet

* Fix formatting

* Fix lint

* Fix lint

* Test for unwind stage5

* Fix formatting
2020-06-06 12:03:29 +01:00
Alex Sharov
d8ffd02b4f
Lmdb and Badger tests (#630)
* try lmddb

* try badger

* try bigger badger

* try lmb

* release

* release

* release

* release

* try badger

* try lmdb low memory

* try lmdb low memory

* lmdb high memory (16Mb not enough for /eth/downloader package)

* badger more memory

* badger more memory

* badger more memory
2020-06-06 11:29:11 +01:00
ledgerwatch
a8b438d21d
Incremental promotion for stage 5 (#628)
* Incremental promotion

* Fix compile errors

* Fix linter

* Transform state keys before loading

* Re-enable and fix the tests

* Fix formatting
2020-06-06 06:36:29 +01:00
Evgeny Danilenko
199ede36a2
Prevent slice make and copy in Pushes (#625)
* prevent slice make and copy in Pushes

* do not pad if the same length

* after CR

* linters

* linters

* linters
2020-06-05 19:46:34 +03:00
Evgeny Danilenko
8d7e603f07
stack pool (#627) 2020-06-05 17:45:56 +03:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger (#624)
* lmdb tests

* trigger ci

* fix tests

* disable parallelism

* disable parallelism

* cleanup resources

* cleanup resources

* reduce concurency

* try run tests on bolt

* try run tests on bolt

* fix downloader test

* run bolt tests

* rely on interface instead of exact instance

* Rename AbstractKV to KV

* don't use separator for badger

* don't initialize badger cursor - because it not used here

* fix linter

* try reduce badger compactors

* compat with master

* try lmdb

* try lmdb

* try lmdb

* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option

* allow to close closed db

* release

* release

* ideal batch size for badger

* ideal batch size for badger
2020-06-05 10:25:33 +01:00
Alex Sharov
318b1b59a2
Bolt: cursor.Put (#622)
* Bolt: cursor.Put

* go mod tidy
2020-06-05 10:25:20 +01:00
Alex Sharov
de76f5a006
Add more info about badger's vlog gc (#620) 2020-06-05 06:33:02 +01:00
Andrew Ashikhmin
05e81184d9
Misc GC optimisations (#619)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray

* Misc GC optimisations

* unsafe experiment with writeByteArray
2020-06-04 19:30:28 +01:00
ledgerwatch
b20cdbe399
Enable staged sync (#618) 2020-06-04 12:34:22 +01:00
Alex Sharov
abf8dfbcf6
implement cursor put (#617)
* implement cursor put

* add bolt and bucket cursor.Put/Del
2020-06-04 12:19:59 +01:00
Alex Sharov
03dde74eca
increase golint confidence level to reduce renaming advises (#616)
* increase golint confidence level to reduce renaming advises

* disable golint because go-ethereum doesn't follow it
2020-06-04 11:21:46 +01:00
Alex Sharov
7aa050ca79
[Breaking change in db!] UI for Buckets stat (#607)
* UI for buckets size
2020-06-04 16:35:42 +07:00
Alex Sharov
717b096afc
[wip] Cache z3 build on CI, up golangci-lint to v1.27.0 (#615)
* z3 build cache

* z3 build cache

* circle 2.1 version

* golangci-lint up to v1.27.0

* separate z3 cache, add architecture to the key

* separate z3 cache, add architecture to the key

* rerun ci

* rerun ci

* rerun ci

* fix z3 cache

* fix z3 cache

* fix z3 cache

* try shorter

* trigger ci
2020-06-04 09:54:17 +01:00
Andrew Ashikhmin
33c28f7fac
[GC] uint256 rather than big.Int in Transaction (#614)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray
2020-06-04 08:43:08 +01:00
Alex Sharov
ff1cde5259
simulated backend: replace .MemCopy by .NewBatch+.Commit (#613)
* simulated backend - replace .MemCopy by .NewBatch and native transactions
close database which createtd inside simulated backend
2020-06-04 11:59:16 +07:00
Alex Sharov
7818d04acc
badger-compatible trie loader (key and value in badger iterator valid only until .Next/.Seek call) (#612) 2020-06-04 11:50:44 +07:00
Igor Mandrigin
0533eed812
Refactor Stagedsync part 1 (#610) 2020-06-03 17:25:44 +03:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments (#608)
* First commit

* Fix the one-off error

* Fix formatting

* Ability to execute stage5 separately

* Clean up intermediate hashes and stage5 progress

* Fix linter

* Print original keys when extracting

* channel

* More logging

* More logging

* Remove excess logging

* fix stage2

* Revert

* Fix stage2

* Add provider exhausted

* Sort sortable buffer

* Fix test

* Another cleanup

* Remove exhaust log
2020-06-03 13:03:40 +01:00
Igor Mandrigin
1e20ed255d
move staged sync to its own package (#605) 2020-06-02 17:52:50 +03:00
ledgerwatch
4ce69916dc
Fix rebuilding the Intermediate Hash bucket (#602)
* Changes

* First attempt to fix

* Sync works

* Fix compile

* Fix linter

* Fix test

* Fix test

* Debug

* Propagate tracing

* Trace default receiver

* More tracing

* Correct unfurl

* Remove tracing

* Try to retain trie for debugging
2020-06-02 08:46:21 +01:00
Igor Mandrigin
c16d3da1b4
etl: startkey for Load + OnLoadCommit callback (#601) 2020-06-01 17:14:40 +03:00
Andrew Ashikhmin
8f031cc85f
Update Secp256k1 and tune its flags for performance (#600)
* Update to 05d315affe

* Changes in the wrapper after the library upgrade

* Faster config for x64

* USE_ENDOMORPHISM
2020-06-01 13:16:17 +01:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes (#597)
* Introduce hashCollector

* Add HashCollector to SubTrieLoader

* Fix linter

* Reset hashed state

* Not to regenerate the hashed state

* Not to delete state

* Fix linter

* Print expected hash in the beginning

* Simplify

* Remove initialTrie

* Use etl to buffer intermediate hashes

* Copy values, not insert empty key

* Compress instead of decompress

* Enhance file buffer logging, fix linter

* Fix compile errors

* Fix log

* Fix logging

* Exclude zero key again

* Add rewind

* Restrict timestamps

* Fix

* Fix formatting

* Incorporate separation

* Extract identityLoadFunction

* Fix formatting
2020-05-31 13:23:34 +01:00
ledgerwatch
04b0dafb3d
Fix TestIndexGenerator_GenerateIndex_SimpleCase (out of file handles) (#596)
* Fix test

* Fix formatting
2020-05-31 09:12:10 +01:00
Igor Mandrigin
786561e65e
don't write files if all data fits into one buffer. (#594) 2020-05-31 08:32:33 +01:00
b00ris
266f9a5208
Plain state index (#595)
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
2020-05-31 07:57:47 +01:00
Evgeny Danilenko
32ec443572
Restore graceful shutdown (#591)
* initial

* gracefull shutdown for staged

* more wg fixes

* fmt

* linters

* remove generalization

* linters

* linters

* fix

* fix

* fmt

* quit into etl

* after CR

* after CR
2020-05-30 16:44:54 +03:00
Igor Mandrigin
a49fa497df
add go mod tidy check (#593) 2020-05-30 09:12:32 +01:00
Alex Sharov
916a1f8b83
[wip] Lmdb: AbstractKV and DB interfaces (#589)
* resetIH from scratch if needed

* lmdb

* add AbstractKV to loader, added new Object accessor around AbstractKV

* add lmdb cli flag

* add requirement of k!=nil on error in docs

* add Size method for compatibility

* read after put tests

* fix multiput nils

* simplify loops

* increase mmap size

* better error messages

* better error messages

* fix tests

* better error messages

* cleanup

* avoid bolt usage in test

* move hardcoded bucket name to dbutils

* register more buckets

* register more buckets

* fix test
2020-05-30 09:12:21 +01:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. (#592) 2020-05-30 10:00:35 +03:00
Andrew Ashikhmin
12d6851be4
[Issue 552] Disable debug protocol by default (#588) 2020-05-28 18:58:08 +01:00
ledgerwatch
dd4db2b36e
Use AbstractKV instead of BoltKV in the SubTrieLoader (#587) 2020-05-28 15:05:35 +01:00
ledgerwatch
f752ac4810
Increase max table size (#586) 2020-05-28 12:52:09 +01:00
Alex Sharov
d74732a662
bolt db stats export to prometheus (#585) 2020-05-28 12:34:37 +01:00
Alex Sharov
daa359c363
Mgr schedule iterator (#566)
* db based version of PrefixByCumulativeWitnessSize

* db based version of PrefixByCumulativeWitnessSize

* retain all in Trie by default

* fix WitnessLen logic in calcTrie roots

* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket

* handle corner cases in WL

* Use correct incarnation for IH bucket

* use name WitnessSize

* save progress towards db-only witness estimation

* results from trie and from db are still different

* less recursion

* correct incarnation in CumulativeSearch

* reuse results from previous Tick, separate concepts of parent and startKey

* experiment: if not including trie structure to WitnessSize will reduce cumulative error

* tool to generate all IH and tool to calculate assessment of cumulative error

* tool to generate all IH

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* better stats

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* calculate ticks size distribution

* estimate cumulative error

* fix linter

* resetIH from scratch if needed

* cleanup

* fix test

* fix test
2020-05-28 12:33:05 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
ledgerwatch
e348dcb846 Move promotion before the shortcut (#582)
* Move promotion before the shortcut

* Correct the logs

* Trying without the encoder

* Revert "Trying without the encoder"

This reverts commit 24c526bc2791fb6806b1c0d4f52a43b87c3959b9.

* Commit to db, not to the mutation

* Fix log more

* No need to log every deleted file
2020-05-27 09:26:56 +01:00
ledgerwatch
5456b2371a
Switch to the latest bolt for hashmap-based freelist (#581) 2020-05-26 18:45:28 +01:00