Commit Graph

12375 Commits

Author SHA1 Message Date
gary rong
1665219f59 trie: extend range proofs with non-existence (#21000)
* trie: implement range proof with non-existent edge proof

* trie: fix cornercase

* trie: consider empty range

* trie: add singleSide test

* trie: support all-elements range proof

* trie: fix typo

* trie: tiny typos and formulations

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	go.sum
#	trie/proof.go
#	trie/proof_test.go
2020-06-15 19:38:12 +03:00
ucwong
b71b1cf73c whisper: use canonical import name of package go-ethereum (#21099)
# Conflicts:
#	whisper/shhclient/client.go
2020-06-15 19:38:12 +03:00
Boqin Qin
defe185e77 accounts: add blockByNumberNoLock() to avoid double-lock (#20983)
* abi/bind/backends: testcase for double-lock

* accounts: add blockByNumberNoLock to avoid double-lock

* backend/simulated: use stateroot, not blockhash for retrieveing state

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
2020-06-15 19:38:12 +03:00
Martin Holst Swende
fd4f6397ee cmd/clef, signer/core: password input fixes (#20960)
* cmd/clef, signer/core: use better terminal input for passwords, make it possible to avoid boot-up warning

* all: move commonly used prompter to isolated (small) package

* cmd/clef: Add new --acceptWarn to clef README

* cmd/clef: rename flag 'acceptWarn' to 'suppress-bootwarn'

Co-authored-by: ligi <ligi@ligi.de>
# Conflicts:
#	cmd/clef/main.go
#	cmd/devp2p/dnscmd.go
#	cmd/ethkey/utils.go
#	cmd/geth/accountcmd.go
#	cmd/geth/chaincmd.go
#	cmd/geth/main.go
#	cmd/wnode/main.go
#	console/bridge.go
#	console/console.go
#	console/console_test.go
#	signer/core/cliui.go
2020-06-15 19:38:12 +03:00
Felix Lange
d019d73e97 console: fix unlockAccount argument count check (#21081) 2020-06-15 19:38:12 +03:00
Marius van der Wijden
88e3533dd5 accounts/abi: simplify reflection logic (#21058)
* accounts/abi: simplified reflection logic

* accounts/abi: simplified reflection logic

* accounts/abi: removed unpack

* accounts/abi: removed comments

* accounts/abi: removed uneccessary complications

* accounts/abi: minor changes in error messages

* accounts/abi: removed unnused code

* accounts/abi: fixed indexed argument unpacking

* accounts/abi: removed superfluous test cases

This commit removes two test cases. The first one is trivially invalid as we have the same
test cases as passing in packing_test.go L375. The second one passes now,
because we don't need the mapArgNamesToStructFields in unpack_atomic anymore.
Checking for purely underscored arg names generally should not be something we do
as the abi/contract is generally out of the control of the user.

* accounts/abi: removed comments, debug println

* accounts/abi: added commented out code

* accounts/abi: addressed comments

* accounts/abi: remove unnecessary dst.CanSet check

* accounts/abi: added dst.CanSet checks
# Conflicts:
#	accounts/abi/reflect.go
2020-06-15 19:38:12 +03:00
Satpal
82ee3c7711 cmd: fix log contexts (#21077) 2020-06-15 19:38:12 +03:00
Péter Szilágyi
17a0766b36 params: begin v1.9.15 release cycle 2020-06-15 19:38:12 +03:00
Igor Mandrigin
25f9f0f34d add to-merge.txt 2020-06-15 19:38:12 +03:00
Alex Sharov
57358730a4
Minor lmdb related improvements (#667)
* don't call initCursor on happy path

* don't call initCursor on happy path

* don't run stale reads goroutine for inMem mode

* don't call initCursor on happy path

* remove buffers from cursor object - they are useful only in Badger implementation

* commit kv benchmark

* remove buffers from cursor object - they are useful only in Badger implementation

* remove buffers from cursor object - they are useful only in Badger implementation

* cancel server before return pipe to pool

* try  to fix test

* set field db in managed tx
2020-06-15 13:30:54 +01:00
Alex Sharov
862296d291
Lmdb cursors not always return to pool (#664)
* set Pooled=true

* don't call initCursor on happy path
2020-06-14 21:05:52 +07:00
ledgerwatch
a8c9910f04
Various fixes to make sync work (#663)
* Fix body fetch

* Reduce spurious reorgs

* Exit the sync cycle after unwinds

* Fix out of range

* No stalling check for staged sync

* Disable failing tests

* Remove duplicate log message

* Fix UnwindTest and add assertions

* Fix formatting

* Cleanup

* Fix off by one error with bodies

* Remove rollback
2020-06-13 22:39:55 +01:00
Igor Mandrigin
96750b3711
Support ETL interruptions while promoting plain state (#662)
* lil etl changez

* fix test compile

* fixups to the unwinds

* add new buffer mode

* support unfinished transitions

* fixups

* fix tests

* linters

* linters
2020-06-13 16:03:38 +01:00
ledgerwatch
d6c89da694
Re-enable skeleton assembly mode (#661) 2020-06-13 15:13:30 +01:00
Alex Sharov
bcac51697b
Enable NoReadahead by default (#660)
* Enable NoReadahead by default

* don't call lmdb.IsNotFound on happy path

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-06-13 14:12:35 +01:00
Alex Sharov
b78a6336d2
dont re-read whole db to calculate file size (#658) 2020-06-13 13:41:38 +01:00
ledgerwatch
99b9c0fa99
Re-enable TestUnwind test and fix issues in the stage sync (#659)
* Fixes to stages

* Fix formatting

* Fixes to stages

* Fix formatting

* Fix linter

* Fix lint

* Updates

* Next update

* Update

* Test works

* Cleanup

* Fix formatting

* Fix formatting

* Cleanup

* Fix core test

* Reinstate randomness for reorgs
2020-06-13 13:39:04 +01:00
Alex Sharov
b05fafc628
Optimize multi put (#657)
* remove unused slice from MultiPut

* mutation: reuse tuples slice and preallocate bucketPuts

* use bucketPool in kv_lmdb

* remove duplicated check of context status

* more benchmarks

* remove reusage of puts
2020-06-12 18:59:06 +01:00
Andrew Ashikhmin
ebdac3a192
AVX2 fnvHash. Bump numOfGoroutines to 8 in stage_senders.go (#656)
* Profile all stages

* Try to recover senders with 8 goroutines

* fix CPU profiling for stage_bodies

* fix out-of-index

* Try full DAG for verfication of header seals

* Try to unroll fnvHash for performance

* SSE2 assembly for fnvHash16

* fnvHash16AVX2

* Revert changes to state.go

* check we're on 64-bit in useAVX2

* Shave a move off fnvHash16AVX2

* asmdecl doesn't know about VMOVD

* disable linter in the right place
2020-06-12 12:48:59 +01:00
Alex Sharov
7c18e6d270
Use same lmdb version as in python bindings: version of 2019 year (#649)
* Upgrade lmdb to version of 2019, which is compatible with https://github.com/jnwatson/py-lmdb (lmdb of 2020 year version - upgaded lock file version from 1 to 2 - and python binding can't open database if lockfile exists - if main app is running).
2020-06-12 18:42:18 +07:00
Alex Sharov
15096f273d
Remove ctx from Open. Stop goroutines on Close. (#650)
* remove ctx from MustOpen

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)

* enable RawReads and add native implementation of Get/Has methods
2020-06-12 10:31:21 +01:00
Alex Sharov
9961fef39e
experiment with rewriting gas_limits to python+lmdb (just to see how it can be) (#654) 2020-06-12 09:25:40 +01:00
Alex Sharov
351c188b02
send bolt buckets stats to prometheus (#653) 2020-06-12 09:24:46 +01:00
Alex Sharov
5d8e147a02
enable RawReads and add native implementation of Get/Has methods (#655) 2020-06-12 09:24:15 +01:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index (#648)
* Debug infos

* bucketStats lmdb by default

* Use bools instead of strings

* Fix linters

* Fix linter

* Finally fix

* Fix compile error

* Fix test

* Fix formatting

* Fix shadowing

* Cleanup

* cleanup
2020-06-11 07:19:15 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
Alex Sharov
fd61af7042
lmdb bucket stats (#645)
* lmdb bucket stats

* fix lint warnings

* add bucket.Size method

* add bucket.Size method
2020-06-10 07:42:26 +01:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
Igor Mandrigin
0ee2d83970
stagedsync: allow long stages/unwinds to be interrupted (#643)
use `s.UpdateWithStageData(db, <block number>, <key>)` to store the key

use `s.StageData` with `etl.NextKey` to restart ETL from where it was interrupted.
2020-06-09 15:49:49 +03:00
Alex Sharov
9f11dabede
Enable golint back but with conditions (#641)
* enable_golint_with_conditions

* enable_golint_with_conditions
2020-06-09 12:00:58 +01:00
Andrew Ashikhmin
24690499f3
Upgrade uint256 to v1.1.0 (#642)
* Upgrade uint256 to v1.1.0

* go mod tidy
2020-06-09 12:00:47 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Alex Sharov
9459a58918
fix lmdb mem leak (#640) 2020-06-09 09:43:33 +07:00
Andrew Ashikhmin
d1509f21b8
No-need to right-pad in makePush (#638) 2020-06-08 16:55:19 +01:00
Alex Sharov
f00f4b5dff
KV: Tx objects pool (#637)
* Lmdb internal objects reuse (built-in feature of lmdb):
- lmdb read transactions pool
- lmdb also support cursors reuse, but not implemented in this PR

And kv abstraction objects reuse:
- lmdbKV pool of all tx objects
- boltKV pool of all tx objects
- badgerKV pool of all tx objects

* switch makefile back to bolt

* switch makefile back to bolt

* lmdb test run

* switch makefile back to bolt

* cursors pool

* run lmdb tests

* make kv objects pool global

* switch makefile back to bolt

* remove badgers GOMAXPROC setup, because our app tunned for sequential read/writes, not for random throughput

* simplify code
2020-06-08 11:53:03 +01:00
Igor Mandrigin
1d740fc78e
refactor stagedsync unwind, add tests(#636)
* some stubs

* post rebase

* fixups

* linter

* linter 2

* Update stages.go
2020-06-08 07:01:26 +03:00
ledgerwatch
0fb37c807e
Introduce intermediate hashes stage (#634)
* Query progress

* Run stage4 offline

* More thorough resetState

* Correct BlockNumber

* Fix formatting

* State loop

* do every 200k blocks

* Shift to 6.6m

* Close dbs in tests

* Stage2 with option of no reset

* every 100k blocks

* Reset state before stage5

* Introduce another stage

* Check compile errors

* Fix linter

* Fix linter

* Disable unreliable test

* Fix test

* Remove unreachable code
2020-06-07 09:56:46 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
Evgeny Danilenko
3134066bba
Fix stage 3 shutdown (#629)
* check stop on every block inside jobs

* stop spawnRecoverSendersStage properly

* fmt

* close recoverSenders goroutines

* fmt
2020-06-06 16:09:57 +01:00
Alex Sharov
ec3a5f6d38
lmdb version up to 2020 (#632) 2020-06-06 13:47:34 +01:00
ledgerwatch
cd9813b82c
Stage5 unwind (#631)
* Unwinding for stage5 - start

* Fix formatting

* Integrated unwind, but no tests yet

* Fix formatting

* Fix lint

* Fix lint

* Test for unwind stage5

* Fix formatting
2020-06-06 12:03:29 +01:00
Alex Sharov
d8ffd02b4f
Lmdb and Badger tests (#630)
* try lmddb

* try badger

* try bigger badger

* try lmb

* release

* release

* release

* release

* try badger

* try lmdb low memory

* try lmdb low memory

* lmdb high memory (16Mb not enough for /eth/downloader package)

* badger more memory

* badger more memory

* badger more memory
2020-06-06 11:29:11 +01:00
ledgerwatch
a8b438d21d
Incremental promotion for stage 5 (#628)
* Incremental promotion

* Fix compile errors

* Fix linter

* Transform state keys before loading

* Re-enable and fix the tests

* Fix formatting
2020-06-06 06:36:29 +01:00
Evgeny Danilenko
199ede36a2
Prevent slice make and copy in Pushes (#625)
* prevent slice make and copy in Pushes

* do not pad if the same length

* after CR

* linters

* linters

* linters
2020-06-05 19:46:34 +03:00
Evgeny Danilenko
8d7e603f07
stack pool (#627) 2020-06-05 17:45:56 +03:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger (#624)
* lmdb tests

* trigger ci

* fix tests

* disable parallelism

* disable parallelism

* cleanup resources

* cleanup resources

* reduce concurency

* try run tests on bolt

* try run tests on bolt

* fix downloader test

* run bolt tests

* rely on interface instead of exact instance

* Rename AbstractKV to KV

* don't use separator for badger

* don't initialize badger cursor - because it not used here

* fix linter

* try reduce badger compactors

* compat with master

* try lmdb

* try lmdb

* try lmdb

* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option

* allow to close closed db

* release

* release

* ideal batch size for badger

* ideal batch size for badger
2020-06-05 10:25:33 +01:00
Alex Sharov
318b1b59a2
Bolt: cursor.Put (#622)
* Bolt: cursor.Put

* go mod tidy
2020-06-05 10:25:20 +01:00
Alex Sharov
de76f5a006
Add more info about badger's vlog gc (#620) 2020-06-05 06:33:02 +01:00
Andrew Ashikhmin
05e81184d9
Misc GC optimisations (#619)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray

* Misc GC optimisations

* unsafe experiment with writeByteArray
2020-06-04 19:30:28 +01:00
ledgerwatch
b20cdbe399
Enable staged sync (#618) 2020-06-04 12:34:22 +01:00