Commit Graph

1757 Commits

Author SHA1 Message Date
Giulio rebuffo
011a15350b
fix stage 2 (#705)
* removed unused actions in stage 2

* fixed stage 2
2020-07-02 19:44:08 +01:00
Giulio rebuffo
3890900dd0
removed unused actions in stage 2 (#701) 2020-07-02 08:38:25 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698)
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
2020-07-01 15:56:56 +01:00
Giulio rebuffo
3ee1435ac2
integration of insertBodies instead of insertChain (#700)
* integration of insertBodies instead of insertChain

* remove seal verification from stage2

* fixed seals
2020-07-01 08:23:15 +01:00
Giulio rebuffo
85ab6ff8aa
added insertBodies (#695) 2020-06-29 17:26:33 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692)
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
2020-06-28 07:10:27 +01:00
ledgerwatch
4c456a649f
Separating HashState and IntermediateHashes stages again, fixing errors (#688)
* TestWatchNoDir not to be parallel

* Print ks and inc

* Print addrHash

* Change the buffer

* Print loading

* Skip

* More logging

* Error out earlier

* Handle empty codes

* Handle empty codes

* Remove logging

* Compare states

* Not do stage5

* compareBucket

* Preimage

* Clearer errors

* No need to clean up contract code

* Restore stage6

* Printing

* Skip the skipping

* Print all

* Change buffer type

* Add limit to stage5

* Always fail

* Remove exessive logging

* Restore buffer type

* Revert

* Print when exception

* Reenable skipping

* Skip storage items for deleted contracts

* not shortcut

* Remove removeAccount

* Re-enable state hashing

* Default to plain state

* Disable hashing state

* Reenable reset5

* Print unfurl list

* Enable removingAccount

* No printing

* Reenable stage5 commit

* Swap order of stages

* Prevent backwards promotion, reset tx lookup

* reset finish

* Introduce storage item replacement

* See if unwind works

* Restore removingAccount

* Don't do removeAccount for unwinding

* Possible fix

* Proper(er) fix

* Don't exclude unwinding

* Remove unwinding flag

* Fix formatting

* Fix lint

* Not to ignore blocks if they cause reorg

* Fix test, separate stages again

* Fix TestUnwind

* Fix stage

* Swap unwinding

* Revert to unwinding flag

* Print unfurl list

* Print

* Print inside receive

* Print after

* No printing

* Cleanup

* Not use blockCache when doing GetBlock
2020-06-26 07:46:05 +01:00
b00ris
e8b617bbf2
Add sort.Search to account changeset (#671)
* fast search

* fix

* remove copy bytes
2020-06-16 15:03:35 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Alex Sharov
8d04689041
close db in tests (#669) 2020-06-16 12:05:40 +07:00
ledgerwatch
dec30b698d
Fix execution for block 10094566 (#665)
* First sketch of the fix

* Re-enable stage4

* Fix formatting

* Park the changes

* Save updates

* Save update

* Cleanup

* Cleanup

* Cleanup

* Fix lint

* Cleanup

* Update

* Tx tracing

* Update

* Cleanup

* Switch tests back to bolt

* Test cleanup

* Update

* Polymorthic CREATE2 test

* Add assertions

* Fix formatting

* Fix test

* Cleanup

* Cleanup

* Cleanup
2020-06-15 22:24:08 +01:00
Igor Mandrigin
93f6c4c01a post-rebase fixups 2020-06-15 19:38:54 +03:00
Ev
f67b3d7d45 core: fix typo in comments (#21181) 2020-06-15 19:38:13 +03:00
Marius van der Wijden
cc69b76852 core/vm, crypt/bls12381: fixed comments in bls (#21182)
* core/vm: crypto/bls12381: minor code comments

* crypto/bls12381: fix comment
2020-06-15 19:38:13 +03:00
Martin Holst Swende
7347d036b0 cmd, core, params: inital support for yolo-v1 testnet (#21154)
* core,params,puppeth: inital support for yolo-v1 testnet

* cmd/geth, core: add yolov1 console flag

* cmd, core, params: YoloV1 bakein fixups

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/vm/interpreter.go
#	params/config.go
2020-06-15 19:38:13 +03:00
kilic
0921861edb core/vm, crypto/bls12381, params: add bls12-381 elliptic curve precompiles (#21018)
* crypto: add bls12-381 elliptic curve wrapper

* params: add bls12-381 precompile gas parameters

* core/vm: add bls12-381 precompiles

* core/vm: add bls12-381 precompile tests

* go.mod, go.sum: use latest bls12381 lib

* core/vm: move point encode/decode functions to base library

* crypto/bls12381: introduce bls12-381 library init function

* crypto/bls12381: import bls12381 elliptic curve implementation

* go.mod, go.sum: remove bls12-381 library

* remove unsued frobenious coeffs

supress warning for inp that used in asm

* add mappings tests for zero inputs

fix swu g2 minus z inverse constant

* crypto/bls12381: fix typo

* crypto/bls12381: better comments for bls12381 constants

* crypto/bls12381: swu, use single conditional for e2

* crypto/bls12381: utils, delete empty line

* crypto/bls12381: utils, use FromHex for string to big

* crypto/bls12381: g1, g2, strict length check for FromBytes

* crypto/bls12381: field_element, comparision changes

* crypto/bls12381: change swu, isogeny constants with hex values

* core/vm: fix point multiplication comments

* core/vm: fix multiexp gas calculation and lookup for g1 and g2

* core/vm: simpler imput length check for multiexp and pairing precompiles

* core/vm: rm empty multiexp result declarations

* crypto/bls12381: remove modulus type definition

* crypto/bls12381: use proper init function

* crypto/bls12381: get rid of new lines at fatal desciprtions

* crypto/bls12-381: fix no-adx assembly multiplication

* crypto/bls12-381: remove old config function

* crypto/bls12381: update multiplication backend

this commit changes mul backend to 6limb eip1962 backend

mul assign operations are dropped

* core/vm/contracts_tests: externalize test vectors for precompiles

* core/vm/contracts_test: externalize failure-cases for precompiles

* core/vm: linting

* go.mod: tiny up sum file

* core/vm: fix goimports linter issues

* crypto/bls12381: build tags for plain ASM or ADX implementation

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/vm/contracts.go
#	core/vm/contracts_test.go
#	go.sum
2020-06-15 19:38:13 +03:00
chenglin
efb387fff9 core: collect NewTxsEvent items without holding reorg lock (#21145)
# Conflicts:
#	core/tx_pool.go
2020-06-15 19:38:13 +03:00
Greg Colvin
57e88afcd4 core/vm: EIP-2315, JUMPSUB for the EVM (#20619)
* core/vm: implement EIP 2315, subroutines for the EVM

* core/vm: eip 2315 - lintfix + check jump dest validity + check ret stack size constraints

  logger: markdown-friendly traces, validate jumpdest, more testcase, correct opcodes

* core/vm: update subroutines acc to eip: disallow walk-into

* core/vm/eips: gas cost changes for subroutines

* core/vm: update opcodes for EIP-2315

* core/vm: define RETURNSUB as a 'jumping' operation + review concerns

Co-authored-by: Martin Holst Swende <martin@swende.se>
# Conflicts:
#	core/vm/contract.go
#	core/vm/instructions_test.go
#	core/vm/interpreter.go
#	core/vm/logger.go
#	core/vm/logger_json.go
#	core/vm/logger_test.go
#	core/vm/runtime/runtime_test.go
#	eth/tracers/tracer.go
2020-06-15 19:38:13 +03:00
Felix Lange
a9f5546d2e eth: interrupt chain insertion on shutdown (#21114)
This adds a new API method on core.BlockChain to allow interrupting
running data inserts, and calls the method before shutting down the
downloader.

The BlockChain interrupt checks are now done through a method instead
of inlining the atomic load everywhere. There is no loss of efficiency from
this and it makes the interrupt protocol a lot clearer because the check is
defined next to the method that sets the flag.

# Conflicts:
#	core/blockchain.go
#	light/lightchain.go
2020-06-15 19:38:13 +03:00
Martin Michlmayr
732a20d468 all: fix typos in comments (#21118)
# Conflicts:
#	accounts/abi/bind/backends/simulated.go
#	cmd/puppeth/module_dashboard.go
#	core/blockchain_test.go
#	core/rawdb/chain_iterator.go
#	core/state/snapshot/difflayer.go
#	core/state/snapshot/iterator.go
#	core/state/snapshot/iterator_fast.go
#	core/state/snapshot/snapshot.go
#	core/state/snapshot/wipe.go
#	core/tx_pool.go
#	les/clientpool.go
#	les/peer.go
#	mobile/doc.go
#	trie/committer.go
#	trie/database.go
2020-06-15 19:38:13 +03:00
Felföldi Zsolt
1ca95e8352 les, les/lespay: implement new server pool (#20758)
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.

Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4

# Conflicts:
#	cmd/utils/flags.go
#	core/forkid/forkid.go
#	les/client.go
#	les/client_handler.go
#	les/commons.go
#	les/distributor.go
#	les/enr_entry.go
#	les/fetcher.go
#	les/lespay/client/valuetracker.go
#	les/metrics.go
#	les/peer.go
#	les/protocol.go
#	les/retrieve.go
#	les/server.go
#	les/serverpool.go
#	les/test_helper.go
#	les/utils/expiredvalue.go
#	les/utils/weighted_select.go
#	les/utils/weighted_select_test.go
#	params/bootnodes.go
2020-06-15 19:38:12 +03:00
ledgerwatch
a8c9910f04
Various fixes to make sync work (#663)
* Fix body fetch

* Reduce spurious reorgs

* Exit the sync cycle after unwinds

* Fix out of range

* No stalling check for staged sync

* Disable failing tests

* Remove duplicate log message

* Fix UnwindTest and add assertions

* Fix formatting

* Cleanup

* Fix off by one error with bodies

* Remove rollback
2020-06-13 22:39:55 +01:00
ledgerwatch
99b9c0fa99
Re-enable TestUnwind test and fix issues in the stage sync (#659)
* Fixes to stages

* Fix formatting

* Fixes to stages

* Fix formatting

* Fix linter

* Fix lint

* Updates

* Next update

* Update

* Test works

* Cleanup

* Fix formatting

* Fix formatting

* Cleanup

* Fix core test

* Reinstate randomness for reorgs
2020-06-13 13:39:04 +01:00
Alex Sharov
b05fafc628
Optimize multi put (#657)
* remove unused slice from MultiPut

* mutation: reuse tuples slice and preallocate bucketPuts

* use bucketPool in kv_lmdb

* remove duplicated check of context status

* more benchmarks

* remove reusage of puts
2020-06-12 18:59:06 +01:00
ledgerwatch
853db762e2
Make GetAsOf work with partially missing history index (#648)
* Debug infos

* bucketStats lmdb by default

* Use bools instead of strings

* Fix linters

* Fix linter

* Finally fix

* Fix compile error

* Fix test

* Fix formatting

* Fix shadowing

* Cleanup

* cleanup
2020-06-11 07:19:15 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
Andrew Ashikhmin
cf7b9a75a7
Issue 635: Stage4 to optionally generate persisted receipts (#644)
* Euphemerally -> Ephemerally

* Move StorageMode to ethdb and pass it to PrepareStagedSync

* linter

* Remove StorageModeThinHistory and move SetStorageModeIfNotExist into storage_mode.go

* Optionally write receipts in the execute stage

* memory profiler

* linter

* proper linter fix

* linter

* typo

* Merge stateDb with changeDb so that all-or-nothing is commited in stage_execute
2020-06-09 14:11:09 +01:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Andrew Ashikhmin
d1509f21b8
No-need to right-pad in makePush (#638) 2020-06-08 16:55:19 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
Alex Sharov
d8ffd02b4f
Lmdb and Badger tests (#630)
* try lmddb

* try badger

* try bigger badger

* try lmb

* release

* release

* release

* release

* try badger

* try lmdb low memory

* try lmdb low memory

* lmdb high memory (16Mb not enough for /eth/downloader package)

* badger more memory

* badger more memory

* badger more memory
2020-06-06 11:29:11 +01:00
Evgeny Danilenko
199ede36a2
Prevent slice make and copy in Pushes (#625)
* prevent slice make and copy in Pushes

* do not pad if the same length

* after CR

* linters

* linters

* linters
2020-06-05 19:46:34 +03:00
Evgeny Danilenko
8d7e603f07
stack pool (#627) 2020-06-05 17:45:56 +03:00
Alex Sharov
f0bc2b2146
Run tests on lmdb and badger (#624)
* lmdb tests

* trigger ci

* fix tests

* disable parallelism

* disable parallelism

* cleanup resources

* cleanup resources

* reduce concurency

* try run tests on bolt

* try run tests on bolt

* fix downloader test

* run bolt tests

* rely on interface instead of exact instance

* Rename AbstractKV to KV

* don't use separator for badger

* don't initialize badger cursor - because it not used here

* fix linter

* try reduce badger compactors

* compat with master

* try lmdb

* try lmdb

* try lmdb

* reduce badger's MaxTableSize, reduce badger's minGoMaxProc for inMem option

* allow to close closed db

* release

* release

* ideal batch size for badger

* ideal batch size for badger
2020-06-05 10:25:33 +01:00
Andrew Ashikhmin
05e81184d9
Misc GC optimisations (#619)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray

* Misc GC optimisations

* unsafe experiment with writeByteArray
2020-06-04 19:30:28 +01:00
Alex Sharov
7aa050ca79
[Breaking change in db!] UI for Buckets stat (#607)
* UI for buckets size
2020-06-04 16:35:42 +07:00
Andrew Ashikhmin
33c28f7fac
[GC] uint256 rather than big.Int in Transaction (#614)
* uint256 in rlp

* uint256 rather than big.Int in Transation

* linters

* more linters

* still linters

* Reduce garbage in writeUint256

* Experiment with GC in writeByteArray
2020-06-04 08:43:08 +01:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments (#608)
* First commit

* Fix the one-off error

* Fix formatting

* Ability to execute stage5 separately

* Clean up intermediate hashes and stage5 progress

* Fix linter

* Print original keys when extracting

* channel

* More logging

* More logging

* Remove excess logging

* fix stage2

* Revert

* Fix stage2

* Add provider exhausted

* Sort sortable buffer

* Fix test

* Another cleanup

* Remove exhaust log
2020-06-03 13:03:40 +01:00
ledgerwatch
4ce69916dc
Fix rebuilding the Intermediate Hash bucket (#602)
* Changes

* First attempt to fix

* Sync works

* Fix compile

* Fix linter

* Fix test

* Fix test

* Debug

* Propagate tracing

* Trace default receiver

* More tracing

* Correct unfurl

* Remove tracing

* Try to retain trie for debugging
2020-06-02 08:46:21 +01:00
ledgerwatch
7ab10c85af
Separate Collector pattern out of ETL, hash collector for rebuilding Intermediate Hashes (#597)
* Introduce hashCollector

* Add HashCollector to SubTrieLoader

* Fix linter

* Reset hashed state

* Not to regenerate the hashed state

* Not to delete state

* Fix linter

* Print expected hash in the beginning

* Simplify

* Remove initialTrie

* Use etl to buffer intermediate hashes

* Copy values, not insert empty key

* Compress instead of decompress

* Enhance file buffer logging, fix linter

* Fix compile errors

* Fix log

* Fix logging

* Exclude zero key again

* Add rewind

* Restrict timestamps

* Fix

* Fix formatting

* Incorporate separation

* Extract identityLoadFunction

* Fix formatting
2020-05-31 13:23:34 +01:00
ledgerwatch
04b0dafb3d
Fix TestIndexGenerator_GenerateIndex_SimpleCase (out of file handles) (#596)
* Fix test

* Fix formatting
2020-05-31 09:12:10 +01:00
b00ris
266f9a5208
Plain state index (#595)
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
2020-05-31 07:57:47 +01:00
Evgeny Danilenko
32ec443572
Restore graceful shutdown (#591)
* initial

* gracefull shutdown for staged

* more wg fixes

* fmt

* linters

* remove generalization

* linters

* linters

* fix

* fix

* fmt

* quit into etl

* after CR

* after CR
2020-05-30 16:44:54 +03:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. (#592) 2020-05-30 10:00:35 +03:00
Alex Sharov
daa359c363
Mgr schedule iterator (#566)
* db based version of PrefixByCumulativeWitnessSize

* db based version of PrefixByCumulativeWitnessSize

* retain all in Trie by default

* fix WitnessLen logic in calcTrie roots

* Rename IntermediateTrieWitnessLenBucket to IntermediateWitnessLenBucket

* handle corner cases in WL

* Use correct incarnation for IH bucket

* use name WitnessSize

* save progress towards db-only witness estimation

* results from trie and from db are still different

* less recursion

* correct incarnation in CumulativeSearch

* reuse results from previous Tick, separate concepts of parent and startKey

* experiment: if not including trie structure to WitnessSize will reduce cumulative error

* tool to generate all IH and tool to calculate assessment of cumulative error

* tool to generate all IH

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* better stats

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* Calculate totalWitnessSize based on DB data - then schedule will not overrun state during MGR cycle

* calculate ticks size distribution

* estimate cumulative error

* fix linter

* resetIH from scratch if needed

* cleanup

* fix test

* fix test
2020-05-28 12:33:05 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
Andrew Ashikhmin
5bbdb6a4b9
Use uint256 for account balance (#580)
* Use uint256 for account balance

* linter
2020-05-26 17:53:50 +01:00
ledgerwatch
810cf13bf8
Explicit incarnation map in the database for deleted contracts (#578)
* Explicit map of incarnations

* Fix linter
2020-05-26 13:27:21 +01:00
Andrew Ashikhmin
b16e560529
Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore (#575)
* Produce less garbage in GetState

* Still playing with mem allocation in GetCommittedState

* Pass key by pointer in GetState as well

* linter

* Avoid a memory allocation in opSload

* Use uint256.Int rather than common.Hash for storage values to reduce memory allocation in opSload & opSstore

* linter

* linters

* small clean up
2020-05-25 12:12:25 +01:00
Igor Mandrigin
db746bba7a
"Unwind" for the execution phase when plain state is selected (#571)
* "Unwind" for the execution phase when plain state is selected

* test stub (fails)

* tests (one with incarnations fails)

* test fixups

* fix tests: cleanup contract code bucket
2020-05-25 12:11:56 +01:00