Commit Graph

55 Commits

Author SHA1 Message Date
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
Alex Sharov
331dcd45eb
Store receipts separately - one record per tx (#1271)
* squash

* add --database flag to integration

* clean

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* split to 2 buckets

* save progress

* save progress

* improve test

* improve test

* save progress

* change app logic

* change app logic

* return err from rawdb package

* don't clean automatically

* don't clean automatically

* clean

* clean

* clean

* don't rely on `make clean`

* improve cbor code

* clean

* clean

* clean

* fix tests

* rebase master

* stop on error: headers stage

* make TxDb walk and multiwalk safe

* Fix panics

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-25 08:38:55 +00:00
Alex Sharov
c2f41d7723
stop on error: headers stage (#1289) 2020-10-24 18:05:12 +01:00
Alex Sharov
77a4b3305d
rawdb to return errors: part 1 (#1286)
* ReadBlockByNumber and ReadChainConfig return error

* Let's make rawdb package to return error. But by small steps.
High-level methods need to move first:

ReadBlockByNumber
ReadChainConfig
ReadBlockByHash
ReadHeaderByHash
WriteChainConfig
WriteDatabaseVersion
WriteBlock
DeleteTd
WriteTd
ReadTd
2020-10-24 07:57:09 +01:00
Alex Sharov
88a099702b
rawdb.ReadCanonicalHash to return err (#1204)
* read_canonical_hash_return_err

* hack hugeFreelist

* fix tests
2020-10-10 07:05:56 +01:00
Péter Szilágyi
bfac425c87 core: define and test chain rewind corner cases (#21409)
* core: define and test chain reparation cornercases

* core: write up a variety of set-head tests

* core, eth: unify chain rollbacks, handle all the cases

* core: make linter smile

* core: remove commented out legacy code

* core, eth/downloader: fix review comments

* core: revert a removed recovery mechanism
# Conflicts:
#	core/blockchain.go
#	core/blockchain_test.go
#	core/headerchain.go
#	core/rawdb/database.go
#	core/rawdb/freezer.go
#	core/rawdb/schema.go
#	eth/downloader/downloader.go
#	eth/sync.go
#	trie/sync.go
2020-08-29 12:29:00 +02:00
Alex Sharov
f037aed6e5
wait for verify_headers method (#793) 2020-07-27 19:26:24 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
ledgerwatch
3359ba7a04
Make cmd/tester initiate staged sync, remove some pools and caches (#746)
* no pools

* Remove excessive logging

* Revert

* Initial commit

* Cleanup

* Fix some compile errors

* Fix core tests

* Fix more tests

* Fix eth/downloader

* go mod tidy

* Fix core/vm

* Fix formatting

* Fix formatting

* Fix lint

* Fix lint

* Switch tests to StagedSync

* Cleanup

* Reuse cache in stage4

* Fix

* Fix formatting

* Try to fix test
2020-07-15 07:15:48 +01:00
Igor Mandrigin
0dae3ade0d post-rebase fixups 2020-05-20 15:26:22 +03:00
icodezjb
83857d4f4a core: mirror full node reorg logic in light client too (#20931)
* core: fix the condition of reorg

* core: fix nitpick to only retrieve head once

* core: don't reorg if received chain is longer at same diff

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
# Conflicts:
#	core/headerchain.go
2020-05-20 15:26:22 +03:00
ledgerwatch
f47d661c34
Proper unwinding in the staged sync (#521)
* introduce fork

* Fix unwind test

* Unwind now really works in the test

* Fix linter

* Fix linter

* Fix linter

* Not to handle block and block hash announced in staged sync
2020-05-06 18:42:10 +01:00
ledgerwatch
c0b87340a2
Introduce staged sync (#482)
* Introducing staged sync

* Fix linter

* Fix linter

* Sync progress in the DB

* Fix linter

* Fix linter

* Not to wake body and receipt downloaders in the header stage

* Next stage

* Fix tests

* Fix linter

* More on block bodies downloading

* More on downloading bodes

* Remove debug

* Fix test compilation

* Fix tests

* Fix linter

* Fix linter

* Turn off the miner during staged sync

* More fixes
2020-04-24 17:57:43 +01:00
Evgeny Danilenko
17a4a56634
Refactor mining. Remove a few mining goroutines (#338)
* add context

* extract chain events

* run commit in goroutines

* mine only on canonical

* typo

* linters

* fmt

* mark unused methods

* restore stress test

* test single miner

* remove unsafe Trie storage

* remove locks from miner

* restore interrupt

* remove result goroutine

* remove unconfirmedBlocks

* cherry-pick 04a1d475ff1a36ad8f92fec80385df18c52bdc1f

* extract uncles

* one miner succeeded

* restore context cancel

* cleanup

* skip an unstable test

* remove pending state

* use context instead of interrupt func

* calculate sealHash only once

* comment out unstable test

* after merge

* fix after merge

Co-authored-by: ledgerwatch <akhounov@gmail.com>
2020-02-03 15:02:26 +03:00
gary rong
f0f1b5237c core, light: write chain data in atomic way (#20287)
* core: write chain data in atomic way

* core, light: address comments

* core, light: fix linter

* core, light: address comments
2020-01-30 13:36:30 +02:00
alex.sharov
76dc061d8b FakePeer - reduce interface size 2019-12-13 09:56:30 +06:00
Alexey Akhunov
fe01bccbb8 Apply Turbo-Geth modifications to go-ethereum codebase 2019-11-01 21:52:03 +01:00
Ryan Schneider
ad03d9801c internal/ethapi: support block number or hash on state-related methods (#19491)
This change adds support for EIP-1898.
2019-09-26 10:47:31 +02:00
Felföldi Zsolt
0ac9bbba6c les: multiple server bugfixes (#20079)
* les: detailed relative cost metrics

* les: filter txpool relative request statistic

* les: initialize price factors

* les: increased connected bias to lower churn rate

* les: fixed clientPool.setLimits

* core: do not use mutex in GetAncestor

* les: bump factor db version again

* les: add metrics

* les, light: minor fixes
2019-09-17 16:28:41 +03:00
Péter Szilágyi
b02958b9c5
core, ethdb, metrics, p2p: expose various counter metrics for grafana 2019-06-11 09:49:13 +03:00
gary rong
37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong
80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
Martin Holst Swende
0b4fe8d192 all: simplify timestamps to uint64 (#19372)
* all: simplify timestamps to uint64

* tests: update definitions

* clef, faucet, mobile: leftover uint64 fixups

* ethash: fix tests

* graphql: update schema for timestamp

* ethash: remove unused variable
2019-04-02 23:28:48 +03:00
Martin Holst Swende
59e1953246 core, ethdb, trie: mode dirty data to clean cache on flush (#19307)
This PR is a more advanced form of the dirty-to-clean cacher (#18995),
where we reuse previous database write batches as datasets to uncache,
saving a dirty-trie-iteration and a dirty-trie-rlp-reencoding per block.
2019-03-26 15:48:31 +01:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
b00ris
769657060e les: implement ultralight client (#16904)
For more information about this light client mode, read
https://hackmd.io/s/HJy7jjZpm
2019-01-24 12:18:26 +01:00
Péter Szilágyi
0f2ba07c41
common, core, light: add block age into info logs 2018-09-20 12:56:35 +03:00
Ha ĐANG
2cffd4ff3c core: fix some small typos on comment code (#17278) 2018-07-30 14:10:48 +03:00
gary rong
a4a2343cdc ethdb, core: implement delete for db batch (#17101) 2018-07-02 11:16:30 +03:00
Felföldi Zsolt
049f5b3572 core, eth, les: more efficient hash-based header chain retrieval (#16946) 2018-06-12 16:52:54 +03:00
Péter Szilágyi
6cf0ab38bd
core/rawdb: separate raw database access to own package (#16666) 2018-05-07 14:35:06 +03:00
Péter Szilágyi
1fae50a199
core: minor evm polishes and optimizations 2018-03-26 12:28:46 +03:00
Martin Holst Swende
f83237573f core: make current*Block atomic, and accessor functions mutex-free (#16171)
* core: make current*Block atomic, and accessor functions mutex-free

* core: fix review concerns

* core: fix error in atomic assignment

* core/light: implement atomic getter/setter for headerchain
2018-02-26 11:53:10 +02:00
Felix Lange
10181b57a9 core, eth/downloader: commit block data using batches (#15115)
* ethdb: add Putter interface and Has method

* ethdb: improve docs and add IdealBatchSize

* ethdb: remove memory batch lock

Batches are not safe for concurrent use.

* core: use ethdb.Putter for Write* functions

This covers the easy cases.

* core/state: simplify StateSync

* trie: optimize local node check

* ethdb: add ValueSize to Batch

* core: optimize HasHeader check

This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.

* core: write fast sync block data in batches

Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.

* eth/downloader: commit larger state batches

Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.

* core: optimize HasBlock check

This avoids a random database read to get the number.

* core: use numberCache in HasHeader

numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.

* core: write imported block data using a batch

Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.

This commit also removes posting of logs when a forked block is imported.

* core: fix DB write error handling

* ethdb: use RLock for Has

* core: fix HasBlock comment
2017-09-09 19:03:07 +03:00
changhong
17f0b11942
core: typos and comments improve
1. fix typos
2. methods recevier of struct should be same
3. comments improve

(cherry picked from commit 1ba979539582a00b7fd1a7c8a37a6852e59eac0d)
2017-05-25 17:14:33 +03:00
Péter Szilágyi
a7b9e484d0 consensus, core, ethstats: use engine specific block beneficiary (#14318)
* consensus, core, ethstats: use engine specific block beneficiary

* core, eth, les, miner: use explicit beneficiary during mining
2017-04-12 16:38:31 +03:00
Péter Szilágyi
158d603528
consensus, core: drop all the legacy custom core error types 2017-04-06 17:34:19 +03:00
Péter Szilágyi
09777952ee core, consensus: pluggable consensus engines (#3817)
This commit adds pluggable consensus engines to go-ethereum. In short, it
introduces a generic consensus interface, and refactors the entire codebase to
use this interface.
2017-04-05 00:16:29 +02:00
Felix Lange
37dd9086ec core: refactor genesis handling
This commit solves several issues concerning the genesis block:

* Genesis/ChainConfig loading was handled by cmd/geth code. This left
  library users in the cold. They could specify a JSON-encoded
  string and overwrite the config, but didn't get any of the additional
  checks performed by geth.
* Decoding and writing of genesis JSON was conflated in
  WriteGenesisBlock. This made it a lot harder to embed the genesis
  block into the forthcoming config file loader. This commit changes
  things so there is a single Genesis type that represents genesis
  blocks. All uses of Write*Genesis* are changed to use the new type
  instead.
* If the chain config supplied by the user was incompatible with the
  current chain (i.e. the chain had already advanced beyond a scheduled
  fork), it got overwritten. This is not an issue in practice because
  previous forks have always had the highest total difficulty. It might
  matter in the future though. The new code reverts the local chain to
  the point of the fork when upgrading configuration.

The change to genesis block data removes compression library
dependencies from package core.
2017-03-23 15:58:43 +01:00
Felföldi Zsolt
525116dbff les: implement request distributor, fix blocking issues (#3660)
* les: implement request distributor, fix blocking issues
* core: moved header validation before chain mutex lock
2017-03-22 20:44:22 +01:00
Péter Szilágyi
94c71c171f Merge pull request #3723 from karalabe/logger-updates-2
Logger updates
2017-02-28 16:55:37 +02:00
Felix Lange
5f7826270c all: unify big.Int zero checks, use common/math in more places (#3716)
* common/math: optimize PaddedBigBytes, use it more

name              old time/op    new time/op    delta
PaddedBigBytes-8    71.1ns ± 5%    46.1ns ± 1%  -35.15%  (p=0.000 n=20+19)

name              old alloc/op   new alloc/op   delta
PaddedBigBytes-8     48.0B ± 0%     32.0B ± 0%  -33.33%  (p=0.000 n=20+20)

* all: unify big.Int zero checks

Various checks were in use. This commit replaces them all with Int.Sign,
which is cheaper and less code.

eg templates:

    func before(x *big.Int) bool { return x.BitLen() == 0 }
    func after(x *big.Int) bool  { return x.Sign() == 0 }

    func before(x *big.Int) bool { return x.BitLen() > 0 }
    func after(x *big.Int) bool  { return x.Sign() != 0 }

    func before(x *big.Int) int { return x.Cmp(common.Big0) }
    func after(x *big.Int) int  { return x.Sign() }

* common/math, crypto/secp256k1: make ReadBits public in package math
2017-02-28 15:09:11 +01:00
Péter Szilágyi
e02883c0a2
core, log: track field length and pad to align 2017-02-28 15:36:51 +02:00
Péter Szilágyi
e588e0ca2b
all: next batch of log polishes to contextual versions 2017-02-28 15:03:20 +02:00
Péter Szilágyi
d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Felix Lange
21f1370d2a core: improve import log alignment 2017-01-10 23:14:08 +01:00
Péter Szilágyi
a59fcc33e6
core: import future blocks one-by-one, enfore chain ancestry 2016-12-13 16:19:45 +02:00
Jeffrey Wilcke
445feaeef5 core, core/state, trie: EIP158, reprice & skip empty account write
This commit implements EIP158 part 1, 2, 3 & 4

1. If an account is empty it's no longer written to the trie. An empty
  account is defined as (balance=0, nonce=0, storage=0, code=0).
2. Delete an empty account if it's touched
3. An empty account is redefined as either non-existent or empty.
4. Zero value calls and zero value suicides no longer consume the 25k
  reation costs.

params: moved core/config to params

Signed-off-by: Jeffrey Wilcke <jeffrey@ethereum.org>
2016-11-13 10:44:04 +01:00
Péter Szilágyi
64500ab0fa
common, core, eth/downloader: adjust import log formatting 2016-10-18 13:16:36 +03:00
Bas van Kervel
bb8059f6aa core: ensure the canonical block is written before the canonical hash is set 2016-08-16 15:21:22 +02:00