Commit Graph

65 Commits

Author SHA1 Message Date
Artem Vorotnikov
e67be2b1be
Remove unused code (#1604)
* Remove graphql, legacy options

* Remove CODEOWNERS

* Remove EWASM and custom EVM interpreter
2021-03-26 09:08:01 +07:00
Artem Vorotnikov
a6742a4cf8
Kill cmd/state/stateless (#1592)
* Move stuff around

* Remove

* tidy

* Fix for writeReceipt printing

* fix lint typo

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2021-03-23 16:33:58 +00:00
Alex Sharov
ba37706681
move remote eth backend from ethdb to core package (#1567) 2021-03-20 10:52:00 +07:00
Alex Sharov
98b36126b2
History bitmap 64 (#1374)
* squash

* squash

* squash

* squash

* squash

* squash

* subscription_doesnt_preserve_shutdown
2020-12-04 21:16:51 +00:00
Alex Sharov
068463dff4
Store transactions individually (#1358)
* Store transactions individually

* Store transactions individually

* save progress

* checkIndex

* merge
2020-11-22 21:25:26 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342)
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-11-16 12:08:28 +00:00
b00ris
6464da7670
Remote snapshot downloader (#1343)
* save state

* save state

* save state

* refactoring

* fix

* save state

* save state

* fmt

* fix lint

* restore torrents for external downloader

* fix lint

* download

* skip debug test

* debug

* remote debug

* small cli fixes

* skip debug test

* external snapshot predownloader

* get rid of remote downloader

* fix lint

* clean makefile

* fix lint

* fix lint

* cleanup

* fix ci

* fmt

* remove proto from interfaces

* Squashed 'interfaces/' content from commit acd02bb94

git-subtree-dir: interfaces
git-subtree-split: acd02bb94c5a421aa8f8d1fd76cd8aad668e9fcb
2020-11-13 16:16:47 +00:00
Horacio Mijail Antón Quiles
ed1819ec58
Opcode tracer (#1320)
* OpcodeTracer

* Cleanup, rename to opcode_tracer

* Bring back check_change_sets, remove repeated vars

* Cleanup, gofmt

* Linter, cleanup

* Rename "segments" to "basic blocks", "bblocks"

* gofmt

* Lintci madness
2020-10-30 19:44:00 +00:00
Alex Sharov
6f27e3043b
lmdb_readonly_dont_create_folder (#1293) 2020-10-25 08:39:09 +00:00
b00ris
e5f6412da2
Update snapshot hash (#1243)
* remove useless bucket

* update snapshot hashes

* remove log
2020-10-17 08:48:20 +01:00
ledgerwatch
cd706d5081
Download/seed block bodies and block headers via BitTorrent (#1218)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

* tidy mods

* Fix compile

* Fix lint

* Fix rpcdaemon running in the docker

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-13 13:56:16 +01:00
ledgerwatch
1af6cbe37b
Revert "Download/seed headers and block bodies via BitTorrent (#1195)" (#1202)
This reverts commit 94360212cb.
2020-10-07 21:49:44 +01:00
ledgerwatch
94360212cb
Download/seed headers and block bodies via BitTorrent (#1195)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-10-07 13:15:49 +01:00
ledgerwatch
bdb0155856
Revert "Add sync by bittorrent snapshots (#1160)" (#1194)
This reverts commit 4ebade3583.
2020-10-06 22:31:57 +01:00
b00ris
4ebade3583
Add sync by bittorrent snapshots (#1160)
* save state

* torrent experiments

* torrent experiment passed

* fixes after merge

* snapshot headers processing passed

* save state

* save state

* download headers works after snapshot processing

* save state

* save state

* save state

* save state

* add lazy load tx to snapshots, increase number of trackers

* save state

* speedup getting info

* change logging

* move to turbo package

* save state

* save state

* save state

* cleanup

* save state

* add test test

* save state

* lmdb debugging

* fix readonly mode

* save state

* fix build

* sync works

* save state

* save state

* save state

* allow cmd stages stageSenders use snapshots

* debugging failed hashing

* remove experimental tests

* remove torrent experimental tests

* fix lint

* extract snapshot wrapper

* metainfo checker

* add remote seeder

* add logs

* update gomod

* remove useless code

* fix lint&remove useless code

* extract verify snapshot to separated command

* skip debug test

* fix test

* change type of seedSnapshot flag

* add eth logger to torrent lib

* skip debug test

* add Close method

* review fixes

* fix lint

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-10-06 20:24:48 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) (#1113)
* added tls auth

* added client side

* put --tls

* fixed flag

* Add key/cert generation instructions, turn off Common Name verification

* Add CLI arguments and Warning

* Lint

* Update the doc about Internal IP

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon (#1089)
* added eth_getStorageAt

* used uint32

* now its 256

* incarnation

* added TLS handshake

* code minified

* lint

* minified client

* Update flags.go

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt (#1085)
* kill bolt

* kill bolt

* fix test

* clean
2020-09-09 11:09:55 +01:00
Alex Sharov
f827f86c11
Rpcdaemon as lib 2 (#943) 2020-08-20 10:52:27 +07:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" (#932) 2020-08-17 07:47:06 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object (#914) 2020-08-14 13:41:18 +07:00
ledgerwatch
6f88e4058f
Fix (#908) 2020-08-11 20:48:37 +01:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb (#788) 2020-07-27 19:15:48 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) (#758)
* First push

* Remove batch from Blockchain

* Fix compile error in cmd/integration

* Fix compile errors in cmd/state

* Fix compile errors

* Save changes

* Fix core tests (skip tx tests)

* More test fixes

* Fix compile error

* Fix formatting

* Fix formatting

* Fix lint

* Fixes

* Skip some miner tests

* Fix tests

* Cleanup
2020-07-19 09:11:53 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670)
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
Alex Sharov
15096f273d
Remove ctx from Open. Stop goroutines on Close. (#650)
* remove ctx from MustOpen

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from Open. Stop goroutines on Close.

* remove ctx from remote open (we have DialTimeout field to manage connection timeouts)

* enable RawReads and add native implementation of Get/Has methods
2020-06-12 10:31:21 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646)
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
2020-06-10 23:07:14 +03:00
ledgerwatch
462edc2345
CheckChangeSets to work with plain state and changeSets, optionally generate receipts (#639)
* Start from 9m7

* Regenerate IH + receipts

* Only stats for iH bucket

* Persist receipts

* Go all in

* Start from block 10m

* Convert DbState to use plain state

* Fix findHistory

* Hard-code export

* More fixes

* Fix test

* Fix formatting

* Introduce PlainDbState

* Actually return PlainDbState

* Fix formatting

* Fix name style lint

* Fix linters

* Fix history_test

* Fix blockchain_test

* Fix compile error

* Bucket stats from all buckets
2020-06-09 06:52:38 +01:00
Evgeny Danilenko
41ee460a27
optimize codeBitmap (#626)
* optimize codeBitmap

* add lru

* remove custom lru

* jumpDests as explocit parameter

* lint

* lint

* linters

* default jumpDestsCache

* remove common.rand
2020-06-06 21:49:06 +01:00
b00ris
266f9a5208
Plain state index (#595)
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
2020-05-31 07:57:47 +01:00
ledgerwatch
dba3363b52
Fix RestAPI for tx_retrace and make it work for local boltdb (#584)
* Not hash, keep the files

* Calculate savings

* Fix

* Fix

* Fix

* Fix

* RestAPI to support local boltdb

* Not error on read-only db

* Changes so far

* Continue

* More

* Roll back a bit

* Restore newline

* something compiles

* Fix restapi

* Fix block number

* Fix reads

* Use plain writer

* Maps for storage reads and writes

* Clean up coersions

* Fix accounts/abi/bind

* Fix tests

* More fixes

* more fixes

* More fixes

* Fixes

* Fixed core/state

* Fixed eth tests

* Move code, fix linter

* Fix test

* Fix linter

* Fix linter

* Fix linter, badger_db to support AbstractKV

* Increase IdealBatchSize for badger

* Fix linter

* Fix linter
2020-05-27 17:24:34 +01:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices (#569)
* save state

* add current index feature

* fix test

* remove logs

* Only execute 1000 blocks

* Reset history index

* Correct action

* Increase batch size

* Increase chunk size, print memory stats

* Fix linter

* Remove unused from

* Split into 2 staged

* Use storage history gen

* remove log

* Not to run tx_cacher in staged mode

* Not to recover during stage 2

* Not to recover during stage 2

* Remove counter

Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
b00ris
cc137af1ce
[WIP] New storage changeset encoding (#493)
* new storage changeset encoding

* fix lint
2020-04-28 08:41:55 +01:00
b00ris
a9f71298aa
Index chunking (#473)
* index generation

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* fix

* save state

* fix
2020-04-20 11:35:33 +01:00
Igor Mandrigin
b8282a0336
Implement a little tool to validate the export file against chaindata (#452)
* verifyExportFile command

* linters
2020-04-14 13:51:20 +01:00
Igor Mandrigin
73b897b2b5
Don't use chaindata where we don't need it. (#449)
* verify_snapshot stuff

* remove unused parameter

* remove unused argument

* make blocksource required
2020-04-14 13:49:25 +01:00
Alex Sharov
9e65f886ea
Stateless prototype - enable metrics collection (#442) 2020-04-12 19:36:14 +01:00
Igor Mandrigin
fb08514b9b
Use geth export generated files in the stateless prototype (#447) 2020-04-12 18:41:06 +03:00
Igor Mandrigin
64a335bb32
Code eviction with code nodes support (#416) 2020-04-08 08:00:31 +03:00
Alex Sharov
9324d83cb2
ethdb.KV interface - pase 2 (#420)
* - handle cursor.Prefix on server
- move state reports to KV interface

* add CmdCursorSeekKey

* tests for abstract_kv

* avoid reading configs of databases

* avoid reading configs of databases

* make linter happy

* make linter happy

* cleanup

* port badger features from original implementation

* try to fix test

* try to fix test

* .Close() don't return error anymore - defer friendly

* try to enable badger now

* try to enable badger now

* badger can't run on CI yet

* badger can't run on CI yet

* re-run ci

* skip ctx cancelation for badger
2020-04-04 08:18:10 +01:00
Alex Sharov
c8cbdb8c47
improve changeSetChecks to accept genesis.json (#415) 2020-03-30 19:56:53 +01:00
Andrew Ashikhmin
82b80c2ebe
Thin history: Fix an error in mutation.getNoLock (#409)
* CheckChangeSets and thin history

* small code de-duplication

* small code clean-up

* Fix an error in mutation.getNoLock

* CheckChangeSets: truly make historyfile = chaindata by default
2020-03-26 21:52:22 +00:00
ledgerwatch
6dcb8ade46
Convenience changes to checkChangeSets and stateless (#406)
* Allow separation of files

* Try next block

* Enable history

* use logging

* No check option

* Option to write history
2020-03-25 20:18:46 +00:00
Alex Sharov
d2286bff1c
Naive implementation of MGR P2P SubProtocol (#403)
* use NoValues cursor where possible

* add ctx

* fix broken logs

* rebase master

* rebase master

* simplify generators

* hack to measure space distribution

* naive epoch and chunking implementation

* make stateless loop cancelable

* make stateless loop cancelable

* remove one rlp layer

* eth64 protocol support - add forkId to status message
2020-03-25 15:40:30 +00:00
Andrew Ashikhmin
aeed1657c7
Issue #340: Re-execute all historical transaction in read-only mode and check ChangeSets (#388)
* Clean up code duplication between IntraBlockState's FinalizeTx & CommitBlock

* checkChangeSets command

* linter

* First attempt at checking account changes

* Reuse runBlock in CheckChangeSets

* linter

* linter

* Optionally include no-changes in the ChangeSets

* linter

* Detect storage changes for account change sets

* Fix post-merge compilation errors

* Use database format compatible with !debug.IsThinHistory()

* PrintChangedAccounts in ChangeSetWriter

* Avoid out-of-bounds access

* Storage changes

* hack FirstContractIncarnation

* Call ChangeSetWriter only once per block
2020-03-11 16:54:09 +01:00
Alex Sharov
c5ffc971c5
[WIP] Badger v2 (#378)
* badger v2 investigation

* buf pool - use native New method and avoid double checks

* db.Open prototype

* db.Tx/Bucket/Cursor prototypes

* Chained config

* Item concept added

* save changes to test on master

* make hack resumable

* Design document v0

* Cursor concept

* less brackets syntax of cursor builder

* benchmarks

* cleanup fs

* test for context cancelations

* test for context cancelations

* test for cursor.Prefix option

* add ForEachKey method

* add ForEachKey method

* add naming explanation

* experiment of non-pointers cursor/bucket

* .Bucket() and .Cursor() doesn't returns error

* .Bucket() and .Cursor() doesn't returns error

* .Bucket() and .Cursor() doesn't returns error

* remove CursorOpts concept

* more test-cases

* simplify open api

* Tx, Bucket, Cursor - now are interfaces

* Tx, Bucket, Cursor - now are interfaces

* switch to interfaces

* rebase master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-11 11:02:37 +00:00
Igor Mandrigin
1115b23832
Stateless Resolver: store and use witnesses for resolved subtries (#326) 2020-01-24 11:58:01 +01:00
ledgerwatch
4f5c858f00
Collect data about Keccak256 invocations for evaluation of STARK proof sizes and performance (#315)
* Gathering start data on witnesses

* Fix number parsing

* Proper filename, actually outputting

* Correct block

* Visualise states

* Code and value fix

* Visualise code hash and storage hash

* Current block number

* Print bucket

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix typo

* Fixes after merging master

* Fix lint

Co-authored-by: b00ris <b00ris@mail.ru>
2020-01-15 17:33:36 +00:00
Alex Sharov
fc7291ec34
Use local db in analytics (#308)
* Use local db in analytics (#308)
* pool of encoders
* incr and decr funcs
2020-01-15 19:47:13 +07:00