Commit Graph

11810 Commits

Author SHA1 Message Date
Igor Mandrigin
397a0fbecc
Nightly tests (#444)
* add a simple staging test

* add buildkite stuff

* buildkite setup like in the [example](https://github.com/buildkite/golang-example)

* yamllint

* add running geth for 1m as a buildkite command

* fixups post go mod

* workaround a stupid go issue

* add 1 hour test

* fix bash maths

* run the job for 5 hours
2020-04-12 19:37:15 +01:00
ledgerwatch
fbe7a2cb06
THIN_HISTORY - Reinstate incarnation encoding in ChangeSets, but remove from index keys (#443)
* Cutting incarnation out of history and changeset keys

* Rollback the remove of incarnation only in changesets

* Fix linter

* Remove debug printing

* Reinstate encoding of incarnations in the ChangeSets

* Fix linter

* Fix linter

* Fix DefaultIncarnation

* Fixes

* Fix lint

* Fix to history writing

* Fix linter
2020-04-12 19:36:27 +01:00
Alex Sharov
9e65f886ea
Stateless prototype - enable metrics collection (#442) 2020-04-12 19:36:14 +01:00
Igor Mandrigin
fb08514b9b
Use geth export generated files in the stateless prototype (#447) 2020-04-12 18:41:06 +03:00
Igor Mandrigin
ef506520d3
Fix geth export and other auxiliary commands. (#446)
They tries to initialized the pruner but the configuration was incorrect.
This commit disabled the pruner completely.
2020-04-12 16:43:18 +03:00
ledgerwatch
2d7832c62e
THIN_HISTORY - implement WalkAsOf and fix corresponding tests (#441)
* Implement WalkAsOf for THIN_HISTORY

* Fix compile

* Fix linter

* Fix linter

* Fix linter

* Fix linter

* Fix linter

* Fix broken tests

* Fix comment
2020-04-11 20:25:19 +01:00
Alex Sharov
535d73be5f
Run CI on bolt with prefix_compression_without_allocation (fixes) (#380) 2020-04-11 20:28:15 +07:00
Alex Sharov
346a79444f
Add support of geth on hostmachine (#437) 2020-04-11 08:22:23 +01:00
Giulio rebuffo
56700620b2
Tx retrace extension for Rest API (#436)
* added extension

* added api

* lint

* more lint

* better response

* added README.md

* README Fixed

* README updated

* fixed lint
2020-04-10 19:47:45 +01:00
Alex Sharov
255a1bcf2f
Add more tests for abstract Kv (#438) 2020-04-10 17:55:31 +07:00
ledgerwatch
ad64ae2085
[THIN_HISTORY] Decouple changesets and history from ethdb.Database (mutation/bolt_db/badger_db) (#432)
* Prepare for refactoring

* Decoupling

* Further cleanup of mutation

* Further cleanup of mutation

* Remove some tests (temporarily)

* Fix linter

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Recover mutation_test cases

* Fix lint

* Fix WalkAsOf test

* Fix lint

* Fix TODO, linter

* Fix lint

* Fix lint

* Fix tests for THIN_HISTORY

* Fix lint

* Fix lint

* Reduce visibility of DecodeStorage

* Revert "Reduce visibility of DecodeStorage"

This reverts commit 2e11e16ea095b75a62cdddb77b9477eabc259e67.
2020-04-09 18:23:29 +01:00
Igor Mandrigin
0589f63d99
Fix a deadlock around block 2.330.000... (#435) 2020-04-08 16:53:22 +01:00
Igor Mandrigin
57ea93f59c
Use the binary witness format to serialize tries. (#434)
* Use the binary witness format to serialize tries.

* test fixups

* support witness diffing of different sizes

* remove hardcoding in the test
2020-04-08 13:59:30 +01:00
Igor Mandrigin
d68cbadb94
Code size in account nodes (#433)
* first stuff + a test

* implement code nodes

* linters 🧐
2020-04-08 12:47:18 +01:00
Alex Sharov
c68f00045a
Revert "remove tombstones" (#430)
* Revert "remove tombstones (#426)"

This reverts commit aa6bab40e8.

* tombstones don't hide storage or account anymore

* auto-format code by prettier (similar to gofmt)

* wow, it works.....

* small simplification, but need make it more clear

* rebase to master

* rebase to master

* rebase to master

* re-run ci

* clean test files
2020-04-08 12:45:51 +01:00
Igor Mandrigin
64a335bb32
Code eviction with code nodes support (#416) 2020-04-08 08:00:31 +03:00
Alex Sharov
64de47642e
Problem: forkFilter variable are catched by goroutine when peer connected (not when status msg received) - it means to change it tester must reconnect. (#429) 2020-04-06 08:53:52 +01:00
ledgerwatch
e5fe539216
Computation of index size + binary search in the indices (#428)
* Calculate index sizes

* Better counting

* Try binary search

* More compact code

* Handle the case of empty index
2020-04-05 18:27:25 +01:00
Alex Sharov
66ce27174d
don't rebuild - because it evicting IH (#427) 2020-04-05 07:33:06 +01:00
Alex Sharov
aa6bab40e8
remove tombstones (#426) 2020-04-05 07:32:28 +01:00
ledgerwatch
19cd0370d3
Fixes in the stateless prototype after error on the block 2675337 (state clearing) (#425)
* Stop

* return

* Bugfixes

* Cleanup
2020-04-04 08:18:22 +01:00
Alex Sharov
9324d83cb2
ethdb.KV interface - pase 2 (#420)
* - handle cursor.Prefix on server
- move state reports to KV interface

* add CmdCursorSeekKey

* tests for abstract_kv

* avoid reading configs of databases

* avoid reading configs of databases

* make linter happy

* make linter happy

* cleanup

* port badger features from original implementation

* try to fix test

* try to fix test

* .Close() don't return error anymore - defer friendly

* try to enable badger now

* try to enable badger now

* badger can't run on CI yet

* badger can't run on CI yet

* re-run ci

* skip ctx cancelation for badger
2020-04-04 08:18:10 +01:00
ledgerwatch
9bb7f3056d
Fix hanging resolver (#424)
* Debugging the cached resolver

* make logic of comparison more explicit

* fix stuck on block 696817

* Remove tracing

* Fix CI

* Search for root hash

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-04-01 17:04:41 +01:00
Giulio rebuffo
c5e503634a
tx lookup File Write (hack.go) (#421)
* tx lookups via file insertions

* cleanup

* added heap + buffer to genTx2

* Validation Process

* fix

* removed profile

* remove batch.Commit() in validation

* separated validation
2020-04-01 17:03:51 +01:00
Alex Sharov
508d497644
commit simulator_genesis.json file generated by go run ./cmd/tester genesis > simulator_genesis.json (#419) 2020-03-31 07:52:13 +01:00
Alex Sharov
20df391732
don't pass common ctx to commit block, to avoid cancelation in the middle of commit (#418) 2020-03-31 09:08:25 +07:00
Andrew Ashikhmin
ac87ee7ed7
Check account in StorageChangeSetBytes.Find (#417)
* Check account in StorageChangeSetBytes.Find

* Remove leftover debug printing
2020-03-30 19:57:53 +01:00
Alex Sharov
c8cbdb8c47
improve changeSetChecks to accept genesis.json (#415) 2020-03-30 19:56:53 +01:00
Alex Sharov
286ef9ea1f
enable intermediate trie hashes by default (#414) 2020-03-30 17:01:24 +01:00
Alex Sharov
c2e71ffc03
[wip] [Tester] - different values in fork and non-fork (#413)
* save progress

* increment in fork only

* skip some steps in fork

* tester switch database to ./simulator file

* correct signer
2020-03-30 13:31:49 +01:00
ledgerwatch
3b202988ef
Fix concurrent map writes from tx_pool (#412)
* Lock when reading code nodes

* Fix double locking
2020-03-28 13:56:24 +00:00
Andrew Ashikhmin
435d56e2fc
Timestamps must be sorted prior to appending into thin history index (#411)
* Timestamps must be sorted prior to appending into thin history index

* Skip unstable test
2020-03-27 16:46:34 +01:00
Alex Sharov
a364cfded6
[Tester] fix fork generator (#410)
fix coinbase
use same genBlock function
switch inside genBlock - now works on blockNumber instead of nonce
fill file in own function
add verbosity cli flag
2020-03-27 20:12:26 +07:00
Andrew Ashikhmin
82b80c2ebe
Thin history: Fix an error in mutation.getNoLock (#409)
* CheckChangeSets and thin history

* small code de-duplication

* small code clean-up

* Fix an error in mutation.getNoLock

* CheckChangeSets: truly make historyfile = chaindata by default
2020-03-26 21:52:22 +00:00
Alex Sharov
8d8e41cd85
Call bufio.Flash() in tester (#407)
* call bufio.Flush

* rerun ci

* handle errors better

* correct sleep
2020-03-26 21:52:05 +00:00
Igor Mandrigin
5ff3948129
Remove error param from NewTrieDbState (#408) 2020-03-26 16:21:15 +03:00
ledgerwatch
6dcb8ade46
Convenience changes to checkChangeSets and stateless (#406)
* Allow separation of files

* Try next block

* Enable history

* use logging

* No check option

* Option to write history
2020-03-25 20:18:46 +00:00
Alex Sharov
d2286bff1c
Naive implementation of MGR P2P SubProtocol (#403)
* use NoValues cursor where possible

* add ctx

* fix broken logs

* rebase master

* rebase master

* simplify generators

* hack to measure space distribution

* naive epoch and chunking implementation

* make stateless loop cancelable

* make stateless loop cancelable

* remove one rlp layer

* eth64 protocol support - add forkId to status message
2020-03-25 15:40:30 +00:00
Alex Sharov
8d7e1e9374
Optimize resolver cached (#404)
* remove allocations related to "remove incarnation" actions

* invalidate startKeyNoInc when startKey changed

* dbutils.RemoveIncarnationFromKey - doesn't do allocation
2020-03-25 09:43:55 +00:00
Alex Sharov
b0f962e3ba
auto-format code by prettier (similar to gofmt) (#405) 2020-03-25 12:45:21 +07:00
Alex Sharov
b490192e67
Use KV Abstraction in RestAPI (#400)
* Introduce NoValuesCursor. From() method is useless because can be replaced by Seek().`
* implement NoValueCursor interface
* use abstract db in restapi
* cleanup .md
2020-03-24 09:12:55 +07:00
Alex Sharov
1fb8749638
Make AccountCreation do determined amount of operations and Move .Put out of .View (#402)
* move .Put out of .View and remove deletion of all tombstones when delete acc

* move .Put out of .View and remove deletion of all tombstones when delete acc
2020-03-23 22:14:05 +00:00
Igor Mandrigin
4f4b395aa4
Introduce code node & get rid of code map (#398)
* introduce code node

* replace codeMap with code touches

* fix a comment

* fixups to tests

* fix compile error

* fix getnodedata tests

* add tests and test stubs

* add more test stubs

* add test method bodies

* add and fix more tests on trie for new codenode

* add test change code between blocks

* fix crash in stateless

* remove unneded files

* remove comment

* fix deleted account code

* fix resolve set builder for code nodes
2020-03-23 22:10:36 +00:00
ledgerwatch
b07ae7667d
Optimise tombstones (#399)
* Optimise tombstones

* Replace Update with View

* remove 1 allocation

* Copy keys

* Fix 1 test (perhaps)

* Fix test

Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-03-22 17:37:55 +00:00
Alex Sharov
965d1101f4
Intermediate hash phase 4 (#396)
* another way to check if account has storage

* cleanup

* v0 of walk by db version

* save progress, to switch to another task. Put tombstone is still not correct.

* place tombstone only if exists something to hide

* db-based implementation

* db-based implementation

* db-based implementation

* fix prop check

* improve prop check logic

* Need custom logic to skip subtree for account and storage buckets because storage bucket has incarnation in key

* rebase to master

* remove all tombstones when account deleted

* remove all tombstones when account deleted

* added db integrity check

* don't rely on account.Root because it valid only about last incarnation

* remove all tombstones when account deleted

* dial with incarnation in MultiWalk2

* dial with incarnation in MultiWalk2

* when fixedbytes=40 resolver did compare cacheKey with storageKey without removing incarnation

* rebase to master

* rebase to master

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 11:30:14 +00:00
Andrew Ashikhmin
abadbdfb80
[Issue 340] Preserve the original when a contract is self-destructed and then its address is touched in the same block (e.g. # 156634) (#397)
* root was unused in BlockChain.StateAt

* TestDoubleAccountRemoval

* Preserve the original when a contract is self-destructed and then its address is touched in the same block (e.g. #
156634)
2020-03-20 10:08:13 +00:00
Alex Sharov
2d7dd025cf
Update AbstractKV.md 2020-03-20 17:08:00 +07:00
Alex Sharov
84ae2eb1f8
Intermediate Hashes UI (#395)
* implement NoValueCursor interface

* cleanup

* fix tests

* add more stats data to ui

* can't display error

* re-open DB low-level net interface when changing db

* re-open DB low-level net interface when changing db

* fix problem with displaying errors

* run ci

* improve prop check logic

* storage tombstones integrity checks UI

* storage page

* make DB configurable

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-20 10:06:14 +00:00
Alex Sharov
5f8b1b75d5
Storage Tombstones UI (#394)
* implement NoValueCursor interface

* cleanup

* fix tests

Co-authored-by: alex.sharov <alex.sharov@lazada.com>
2020-03-16 22:00:48 +00:00
Igor Mandrigin
34890543e8 post-rebase: go mod tidy 2020-03-16 16:40:19 +03:00