Commit Graph

23 Commits

Author SHA1 Message Date
Raul Jordan
a9a5973b98
Add Getter for Genesis Block (#4271)
* test passing

* kafka
2019-12-12 16:27:22 -06:00
Preston Van Loon
7a46cc0681 Enforce stronger head state operations (#4216)
* Enforce stronger head state operations

* fix genesis state generation

* one test left to fix

* all tests passing now

* gofmt

* Update beacon-chain/db/kv/state_test.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* Update beacon-chain/db/kv/state.go

Co-Authored-By: Raul Jordan <raul@prysmaticlabs.com>

* fix tests
2019-12-09 15:35:18 +08:00
Nishant Das
c36a852329 Swap to Ristretto Cache (#4070)
* add new cache
* change to larger size
* Merge branch 'master' into swapCache
* Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding into swapCache
* remove imports
* cache fixes
* Merge branch 'master' into swapCache
* add better costing
* Merge branch 'swapCache' of https://github.com/prysmaticlabs/geth-sharding into swapCache
* comment
* change back to var
* Merge branch 'master' into swapCache
* Merge branch 'master' into swapCache
* Merge branch 'master' into swapCache
* Merge branch 'master' into swapCache
* Merge refs/heads/master into swapCache
2019-12-05 19:13:11 +00:00
Raul Jordan
3be4894b8a
Update Ethereum APIs, Allow Genesis Data Retrieval for Blocks + Attestations (#4150)
* update apis

* include block filter genesis

* genesis atts

* add in workspace file

* include proper diff targeting master of ethereum apis

* genesis block fetching fixes

* remove fmt

* tests for genesis list blocks passing

* fixed up container tests

* tests now passing

* fix up tests
2019-11-30 22:30:48 -06:00
Preston Van Loon
e203f66fe0 DB Improvements: Snappy compression, remove some unnecessary batch / goroutines (#4125)
* do not use batch for SaveAttestations
* use snappy compression
* Encode / decode everything with snappy
* Add snappy migration path
* batch is probably fine...
* fix test
* gofmt
* Merge branch 'master' of github.com:prysmaticlabs/prysm into remove-batch-attestations
* add sanity check
* remove that thing
* gaz
* Merge branch 'master' of github.com:prysmaticlabs/prysm into remove-batch-attestations
2019-11-27 06:32:56 +00:00
Raul Jordan
0326be86b5 Apply Patch Rules to Use EthereumAPIs Generated Protos in Prysm (#4112)
* starting on patch
* finish determining all required patches
* properly redefine the patch rules
* new patch
* rem double semicolon
* fix patch file
* Merge branch 'master' of github.com:prysmaticlabs/prysm into deprecate-eth-protos
* building the deps
* test target passes using ethereumapis
* compile gateway
* attempting to build everything
* e2e use ethereumapis
* more fixes for slasher
* other item
* getting closer to compiling slasher
* build slasher package
* Merge branch 'master' into deprecate-eth-protos
* Merge branch 'master' into deprecate-eth-protos
* fix benches
* lint gazelle
* Merge branch 'deprecate-eth-protos' of github.com:prysmaticlabs/prysm into deprecate-eth-protos
* proper gateway
* lint
* Merge branch 'master' into deprecate-eth-protos
* fix build
* Merge branch 'deprecate-eth-protos' of github.com:prysmaticlabs/prysm into deprecate-eth-protos
* use swag
* resolve
* ignore change
* include new patch changes
* fix test
* builds
* fix e2e
* gaz
2019-11-27 05:08:18 +00:00
Nishant Das
b88e6dc918 Speed Up Block Processing In Sync (#4075)
* fix proto
* make them non-batched
* gate behind flag
* fix refs
* fix refs
* Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding into speedUpProcessing
* Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding into speedUpProcessing
* use global archiver flags
* lint
* Merge branch 'master' into speedUpProcessing
* Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding into speedUpProcessing
* preston's review
* Merge branch 'speedUpProcessing' of https://github.com/prysmaticlabs/geth-sharding into speedUpProcessing
* Merge branch 'master' into speedUpProcessing
* Merge branch 'master' into speedUpProcessing
* Merge branch 'master' into speedUpProcessing
2019-11-26 07:15:54 +00:00
terence tsao
9a0fb5dca1
Delete DB states older than finalized checkpoint (#3824) 2019-10-23 08:30:21 -07:00
Raul Jordan
1a94ef12b9
Ensure Blocks Are Not Duplicated When Saved to DB (#3739)
* dedup

* tests pass when using the fallback
2019-10-08 13:47:48 +09:00
Raul Jordan
8d215feb25
Large Prysm Performance Improvements (#3622)
* skip bls verification with a feature flag at runtime

* gazelle

* more bls mocks

* block roots efficiency

* db block roots now does not show up on the flame graphs

* save validator latest votes batch

* batch save att

* misc improvements to pprof

* import

* include validator index cache

* error if no filter criteria

* resolved comments

* build fix

* lint

* remove delay global

* attestation and block test fixes

* preston suggestions

* fix db tests

* fix missing broken tests

* tests pass
2019-09-30 15:45:53 -05:00
Raul Jordan
4bd2730c5e
Batch Deletions for Blocks and Attestations (#3496)
* batch deletions for blocks and attestations

* test for atts delete

* test for blocks delete

* better naming of args in iface methods

* modify a bit

* convert to batch

* blocks batch delete

* batch fixes

* suspecting deadlock

* blocks batch delete tests pass

* more complex test
2019-09-27 11:11:10 -05:00
terence tsao
f0332e1131 Save genesis state in DB (#3359)
* Done

* Test

* Fixed test

* emit state initialized
2019-08-29 15:32:35 -07:00
Raul Jordan
8f01b76366
Integrate DB Refactor Into Ethereum APIs Beacon Chain Server (#3245)
* deprecate db

* fix build

* begin integrating new db

* gaz

* use more of the new db

* newest implementation uses head state

* remove more deprecated items

* setup validators in state helper

* fix up some tests with the new db

* resolve broken build

* gaz

* begin ensuring tests pass

* optional idx

* list validator balances passing

* default page size passing

* only two failing

* fixed most tests, found edge case

* allow nil return and add proper tests

* pass tests

* fix head block root problem

* working with the new db

* every ethereumapis method now compliant with both dbs

* pass in db into server
2019-08-22 15:28:53 -05:00
Raul Jordan
bb542d2032
Basic Block/LatestVote Caching in DB Refactor (#3249)
* add block caching layer

* runlock

* lockinggg

* latest votes map

* validator latest vote deletion method added to interface

* seguin cache working

* cache size

* impl interface

* initialize caches at struct layer
2019-08-22 10:04:13 -05:00
terence tsao
1c4b7329f2
Part 2 of fork choice update - benchmark tests (#3212) 2019-08-20 16:13:20 -06:00
Raul Jordan
900b550864
add all proper spans to methods (#3248) 2019-08-20 14:35:48 -05:00
Raul Jordan
8f967d26d7
Allow Nil Return from DB Methods (#3247)
* allow nil return and add proper tests

* pass tests
2019-08-20 12:24:29 -05:00
Raul Jordan
a7d336a7d0
Misc Blocks DB Improvements (#3244)
* allow string constructions

* fix all other cases with formatting
2019-08-20 10:09:10 -05:00
Raul Jordan
1c8ac6658e
Implement Block DB Methods (#3221)
* include new getter for block

* create block filters from indices

* give every block index a unique bucket

* construct block indices by bucket mmap

* almost done save for the block filters

* include block filters, need a few more small touches for fetching the proper indices by bucket

* full functionality to filter by parent root

* tests pass when using the same logic as attestations

* todo

* proper todo formatting

* first minimum slot range filter

* slot range filters pass

* more filter criteria passing

* tests passing

* add todos

* all block tests pass and work

* rem fmt

* range retrieval test

* fixed test conditions

* instantiate the other buckets

* simplify bucket lookups

* deprecate non map code

* revamp to remove old index prefixes

* create indices from data

* create indices from data

* fetch block roots by slot range

* better abstractions

* simpler abstractions

* roots rename

* comment

* preston feedback

* allow blocks without parent root
2019-08-19 19:34:53 -05:00
Raul Jordan
6ec9d7e6e2
Utilize Indices for Key Lookup and Filtering Attestations in DB (#3202)
* begin indices approach

* use shard bucket

* continue the indices approach

* eliminate the filter checkers in favor of the single loop of root lookups

* elim extraneous println statement

* continue the indices approach

* intersection for multiple filter types works, but is complex, verbose, and nearly unreadable

* remove unused code

* table drive tests for byte slice intersections

* include all table driven tests

* gazelle imports

* better abstractions

* better comments

* variadic approach working

* transform to variadic

* comments

* comments

* separate bucket for indices for faster range scans

* attestation key as hash tree root of data and different indices buckets

* test pass

* default behavior without filter

* appropriate filter criterion errors if criterion does not apply to type

* better abstractions and prune keys on deletion

* better naming

* fix build

* fix build

* rem extraneous code
2019-08-15 19:57:43 -05:00
Raul Jordan
655f5830f4 Implement Blocks DB Methods (#3195) 2019-08-13 09:49:27 -07:00
terence tsao
715b9cd5ba Save head block root for new DB refactor (#3182)
* Save head block root instead of save head state

* Revert state
2019-08-12 12:13:30 -04:00
Raul Jordan
dce9c41094 Define Interface Stubs for New DB Interface (#3164)
* begin db interface

* define the database interface

* interface definition simplifications

* include latest message proto

* modify pbs

* rem kv folder

* add filter interface

* lint

* ctx package is great

* interface getting better

* ctx everywhere...it's everywhere!

* block roots method

* new kv store initialization

* comments

* gaz

* implement interface

* refactor for proper naming conventions

* add todos

* proper comments

* rem unused
2019-08-10 20:50:10 -04:00