Alex Sharov
e02d6acc7d
bitmap indices for logs ( #1124 )
...
* save progress
* try now
* don't create bloom inside rlpDecode
* don't create bloom inside ApplyTransaction
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* rename method
* print timings
* print timings
* print timings
* sort before flush
* fix err lint
* clean
* move tests to transactions
* compressed version
* up bound
* up bound
* more tests
* more tests
* more tests
* more tests
* better removal
* clean
* better performance of get/put methods
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* optimize rpcdaemon
* fix test
* fix rpcdaemon
* fix test
* simplify
* simplify
* fix nil pointer
* clean
* revert some changes
* add some logs
* clean
* try without optimize
* clean
* clean
* clean
* clean
* try
* move log_index to own stage
* move log_index to own stage
* integration add log_index stage
* integration add log_index stage
* clean
* clean
* print timing
* remove duplicates at unwind
* extract truncateBitmaps func
* try detect
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* add blackList of topics
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* sharding 1
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 3
* sharded 3
* sharded 3
* speedup things by putCurrent and putReserve
* clean
* optimize trim
* clean
* remove blacklist
* add more info to err
* ?
* clean
* clean
* clean
* clean
* clean
* working version
* switch to cgo version of roaring bitmaps
* clean
* clean
* clean
* clean
* more docs
* clean
* clean
* fix logs bloom field
* Fix debug_getModifiedAccountsByNumber
* Try to fix crash
* fix problem with "absent block"
* fix problem with "absent block"
* remove optimize method call
* remove roaring iterator
* fix problem with rebuild indicess
* remove debug prints
* tests for eth_getLogs involving topics
* add tests for new stage, speparate topics into 2 buckets
* version up
* remove debug logs
* remove debug logs
* remove bloom filter implementation
* Optimisation
* Optimisatin not required, make rpctest lenient to geth errors
* Lenient to geth failures
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Alex Sharov
da31494cc3
transactional migrations ( #1079 )
2020-09-08 20:39:43 +01:00
Alex Sharov
9091ce62bb
KV reduce features amount ( #975 )
...
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize
* remove bolt_db.go file
* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
Alex Sharov
2facf92533
Better logs2 ( #964 )
2020-08-23 06:55:17 +07:00
Alex Sharov
a6be18b915
ticker-based logs ( #954 )
...
* timer-based logs
* timer-based logs
* delegate progress calculation to user
* delegate progress calculation to user
* delegate progress calculation to user
* clear
* add logs to senders recovery
* use default dir in integration
* more logs
* more logs
2020-08-22 12:12:33 +02:00
Igor Mandrigin
2f881f3865
Use batches when unwinding indexes ( #931 )
2020-08-16 18:54:33 +02:00
Alex Sharov
90fa69a9e5
Swap IH and HashState stages ( #922 )
...
* move ih stage v0
* save progress
* save working version
* baby step
* baby step
* baby step
* non-baby step
* cleanup
2020-08-15 14:49:36 +01:00
Alex Sharov
d9d9e14f45
change bucket type to string ( #894 )
2020-08-11 06:55:32 +07:00
Igor Mandrigin
3aedcbcd7d
Use datadir for temp files when generating indexes
2020-08-02 12:32:41 +02:00
Alex Sharov
f500148b86
remove BoltDatabase, BadgerDatabase, RemoteDatabase classes (replaced by objectDb) ( #794 )
2020-07-27 22:59:54 +01:00
b00ris
12aa55bcbb
Remove incarnation inversion ( #755 )
...
* remove incarnation inversion
* fix lint
* remove incarnation inversion
* remove comment
* remove inversion from new code
* revert for unwind
2020-07-17 15:18:31 +01:00
Alex Sharov
8335700331
Integration: compare to reference_db after all cycles, better cancelation, add to makefile ( #733 )
...
* cmp to reference db if --reference_chaindata provided
* graceful shutdown
* add ./cmd/integration to makefile
* add interruption support for different stages
* don't compare errors directly
2020-07-10 07:03:18 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf ( #724 )
...
* save state
* walkAsOf by cs test passed
* cs search for plain state
* save state
* fix accounts tests
* refactor walkAsOf account tests
* fix storage test
* refactor walkAsOf storage tests
* fix lint
* fix test
* save state
* save state
* add test with fixed bits
* fmt
* add stages check
* fix lint
* fix lint
* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
305f8aff02
Integration tests2 ( #714 )
...
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out // to file
--pprof --pprof.port=6060 // launch pprof server
--metrics // sends to prometheus
```
2020-07-07 11:00:25 +07:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes ( #716 )
...
* Fixing history generation
* Try fixes
* Reset from 0
* Cleanup
2020-07-06 07:34:24 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets ( #698 )
...
* Debugging changesets
* Kinda works
* Fix compile error
* Fix formatting
* Fix lint
* Duplicate entries kludge
* Fix compile error
* Cleanup
2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon ( #692 )
...
* Fixing history index
* Remove chunk generation, fix formatting
* Fix compile error, clean up hack.go
* Fix output tests
* Fix index generator test
* Fixed checkChangeSets
* Fix linter
2020-06-28 07:10:27 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change ( #670 )
...
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods
* choose db based on file suffix
* implement db.id method
* implement db.id method
* use ethdb.NewDatabase method
* use ethb.MustOpen method
* cleanup
* support TEST_DB env flag
* create db path automatically needed
* bolt - don't change prefix on happy path
2020-06-16 14:36:16 +01:00
ledgerwatch
99b9c0fa99
Re-enable TestUnwind test and fix issues in the stage sync ( #659 )
...
* Fixes to stages
* Fix formatting
* Fixes to stages
* Fix formatting
* Fix linter
* Fix lint
* Updates
* Next update
* Update
* Test works
* Cleanup
* Fix formatting
* Fix formatting
* Cleanup
* Fix core test
* Reinstate randomness for reorgs
2020-06-13 13:39:04 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage ( #646 )
...
* save state
* txlookup full results
* save state
* save state
* remove experiments
* some fix&lint
* add end key to txLookup and index generation
* change log message
* change log
* fix lint
* lint
* fix test
2020-06-10 23:07:14 +03:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments ( #608 )
...
* First commit
* Fix the one-off error
* Fix formatting
* Ability to execute stage5 separately
* Clean up intermediate hashes and stage5 progress
* Fix linter
* Print original keys when extracting
* channel
* More logging
* More logging
* Remove excess logging
* fix stage2
* Revert
* Fix stage2
* Add provider exhausted
* Sort sortable buffer
* Fix test
* Another cleanup
* Remove exhaust log
2020-06-03 13:03:40 +01:00
b00ris
266f9a5208
Plain state index ( #595 )
...
* plain state indexes
* generalize index generation&tests
* add plain state tests
* fix regenerate index
* сoncurrent changeset chunks processing
* remove concurrency
* fix lint
* remove comments
* add test to truncate
* fix conflicts
* fmt
* remove shadowing
2020-05-31 07:57:47 +01:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. ( #592 )
2020-05-30 10:00:35 +03:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices ( #569 )
...
* save state
* add current index feature
* fix test
* remove logs
* Only execute 1000 blocks
* Reset history index
* Correct action
* Increase batch size
* Increase chunk size, print memory stats
* Fix linter
* Remove unused from
* Split into 2 staged
* Use storage history gen
* remove log
* Not to run tx_cacher in staged mode
* Not to recover during stage 2
* Not to recover during stage 2
* Remove counter
Co-authored-by: b00ris <b00ris@mail.ru>
2020-05-23 10:19:56 +01:00
Alex Sharov
b3a5c62147
sort buckets then sort keys inside buckets, then send data to MultiPut ( #553 )
2020-05-17 05:46:30 +01:00
ledgerwatch
0c6a44b5e5
Index bits to mark creation of accounts or storage items ( #483 )
...
* Index bits
* Fix linter
* Fix linter
* Not to call getModifiedAccountsByNumber for go-ethereum
* Remove the 0x000 kludge
2020-04-25 15:50:32 +01:00
ledgerwatch
b3ae0cfe13
Not to repeat index chunks ( #479 )
...
* Fix for replicating chunks
* More fixes
* Fix getBalance
* Fix linter
* Fix tests
* Ping
2020-04-21 18:43:57 +01:00
ledgerwatch
623545a47c
Fix index chunks ( #478 )
...
* First stab at fixing index chunks
* Fix compilation
* Fix tests
* Fix walkAsOf for storage
* Fix
* Fix for walkAsOf accounts
* Fix linter
2020-04-21 09:15:40 +01:00
b00ris
a9f71298aa
Index chunking ( #473 )
...
* index generation
* save state
* save state
* save state
* save state
* save state
* save state
* save state
* save state
* fix
* save state
* fix
2020-04-20 11:35:33 +01:00
