mirror/erigon-pulse
1
0
Fork 0
mirror of https://gitlab.com/pulsechaincom/erigon-pulse.git synced 2025-01-04 01:54:28 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
12,953 Commits 5 Branches 14 Tags 294 MiB
a8b7a59977
Commit Graph

35 Commits

Author SHA1 Message Date
Alex Sharov
583fada5ad
history_last_but_one_chunk_dissapear (#1355) 2020-11-16 14:49:11 +00:00
Alex Sharov
b3f1915d09
ChangeSets dupsort (#1342) 
* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* change_set_dup

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* working version

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* aa

* squash

* squash

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* history_early_stop

* history_early_stop

* vmConfig with ReadOnly false

* auto_increment

* auto_increment

* rebase master

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-11-16 12:08:28 +00:00
Alex Sharov
9712e858e2
Method delete to accept second param: allow delete dupsort values (#1297) 
* aloow_delete_dupsort_values

* check canonical receipt

* exclusive lock for mdbx

* readme_load_dump_tables

* readme_load_dump_tables

* readme_load_dump_tables

* Remove IdealBatchSize change

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-10-29 13:19:31 +00:00
Alex Sharov
76f1b05cb2
Move tmpdir definition to app-start, move migrations folder inside tmpdir (#1282) 
* extract tmpdir to app-level-code

* extract tmpdir to app-level-code

* save progresss
 2020-10-23 12:18:45 +01:00
ledgerwatch
88acfc55ca
Vary batch size for execution stage (#1274) 
* Vary batch size for execution stage

* Fix in logging

* Improve log of timings

* Log improvements

* Log improvements

* Fix tests

* More log improvments, fix tests

* Fix tests

* Better logging for Bodies and Interhash

* Fix tests

* Minor fixes

* Fix test
 2020-10-21 18:01:40 +01:00
Alex Sharov
5dbe1724be
Bitmap etl (#1227) 
* save progress

* save progress

* save progress

* save progress

* etl bimaps

* etl bimaps

* etl bimaps

* etl bimaps

* a

* a

* a

* a

* a

* a

* a

* save progress

* save progress

* save progress

* a

* fix_race_condition_on_zstd_build

* clean

* clean

* clean

* clean

* clean

* go mod tidy

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* fix corner case

* commit every

* call traces etl

* call traces etl

* call traces etl

* return missed error

* go mod tidy

* go mod tidy

* go mod tidy

* rebase to master

* remove trash

* print full key if it has 8 leading zeroes
 2020-10-19 14:11:01 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs (#1124) 
* save progress

* try now

* don't create bloom inside rlpDecode

* don't create bloom inside ApplyTransaction

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* rename method

* print timings

* print timings

* print timings

* sort before flush

* fix err lint

* clean

* move tests to transactions

* compressed version

* up bound

* up bound

* more tests

* more tests

* more tests

* more tests

* better removal

* clean

* better performance of get/put methods

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* optimize rpcdaemon

* fix test

* fix rpcdaemon

* fix test

* simplify

* simplify

* fix nil pointer

* clean

* revert some changes

* add some logs

* clean

* try without optimize

* clean

* clean

* clean

* clean

* try

* move log_index to own stage

* move log_index to own stage

* integration add log_index stage

* integration add log_index stage

* clean

* clean

* print timing

* remove duplicates at unwind

* extract truncateBitmaps func

* try detect

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* add blackList of topics

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* clean

* sharding 1

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 2

* sharded 3

* sharded 3

* sharded 3

* speedup things by putCurrent and putReserve

* clean

* optimize trim

* clean

* remove blacklist

* add more info to err

* ?

* clean

* clean

* clean

* clean

* clean

* working version

* switch to cgo version of roaring bitmaps

* clean

* clean

* clean

* clean

* more docs

* clean

* clean

* fix logs bloom field

* Fix debug_getModifiedAccountsByNumber

* Try to fix crash

* fix problem with "absent block"

* fix problem with "absent block"

* remove optimize method call

* remove roaring iterator

* fix problem with rebuild indicess

* remove debug prints

* tests for eth_getLogs involving topics

* add tests for new stage, speparate topics into 2 buckets

* version up

* remove debug logs

* remove debug logs

* remove bloom filter implementation

* Optimisation

* Optimisatin not required, make rpctest lenient to geth errors

* Lenient to geth failures

Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
 2020-09-28 18:18:36 +01:00
Alex Sharov
da31494cc3
transactional migrations (#1079) 2020-09-08 20:39:43 +01:00
Alex Sharov
9091ce62bb
KV reduce features amount (#975) 
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize

* remove bolt_db.go file

* move method NoValueCursor to tx object
 2020-08-26 07:03:50 +01:00
Alex Sharov
2facf92533
Better logs2 (#964) 2020-08-23 06:55:17 +07:00
Alex Sharov
a6be18b915
ticker-based logs (#954) 
* timer-based logs

* timer-based logs

* delegate progress calculation to user

* delegate progress calculation to user

* delegate progress calculation to user

* clear

* add logs to senders recovery

* use default dir in integration

* more logs

* more logs
 2020-08-22 12:12:33 +02:00
Igor Mandrigin
2f881f3865
Use batches when unwinding indexes (#931) 2020-08-16 18:54:33 +02:00
Alex Sharov
90fa69a9e5
Swap IH and HashState stages (#922) 
* move ih stage v0

* save progress

* save working version

* baby step

* baby step

* baby step

* non-baby step

* cleanup
 2020-08-15 14:49:36 +01:00
Alex Sharov
d9d9e14f45
change bucket type to string (#894) 2020-08-11 06:55:32 +07:00
Igor Mandrigin
3aedcbcd7d Use datadir for temp files when generating indexes 2020-08-02 12:32:41 +02:00
Alex Sharov
f500148b86
remove BoltDatabase, BadgerDatabase, RemoteDatabase classes (replaced by objectDb) (#794) 2020-07-27 22:59:54 +01:00
b00ris
12aa55bcbb
Remove incarnation inversion (#755) 
* remove incarnation inversion

* fix lint

* remove incarnation inversion

* remove comment

* remove inversion from new code

* revert for unwind
 2020-07-17 15:18:31 +01:00
Alex Sharov
8335700331
Integration: compare to reference_db after all cycles, better cancelation, add to makefile (#733) 
* cmp to reference db if --reference_chaindata provided

* graceful shutdown

* add ./cmd/integration to makefile

* add interruption support for different stages

* don't compare errors directly
 2020-07-10 07:03:18 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf (#724) 
* save state

* walkAsOf by cs test passed

* cs search for plain state

* save state

* fix accounts tests

* refactor walkAsOf account tests

* fix storage test

* refactor walkAsOf storage tests

* fix lint

* fix test

* save state

* save state

* add test with fixed bits

* fmt

* add stages check

* fix lint

* fix lint

* remove obsoleted methods
 2020-07-09 07:13:45 +01:00
Alex Sharov
305f8aff02
Integration tests2 (#714) 
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages  --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000 
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out   // to file
--pprof --pprof.port=6060     // launch pprof server
--metrics                  //  sends to prometheus 
```
 2020-07-07 11:00:25 +07:00
ledgerwatch
d713de4713
Fix index generation for storage when incarnation changes (#716) 
* Fixing history generation

* Try fixes

* Reset from 0

* Cleanup
 2020-07-06 07:34:24 +01:00
ledgerwatch
137daa6c67
Fixes in generated changeSet and checkChangeSets (#698) 
* Debugging changesets

* Kinda works

* Fix compile error

* Fix formatting

* Fix lint

* Duplicate entries kludge

* Fix compile error

* Cleanup
 2020-07-01 15:56:56 +01:00
ledgerwatch
ed866e6934
non-concurrent ETL, debug_traceTransaction in rcpdaemon (#692) 
* Fixing history index

* Remove chunk generation, fix formatting

* Fix compile error, clean up hack.go

* Fix output tests

* Fix index generator test

* Fixed checkChangeSets

* Fix linter
 2020-06-28 07:10:27 +01:00
Alex Sharov
57777e7c60
Prepare codebase for future default DB change (#670) 
* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* Add kv.tx.bucket.Clear() and db.ClearBuckets() methods

* choose db based on file suffix

* implement db.id method

* implement db.id method

* use ethdb.NewDatabase method

* use ethb.MustOpen method

* cleanup

* support TEST_DB env flag

* create db path automatically needed

* bolt - don't change prefix on happy path
 2020-06-16 14:36:16 +01:00
ledgerwatch
99b9c0fa99
Re-enable TestUnwind test and fix issues in the stage sync (#659) 
* Fixes to stages

* Fix formatting

* Fixes to stages

* Fix formatting

* Fix linter

* Fix lint

* Updates

* Next update

* Update

* Test works

* Cleanup

* Fix formatting

* Fix formatting

* Cleanup

* Fix core test

* Reinstate randomness for reorgs
 2020-06-13 13:39:04 +01:00
b00ris
b4ba764fb1
[WIP] TxLookup stage (#646) 
* save state

* txlookup full results

* save state

* save state

* remove experiments

* some fix&lint

* add end key to txLookup and index generation

* change log message

* change log

* fix lint

* lint

* fix test
 2020-06-10 23:07:14 +03:00
ledgerwatch
e5692d1726
Various fixes to staged sync and experiments (#608) 
* First commit

* Fix the one-off error

* Fix formatting

* Ability to execute stage5 separately

* Clean up intermediate hashes and stage5 progress

* Fix linter

* Print original keys when extracting

* channel

* More logging

* More logging

* Remove excess logging

* fix stage2

* Revert

* Fix stage2

* Add provider exhausted

* Sort sortable buffer

* Fix test

* Another cleanup

* Remove exhaust log
 2020-06-03 13:03:40 +01:00
b00ris
266f9a5208
Plain state index (#595) 
* plain state indexes

* generalize index generation&tests

* add plain state tests

* fix regenerate index

* сoncurrent changeset chunks processing

* remove concurrency

* fix lint

* remove comments

* add test to truncate

* fix conflicts

* fmt

* remove shadowing
 2020-05-31 07:57:47 +01:00
Igor Mandrigin
ebe7aec14e
Generalize "insert into DB though sorted files" pattern. (#592) 2020-05-30 10:00:35 +03:00
ledgerwatch
f06db2f37b
Stages 6 and 7 for generating history indices (#569) 
* save state

* add current index feature

* fix test

* remove logs

* Only execute 1000 blocks

* Reset history index

* Correct action

* Increase batch size

* Increase chunk size, print memory stats

* Fix linter

* Remove unused from

* Split into 2 staged

* Use storage history gen

* remove log

* Not to run tx_cacher in staged mode

* Not to recover during stage 2

* Not to recover during stage 2

* Remove counter

Co-authored-by: b00ris <b00ris@mail.ru>
 2020-05-23 10:19:56 +01:00
Alex Sharov
b3a5c62147
sort buckets then sort keys inside buckets, then send data to MultiPut (#553) 2020-05-17 05:46:30 +01:00
ledgerwatch
0c6a44b5e5
Index bits to mark creation of accounts or storage items (#483) 
* Index bits

* Fix linter

* Fix linter

* Not to call getModifiedAccountsByNumber for go-ethereum

* Remove the 0x000 kludge
 2020-04-25 15:50:32 +01:00
ledgerwatch
b3ae0cfe13
Not to repeat index chunks (#479) 
* Fix for replicating chunks

* More fixes

* Fix getBalance

* Fix linter

* Fix tests

* Ping
 2020-04-21 18:43:57 +01:00
ledgerwatch
623545a47c
Fix index chunks (#478) 
* First stab at fixing index chunks

* Fix compilation

* Fix tests

* Fix walkAsOf for storage

* Fix

* Fix for walkAsOf accounts

* Fix linter
 2020-04-21 09:15:40 +01:00
b00ris
a9f71298aa
Index chunking (#473) 
* index generation

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* save state

* fix

* save state

* fix
 2020-04-20 11:35:33 +01:00