Commit Graph

762 Commits

Author SHA1 Message Date
Alex Sharov
5be29adf1a
Grpc err utils (#334)
* save

* save
2022-02-18 09:40:11 +07:00
Alex Sharov
d94c4ada78
dir package - to enforce dir RW lock by types (#332)
* save

* save
2022-02-18 09:24:30 +07:00
Alex Sharov
d309d52fe2
rpcdaemon embed fixes (#333) 2022-02-17 11:24:13 +07:00
alex.sharov
a2ca4cf55f save 2022-02-17 10:10:23 +07:00
ledgerwatch
43b38539b3
[erigon2] Remove empty commitments and add comments (#331)
* Remove empty commitments and add comments

* Remove logging

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-16 22:38:37 +00:00
ledgerwatch
1c6e82c2b6
[erigon2] Thin commitment (2nd attempt) (#329)
* Another fix for history files

* Half way through

* Another fix

* Correct closing sequence

* Remove first byte insert marker

* More on think commitments

* Fixes

* Fixes

* Print

* Skip touchMap

* Merge branchData from trees and from files

* Fill branch commitment

* Fill branch commitment

* Print

* Fix?

* Merge branchData when updating in the tree

* Better panic

* Prints

* Prints

* Prints

* Create complete branch data if it did not exist before

* Cleanup printing

* Fix merge

* Fix merge use

* Fix transform

* Better startBlock panic

* Preserve touchMap

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Include fieldbits during transform

* Fix history reads

* Print

* Print

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Print largestMerge, lock files in branchFn

* Add storage lock

* Prints

* prefixLen fix

* prefixLen fix

* Fixes

* Remove print

* Remove print

* Set changesets and commitments flags upfront

* Logging instead of printing

* Fix history merge, recsplit panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-16 16:44:00 +00:00
Alex Sharov
0c69ccd29d
rpcdaemon embed (#330)
* save

* save

* save
2022-02-16 10:56:50 +07:00
Alex Sharov
9789f53041
aggregator: close idx after build (#327) 2022-02-14 18:38:39 +00:00
Alex Sharov
1f5a1ab9cd
fuzz cases (#328) 2022-02-14 11:53:20 +07:00
ledgerwatch
1363859958
[erigon2] Fix history bug, optimise history search (#326)
* Another fix for DeleteAccount

* Optimise bitmap search

* Optimise bitmap search

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-13 18:56:36 +00:00
Alex Sharov
79aa17d297
Recsplit: use crypto rand seed if user not set (#325) 2022-02-13 16:14:04 +07:00
Alex Sharov
39566b5e50
return EOF on direct stream end (#324)
* save

* save
2022-02-13 09:33:54 +07:00
ledgerwatch
d9367b582a
[erigon2] fix change files for commitments (#318)
* fix change files for commitments

* Not create commitment files unless enabled

* Print loaded files

* More fixes to commitment changes

* More fixes to commitment changes

* Fix for DeleteAccount storage values

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-12 14:51:23 +00:00
Alex Sharov
6f85066c7e
path -> filepath (path package is for urls) (#321) 2022-02-12 20:11:30 +07:00
alex.sharov
b041b959ca direct: txpool, mining 2022-02-12 19:33:47 +07:00
Alex Sharov
31982dacc8
direct: more compact naming (#320) 2022-02-12 18:48:42 +07:00
Alex Sharov
51dda9de27
direct: txpool, mining (#319) 2022-02-12 18:41:17 +07:00
Alex Sharov
e16bce5233
eth backend: direct (#317) 2022-02-12 16:55:33 +07:00
Alex Sharov
ba6af37677
eth backend: direct (#316)
* eth backend: direct

* eth backend: direct
2022-02-12 16:50:06 +07:00
Alex Sharov
57d026dcca
grpc: disabled server txpool (#314)
* grpc disabled server txpool

* grpc disabled server txpool
2022-02-12 12:24:17 +07:00
ledgerwatch
5f7283d0f1
[erigon2] History reader (#312)
* Initial code for history reader

* Print

* Print

* Fix npe

* Open files of all types

* don't close file in the constructor

* Print foundTxNum

* Print foundTxNum

* Print adding to bitmap

* Print when reducing

* Print when reducing

* Print when reducing

* Enum false for index

* Remove prints

* Remove prints

* Print loaded files

* Tracing

* Tracing

* Tracing

* Tracing

* Tracing

* Optinal commitments

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-11 23:44:30 +00:00
Alex Sharov
7aec036bdd
remotedb server: ro db (#313) 2022-02-11 16:02:57 +07:00
ledgerwatch
441a4c3cde
[erigon2] Chain history and bitmap indices - part 2 (#308)
* corretly shutdown history goroutine

* Different final merge for history files

* Skip value

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Bitmap production bug

* Handle collision

* Handle collision

* Debug

* Debug

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-10 15:30:55 +00:00
primal_concrete_sledge
c4805e0262
WIP: issue/issue-281-create_binding_to_pedersen_hash (#301)
* issue/issue-281-create_binding_to_pedersen_hash

* Add //nolint

* Add more nolints

* move nolint

* Remove nolit

* Add gcc install

* Upd .ci

* Remove staticcheck

* Add envs

* try to exclude pedersen_hash from test

* try to fix mac os build

* Add include for mac os

* Add include for mac os

* Fix runner_os

* remove test for macos

* Change restrictions

* restrict tests to ubuntu

* Try test windows

* Add build constraint
2022-02-10 14:47:28 +00:00
Alex Sharov
3c4f8a759c
recsplit: reset offset collector, etl: faster flush and load (#310) 2022-02-10 14:40:24 +07:00
Alex Sharov
8e4618c57d
state and newHead: 1 goroutine per subscriber #309 2022-02-10 08:26:08 +07:00
Andrew Ashikhmin
2e162c70ba
Update Merge Engine API (Kiln v1) (#268)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit 2daa726

git-subtree-dir: interfaces
git-subtree-split: 2daa72683ceaacfe0476005c11a6fc7b6d945780

* Update Merge Engine API

* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit 0e25bcc

git-subtree-dir: interfaces
git-subtree-split: 0e25bcc642c90fcc63e12930032a7e0f2260e865

* Engine API updated

* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit ee8057d

git-subtree-dir: interfaces
git-subtree-split: ee8057d3e8ab57059fac55434f3f55d189b00242

* Add INVALID_TERMINAL_BLOCK to EngineStatus

* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit 50ece30

git-subtree-dir: interfaces
git-subtree-split: 50ece307faac31b7d7fe9cd06e11ed82c09064b9
2022-02-09 07:49:31 +01:00
Alex Sharov
e649f7ea91
Less alloc etl recsplit (#307)
* less allocs recsplit

* save

* save
2022-02-09 13:22:45 +07:00
ledgerwatch
f5deff3ac9
[erigon2] accumulating chain history and bitmap indices (#306)
* Aggregator refactoring

* Restore the test

* Simplify commitment

* Lint

* onBefore

* Wip smaller commitments

* Fixes

* Fix lint

* Check for zero len

* Print

* Remove restriction

* Correct restrictions

* Correct restrictions

* More merging

* delBitmap

* Switch off merging

* Reintroduce background merge

* Try to fix

* Slight optimisation

* Produce initial changeset files

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Print changeset keys

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

* Merge for history and bitmaps

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-08 21:19:29 +00:00
ledgerwatch
92fc7215c8
[erigon2] fix commitment error (#304)
* Try to fix

* Fix

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-04 18:48:50 +00:00
Alex Sharov
567d9ddfed
ParallelCompressor: Remove intermediate ETL collectors (#302) 2022-02-04 16:48:02 +07:00
alex.sharov
d4b9053aed save 2022-02-04 11:20:02 +07:00
ledgerwatch
55080d5c01
Proper reset of decompressor getter (#299)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-03 17:58:56 +00:00
ledgerwatch
11615db276
Fix memory leak in txpool (#298)
* Fix memory leak in txpool

* Cannot delete during iteration

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-03 17:52:20 +00:00
alex.sharov
d814adb13e save 2022-02-03 16:28:43 +07:00
alex.sharov
02c407a14e save 2022-02-03 16:28:32 +07:00
ledgerwatch
cbfd733672
[erigon2] optimisations (#297)
* Reuse getter and indexReader

* Properly reset Getter

* Use separate getter and indexReader for merge thread

* Remove DB

* Fixes

* Print

* Fixes

* Cleanup signatures

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-03 08:49:47 +00:00
Alex Sharov
ec354d1615
Fuzz fixes (#295)
* fuzz tests fixes

* fuzz tests fixes
2022-02-02 09:18:04 +00:00
Alex Sharov
a53f232322
pool: fix race in async broadcast (#294)
* pool: better async broadcast

* save

* save
2022-02-02 14:49:26 +07:00
Alex Sharov
74d49a9de1
non blocking broadcast of new txs (#292) 2022-02-01 12:16:43 +00:00
ledgerwatch
1cc07746a4
[erigon2] Try to fix deadlock (#291)
* Try to fix deadlock

* Another lock

* Try to optimise

* comments and cleanup

* Prevent deadlock on panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-01 10:28:05 +00:00
Alex Sharov
0feb7fd591
Decompressor.WithReadAhead (#290) 2022-02-01 11:19:11 +07:00
ledgerwatch
4e8840256e
[erigon2] Use shorter references instead of full plain keys in the commitment files (#289)
* Rearrange aggregations

* More rearranging before introducing 3 threads

* Background aggregation|

* Concurrency fixes

* Remove files under lock

* Better logging

* Remove files without lock

* Fix lint

* Fix locking

* Try

* Fix background Merge

* Log merging

* Log merging

* Less logging

* Millisecond

* Add Stats function

* Log merge only after 1m

* Wrong counting

* plain key extract and replace functions

* Insert valTransform function

* Not parse first byte

* Not parse first byte

* Fix lint

* Switch to thin state references

* Fix lint

* Fix lint

* Debug print|

* Fix decoding

* Turn off valTransform

* Not to reuse transformer

* Print

* Print

* Print

* Derive hashed keys later

* Fix

* Fix log

* Fix

* Debug

* Another fix

* Fix

* Fix

* Print

* Print

* Data race

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-31 22:32:00 +00:00
ledgerwatch
586ab3e6b3
Separate state btree files (#287)
* Separate state file btrees, fix Match in the decompressor

* fix match

* Fix to match

* Switch back from Match

* Try to use match, close indices

* Fixing Match

* Use Skip

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-29 11:12:38 +00:00
Enrique Jose Avila Asapche
a5c2d83108
Added bor tables (#286)
Co-authored-by: Krishna Upadhyaya <krishnau1604@gmail.com>
2022-01-28 21:34:03 +00:00
primal_concrete_sledge
24098d94d8
WIP: issue/issue-250-use_compressed_keys (#285)
* issue/issue-250-use_compressed_keys

* Step into the twilight zone

* fix zero byte

* Add zerolen check

* Remove branhc data check

* Check lengths
2022-01-28 19:30:32 +00:00
Alex Sharov
c8397a0433
Mdbx: more 4tb fixes (#282) 2022-01-27 23:10:30 +07:00
Alex Sharov
dfdf7c8a66
[wip] parallel compress: less read of dat file (#284)
* save

* save

* save
2022-01-27 17:13:26 +07:00
Alex Sharov
ec11eb3d91
parallel compressor: don't save dict (#283)
* save

* save
2022-01-27 12:54:38 +07:00
ledgerwatch
1d4022960a
Fix RLP inconsistency in txpool (#280)
* Fix RLP inconsistency in txpool

* Add debug

* hasEnvelope argument when parsing transaction

* Handle incorrect packets without panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-26 16:09:02 +00:00