Commit Graph

791 Commits

Author SHA1 Message Date
ledgerwatch
77eb94b53e
Elias fano search and merge (#357)
* Elias fano search and merge

* Add first cut of search

* Iterator and test

* Changes in aggregator

* Elias fano bitmap

* Fix uncompress decompress

* Print

* Print

* No print

* Print

* Print

* Print

* Change to AppendBytes

* Print

* Fix NextUncompressed

* Remove print

* Fix history search

* Fix in history search

* More tracing

* More tracing

* Fix

* Print

* Print key

* More print

* Print

* No deletion for history records

* Remove print

* Fix

* Fix

* Fix test

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-03-13 22:46:17 +00:00
Alex Sharov
c1f1365f92
cancel compress (#362) 2022-03-12 16:34:58 +07:00
Alex Sharov
c0fcdabf91
compress: less allocs (#361) 2022-03-12 15:33:01 +07:00
ledgerwatch
d3e05fb352
Subscriptions for logs (#279)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit 4b4b765

git-subtree-dir: interfaces
git-subtree-split: 4b4b76598f956abcdeae67c50b7c37f055d11b17

* Regenerate bindings

* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit a0b1e1c

git-subtree-dir: interfaces
git-subtree-split: a0b1e1cdbce09fd62c369f25e8ef35cf3972357e

* Rebuild gointerfaces

* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit eea9297

git-subtree-dir: interfaces
git-subtree-split: eea92977065e217c385db8eefe261671fb72c529

* Update to new interfaces

* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit 548970d

git-subtree-dir: interfaces
git-subtree-split: 548970d40b288ed7ef80412e5fdb1144c8ffe129

* Update to gointerfaces

* Update pb

* Add subslogs

* Upd pb

* Add direct for logs

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Andrey Rybnov <ryban92@gmail.com>
2022-03-11 16:33:47 +00:00
Alex Sharov
fb9214d8bb
lint up (#360) 2022-03-11 08:49:54 +07:00
Enrique Jose Avila Asapche
c82e350e63
fixed marshalling error with nil (#354)
* fixed marshalling error with nil

* returning empty byte slice

* returning empty byte slice pt2
2022-03-10 20:24:04 +00:00
battlmonstr
3123b6d895
kv: add NodeRecords nodedb table (#359) 2022-03-10 13:15:15 +01:00
battlmonstr
f54e4c7eae
kv: use SetSyncPeriod from mdbx-go v0.22.16 (#358) 2022-03-10 13:04:08 +01:00
Alex Sharov
6512e3c941
add emptyWordsCount field to .seg file header (breaking .seg format) (#355)
* up torrent

* save

* save

* save

* save

* save

* save

* save
2022-03-10 07:48:37 +00:00
battlmonstr
009358ab2f
kv: expose mdbx SafeNoSync and OptSyncPeriod options (#356) 2022-03-10 10:25:11 +07:00
ledgerwatch
75b52ac25e
[compress] Allow uncompressed words (#350)
* Intermediate work

* Allow uncompressed words

* Fix

* Fix tests

* Add NextUncompressed, remove g.word buffer

* Code simplifications, no goroutines when workers == 1

* Fix lint|

* Add test for MatchPrefix

* Work on patricia

* Beginning of new matcher

* Fuzz test for new longest match

* No skip

* Fixes

* Fixes

* More tracing

* Fixes

* Fixes

* Change back to old FindLongestMatches

* Switch to old match finder

* Print mismatches

* Fix

* After fix

* After fix

* After fix

* Print pointers

* Fixes and tests

* Print

* Print

* Print

* More tests

* Intermediate

* Fix

* Fix

* Prints

* Fix

* Fix

* Initialise matchStack

* Compute only once

* Compute only once

* Switch back

* Switch to old Find

* Introduce sais

* Switch patricia to sais

* Use sais in compressor

* Use sais in compressor

* Remove unused code

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-03-09 17:25:22 +00:00
Alex Sharov
a2cbecf373
[wip] KV: limit amount of parallel read txs (#352)
* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* a

* a

* a

* a

* save

* save

* save

* save

* save
2022-03-07 12:21:59 +00:00
Alex Sharov
56d528b1f0
save (#353) 2022-03-04 09:13:15 +07:00
Alex Sharov
ebfbf92936
mdbx_v0.11.5 (#351) 2022-02-25 07:42:48 +00:00
Giulio rebuffo
2fc62cd2f5
prevrandao (#349) 2022-02-24 16:18:01 +01:00
racytech
7763945374
Reverted 3 last commits (#348)
* Revert "unnecessary includes removed"

This reverts commit 76406bb78b.

* Revert "local dev setup"

This reverts commit ac06fd9400.

* Revert "compress/cgo-addition"

This reverts commit fae7683d46, reversing
changes made to e3e108c6c4.
2022-02-24 14:39:42 +00:00
Kairat Abylkasymov
fae7683d46 compress/cgo-addition 2022-02-24 06:28:01 -05:00
Kairat Abylkasymov
76406bb78b unnecessary includes removed 2022-02-24 06:21:25 -05:00
Kairat Abylkasymov
ac06fd9400 local dev setup 2022-02-24 06:15:14 -05:00
Alex Sharov
e3e108c6c4
logger version up (#347) 2022-02-24 15:14:37 +07:00
Alex Sharov
3205770ee0
snapshots: fix test (#346) 2022-02-24 08:35:13 +07:00
ledgerwatch
27a61d4611
[txpool] Poking only caused by local transactions (#345)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-24 00:18:46 +00:00
ledgerwatch
efdeb69af0
[txpool] Poke transactions that cannot be replaced (#343)
* Poke transactions that cannot be replaced

* Remove printing

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-23 23:48:05 +00:00
ledgerwatch
50082e3553
[txpool] Remove unsafe parameter for onSenderStateChange (#341)
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-22 08:53:29 +00:00
ledgerwatch
c71ac02a0f
[erigon2] Optimisations in etl collector and compressor (#339)
* Optimisations in etl collector and compressor

* Not copy k and v in the collector

* Fix lint

* Optimisations

* Change Load1 back to Load

* Reduce allocations for tests

* preallocate inv

* counting hits and misses

* Try to fix

* Try to fix

* Relaxation 1

* Relaxation 2

* Add arch tables

* Fix

* Update arch tables and use them

* Not to override larger value

* Increase arch table size

* Increase arch table size

* Fixes to arch

* Print

* Off by one

* Print

* Fix

* Remove print

* Perform update of arch in the background

* Build up huffman tree

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-20 22:14:06 +00:00
EXEC
143cd510bd
Fix typo in README.md (#340)
`currenk` -> `current`
2022-02-20 14:12:06 +00:00
ledgerwatch
91c51b20ad
[erigon2] History reader fixes (#338)
* More tracing for history

* More tracing for history

* More tracing for history

* only under tracing

* Trace

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-18 22:19:38 +00:00
Alex Sharov
c1c45b94bf
last version of kanzi-go dep (#337) 2022-02-18 10:11:29 +07:00
Alex Sharov
7747be5228
more stable grpc err check (#336) 2022-02-18 09:54:27 +07:00
Alex Sharov
5be29adf1a
Grpc err utils (#334)
* save

* save
2022-02-18 09:40:11 +07:00
Alex Sharov
d94c4ada78
dir package - to enforce dir RW lock by types (#332)
* save

* save
2022-02-18 09:24:30 +07:00
Alex Sharov
d309d52fe2
rpcdaemon embed fixes (#333) 2022-02-17 11:24:13 +07:00
alex.sharov
a2ca4cf55f save 2022-02-17 10:10:23 +07:00
ledgerwatch
43b38539b3
[erigon2] Remove empty commitments and add comments (#331)
* Remove empty commitments and add comments

* Remove logging

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-16 22:38:37 +00:00
ledgerwatch
1c6e82c2b6
[erigon2] Thin commitment (2nd attempt) (#329)
* Another fix for history files

* Half way through

* Another fix

* Correct closing sequence

* Remove first byte insert marker

* More on think commitments

* Fixes

* Fixes

* Print

* Skip touchMap

* Merge branchData from trees and from files

* Fill branch commitment

* Fill branch commitment

* Print

* Fix?

* Merge branchData when updating in the tree

* Better panic

* Prints

* Prints

* Prints

* Create complete branch data if it did not exist before

* Cleanup printing

* Fix merge

* Fix merge use

* Fix transform

* Better startBlock panic

* Preserve touchMap

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments during aggregation

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Merge commitments

* Include fieldbits during transform

* Fix history reads

* Print

* Print

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Prints

* Print largestMerge, lock files in branchFn

* Add storage lock

* Prints

* prefixLen fix

* prefixLen fix

* Fixes

* Remove print

* Remove print

* Set changesets and commitments flags upfront

* Logging instead of printing

* Fix history merge, recsplit panic

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-02-16 16:44:00 +00:00
Alex Sharov
0c69ccd29d
rpcdaemon embed (#330)
* save

* save

* save
2022-02-16 10:56:50 +07:00
Alex Sharov
9789f53041
aggregator: close idx after build (#327) 2022-02-14 18:38:39 +00:00
Alex Sharov
1f5a1ab9cd
fuzz cases (#328) 2022-02-14 11:53:20 +07:00
ledgerwatch
1363859958
[erigon2] Fix history bug, optimise history search (#326)
* Another fix for DeleteAccount

* Optimise bitmap search

* Optimise bitmap search

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-13 18:56:36 +00:00
Alex Sharov
79aa17d297
Recsplit: use crypto rand seed if user not set (#325) 2022-02-13 16:14:04 +07:00
Alex Sharov
39566b5e50
return EOF on direct stream end (#324)
* save

* save
2022-02-13 09:33:54 +07:00
ledgerwatch
d9367b582a
[erigon2] fix change files for commitments (#318)
* fix change files for commitments

* Not create commitment files unless enabled

* Print loaded files

* More fixes to commitment changes

* More fixes to commitment changes

* Fix for DeleteAccount storage values

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-12 14:51:23 +00:00
Alex Sharov
6f85066c7e
path -> filepath (path package is for urls) (#321) 2022-02-12 20:11:30 +07:00
alex.sharov
b041b959ca direct: txpool, mining 2022-02-12 19:33:47 +07:00
Alex Sharov
31982dacc8
direct: more compact naming (#320) 2022-02-12 18:48:42 +07:00
Alex Sharov
51dda9de27
direct: txpool, mining (#319) 2022-02-12 18:41:17 +07:00
Alex Sharov
e16bce5233
eth backend: direct (#317) 2022-02-12 16:55:33 +07:00
Alex Sharov
ba6af37677
eth backend: direct (#316)
* eth backend: direct

* eth backend: direct
2022-02-12 16:50:06 +07:00
Alex Sharov
57d026dcca
grpc: disabled server txpool (#314)
* grpc disabled server txpool

* grpc disabled server txpool
2022-02-12 12:24:17 +07:00
ledgerwatch
5f7283d0f1
[erigon2] History reader (#312)
* Initial code for history reader

* Print

* Print

* Fix npe

* Open files of all types

* don't close file in the constructor

* Print foundTxNum

* Print foundTxNum

* Print adding to bitmap

* Print when reducing

* Print when reducing

* Print when reducing

* Enum false for index

* Remove prints

* Remove prints

* Print loaded files

* Tracing

* Tracing

* Tracing

* Tracing

* Tracing

* Optinal commitments

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-11 23:44:30 +00:00