Commit Graph

1014 Commits

Author SHA1 Message Date
Alex Sharov
ceafdded8f
Compress: reduce etl buffers to save RAM (#502) 2022-06-25 19:39:36 +06:00
Giulio rebuffo
b2c211d9bf
Refactor sequence handling in mutation 2022-06-25 01:05:27 +02:00
Giulio rebuffo
541dbe8c47
SeekBothRange instead of FirstDup 2022-06-25 01:04:51 +02:00
Giulio rebuffo
8e0df524d8
Added more functionality to the mutation. 2022-06-24 22:10:45 +02:00
ledgerwatch
46bebb3317
[erigon2.2] Add ReadIndices aggregator to collect data (#500)
* [erigon2.2] Add ReadIndices aggregator to collect data

* Try

* Fix for history access

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-20 08:39:29 +01:00
Alex Sharov
5574d68a87
save (#499) 2022-06-19 06:35:29 +01:00
Giulio rebuffo
addc05bae3
Plaxeholder values for dbsize and pagesize 2022-06-19 00:46:31 +02:00
ledgerwatch
234be664fc
Optimise history access for multiple files (#498)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-18 22:54:36 +01:00
Giulio rebuffo
d57a16c7f6
Implemented ForAmount and MDBX sequencing (#497) 2022-06-18 21:24:30 +02:00
Giulio rebuffo
988a4b5510
Added update txn function 2022-06-17 23:59:04 +02:00
ledgerwatch
945b0e9e0f
Fix merge of code files (#495)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-17 19:24:56 +01:00
ledgerwatch
d7693ce094
[txpool] Prevent clogging up the pending sub pool (#494)
* Debug txpool Best

* Print Best result

* Print

* Trace removeMined

* Remove mined tx from pending subpool

* Remove debug

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-17 15:46:01 +01:00
ledgerwatch
df49481ddc
[erigon 2.2] Make keys always uncompressed, values compressed only for code (#492)
* Reduce allocations in domain and aggregator

* Make keys always uncompressed, values compressed only for code

* Functions to remake index

* Fix index recreation

* Test for reindex, fix

* Use uncompress vals in history

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-17 12:39:49 +01:00
Giulio rebuffo
280c5e9dcc
Added memory mutation to Erigon lib (#493) 2022-06-16 08:01:48 +01:00
ledgerwatch
bbf96d0580
Close compressor (#491)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-14 22:38:18 +01:00
ledgerwatch
a450f5c341
MDBX patch for txid to try to reduce GC work on commit (#490)
* MDBX patch for txid to try to reduce GC work on commit

* Update to mdbx-go master

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-14 11:42:14 +01:00
ledgerwatch
e2c6ef0058
[erigon2.2] Fixes for inverted indices and domains for the prototype (#489)
* Better control of compress/uncompressed

* Add new function

* more careful pruning

* Printf

* Printf

* Fix DupSort

* Remove copying in prune

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-13 19:32:13 +01:00
Giulio rebuffo
dba877998b
Fix pool giving too many entries (#488) 2022-06-13 09:00:31 +01:00
ledgerwatch
6cad65e62b
[erigon2.2] Parallel build files and merge, change file names (#487)
* Parallel build files and merge, change file names

* Update ci.yml

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-12 10:14:18 +01:00
ledgerwatch
45d4c21490
Expose inverted index ranges in aggregator (#486)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-11 12:09:24 +01:00
ledgerwatch
7ce8bd589f
[erigon 2.2] Add functions for traces and event logs (#485)
* Add functions for traces and event logs

* Add functions for traces and event logs

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-06-10 06:51:00 +01:00
ledgerwatch
74ea75f9b8
[erigon2.2] Merge fixes, add historical access (#482)
* Merge fixes, add historical access

* Change API from AfterTxNum to BeforeTxNum

* Change API functions

* Change API functions

* Print

* Fix for non-existent items

* Remove prints

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-09 14:48:16 +01:00
Artem Tsebrovskiy
f8bdadf3e0
HPH with direct reading from state by plainKey (#472)
* dirty trie with direct reading of account/storage data from state

run with fixes

implemented trie with direct reading from state

* cleaner version without updates
2022-06-09 13:46:11 +01:00
Alex Sharov
e4b9cc3884
Reclaimable space in logs (#484)
* save

* save

* save

* save

* save

* save
2022-06-09 13:37:28 +01:00
Alex Sharov
19086759eb
methods to configure db #483 2022-06-09 09:43:12 +07:00
leonardchinonso
13529121f3
Chenged lowercase b meaning bits to uppercase B meaning Bytes (#481) 2022-06-08 19:19:17 +07:00
Andrew Ashikhmin
5278815cd0
Start txpool in AddLocalTxs (#480)
* Start txpool in AddLocalTxs

* Correct tx in fromDB called by AddLocalTxs
2022-06-08 10:39:22 +02:00
ledgerwatch
a77e6425eb
Fixes for the Erigon 2 upgrade 2 prototype (#479)
* Print

* Remove print

* Remove print

* Fix one panic

* Fix duplicate collation

* Print

* Fix print

* fix maxSpan

* Reduce maxSpan

* Remove duplicate join

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-05 22:32:34 +01:00
ledgerwatch
f16b285631
Adjustments for erigon 2 upgrade prototype (#477)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-06-05 16:34:38 +01:00
Alex Sharov
fdf7c6598b
compress.Count() method (#478) 2022-06-03 12:14:58 +07:00
ledgerwatch
157b4299e4
[erigon2] Continuation on domains and inverted indices, putting things together (#476)
* Add scan files tests, create new aggregator type

* Fix lint

* windows test fix

* Add delelte test

* AggCollation

* More functions to Aggregator

* More aggregator functions

* Update

* More functions

* More functions

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-06-02 21:40:58 +01:00
ledgerwatch
c5a10975ab
[erigon2] Introduce inverted index type (#473)
* [erigon2] Introduce inverted index type

* More inverted index code

* More tests for inverted index

* Think about public and non-public APIs

* Minimise DB access when accessing history

* Work on iterator

* Implementation of inverted iterator

* Test for inverted index

* Assert end of iterators

* Merge of inverted index files and test

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-05-31 18:42:04 +01:00
Alex Sharov
7e00e77ae5
match roaring version with erigon #475 2022-05-30 09:17:47 +07:00
Alex Sharov
b995bd7540
recsplit: configurable etl limit #474 2022-05-30 09:06:11 +07:00
ledgerwatch
990e586823
[erigon2] Continuation of work on domains - merge static files (#466)
* Iteration over files - initial

* Fix interator for multistep

* Add function

* More functions for merge

* Merge files

* More work on the merge

* Fix buildIndex

* Fix history test for test of not completely pruned db

* Prepare for merge test

* Merge file test

* Close files

* Move functions into separate file

* Print

* fix for closing index

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp.lan>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-29 19:57:09 +01:00
Artem Tsebrovskiy
56a6afc8c9
commitment: fixed hash evaluation for hex patricia trie (#471) 2022-05-28 07:52:24 +01:00
Artem Tsebrovskiy
49e3522a05
added print of decompressed file at panic (#468)
* added print of decompressed file at panic

* more info for recovered decompressing
2022-05-27 08:20:53 +07:00
Andrew Ashikhmin
28d6b37a22
Remove INVALID_TERMINAL_BLOCK (#470)
* Remove interfaces for replacement

* Squashed 'interfaces/' content from commit 981f4ba

git-subtree-dir: interfaces
git-subtree-split: 981f4bae6619ab7100c7222852fd4eae3f837ad4

* regenerate go proto
2022-05-26 17:01:43 +02:00
battlmonstr
8c81bd094c
sentry: grpc.ClientStream.RecvMsg for direct sentry streams (#469) 2022-05-26 20:53:16 +07:00
battlmonstr
da10d41489
sentry: PeerById method (#467) 2022-05-25 19:50:19 +07:00
Alex Sharov
dd8136cde9
save (#465) 2022-05-24 19:00:09 +01:00
ledgerwatch
37d9944da9
[erigon2] State domains (move functionality out of aggregator) (#436)
* Domain

* First functions

* change year

* More on domain

* More to test

* More on test

* More on domains

* buildFiles

* More on domains

* Collation test

* Fix collate

* Add test for decompressors

* Restructure history tables

* Split history into 2 tables

* Fix lint

* Check index files in the test

* Close files

* Add file scanning

* Fix lint

* Fix lint

* Add readFromFiles

* Add ef history idx file

* Start cleanup

* More to cleanup, test for ef history

* More test

* Add prune to test

* Test for prune and fix

* Start history access

* History test

* Test for LastDup

* Fix one lint

* Workaround

* History tests

* Debug

* Fix

* Fix in history

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-macbook-pro.home>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alex Sharp <alexsharp@alexs-mbp.lan>
2022-05-24 18:59:57 +01:00
Alex Sharov
9821dcdff5
kv_mdbx: atomic closed flag (#464) 2022-05-24 12:20:19 +07:00
Alex Sharov
2bfb6d70bb
a bit more docs to etl #463 2022-05-24 11:37:27 +07:00
Alex Sharov
26e4d01d16
save (#462) 2022-05-23 14:47:49 +07:00
ledgerwatch
0497aa6aa2
ExtractEndKey should be exclusive bound in etl (#461)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-05-22 17:26:39 +01:00
Artem Vorotnikov
15d2a37c0b
Typify commitment (#454) 2022-05-20 15:01:44 +07:00
Artem Tsebrovskiy
6de4ac4ba9
reduced memory footprint on building huffman table (#459) 2022-05-20 11:23:05 +07:00
Artem Tsebrovskiy
59d810650c
[erigon2] cleanup obsolete code for bin-commitment-trie (#451)
* cleanup obsolete code for bin-commitment-trie

* renamed back test functions for hex, renamed test file for bin trie for consistency
2022-05-19 21:18:03 +01:00
Alex Sharov
7908982ed9
MatchPrefix: limit 2nd loop iterations (#458)
* sf

* sf

* save
2022-05-19 12:27:36 +07:00