25 Commits

Author	SHA1	Message	Date
primal_concrete_sledge	d8a33270e8	issue/issue-249-add_index_reader (#273) ... * issue/issue-249-add_index_reader * Add licence	2022-01-24 20:39:04 +00:00
Alex Sharov	79eb27d3f1	Helper to prohibit cli flag changes (#262)	2022-01-22 10:48:22 +07:00
ledgerwatch	e5c07ec901	[erigon2] Resumable prototype (#236) ... * Not warn about files that don't match at all * Not warn about files that don't match at all * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Add trace * Cleanup * Cleanup * Cleanup * Cleanup * Cleanup * Cleanup * Cleanup * Cleanup Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>	2022-01-15 22:09:06 +00:00
alex.sharov	8a0d41693f	create idx in tmpdir	2022-01-07 14:38:38 +07:00
alex.sharov	6aa0a5f08e	create idx in tmpdir	2022-01-07 14:37:27 +07:00
alex.sharov	68b0fe6030	create dir automatically	2022-01-07 14:27:26 +07:00
Alex Sharov	bb3f510d16	RecSplit: store BaseDataID in .idx file (helps to navigate over non blockNum-based entries) (#180) ... * save * save * save * save	2021-11-21 14:52:23 +00:00
Alex Sharov	bb1d712834	Hack: dump bodies and headers (#177) ... * save * save * save * save	2021-11-19 22:00:55 +07:00
Alex Sharov	5b7f67deae	Snapshot naming (#163) ... * save * save * save * save * save * save	2021-11-15 14:19:56 +00:00
ledgerwatch	fd19ad8148	State aggregator (#114) ... * State aggregator * Compile fix * More * Add * More * More on aggregator * Writes (still incorrect) * Move table names * More * Start of aggregation * Change files instead of db * More on change files * More * More * Dealing with state and change files * More * More * More boilerplate * More * More * Iteration over storage * More boilerplate * More fixes * Insert flag * More * Unit test * Add more to the test * Expand the test a bit * More testing * Keep fixing the test * More fixes to the test * Clean up DB tables upon aggregation * More fixes * Remove update/insert indicator from returned values * Add assertions * close files before deleting * close files before deleting * close files before deleting Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>	2021-11-13 12:12:29 +00:00
Alex Sharov	d79f87a0e9	Recsplit: single offset bucket (#152)	2021-11-08 14:27:21 +07:00
alex.sharov	377bc94675	save	2021-11-08 07:59:18 +07:00
Alex Sharov	f6b0a0c969	Recsplit: collision typed error (#150)	2021-11-07 09:54:48 +07:00
Alex Sharov	3c86aa6290	ETL: use logPrefix as suffix of tmp files (#146)	2021-11-05 17:04:17 +07:00
Alex Sharov	47f8ac208a	EliasFano: fix jump calculation, fuzzing to trigger jump logic (#145)	2021-11-04 13:25:23 +07:00
Alex Sharov	b50cb37fa8	Recsplit: call ef.Build and set ef.prevOffset (#140)	2021-11-01 09:23:38 +07:00
Alex Sharov	8213f020f0	recsplit add MustOpen method (#138)	2021-10-31 09:38:10 +07:00
Alex Sharov	78e3f747f4	recsplit: bigger bufio buffer (#129)	2021-10-26 11:19:26 +07:00
Alex Sharov	ba51a5966a	etl.collector - move logPrefix to constructor (#128)	2021-10-25 09:12:00 +07:00
Alex Sharov	70c39cd195	Pool intrinsic gas check (#126)	2021-10-25 08:49:04 +07:00
ledgerwatch	967937151d	Fixes for compress, decompressor, and tests (#110) ... * Fixes for compress, and first test * Add decompressor and memory mapping * Add decompressor and memory mapping * Fix for windows * Fix lint * Fix compile for windows * More on decompressor * Fix lint * Decompress * Fix lint * Use decompressor in tests, fixes * Introduce Index for RecSplit * Fix compilation on Windows * close index file on failure * Fixes to the tests * Add single Elias Fano, fix recsplit fuzz test * Fix elias fano * Add two layer index * Add two level index to the tests Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local> Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>	2021-10-16 10:43:41 +01:00
ledgerwatch	47490aa942	Optimise RecSplit (#82) ... * not allocate count * Print timings * More time measurement * See time with fanout=2 * Less branching? * Revert * Split functions for fingeprint and bucket separation * Save indices * Fix limits * Use original split formula * Revert * uint16 * Correctly measure 2 * Less branching again? * No time measurements * Cleanup * Fix lint Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>	2021-09-20 17:39:32 +01:00
ledgerwatch	f2549ad6ec	Integration of recsplit (#79) ... * Integration of recsplit * Add tables * Print bucket by bucket * Not to print all keys * Print correct bitSize * switch to []byte * Optimisation * Fix * Fix lint * Performance improvements * Print bucket info * Add tracing * Fixed split Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>	2021-09-20 12:14:49 +01:00
ledgerwatch	312d43aa88	Recsplit encoding (#69) ... * Recsplit encoding * Added Golomb-Rice encoding * More on encoding * More * Fix compile errors * Fix fuzz test, add corpus * Integrated Elias-Fano * Fix lint * Add select64 * More * Add fuzz test for elias fano * Debugging elias fano * Fuzz test for elias fano * More elias fano debugging * Fix elias fano * More fixes * Fix to golombRiceLength Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>	2021-09-18 22:59:27 +01:00
ledgerwatch	6dce34ac32	Initial recsplit (#67) ... * Initial recsplit * Move licence * Fix bucket count and key count * Check for duplicate kwys * More recsplit implementation * Skeleton of recsplit, fuzz test * Fix lint Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local> Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>	2021-09-13 18:31:09 +01:00