Evgeny Danilenko
ec6be820ac
Don't force sync in case InMemDb ( #1174 )
...
* initial
* fix
* fix chain length
2020-10-02 14:08:28 +01:00
Alex Sharov
48cf5fb43e
History-style sharding for bitmap indices ( #1159 )
2020-10-02 10:54:11 +07:00
Alex Sharov
e02d6acc7d
bitmap indices for logs ( #1124 )
...
* save progress
* try now
* don't create bloom inside rlpDecode
* don't create bloom inside ApplyTransaction
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* rename method
* print timings
* print timings
* print timings
* sort before flush
* fix err lint
* clean
* move tests to transactions
* compressed version
* up bound
* up bound
* more tests
* more tests
* more tests
* more tests
* better removal
* clean
* better performance of get/put methods
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* optimize rpcdaemon
* fix test
* fix rpcdaemon
* fix test
* simplify
* simplify
* fix nil pointer
* clean
* revert some changes
* add some logs
* clean
* try without optimize
* clean
* clean
* clean
* clean
* try
* move log_index to own stage
* move log_index to own stage
* integration add log_index stage
* integration add log_index stage
* clean
* clean
* print timing
* remove duplicates at unwind
* extract truncateBitmaps func
* try detect
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* add blackList of topics
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* sharding 1
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 3
* sharded 3
* sharded 3
* speedup things by putCurrent and putReserve
* clean
* optimize trim
* clean
* remove blacklist
* add more info to err
* ?
* clean
* clean
* clean
* clean
* clean
* working version
* switch to cgo version of roaring bitmaps
* clean
* clean
* clean
* clean
* more docs
* clean
* clean
* fix logs bloom field
* Fix debug_getModifiedAccountsByNumber
* Try to fix crash
* fix problem with "absent block"
* fix problem with "absent block"
* remove optimize method call
* remove roaring iterator
* fix problem with rebuild indicess
* remove debug prints
* tests for eth_getLogs involving topics
* add tests for new stage, speparate topics into 2 buckets
* version up
* remove debug logs
* remove debug logs
* remove bloom filter implementation
* Optimisation
* Optimisatin not required, make rpctest lenient to geth errors
* Lenient to geth failures
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Alex Sharov
e2acbfecda
try to remove sleep ( #1126 )
2020-09-23 12:10:13 +01:00
Alex Sharov
b76661cd25
Revert "integration_write_receipts ( #1096 )" ( #1097 )
...
This reverts commit 7a1285bf3c
.
2020-09-11 15:32:45 +07:00
Alex Sharov
7a1285bf3c
integration_write_receipts ( #1096 )
2020-09-11 15:28:28 +07:00
Alex Sharov
62fe81e4be
IH stage speedup and lmdb custom comparators support ( #1080 )
...
* etl.Loader - allow use of custom comparator
* log timing
* try now
* try now
* more performance
* etl.Loader - allow use of custom comparator
* working version
* simplify IH cursor
* clean
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* clean
* add only unwind support
* squash
* squash
* clean
* fix test
* clean
* clean
* clean
2020-09-10 13:35:58 +01:00
b00ris
bdb7832a59
[lmdb] Stop force sync for ReadOnly mode ( #1088 )
...
* fix warning
* fix log
2020-09-09 21:07:19 +01:00
Alex Sharov
c45a710ce6
Warmup logs, less overhead, warmup code bucket ( #1054 )
...
* warmup logs and less overhead.
* warmup logs and less overhead.
* move WarmUp to common func
2020-09-08 08:28:37 +01:00
b00ris
23e995838f
Fix readonly mode ( #1077 )
...
* fix readonly mode
* fix vet
2020-09-08 07:19:06 +01:00
Alex Sharov
b9bdd003a7
KV: Raw (no custom logic) cursors for DupSort and DupFixed buckets ( #1020 )
2020-09-04 10:54:15 +07:00
Alex Sharov
e02f2f6dfc
Revert "simplify cursor.Next and cursor.First implementation ( #1019 )" ( #1039 )
...
This reverts commit f19bb0a345
.
2020-09-03 13:21:11 +07:00
Alex Sharov
f19bb0a345
simplify cursor.Next and cursor.First implementation ( #1019 )
2020-09-01 12:50:23 +01:00
Alex Sharov
49351b0091
KV: to support multiple databases and multiple set of buckets ( #1002 )
...
* don't use global variables
* up deps
* clean
* use joined flags
* use single non-global buckets config
* clean
* copy map
* fix tests
* fix tests
* clean
* clean
2020-08-30 18:34:18 +01:00
Alex Sharov
df05a581dd
Add tx pool stage to transaction ( #981 )
...
* add tx_pool stage to tx
* clean
* don't call commit in read tx
* don't call commit in read tx
* clean
* more logs
* fix tx pool race
* more logs
* clean
2020-08-28 07:27:48 +01:00
Alex Sharov
714e5041e2
Better logs3 ( #976 )
2020-08-26 19:06:51 +07:00
Alex Sharov
9091ce62bb
KV reduce features amount ( #975 )
...
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize
* remove bolt_db.go file
* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
Alex Sharov
522287ac18
Transactional cycle ( #966 )
...
* v0
* v1
* v3
* v4
* clean
* temporary fix of txpool
* Add debug logs about tx start/commit
* Add debug logs about tx start/commit
* Add debug logs about tx start/commit
* add condition
* tx pool to not hold own db
* try to enable TxPool in integration
* exclude tx pool from tx
* exclude tx pool from integration
* reduce limit
* fix integration
* clean
* clean
* clean
* clean
* clean
* exclude tx pool unwind
* exclude tx pool unwind in integration
* fix integration tx pool
* fix commit
* fix current stage after unwind
* fix current stage after unwind
* fix linter
* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.
* move body and headers unwind out of tx
* fix unwind order after reboot
* add support external tx to exec stage
* clean
* clean
* clean
* clean
* clean
* add logs
* better id check
* better id check
2020-08-26 07:02:10 +01:00
Alex Sharov
43a4e34a6e
[to discuss] exec blocks without ObjectDb: Mutation over TxDb ( #947 )
...
* use mutation over tx
* clean
* clear
* add .CommitAndBegin() method
* clean
* increase timings for logging
* return ideal batch size
2020-08-24 12:07:59 +01:00
Alex Sharov
ebbaa55672
sub-transaction to not call fsync, sub-transactions to not call runtime.LockThread() ( #945 )
2020-08-21 07:31:43 +01:00
b00ris
eb46dd68df
Account range testing ( #938 )
...
* save state
* save state
* save new timeouts
* compare passed
* save state
* cleanup
2020-08-19 07:33:49 +01:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction ( #928 )
...
* drop Bucket object from KV abstraction
* add some non-public methods
* rebase master
* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
0e253e7336
lmdb transactions of unlimited size ( #918 )
...
* add logging to loader
* use pure tx in etl loading, logs in mutation commit
* clean
* bletter logging and more cleanup
* bletter logging and more cleanup
* increase batch size to 500M
* better batch commit logging
* async fsync
* sync fsync
* sync fsync
* unify logging
* fix corner-case when etl can use empty bucket name
* fix tests
* better logging
* better logging
* rebase master
* remove lmdb.NoMetaSync flag for now
* consistent walk and multi-walk
* clean
* sub tx
* add consistent multi-put
* implement dupsort support in one new cursor method
* clear
2020-08-17 07:45:52 +01:00
Alex Sharov
bf596c26f3
[merge after release] dupsort of plain state ( #913 )
...
* dupsort of plain state
* rebase master
2020-08-15 08:11:40 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object ( #914 )
2020-08-14 13:41:18 +07:00
Alex Sharov
16f09be0a8
add method .Last() ( #909 )
2020-08-12 10:49:52 +07:00
Alex Sharov
760fe0f0b6
DupSort of hash state ( #896 )
2020-08-12 09:57:55 +07:00
Alex Sharov
6de7eb1bec
correct_thread_lock_on_non_managed_tx ( #905 )
2020-08-11 18:53:47 +07:00
Alex Sharov
7addf34f64
fix wrong merge ( #907 )
2020-08-11 18:24:39 +07:00
Alex Sharov
e89ad5d6aa
sub tx support ( #895 )
2020-08-11 17:35:59 +07:00
Alex Sharov
695f8f8904
allow multi-rollback ( #904 )
2020-08-11 17:35:19 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string ( #894 )
2020-08-11 06:55:32 +07:00
Alex Sharov
7a1b892022
remove .SeekTo() method ( #901 )
2020-08-11 06:41:15 +07:00
Igor Mandrigin
8600616c3d
geth 1.9.18 post-rebase fixups
2020-08-08 17:33:35 +02:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key ( #868 )
2020-08-05 17:13:35 +07:00
Igor Mandrigin
e732ccd438
add lmdb.mapSize param
2020-08-05 10:17:50 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key ( #858 )" ( #867 )
...
This reverts commit 7290bf519e
.
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key ( #858 )
2020-08-04 16:25:28 +07:00
Alex Sharov
a5c1089d58
optimize .get ( #863 )
2020-08-04 10:31:37 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) ( #854 )
...
* return error from rawdb
* squash
* v14
* improve performance of put
* clean
* clean
* hide feature behind ENV variable
* cleanup
* cleanup
* disable ipc and make Readme less confusing (people thought points are depend on each-other)
* fix test
* cleanup
* cleanup
2020-08-02 12:53:08 +01:00
Alex Sharov
9a5c2a1242
reduce mmap limit size (raspbery-pi failed on 4TB) ( #840 )
2020-08-01 11:59:40 +07:00
Alex Sharov
6c4ec563e1
fix - change private api from ui ( #829 )
2020-07-30 11:17:09 +01:00
Alex Sharov
0a367001d1
implement db stats page ( #803 )
2020-07-29 05:31:46 +01:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb ( #788 )
2020-07-27 19:15:48 +07:00
Alex Sharov
79a49e9e44
avoid touching closed db ( #789 )
...
* avoid touching closed db
* fix tx rollback
2020-07-27 07:10:50 +01:00
Alex Sharov
b2bcbaa895
Compact flag ( #783 )
...
* compact flag
* compact flag
* compact flag
* compact flag
* compact flag
2020-07-25 08:35:08 +01:00
Alex Sharov
7cc0b7e391
On mac lmdb can't start with error "cannot allocate memory" until I not reduced magic number ( #779 )
2020-07-24 14:47:03 +01:00
Alex Sharov
1a075b10f1
re-create bucket on b.Clear - to easier change bucket settings (for example by releasing new version or just by switching git branches) ( #766 )
2020-07-21 15:33:30 +07:00
Alex Sharov
621e3a2074
Ci lmdb - reduce memory usage ( #762 )
2020-07-21 08:58:00 +07:00
Alex Sharov
7f60d1b535
Remove load start key from etl ( #757 )
...
* limit incremental step size of stages 5 and 6
* remove limit, make incremental
* remove_load_start_key_from_etl
* switch to "struct of arrays" from "array of structs"
* switch to "struct of arrays" from "array of structs"
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-18 10:17:04 +01:00