Alex Sharov
6ccf57dace
switch to go implementation of roaring bitmaps for alpine support ( #1165 )
2020-10-02 15:16:21 +07:00
Alex Sharov
61fbe34024
Better cbor support ( #1163 )
2020-10-02 10:56:13 +07:00
Alex Sharov
48cf5fb43e
History-style sharding for bitmap indices ( #1159 )
2020-10-02 10:54:11 +07:00
Alex Sharov
0559d56dae
Pin protoc version and store it in ./build/bin ( #1154 )
...
* use ./build/bin folder to store protobuf binaries
* use ./build/bin folder to store protobuf binaries
2020-09-29 15:09:50 +01:00
ledgerwatch
7bb9cb4274
Remove BloomStatus from the API ( #1155 )
...
* Remove BloomStatus from the API
* Trying to find missing space
* Fix lint
2020-09-29 15:09:29 +01:00
Alex Sharov
e02d6acc7d
bitmap indices for logs ( #1124 )
...
* save progress
* try now
* don't create bloom inside rlpDecode
* don't create bloom inside ApplyTransaction
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* rename method
* print timings
* print timings
* print timings
* sort before flush
* fix err lint
* clean
* move tests to transactions
* compressed version
* up bound
* up bound
* more tests
* more tests
* more tests
* more tests
* better removal
* clean
* better performance of get/put methods
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* optimize rpcdaemon
* fix test
* fix rpcdaemon
* fix test
* simplify
* simplify
* fix nil pointer
* clean
* revert some changes
* add some logs
* clean
* try without optimize
* clean
* clean
* clean
* clean
* try
* move log_index to own stage
* move log_index to own stage
* integration add log_index stage
* integration add log_index stage
* clean
* clean
* print timing
* remove duplicates at unwind
* extract truncateBitmaps func
* try detect
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* add blackList of topics
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* clean
* sharding 1
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 2
* sharded 3
* sharded 3
* sharded 3
* speedup things by putCurrent and putReserve
* clean
* optimize trim
* clean
* remove blacklist
* add more info to err
* ?
* clean
* clean
* clean
* clean
* clean
* working version
* switch to cgo version of roaring bitmaps
* clean
* clean
* clean
* clean
* more docs
* clean
* clean
* fix logs bloom field
* Fix debug_getModifiedAccountsByNumber
* Try to fix crash
* fix problem with "absent block"
* fix problem with "absent block"
* remove optimize method call
* remove roaring iterator
* fix problem with rebuild indicess
* remove debug prints
* tests for eth_getLogs involving topics
* add tests for new stage, speparate topics into 2 buckets
* version up
* remove debug logs
* remove debug logs
* remove bloom filter implementation
* Optimisation
* Optimisatin not required, make rpctest lenient to geth errors
* Lenient to geth failures
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-28 18:18:36 +01:00
Alex Sharov
e2acbfecda
try to remove sleep ( #1126 )
2020-09-23 12:10:13 +01:00
Giulio rebuffo
d33302c2be
Mutual TLS authentication (for now without verifying the Common Name) ( #1113 )
...
* added tls auth
* added client side
* put --tls
* fixed flag
* Add key/cert generation instructions, turn off Common Name verification
* Add CLI arguments and Warning
* Lint
* Update the doc about Internal IP
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-19 15:16:04 +01:00
ledgerwatch
0f24a01677
Close the ticker ( #1104 )
2020-09-11 21:44:24 +01:00
Giulio rebuffo
23d89c0941
Added TLS handshake to RPCDaemon ( #1089 )
...
* added eth_getStorageAt
* used uint32
* now its 256
* incarnation
* added TLS handshake
* code minified
* lint
* minified client
* Update flags.go
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-09-11 21:17:37 +01:00
Alex Sharov
b76661cd25
Revert "integration_write_receipts ( #1096 )" ( #1097 )
...
This reverts commit 7a1285bf3c
2020-09-11 15:32:45 +07:00
Alex Sharov
7a1285bf3c
integration_write_receipts ( #1096 )
2020-09-11 15:28:28 +07:00
Alex Sharov
62fe81e4be
IH stage speedup and lmdb custom comparators support ( #1080 )
...
* etl.Loader - allow use of custom comparator
* log timing
* try now
* try now
* more performance
* etl.Loader - allow use of custom comparator
* working version
* simplify IH cursor
* clean
* squash
* squash
* squash
* squash
* squash
* squash
* squash
* clean
* add only unwind support
* squash
* squash
* clean
* fix test
* clean
* clean
* clean
2020-09-10 13:35:58 +01:00
b00ris
bdb7832a59
[lmdb] Stop force sync for ReadOnly mode ( #1088 )
...
* fix warning
* fix log
2020-09-09 21:07:19 +01:00
Alex Sharov
fd9c7c3b2d
kill bolt ( #1085 )
...
* kill bolt
* kill bolt
* fix test
* clean
2020-09-09 11:09:55 +01:00
Alex Sharov
da31494cc3
transactional migrations ( #1079 )
2020-09-08 20:39:43 +01:00
Alex Sharov
c45a710ce6
Warmup logs, less overhead, warmup code bucket ( #1054 )
...
* warmup logs and less overhead.
* warmup logs and less overhead.
* move WarmUp to common func
2020-09-08 08:28:37 +01:00
b00ris
23e995838f
Fix readonly mode ( #1077 )
...
* fix readonly mode
* fix vet
2020-09-08 07:19:06 +01:00
Alex Sharov
afe808c88e
time based reopen tx ( #1069 )
2020-09-07 12:47:08 +07:00
ledgerwatch
3a92b2b39d
Fix for RPC daemon leak ( #1059 )
...
* Start memory prof
* Fix rpctest
* Fix rpctest
* Attempt to fix the leak
* Remove http pprof
2020-09-05 21:58:51 +01:00
Alex Sharov
e690103c2a
Docs for Cursors and KV ( #1046 )
2020-09-05 14:26:24 +07:00
Alex Sharov
7e71c4512e
restrict metric by 1 bucket ( #1052 )
2020-09-05 13:09:47 +07:00
Alex Sharov
95e985ef8f
add commit event to dashboard ( #1050 )
2020-09-05 10:10:17 +07:00
Alex Sharov
0c75304261
Os page cache monitoring ( #1049 )
2020-09-05 09:36:42 +07:00
Alex Sharov
b9bdd003a7
KV: Raw (no custom logic) cursors for DupSort and DupFixed buckets ( #1020 )
2020-09-04 10:54:15 +07:00
Evgeny Danilenko
e4f495fa44
Get logs ( #1028 )
...
* it compiles
* after recent master
* fix linters warnings
* grpcV7
* go mod tidy
* unmarshall adresses or adress
* fix linters
* after cr
* after cr
* after cr
* after cr
* fix tests
* remove dev version
* it compiles
* mod tidy
* fix bin deps
* use stable version of grpc
* switch back to master constructor
* switch back to master constructor
* add a bit docs
* add a bit docs
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
2020-09-03 08:51:19 +01:00
Alex Sharov
e02f2f6dfc
Revert "simplify cursor.Next and cursor.First implementation ( #1019 )" ( #1039 )
...
This reverts commit f19bb0a345
2020-09-03 13:21:11 +07:00
Alex Sharov
f19bb0a345
simplify cursor.Next and cursor.First implementation ( #1019 )
2020-09-01 12:50:23 +01:00
ledgerwatch
c06dbdadf9
Use Append for changesets and receipts ( #1018 )
...
* Use Append for changesets and receipts
* Fix lint
* Delete all receipts on unwind
* Fix lint
* Fix unwind keys for changeset
* Fix test
* Use writeReceipts flag
* Revert to Append
* Delete receipts too when resetting
2020-09-01 07:48:25 +01:00
Alex Sharov
49351b0091
KV: to support multiple databases and multiple set of buckets ( #1002 )
...
* don't use global variables
* up deps
* clean
* use joined flags
* use single non-global buckets config
* clean
* copy map
* fix tests
* fix tests
* clean
* clean
2020-08-30 18:34:18 +01:00
Alex Sharov
5ce73d438a
fix integration ( #999 )
2020-08-29 15:27:20 +07:00
Alex Sharov
07ca5c643a
eth_syncing ( #991 )
...
* eth_syncing
* linters
* fix readme
* cleanup cli params
* go mod tidy
* remove memsize ui dep
* remove bloomfilter
* implement net_peerCount
* remove legacy flags
2020-08-29 08:24:50 +01:00
Alex Sharov
df05a581dd
Add tx pool stage to transaction ( #981 )
...
* add tx_pool stage to tx
* clean
* don't call commit in read tx
* don't call commit in read tx
* clean
* more logs
* fix tx pool race
* more logs
* clean
2020-08-28 07:27:48 +01:00
Alex Sharov
714e5041e2
Better logs3 ( #976 )
2020-08-26 19:06:51 +07:00
Alex Sharov
9091ce62bb
KV reduce features amount ( #975 )
...
* move IdealBatchSize method to Batch object :-)
remove cursor.Walk
remove kv.IdealBatchSize
* remove bolt_db.go file
* move method NoValueCursor to tx object
2020-08-26 07:03:50 +01:00
Alex Sharov
522287ac18
Transactional cycle ( #966 )
...
* v0
* v1
* v3
* v4
* clean
* temporary fix of txpool
* Add debug logs about tx start/commit
* Add debug logs about tx start/commit
* Add debug logs about tx start/commit
* add condition
* tx pool to not hold own db
* try to enable TxPool in integration
* exclude tx pool from tx
* exclude tx pool from integration
* reduce limit
* fix integration
* clean
* clean
* clean
* clean
* clean
* exclude tx pool unwind
* exclude tx pool unwind in integration
* fix integration tx pool
* fix commit
* fix current stage after unwind
* fix current stage after unwind
* fix linter
* move unwind of tx_pool after unwind of unwind of senders, then all stages from body to tx_pool will be inside tx.
* move body and headers unwind out of tx
* fix unwind order after reboot
* add support external tx to exec stage
* clean
* clean
* clean
* clean
* clean
* add logs
* better id check
* better id check
2020-08-26 07:02:10 +01:00
ledgerwatch
8728e54a72
Another step in migrating tests to staged sync ( #970 )
...
* Log in GetAsOf
* Default plain state
* trace
* More test fixes
* Fix lint
* Fix
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix for events
* Repair events
* Fix lint
* Fix lint
* Change back to InsertChain
2020-08-24 18:00:40 +01:00
Alex Sharov
43a4e34a6e
[to discuss] exec blocks without ObjectDb: Mutation over TxDb ( #947 )
...
* use mutation over tx
* clean
* clear
* add .CommitAndBegin() method
* clean
* increase timings for logging
* return ideal batch size
2020-08-24 12:07:59 +01:00
Alex Sharov
a6be18b915
ticker-based logs ( #954 )
...
* timer-based logs
* timer-based logs
* delegate progress calculation to user
* delegate progress calculation to user
* delegate progress calculation to user
* clear
* add logs to senders recovery
* use default dir in integration
* more logs
* more logs
2020-08-22 12:12:33 +02:00
Alex Sharov
ebbaa55672
sub-transaction to not call fsync, sub-transactions to not call runtime.LockThread() ( #945 )
2020-08-21 07:31:43 +01:00
ledgerwatch
01f211f3b8
Revert "IH: no retain list ( #946 )" ( #948 )
...
This reverts commit f86ed64288
2020-08-20 13:37:34 +01:00
Alex Sharov
f86ed64288
IH: no retain list ( #946 )
...
* experiment of not using retain list
* better handle sub-tx
* save progress
* don't touch collector
* don't run collector on sub-tx because of `mdb_cursor_put: MDB_TXN_FULL: Transaction has too many dirty pages - transaction too big`
* clear
* clear
2020-08-20 11:42:43 +01:00
b00ris
eb46dd68df
Account range testing ( #938 )
...
* save state
* save state
* save new timeouts
* compare passed
* save state
* cleanup
2020-08-19 07:33:49 +01:00
Evgeny Danilenko
3b40819444
grpc regenerate ( #936 )
...
* grpc regenerate
* use string type for id in net_version
2020-08-18 20:22:49 +03:00
Alex Sharov
081d02920d
Remove "Bucket" class from KV abstraction ( #928 )
...
* drop Bucket object from KV abstraction
* add some non-public methods
* rebase master
* rebase master
2020-08-17 08:24:59 +01:00
Alex Sharov
b2e35eb63f
rename "remote2" to "remote" ( #932 )
2020-08-17 07:47:06 +01:00
Alex Sharov
0e253e7336
lmdb transactions of unlimited size ( #918 )
...
* add logging to loader
* use pure tx in etl loading, logs in mutation commit
* clean
* bletter logging and more cleanup
* bletter logging and more cleanup
* increase batch size to 500M
* better batch commit logging
* async fsync
* sync fsync
* sync fsync
* unify logging
* fix corner-case when etl can use empty bucket name
* fix tests
* better logging
* better logging
* rebase master
* remove lmdb.NoMetaSync flag for now
* consistent walk and multi-walk
* clean
* sub tx
* add consistent multi-put
* implement dupsort support in one new cursor method
* clear
2020-08-17 07:45:52 +01:00
Igor Mandrigin
d848233275
Merge branch 'master' into geth-1.9.19
...
# Conflicts:
# cmd/geth/consolecmd.go
# cmd/rpcdaemon/service/service.go
# cmd/utils/flags.go
# eth/backend.go
2020-08-15 21:55:09 +02:00
Igor Mandrigin
201524c33e
fix node tests
2020-08-15 20:21:30 +02:00
Alex Sharov
bf596c26f3
[merge after release] dupsort of plain state ( #913 )
...
* dupsort of plain state
* rebase master
2020-08-15 08:11:40 +01:00
Alex Sharov
e04c4ebe58
KV: move bucket.Get and bucket.Cursor methods to Tx object ( #914 )
2020-08-14 13:41:18 +07:00
Alex Sharov
f7deef79bb
drop bucket log message ( #917 )
2020-08-13 13:40:28 +07:00
Evgeny Danilenko
5d39fe8873
RPCDeamon coinbase, net_version ( #911 )
...
* use eth.backend
* net_version
* add coinbase to the console test
2020-08-12 14:47:59 +01:00
Alex Sharov
16f09be0a8
add method .Last() ( #909 )
2020-08-12 10:49:52 +07:00
Alex Sharov
760fe0f0b6
DupSort of hash state ( #896 )
2020-08-12 09:57:55 +07:00
Evgeny Danilenko
3980fa7d45
Grps eth_sendTransaction ( #882 )
...
* implementation
* tidy gomod
* linters
* fix cmd test
* fix
* fix lint
2020-08-11 22:09:30 +01:00
Alex Sharov
6de7eb1bec
correct_thread_lock_on_non_managed_tx ( #905 )
2020-08-11 18:53:47 +07:00
Alex Sharov
7addf34f64
fix wrong merge ( #907 )
2020-08-11 18:24:39 +07:00
Alex Sharov
e89ad5d6aa
sub tx support ( #895 )
2020-08-11 17:35:59 +07:00
Alex Sharov
695f8f8904
allow multi-rollback ( #904 )
2020-08-11 17:35:19 +07:00
Alex Sharov
d9d9e14f45
change bucket type to string ( #894 )
2020-08-11 06:55:32 +07:00
Alex Sharov
7a1b892022
remove .SeekTo() method ( #901 )
2020-08-11 06:41:15 +07:00
Alex Sharov
091819a51c
adopt --metrics.addr flag in integration ( #889 )
2020-08-11 06:38:34 +07:00
Igor Mandrigin
8600616c3d
geth 1.9.18 post-rebase fixups
2020-08-08 17:33:35 +02:00
Alex Sharov
f41959f030
drop badger support ( #869 )
...
* drop badger support
* rebase master
2020-08-05 16:33:45 +01:00
Alex Sharov
d458c4cc1e
Migrations: use stage name as db key ( #868 )
2020-08-05 17:13:35 +07:00
Igor Mandrigin
e732ccd438
add lmdb.mapSize param
2020-08-05 10:17:50 +02:00
ledgerwatch
0514501937
Revert "Use stage name as db key ( #858 )" ( #867 )
...
This reverts commit 7290bf519e
2020-08-04 11:05:27 +01:00
Alex Sharov
7290bf519e
Use stage name as db key ( #858 )
2020-08-04 16:25:28 +07:00
Alex Sharov
a5c1089d58
optimize .get ( #863 )
2020-08-04 10:31:37 +07:00
Alex Sharov
ecc94a63f0
DupSort for Plain and Hashed state buckets (behind feature flags) ( #854 )
...
* return error from rawdb
* squash
* v14
* improve performance of put
* clean
* clean
* hide feature behind ENV variable
* cleanup
* cleanup
* disable ipc and make Readme less confusing (people thought points are depend on each-other)
* fix test
* cleanup
* cleanup
2020-08-02 12:53:08 +01:00
b00ris
a84230d45c
fix streams leak ( #847 )
2020-08-01 16:18:31 +01:00
Alex Sharov
9a5c2a1242
reduce mmap limit size (raspbery-pi failed on 4TB) ( #840 )
2020-08-01 11:59:40 +07:00
Alex Sharov
6c4ec563e1
fix - change private api from ui ( #829 )
2020-07-30 11:17:09 +01:00
Alex Sharov
b6399a2256
better error messages ( #824 )
2020-07-30 13:55:40 +07:00
ledgerwatch
7bc45fab80
Remove preimage storage mode, write receipts by default ( #809 )
2020-07-29 14:00:39 +01:00
Alex Sharov
0a367001d1
implement db stats page ( #803 )
2020-07-29 05:31:46 +01:00
Alex Sharov
47b7b2cf40
lmdb_is_not_friendly_to_empty_values ( #795 )
2020-07-29 05:30:29 +01:00
Alex Sharov
f500148b86
remove BoltDatabase, BadgerDatabase, RemoteDatabase classes (replaced by objectDb) ( #794 )
2020-07-27 22:59:54 +01:00
Alex Sharov
c4a1ffa4f0
Grpc-based remoteDb ( #788 )
2020-07-27 19:15:48 +07:00
Alex Sharov
79a49e9e44
avoid touching closed db ( #789 )
...
* avoid touching closed db
* fix tx rollback
2020-07-27 07:10:50 +01:00
Alex Sharov
b2bcbaa895
Compact flag ( #783 )
...
* compact flag
* compact flag
* compact flag
* compact flag
* compact flag
2020-07-25 08:35:08 +01:00
Alex Sharov
7cc0b7e391
On mac lmdb can't start with error "cannot allocate memory" until I not reduced magic number ( #779 )
2020-07-24 14:47:03 +01:00
Alex Sharov
1a075b10f1
re-create bucket on b.Clear - to easier change bucket settings (for example by releasing new version or just by switching git branches) ( #766 )
2020-07-21 15:33:30 +07:00
Alex Sharov
e65b743878
Prometheus dashboard revisit ( #767 )
2020-07-21 15:33:03 +07:00
Alex Sharov
a7e5c4b453
Remove remote_chain package, fix rpc methods: eth_blockNumber and eth_getBlockByNumber ( #769 )
2020-07-21 15:19:04 +07:00
Alex Sharov
621e3a2074
Ci lmdb - reduce memory usage ( #762 )
2020-07-21 08:58:00 +07:00
Alex Sharov
1faa9dbcd4
add senders bucket to py ( #763 )
2020-07-20 16:11:47 +07:00
ledgerwatch
b765ff0e33
cmd/tester initiates staged sync, and unwind (but not reorg) ( #758 )
...
* First push
* Remove batch from Blockchain
* Fix compile error in cmd/integration
* Fix compile errors in cmd/state
* Fix compile errors
* Save changes
* Fix core tests (skip tx tests)
* More test fixes
* Fix compile error
* Fix formatting
* Fix formatting
* Fix lint
* Fixes
* Skip some miner tests
* Fix tests
* Cleanup
2020-07-19 09:11:53 +01:00
Alex Sharov
7f60d1b535
Remove load start key from etl ( #757 )
...
* limit incremental step size of stages 5 and 6
* remove limit, make incremental
* remove_load_start_key_from_etl
* switch to "struct of arrays" from "array of structs"
* switch to "struct of arrays" from "array of structs"
Co-authored-by: Alexey Akhunov <akhounov@gmail.com>
2020-07-18 10:17:04 +01:00
Alex Sharov
03e136c7b8
remove all pools from bolt, badger, lmdb ( #750 )
2020-07-18 09:39:18 +01:00
Alex Sharov
c8a594dec6
Limit incremental step size of stages 5 and 6. ( #754 )
2020-07-17 12:52:09 +07:00
ledgerwatch
887fc4f99c
GenerateChain without using TrieDbState ( #726 )
...
* Different chain generation
* More fixes
* Fix TestSideLogRebirth
* Fix the rest of core tests
* Fix accounts/abi/bind
* Fix compile errors for cmd/pics
* Fix consensus/clique tests
* More fixes
* Fix core/state tests
* Fix eth tests
* Fix eth/downloader tests
* Fix more tests
* Fix compile errors
* Some more fixes
* Save
* Exists
* Fix last tests
* Clean up
* Fix formatting
* Fix linter
* Fix formatting
* Fix linter
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
* Fix lint
2020-07-09 07:15:28 +01:00
b00ris
eca0f233ec
[WIP] WalkAsOf ( #724 )
...
* save state
* walkAsOf by cs test passed
* cs search for plain state
* save state
* fix accounts tests
* refactor walkAsOf account tests
* fix storage test
* refactor walkAsOf storage tests
* fix lint
* fix test
* save state
* save state
* add test with fixed bits
* fmt
* add stages check
* fix lint
* fix lint
* remove obsoleted methods
2020-07-09 07:13:45 +01:00
Alex Sharov
fb933bc9e3
move lmdb-go under ledgerwatch org ( #720 )
2020-07-07 16:18:09 +07:00
Alex Sharov
305f8aff02
Integration tests2 ( #714 )
...
```
go run ./cmd/integration reset --chaindata=...
go run ./cmd/integration state_stages -h
go run ./cmd/integration state_stages --chaindata=... --verbosity=3 --block=2_000_000 --unwind=10 --unwind_every=1_000
```
Also, it inherits flags from geth:
```
--pprof.cpuprofile=./cpu.out // to file
--pprof --pprof.port=6060 // launch pprof server
--metrics // sends to prometheus
```
2020-07-07 11:00:25 +07:00
Alex Sharov
2aadd94acb
remove mgr feature and stateSize calculation feature ( #711 )
2020-07-05 07:17:27 +01:00
ledgerwatch
49181a298b
Special bucket for transaction senders ( #707 )
...
* Senders in the separate bucket
* Pre-create txSenders bucket
* Fix compile error
* Fix compile error
* Fix compile error, remove block transformations
* Fix formatting
* Fix test
2020-07-03 08:23:49 +01:00
Alex Sharov
a7f5b1ca08
KV: defer friendly rollback ( #696 )
...
* AbstractKV: defer friendly rollback
* AbstractKV: defer friendly rollback
2020-06-30 04:48:46 +01:00
Alex Sharov
63c998c91f
Only useful IH cursor ( #694 )
...
* cursor which returns only useful IH
* cursor which returns only useful IH
2020-06-28 13:51:18 +01:00
