Commit Graph

37 Commits

Author SHA1 Message Date
Håvard Anda Estensen
5a1726b107
Avoid leaking the popped item (#760)
Set the deleted item to its zero val so it can be garbage collected.
2022-12-04 16:19:09 +07:00
Alex Sharov
4fec09660e
e3: better prune logs (#703) 2022-10-25 16:06:40 +07:00
Håvard Anda Estensen
222bb666e9
Fieldalign (#695)
```
➜  erigon-lib git:(fieldalign) ✗ fieldalignment -fix ./...
/Users/estensen/Developer/erigon-lib/commitment/bin_patricia_hashed.go:81:16: struct of size 1065120 could be 1065112
/Users/estensen/Developer/erigon-lib/commitment/bin_patricia_hashed.go:1063:14: struct of size 1032 could be 1024
/Users/estensen/Developer/erigon-lib/commitment/hex_patricia_hashed.go:62:24: struct of size 952776 could be 952768
/Users/estensen/Developer/erigon-lib/commitment/hex_patricia_hashed.go:98:12: struct of size 1832 could be 1824
/Users/estensen/Developer/erigon-lib/commitment/hex_patricia_hashed.go:113:12: struct with 208 pointer bytes could be 152
/Users/estensen/Developer/erigon-lib/commitment/hex_patricia_hashed.go:143:11: struct of size 464 could be 456
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:24:11: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:79:11: struct of size 56 could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:134:11: struct with 56 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:189:11: struct with 56 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:244:12: struct with 56 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:299:12: struct with 56 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:355:19: struct of size 56 could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:418:23: struct with 168 pointer bytes could be 128
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:571:20: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:626:20: struct with 136 pointer bytes could be 128
/Users/estensen/Developer/erigon-lib/gointerfaces/types/types.pb.go:721:15: struct of size 168 could be 160
/Users/estensen/Developer/erigon-lib/etl/buffers.go:75:21: struct with 88 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/etl/buffers.go:182:27: struct with 56 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/etl/buffers.go:274:32: struct with 56 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/etl/collector.go:41:16: struct with 72 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/etl/etl.go:66:20: struct with 96 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/etl/heap.go:25:15: struct with 40 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/patricia/patricia.go:29:11: struct with 40 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/patricia/patricia.go:347:12: struct with 32 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/patricia/patricia.go:367:18: struct with 48 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/patricia/patricia.go:377:19: struct with 168 pointer bytes could be 144
/Users/estensen/Developer/erigon-lib/compress/compress.go:52:17: struct with 176 pointer bytes could be 136
/Users/estensen/Developer/erigon-lib/compress/compress.go:241:24: struct with 48 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/compress/compress.go:327:14: struct with 40 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/compress/compress.go:353:18: struct with 48 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/compress/compress.go:450:19: struct with 48 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/compress/compress.go:670:21: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/compress/compress.go:734:23: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/compress/decompress.go:31:15: struct with 32 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/compress/decompress.go:39:19: struct with 40 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/compress/decompress.go:117:15: struct with 64 pointer bytes could be 56
/Users/estensen/Developer/erigon-lib/compress/decompress.go:125:19: struct with 96 pointer bytes could be 80
/Users/estensen/Developer/erigon-lib/compress/decompress.go:386:13: struct with 64 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/compress/parallel_compress.go:208:22: struct with 16 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/recsplit/golomb_rice.go:32:17: struct with 16 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/recsplit/index.go:35:12: struct of size 432 could be 424
/Users/estensen/Developer/erigon-lib/recsplit/index_reader.go:26:18: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/recsplit/recsplit.go:64:15: struct of size 704 could be 680
/Users/estensen/Developer/erigon-lib/recsplit/recsplit.go:111:19: struct of size 104 could be 96
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:171:17: struct of size 640 could be 632
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:198:17: struct with 168 pointer bytes could be 144
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:389:14: struct with 584 pointer bytes could be 568
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:921:21: struct with 72 pointer bytes could be 56
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:1195:22: struct with 2432 pointer bytes could be 2416
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:2123:13: struct with 2448 pointer bytes could be 2416
/Users/estensen/Developer/erigon-lib/aggregator/aggregator.go:2634:17: struct with 96 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/aggregator/history.go:39:14: struct with 96 pointer bytes could be 88
/Users/estensen/Developer/erigon-lib/bptree/binary_file.go:33:17: struct with 40 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/bptree/node.go:79:13: struct of size 88 could be 80
/Users/estensen/Developer/erigon-lib/chain/chain_config.go:28:13: struct with 136 pointer bytes could be 120
/Users/estensen/Developer/erigon-lib/common/background/progress.go:26:18: struct with 40 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/gointerfaces/downloader/downloader.pb.go:25:19: struct with 64 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/downloader/downloader.pb.go:80:22: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/downloader/downloader.pb.go:127:20: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/downloader/downloader.pb.go:165:19: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/downloader/downloader.pb.go:203:17: struct of size 104 could be 96
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:135:23: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:173:21: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:220:24: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:258:22: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:305:26: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:343:24: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:390:30: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:437:26: struct of size 72 could be 64
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:500:30: struct with 64 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:563:28: struct with 64 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:626:37: struct with 56 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:681:35: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:736:29: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:774:27: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:821:27: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:859:25: struct with 48 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:906:23: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:953:21: struct of size 72 could be 64
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1008:24: struct of size 104 could be 88
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1079:25: struct of size 144 could be 136
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1190:19: struct with 56 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1245:17: struct with 72 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1300:23: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1347:21: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1394:23: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1441:21: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1488:17: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/ethbackend.pb.go:1535:24: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:220:13: struct of size 120 could be 112
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:299:11: struct of size 104 could be 96
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:370:20: struct with 56 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:425:20: struct of size 136 could be 128
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:513:23: struct with 56 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:585:18: struct of size 112 could be 104
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:664:25: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:719:23: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/remote/kv.pb.go:757:21: struct with 72 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:300:26: struct of size 72 could be 64
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:355:35: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:410:29: struct with 56 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:465:38: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:520:16: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:567:26: struct of size 56 could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:622:26: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:677:21: struct of size 80 could be 72
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:740:12: struct with 56 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:795:17: struct with 72 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:874:21: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:912:21: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:959:22: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1006:17: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1053:23: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1091:21: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1138:22: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1185:20: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1232:24: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/sentry/sentry.pb.go:1270:16: struct of size 56 could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:25:28: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:63:26: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:110:26: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:148:24: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:195:27: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:233:25: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:280:21: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:318:19: struct with 96 pointer bytes could be 80
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:389:24: struct with 96 pointer bytes could be 88
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:452:22: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:499:28: struct with 56 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:554:26: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:601:22: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:639:20: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:686:20: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/mining.pb.go:724:18: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:132:15: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:179:17: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:226:15: struct with 72 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:281:26: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:328:24: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:375:19: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:413:17: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:460:17: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:498:15: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:545:19: struct with 48 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:592:20: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:630:18: struct of size 56 could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:693:19: struct with 48 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:740:17: struct of size 56 could be 48
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:795:18: struct of size 80 could be 72
/Users/estensen/Developer/erigon-lib/gointerfaces/txpool/txpool.pb.go:858:22: struct of size 80 could be 72
/Users/estensen/Developer/erigon-lib/direct/sentry_client.go:171:25: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/kv/mdbx/kv_mdbx.go:50:15: struct with 104 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/kv/mdbx/kv_mdbx.go:355:13: struct with 160 pointer bytes could be 152
/Users/estensen/Developer/erigon-lib/kv/memdb/memory_mutation_cursor.go:39:27: struct with 200 pointer bytes could be 184
/Users/estensen/Developer/erigon-lib/kv/remotedb/kv_remote.go:22:17: struct with 72 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/kv/remotedb/kv_remote.go:38:15: struct with 80 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/kv/remotedbserver/server.go:314:24: struct with 40 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/state/aggregator.go:44:17: struct with 192 pointer bytes could be 128
/Users/estensen/Developer/erigon-lib/state/aggregator.go:422:13: struct of size 384 could be 360
/Users/estensen/Developer/erigon-lib/state/aggregator.go:455:26: struct with 424 pointer bytes could be 368
/Users/estensen/Developer/erigon-lib/state/aggregator.go:853:22: struct with 24 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/state/aggregator22.go:37:19: struct with 176 pointer bytes could be 136
/Users/estensen/Developer/erigon-lib/state/aggregator22.go:643:15: struct of size 240 could be 216
/Users/estensen/Developer/erigon-lib/state/aggregator22.go:674:28: struct with 272 pointer bytes could be 224
/Users/estensen/Developer/erigon-lib/state/aggregator22.go:1126:26: struct with 104 pointer bytes could be 88
/Users/estensen/Developer/erigon-lib/state/aggregator22.go:1156:23: struct with 24 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/state/domain.go:52:16: struct with 32 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/state/domain.go:89:13: struct with 120 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/state/domain.go:399:17: struct with 96 pointer bytes could be 64
/Users/estensen/Developer/erigon-lib/state/domain.go:443:14: struct with 32 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/state/domain.go:594:16: struct with 72 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/state/history.go:48:14: struct with 72 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/state/history.go:435:20: struct with 48 pointer bytes could be 32
/Users/estensen/Developer/erigon-lib/state/history.go:529:23: struct with 40 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/state/history.go:1150:23: struct of size 344 could be 336
/Users/estensen/Developer/erigon-lib/state/inverted_index.go:46:20: struct with 128 pointer bytes could be 88
/Users/estensen/Developer/erigon-lib/state/inverted_index.go:389:23: struct with 136 pointer bytes could be 88
/Users/estensen/Developer/erigon-lib/state/inverted_index.go:541:24: struct with 184 pointer bytes could be 152
/Users/estensen/Developer/erigon-lib/state/merge.go:69:19: struct of size 72 could be 56
/Users/estensen/Developer/erigon-lib/state/merge.go:143:20: struct of size 48 could be 40
/Users/estensen/Developer/erigon-lib/state/read_indices.go:29:18: struct with 64 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/state/read_indices.go:211:14: struct of size 72 could be 56
/Users/estensen/Developer/erigon-lib/state/read_indices.go:233:27: struct with 72 pointer bytes could be 56
/Users/estensen/Developer/erigon-lib/state/state_recon.go:61:16: struct with 56 pointer bytes could be 16
/Users/estensen/Developer/erigon-lib/state/state_recon.go:112:19: struct with 136 pointer bytes could be 112
/Users/estensen/Developer/erigon-lib/state/state_recon.go:190:22: struct with 128 pointer bytes could be 112
/Users/estensen/Developer/erigon-lib/types/testdata.go:81:26: struct with 40 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/types/txn.go:46:21: struct with 400 pointer bytes could be 40
/Users/estensen/Developer/erigon-lib/types/txn.go:82:13: struct with 200 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/types/txn.go:691:18: struct with 32 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/txpool/fetch.go:43:12: struct with 136 pointer bytes could be 112
/Users/estensen/Developer/erigon-lib/txpool/pool.go:69:13: struct with 104 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/txpool/pool.go:221:13: struct of size 104 could be 96
/Users/estensen/Developer/erigon-lib/txpool/pool.go:291:13: struct with 336 pointer bytes could be 288
/Users/estensen/Developer/erigon-lib/txpool/pool.go:1789:29: struct with 32 pointer bytes could be 8
/Users/estensen/Developer/erigon-lib/txpool/pool.go:1819:19: struct with 32 pointer bytes could be 24
/Users/estensen/Developer/erigon-lib/txpool/pool.go:1994:18: struct of size 64 could be 56
/Users/estensen/Developer/erigon-lib/txpool/pool.go:2102:14: struct of size 64 could be 56
/Users/estensen/Developer/erigon-lib/txpool/send.go:37:11: struct with 64 pointer bytes could be 48
/Users/estensen/Developer/erigon-lib/txpool/test_util.go:31:17: struct with 72 pointer bytes could be 40
```
2022-10-21 15:31:23 +07:00
Alex Sharov
12f8c4be0f
e3: parallel build missed indices (#693) 2022-10-21 13:31:03 +07:00
Alex Sharov
0eab2a3dd1
e3: prevent files ranges overlap (kill -9 during merge handle) (#674) 2022-10-12 10:18:51 +07:00
Alex Sharov
b683ed435c
Compress params change (#651)
Main Target: reduce RAM usage of huffman tables. If possible - improve
decompression speed. Compression speed not so important.

Experiments on 74Gb uncompressed file (bsc
012500-013000-transactions.seg)
Ram - needed just to open compressed file (Huff tables, etc...)
dec_speed - loop with `word, _ = g.Next(word[:0])`
skip_speed - loop with `g.Skip()` 
```
| DictSize | Ram  | file_size | dec_speed | skip_speed |
| -------- | ---- | --------- | --------- | ---------- |
| 1M       | 70Mb | 35871Mb   | 4m06s     | 1m58s      |
| 512K     | 42Mb | 36496Mb   | 3m49s     | 1m51s      |
| 256K     | 21Mb | 37100Mb   | 3m44s     | 1m48s      |
| 128K     | 11Mb | 37782Mb   | 3m25s     | 1m44s      |
| 64K      | 7Mb  | 38597Mb   | 3m16s     | 1m34s      |
| 32K      | 5Mb  | 39626Mb   | 3m0s      | 1m29s      |
```
 
Also about small sampling: skip superstrings if superstringNumber % 4 !=
0 does reduce compression ratio by 1% - checked on big BSC file and
small (1gb) goerli file.

so, I feel it's not so bad idea to use:
maxDictPatterns=64k
samplingFactor=4

Tradeoffs: sacrify 5% compression ratio to 4x compression speedup (i
think even more), 30% decompression speedup, 10x RAM reduction

Release: I will not change existing snapshots - now will focus on
releasing new block snapshots and releasing new history snapshots
(Erigon3). If have time will re-compress existing snapshots later.
2022-10-05 17:54:48 +07:00
Alex Sharov
0d6bc2eca4
Madv helpers (#667) 2022-10-04 10:51:51 +01:00
Alex Sharov
f23061eed9
compressor: generic sort (#524) 2022-07-18 17:12:39 +07:00
Alex Sharov
ceafdded8f
Compress: reduce etl buffers to save RAM (#502) 2022-06-25 19:39:36 +06:00
Alex Sharov
a8ce14e8cc
option to disable runtime.ReadMemStats (#457)
* save

* save

* save

* save
2022-05-19 11:46:55 +07:00
Alex Sharov
91f7d84e60
Generic sort of slices (no allocs, inlinable) (#449)
* save

* save
2022-05-16 08:23:43 +01:00
Alex Sharov
04337fd090
Compress: reduce maxlen to 512 (#416) 2022-04-17 07:59:29 +07:00
ledgerwatch
f18e05186d
Compact huffman representation in files (#414)
* More compact huffman represenation

* Intermediate

* Intermediate

* fix

* Fix lint

* Fix lint

* Fix lint

* Change min file size

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-04-13 12:55:15 +01:00
Alex Sharov
75b64f01a3
compressor: log lvl #408 2022-04-01 10:44:25 +07:00
Alex Sharov
83951a1d62
Enable more linters (#381) 2022-03-19 11:38:37 +07:00
ledgerwatch
77eb94b53e
Elias fano search and merge (#357)
* Elias fano search and merge

* Add first cut of search

* Iterator and test

* Changes in aggregator

* Elias fano bitmap

* Fix uncompress decompress

* Print

* Print

* No print

* Print

* Print

* Print

* Change to AppendBytes

* Print

* Fix NextUncompressed

* Remove print

* Fix history search

* Fix in history search

* More tracing

* More tracing

* Fix

* Print

* Print key

* More print

* Print

* No deletion for history records

* Remove print

* Fix

* Fix

* Fix test

* Fix lint

Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-03-13 22:46:17 +00:00
Alex Sharov
c1f1365f92
cancel compress (#362) 2022-03-12 16:34:58 +07:00
Alex Sharov
c0fcdabf91
compress: less allocs (#361) 2022-03-12 15:33:01 +07:00
Alex Sharov
6512e3c941
add emptyWordsCount field to .seg file header (breaking .seg format) (#355)
* up torrent

* save

* save

* save

* save

* save

* save

* save
2022-03-10 07:48:37 +00:00
ledgerwatch
75b52ac25e
[compress] Allow uncompressed words (#350)
* Intermediate work

* Allow uncompressed words

* Fix

* Fix tests

* Add NextUncompressed, remove g.word buffer

* Code simplifications, no goroutines when workers == 1

* Fix lint|

* Add test for MatchPrefix

* Work on patricia

* Beginning of new matcher

* Fuzz test for new longest match

* No skip

* Fixes

* Fixes

* More tracing

* Fixes

* Fixes

* Change back to old FindLongestMatches

* Switch to old match finder

* Print mismatches

* Fix

* After fix

* After fix

* After fix

* Print pointers

* Fixes and tests

* Print

* Print

* Print

* More tests

* Intermediate

* Fix

* Fix

* Prints

* Fix

* Fix

* Initialise matchStack

* Compute only once

* Compute only once

* Switch back

* Switch to old Find

* Introduce sais

* Switch patricia to sais

* Use sais in compressor

* Use sais in compressor

* Remove unused code

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-03-09 17:25:22 +00:00
racytech
7763945374
Reverted 3 last commits (#348)
* Revert "unnecessary includes removed"

This reverts commit 76406bb78b.

* Revert "local dev setup"

This reverts commit ac06fd9400.

* Revert "compress/cgo-addition"

This reverts commit fae7683d46, reversing
changes made to e3e108c6c4.
2022-02-24 14:39:42 +00:00
Kairat Abylkasymov
ac06fd9400 local dev setup 2022-02-24 06:15:14 -05:00
ledgerwatch
c71ac02a0f
[erigon2] Optimisations in etl collector and compressor (#339)
* Optimisations in etl collector and compressor

* Not copy k and v in the collector

* Fix lint

* Optimisations

* Change Load1 back to Load

* Reduce allocations for tests

* preallocate inv

* counting hits and misses

* Try to fix

* Try to fix

* Relaxation 1

* Relaxation 2

* Add arch tables

* Fix

* Update arch tables and use them

* Not to override larger value

* Increase arch table size

* Increase arch table size

* Fixes to arch

* Print

* Off by one

* Print

* Fix

* Remove print

* Perform update of arch in the background

* Build up huffman tree

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-02-20 22:14:06 +00:00
Alex Sharov
567d9ddfed
ParallelCompressor: Remove intermediate ETL collectors (#302) 2022-02-04 16:48:02 +07:00
Alex Sharov
ec11eb3d91
parallel compressor: don't save dict (#283)
* save

* save
2022-01-27 12:54:38 +07:00
ledgerwatch
7ec016b160
Fixes in compress (#260)
* Fixes in compress

* Reuse outputFile also as uncompressed file

* Close file before renaming

* Trace

* Untrace

* Use 8 threads

* Print aggregations

* Print merge and timing

* Print merge and timing

* readonly mode for patricia

* Fix to infinite loop

* Fix file names

* Cleanup

* Cleanup

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
Co-authored-by: Alex Sharp <alexsharp@Alexs-MacBook-Pro.local>
2022-01-24 22:13:48 +00:00
primal_concrete_sledge
d8a33270e8
issue/issue-249-add_index_reader (#273)
* issue/issue-249-add_index_reader

* Add licence
2022-01-24 20:39:04 +00:00
ledgerwatch
340195df93
Less verbose parallel compressor (#247)
Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 14:20:05 +00:00
Alex Sharov
11ab5bdbb8
Parallel compressor - allow empty words (#245)
* save

* save

* Fix lint

Co-authored-by: Alexey Sharp <alexeysharp@Alexeys-iMac.local>
2022-01-18 13:57:35 +00:00
Alex Sharov
0f80e9941f
Switch to parallel compressor (#244) 2022-01-18 12:55:20 +07:00
Alex Sharov
51220cfe43
ParallelCompressor class, DecompressedFile class (#234)
* save

* save

* save

* remove major jump check

* remove major jump check

* log

* log

* save

* format docs

* format docs

* issue-260

* issue-260

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save

* save
2022-01-17 08:50:42 +00:00
Alex Sharov
01a6417505
snapshots: same workers amount #233 2022-01-15 11:23:19 +07:00
Alex Sharov
1647faec37
Fix bigChunk helper (#229) 2022-01-12 10:46:26 +07:00
alex.sharov
a8c2481967 create huffman_codes.txt in tmpdir 2022-01-09 14:52:52 +07:00
alex.sharov
8bc0f26a49 create .seg in tmpdir 2022-01-09 14:49:56 +07:00
Alex Sharov
0d5d8975d9
Snapshots: create .dat in tmpdir (#225) 2022-01-09 14:43:55 +07:00
Alex Sharov
f5733d438f
Parallel compression (#223) 2022-01-06 14:13:03 +07:00