milvus-2.5.1集群150w数据，三台机器，IVF_FLAT索引，30路压测的时候，queryNode节点CPU利用率波动幅度很大 #41341

jhyever · 2025-04-16T02:57:42Z

jhyever
Apr 16, 2025

milvus-2.5.1集群150w数据，三台机器，IVF_FLAT索引，30路压测的时候，queryNode节点CPU利用率波动幅度很大，这是为啥 @yhmo 麻烦大佬帮忙解答一下感谢

jhyever · 2025-04-16T03:32:26Z

jhyever
Apr 16, 2025
Author

queryNode日志

TotalMem: 17179869184
UsedMem: 33386496

open pid file: /run/milvus/querynode.pid
lock pid file: /run/milvus/querynode.pid
[2025/04/16 03:04:33.562 +00:00] [INFO] [roles/roles.go:345] ["starting running Milvus components"]
[2025/04/16 03:04:33.562 +00:00] [INFO] [roles/roles.go:187] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2025/04/16 03:04:33.571 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/04/16 03:04:33.572 +00:00] [DEBUG] [paramtable/base_table.go:213] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL" :false,"Endpoints":["my-release-etcd:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pe m","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2025/04/16 03:04:33.573 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd:2379]"] [minVersion=1.3]
[2025/04/16 03:04:33.576 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[my-release-etcd:2379]"]
[2025/04/16 03:04:33.576 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=EtcdSource]
[2025/04/16 03:04:33.577 +00:00] [INFO] [paramtable/component_param.go:4471] ["DeployModeEnv is not set, use default"] [default=0.5]
[2025/04/16 03:04:33.579 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2025/04/16 03:04:33.579 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]
[2025/04/16 03:04:33.580 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2025/04/16 03:04:33.580 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2025/04/16 03:04:33.580 +00:00] [INFO] [roles/roles.go:294] [setupPrometheusHTTPServer]
[2025/04/16 03:04:33.580 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]
[2025/04/16 03:04:33.581 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=15461882265]
[2025/04/16 03:04:33.582 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]
[2025/04/16 03:04:33.582 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd:2379]"] [minVersion=1.3]
[2025/04/16 03:04:33.585 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]
[2025/04/16 03:04:33.685 +00:00] [INFO] [querynodev2/server.go:294] ["register metrics actions finished"]
[2025/04/16 03:04:33.685 +00:00] [INFO] [querynodev2/server.go:304] ["QueryNode session info"] [metaPath=by-dev/meta]
[2025/04/16 03:04:33.685 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd:2379]"] [minVersion=1.3]
[2025/04/16 03:04:33.697 +00:00] [INFO] [sessionutil/session_util.go:296] ["start server"] [name=querynode] [address=100.82.231.16:21123] [id=1981] [server_labels={}]
[2025/04/16 03:04:33.697 +00:00] [INFO] [sessionutil/session_util.go:1264] ["save server info into file"] [content="querynode-1981\n"] [filePath=/tmp/milvus/server_id_8]
[2025/04/16 03:04:33.697 +00:00] [INFO] [querynodev2/server.go:168] ["QueryNode init session"] [nodeID=1981] ["node address"=100.82.231.16:21123]
[2025/04/16 03:04:33.698 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2025/04/16 03:04:33.704 +00:00] [INFO] [storage/remote_chunk_manager.go:97] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-bucket] [root=file]
[2025/04/16 03:04:33.704 +00:00] [INFO] [scheduler/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=10]
[2025/04/16 03:04:33.705 +00:00] [INFO] [querynodev2/server.go:345] ["queryNode init scheduler"] [policy=fifo]
[2025/04/16 03:04:33.705 +00:00] [INFO] [segments/segment_loader.go:168] ["SegmentLoader created"] [ioPoolSize=80]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250416 03:04:33.705492 56 knowhere_config.cc:107] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384
I20250416 03:04:33.705684 56 knowhere_config.cc:118] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0
I20250416 03:04:33.705736 56 knowhere_config.cc:54] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.5.0
I20250416 03:04:33.706166 56 thread_pool.h:198] [KNOWHERE][InitGlobalSearchThreadPool][milvus] Init global search thread pool with size 40
I20250416 03:04:33.706274 56 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
I20250416 03:04:33.706298 56 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512
I20250416 03:04:33.706327 56 Common.cpp:35] [SERVER][SetIndexSliceSize][milvus][]set config index slice size (byte): 16777216
I20250416 03:04:33.706413 56 Common.cpp:41] [SERVER][SetHighPriorityThreadCoreCoefficient][milvus][]set high priority thread pool core coefficient: 10
I20250416 03:04:33.706436 56 Common.cpp:48] [SERVER][SetMiddlePriorityThreadCoreCoefficient][milvus][]set middle priority thread pool core coefficient: 5
I20250416 03:04:33.706454 56 Common.cpp:55] [SERVER][SetLowPriorityThreadCoreCoefficient][milvus][]set low priority thread pool core coefficient: 1
[2025/04/16 03:04:33.706 +00:00] [INFO] [querynodev2/server.go:235] ["set up knowhere build pool size"] [pool_size=5]
I20250416 03:04:33.706553 56 thread_pool.h:178] [KNOWHERE][InitGlobalBuildThreadPool][milvus] Init global build thread pool with size 5
I20250416 03:04:33.706599 56 Common.cpp:62] [SERVER][SetDefaultExecEvalExprBatchSize][milvus][]set default expr eval batch size: 8192
I20250416 03:04:33.706735 56 MinioChunkManager.cpp:161] [SERVER][InitSDKAPIDefault][milvus][]init aws with log level:fatal
I20250416 03:04:35.738305 56 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus][]start to precheck chunk manager with configuration: [address=my-release-minio:9000, bucket_name=milvus-b ucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000 , gcp_native_without_auth=false]
I20250416 03:04:35.747589 56 ChunkManager.cpp:112] [SERVER][AwsChunkManager][milvus][]init AwsChunkManager with parameter[endpoint=my-release-minio:9000][bucket_name=milvus-bucket][root_pa th=file][use_secure=false]
I20250416 03:04:35.748157 56 MmapChunkManager.cpp:320] [SERVER][MmapChunkManager][milvus][]Init MappChunkManager with: Path /var/lib/milvus/data/mmap/mmap_chunk_manager, MaxDiskSize 50682 MB, FixedFileSize 1 MB.
I20250416 03:04:35.748193 56 ChunkCache.h:42] [SERVER][ChunkCache][milvus][]Init ChunkCache with read_ahead_policy: willneed
I20250416 03:04:35.748203 56 MmapManager.h:74] [SERVER][Init][milvus][]Init MmapConfig with MmapConfig: [cache_read_ahead_policy=willneed, mmap_path=/var/lib/milvus/data/mmap/mmap_chunk_ma nager, disk_limit=50682MB, fix_file_size=1MB, growing_enable_mmap=false, scalar_index_enable_mmap=false]
I20250416 03:04:35.748277 56 Tracer.cpp:81] [SERVER][initTelemetry][milvus][]Empty Trace
I20250416 03:04:35.748442 56 FunctionFactory.cpp:59] [SERVER][RegisterAllFunctions][milvus][]2 functions registered
[2025/04/16 03:04:35.748 +00:00] [INFO] [querynodev2/server.go:385] ["query node init successfully"] [queryNodeID=1981] [Address=100.82.231.16:21123]
[2025/04/16 03:04:35.748 +00:00] [INFO] [querynodev2/server.go:413] ["query node start successfully"] [queryNodeID=1981] [Address=100.82.231.16:21123] [mmapEnabled=false] [growingmmapEnable=f alse] [mmapVectorIndex=false] [mmapVectorField=false] [mmapScalarIndex=false] [mmapScalarField=false] [mmapChunkCache=true]
[2025/04/16 03:04:35.748 +00:00] [INFO] [scheduler/concurrent_safe_scheduler.go:213] ["start execute loop"]
[2025/04/16 03:04:35.761 +00:00] [INFO] [sessionutil/session_util.go:494] ["put session key into etcd"] [key=by-dev/meta/session/querynode-1981] [value="{"ServerID":1981,"ServerName":"qu erynode","Address":"100.82.231.16:21123","TriggerKill":true,"Version":"2.5.1","IndexEngineVersion":{"CurrentIndexVersion":6},"LeaseID":1276654206192162661,"HostName":"my-r elease-milvus-querynode-7df5777fb7-97z6x"}"]
[2025/04/16 03:04:35.761 +00:00] [INFO] [sessionutil/session_util.go:504] ["Service registered successfully"] [ServerName=querynode] [serverID=1981]
[2025/04/16 03:04:35.764 +00:00] [INFO] [components/query_node.go:63] ["QueryNode successfully started"]
[2025/04/16 03:04:35.765 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]
[2025/04/16 03:04:35.801 +00:00] [INFO] [sessionutil/session_util.go:944] ["register session success"] [role=querynode] [key=by-dev/meta/session/querynode-1981]
[2025/04/16 03:05:03.562 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=66]
[2025/04/16 03:05:33.563 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=66]
[2025/04/16 03:06:03.562 +00:00] [DEBUG] [metrics/thread.go:53] ["thread watcher observe thread num"] [threadNum=66]
[2025/04/16 03:06:15.934 +00:00] [INFO] [querynodev2/services.go:205] ["received watch channel request"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel= by-dev-rootcoord-dml_1_456275818148801950v1] [currentNodeID=1981] [version=1744772775931851176]
[2025/04/16 03:06:15.934 +00:00] [INFO] [segments/collection.go:98] ["put new collection"] [collectionID=456275818148801950] [schema="name:"qqxlmx1" description:"Test content search" fi elds:{name:"RowID" description:"row id" data_type:Int64} fields:{fieldID:1 name:"Timestamp" description:"time stamp" data_type:Int64} fields:{fieldID:100 name:"id" is_prima ry_key:true data_type:Int64} fields:{fieldID:101 name:"lib_id" data_type:VarChar type_params:{key:"max_length" value:"60"}} fields:{fieldID:102 name:"part_id" data_type:VarCh ar type_params:{key:"max_length" value:"60"}} fields:{fieldID:103 name:"doc_id" data_type:VarChar type_params:{key:"max_length" value:"60"}} fields:{fieldID:104 name:"cate gory" data_type:VarChar type_params:{key:"max_length" value:"100"}} fields:{fieldID:105 name:"user" data_type:VarChar type_params:{key:"max_length" value:"60"}} fields:{fi eldID:106 name:"source_content" data_type:VarChar type_params:{key:"max_length" value:"4096"}} fields:{fieldID:107 name:"versionList" data_type:Array type_params:{key:"max_ca pacity" value:"100"} type_params:{key:"max_length" value:"100"} element_type:Int16} fields:{fieldID:108 name:"labels" data_type:Array type_params:{key:"max_capacity" value :"100"} type_params:{key:"max_length" value:"100"} element_type:VarChar} fields:{fieldID:109 name:"embedding" data_type:FloatVector type_params:{key:"dim" value:"1024"}} fields:{fieldID:110 name:"$meta" description:"dynamic schema" data_type:JSON is_dynamic:true} enable_dynamic_field:true"]
[2025/04/16 03:06:15.934 +00:00] [INFO] [vecindexmgr/vector_index_mgr.go:124] ["init vector indexes with features : BINFLAT : 720897,BIN_FLAT : 720897,BIN_IVF_FLAT : 524289,DISKANN : 2097166, FLAT : 720910,HNSW : 1572910,HNSWLIB_DEPRECATED : 1572879,HNSW_PQ : 524334,HNSW_PRQ : 524334,HNSW_SQ : 524334,IVFBIN : 524289,IVFFLAT : 524302,IVFFLATCC : 14,IVFPQ : 524302,IVFSQ : 524302,IVF _FLAT : 524302,IVF_FLAT_CC : 14,IVF_PQ : 524302,IVF_SQ : 524302,IVF_SQ8 : 524302,IVF_SQ_CC : 14,SCANN : 524302,SPARSE_INVERTED_INDEX : 524304,SPARSE_INVERTED_INDEX_CC : 524304,SPARSE_WAND : 5 24304,SPARSE_WAND_CC : 524304,"]
I20250416 03:06:15.935715 56 Collection.cpp:64] [SERVER][parseIndexMeta][milvus][]index meta info: maxRowCount : {20609} FieldId : {109 IndexParams : { index_type : IVF_FLAT, metric_type : IP, nlist : 1024, }TypeParams : {dim : 1024, }}
[2025/04/16 03:06:15.935 +00:00] [INFO] [segments/collection.go:229] ["collection ref increment"] [nodeID=1981] [collectionID=456275818148801950] [refCount=1]
[2025/04/16 03:06:15.935 +00:00] [INFO] [delegator/delegator.go:885] ["Init delete cache with list delete buffer"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [replicaID=457252108674793478] [channel=by-dev-rootcoord-dml_1_456275818148801950v1] [version=1744772775931851176] [startTs=457381644842041348] [sizePerBlock=1048576] [startTime=2025/04/16 0 3:01:49.926 +00:00]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator.go:890] ["shard delegator setup l0 forward policy"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [ replicaID=457252108674793478] [channel=by-dev-rootcoord-dml_1_456275818148801950v1] [version=1744772775931851176] [startTs=457381644842041348] [policy=FilterByBF]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator.go:937] ["finish build new shardDelegator"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [replicaI D=457252108674793478] [channel=by-dev-rootcoord-dml_1_456275818148801950v1] [version=1744772775931851176] [startTs=457381644842041348]
[2025/04/16 03:06:15.936 +00:00] [INFO] [pipeline/manager.go:68] ["start create pipeine"] [collectionID=456275818148801950] [channel=by-dev-rootcoord-dml_1_456275818148801950v1]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator_data.go:354] ["loading growing segments..."] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel =by-dev-rootcoord-dml_1_456275818148801950v1] [replicaID=457252108674793478] [segmentIDs="[]"]
[2025/04/16 03:06:15.936 +00:00] [INFO] [segments/segment_loader.go:231] ["no segment to load"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [segmentType=Growi ng]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator_data.go:377] ["load growing segments done"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel= by-dev-rootcoord-dml_1_456275818148801950v1] [replicaID=457252108674793478] [segmentIDs="[]"]
[2025/04/16 03:06:15.936 +00:00] [INFO] [delegator/delegator_data.go:343] ["add growing segments to delegator"] [collectionID=456275818148801950] [channel=by-dev-rootcoord-dml_1_4562758181488 01950v1] [replicaID=457252108674793478] [segmentIDs="[]"]
[2025/04/16 03:06:15.936 +00:00] [INFO] [msgdispatcher/manager.go:66] ["create new dispatcherManager"] [role=querynode] [nodeID=1981] [pchannel=by-dev-rootcoord-dml_1]
[2025/04/16 03:06:15.936 +00:00] [INFO] [msgdispatcher/dispatcher.go:97] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_1] [subName=querynode-1981-by-dev-rootcoord-dml_1_4562758181 48801950v1-true] [isMain=true]
[2025/04/16 03:06:15.936 +00:00] [INFO] [msgdispatcher/manager.go:167] ["dispatcherManager is running..."] [role=querynode] [nodeID=1981] [pchannel=by-dev-rootcoord-dml_1]
[2025/04/16 03:06:15.937 +00:00] [INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[2025/04/16 03:06:16.021 +00:00] [INFO] [msgstream/mq_msgstream.go:922] ["MsgStream begin to seek start msg: "] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [channel=by-dev-rootcoord-dml_1] [Me ssageID="CM4hEIWcAhgAIAAwAQ=="]
[2025/04/16 03:06:16.154 +00:00] [INFO] [msgstream/mq_msgstream.go:932] ["MsgStream seek finished"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [channel=by-dev-rootcoord-dml_1]
[2025/04/16 03:06:16.169 +00:00] [INFO] [msgdispatcher/dispatcher.go:126] ["seek successfully"] [pchannel=by-dev-rootcoord-dml_1] [subName=querynode-1981-by-dev-rootcoord-dml_1_45627581814880 1950v1-true] [isMain=true] [posTs=457381644842041348] [posTime=2025/04/16 03:01:49.926 +00:00] [tsLag=4m26.243846458s]
[2025/04/16 03:06:16.169 +00:00] [INFO] [msgdispatcher/dispatcher.go:162] ["add new target"] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1] [isMain=true]
[2025/04/16 03:06:16.169 +00:00] [INFO] [msgdispatcher/manager.go:114] ["add main dispatcher"] [role=querynode] [nodeID=1981] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/dispatcher.go:190] ["get signal"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=true]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/dispatcher.go:213] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_1] [signal=start] [isMain=true]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/client.go:103] ["register done"] [role=querynode] [nodeID=1981] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1]
[2025/04/16 03:06:16.170 +00:00] [INFO] [pipeline/stream_pipeline.go:133] ["stream pipeline seeks from position with msgDispatcher"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [pchannel=by-de v-rootcoord-dml_1_456275818148801950v1] [vchannel=by-dev-rootcoord-dml_1_456275818148801950v1] [checkpointTs=2025/04/16 03:01:49.926 +00:00] [tsLag=4m26.244046407s] [elapse=233.583722ms]
[2025/04/16 03:06:16.170 +00:00] [INFO] [querynodev2/services.go:341] ["watch dml channel success"] [traceID=ceccef9ad6063a6e648ff51f0bcaaf32] [collectionID=456275818148801950] [channel=by-de v-rootcoord-dml_1_456275818148801950v1] [currentNodeID=1981]
[2025/04/16 03:06:16.170 +00:00] [INFO] [msgdispatcher/dispatcher.go:218] ["begin to work"] [pchannel=by-dev-rootcoord-dml_1] [isMain=true]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:417] ["received load segments request"] [traceID=39ceeafe2ec6438a01f8e842ffdba5ec] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627882600] [level=L1] [currentNodeID=1981] [dstNodeID=1978] [version=1744772776429185431] [need Transfer=true] [loadScope=Full]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:417] ["received load segments request"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [level=L1] [currentNodeID=1981] [dstNodeID=1981] [version=1744772776430590177] [need Transfer=true] [loadScope=Full]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:417] ["received load segments request"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [level=L1] [currentNodeID=1981] [dstNodeID=1981] [version=1744772776430590177] [need Transfer=false] [loadScope=Full]
[2025/04/16 03:06:16.431 +00:00] [INFO] [segments/collection.go:229] ["collection ref increment"] [nodeID=1981] [collectionID=456275818148801950] [refCount=2]
[2025/04/16 03:06:16.431 +00:00] [INFO] [querynodev2/services.go:489] ["start to load segments..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitionID=4 56275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [level=L1] [currentNodeID=1981] [dstNodeID=1981]
[2025/04/16 03:06:16.431 +00:00] [INFO] [segments/segment_loader.go:252] ["start loading..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [segmentType=Sealed] [requestSegments="[457204142627881297]"] [preparedSegments="[457204142627881297]"] [segmentNum=1] [afterFilter=1]
[2025/04/16 03:06:16.436 +00:00] [INFO] [segments/pool.go:108] ["init dynamicPool done"] [size=10]
[2025/04/16 03:06:16.438 +00:00] [INFO] [segments/segment_loader.go:1414] ["predict memory and disk usage while loading (in MiB)"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456 275818148801950] [maxSegmentSize(MB)=0.2816162109375] [committedMemSize(MB)=0] [memLimit(MB)=16384] [memUsage(MB)=56.57421875] [committedDiskSize(MB)=0] [diskUsage(MB)=0] [predictMemUsage(MB) =56.8558349609375] [predictDiskUsage(MB)=0] [mmapFieldCount=0]
[2025/04/16 03:06:16.438 +00:00] [INFO] [segments/segment_loader.go:482] ["request resource for loading segments (unit in MiB)"] [traceID=1c46417f55a81e976ca6025d2ec40028] [segmentIDs="[45720 4142627881297]"] [memory=0.2816162109375] [committedMemory=0.2816162109375] [disk=0] [committedDisk=0]
[2025/04/16 03:06:16.441 +00:00] [INFO] [segments/segment.go:337] ["create segment done"] [traceID=1c46417f55a81e976ca6025d2ec40028]
[2025/04/16 03:06:16.441 +00:00] [INFO] [segments/segment_loader.go:355] ["start to load segments in parallel"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [s egmentType=Sealed] [requestSegments="[457204142627881297]"] [preparedSegments="[457204142627881297]"] [segmentNum=1] [concurrencyLevel=1]
[2025/04/16 03:06:16.442 +00:00] [INFO] [segments/segment_loader.go:328] ["load segment..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [segmentType=Sealed] [requestSegments="[457204142627881297]"] [preparedSegments="[457204142627881297]"] [partitionID=456275818148801951] [segmentID=457204142627881297] [segmentType=L1]
[2025/04/16 03:06:16.442 +00:00] [INFO] [segments/segment_loader.go:842] ["start loading segment files"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [shard=by-dev-rootcoord-dml_1_456275818148801950v1] [segmentID=457204142627881297] [rowNum=15] [segmentType=Sealed]
[2025/04/16 03:06:16.442 +00:00] [INFO] [segments/pool.go:129] ["init loadPool done"] [size=50]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:846] ["add field data info done"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitionID=456275 818148801951] [segmentID=457204142627881297] ["row count"=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment_loader.go:761] ["Start loading fields..."] [traceID=1c46417f55a81e976ca6025d2ec40028] [segmentID=457204142627881297] [indexedFields=" []"] ["indexed text fields"="[]"] ["unindexed text fields"="[]"]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=108] [rowCount=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=0] [rowCount=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=102] [rowCount=15]
[2025/04/16 03:06:16.443 +00:00] [INFO] [segments/segment.go:773] ["start loading field data for field"] [traceID=1c46417f55a81e976ca6025d2ec40028] [collectionID=456275818148801950] [partitio nID=456275818148801951] [segmentID=457204142627881297] [fieldID=103] [rowCount=15]

0 replies

xiaofan-luan · 2025-04-16T03:58:11Z

xiaofan-luan
Apr 16, 2025
Maintainer

IVF_FLAT

cpu已经打满了，建议用hnsw索引

3 replies

jhyever Apr 16, 2025
Author

CPU没打满啊，整机才用了30%左右，queryNode节点pod用了最多才8核，，，是queryNode有什么限流麽，为啥会出现波动呀

xiaofan-luan Apr 16, 2025
Maintainer

波动时因为你的压力不够大吧。

很大概率你的查询都是batch，每次milvus会对batch进行一些并发。然后中间空的地方就是没有新的batch进来。
如果是看吞吐，那么你可以尝试并发更多的请求
如果是看延迟，那么要做的就是减小batch

xiaofan-luan Apr 16, 2025
Maintainer

然后你看一下你这几台的机器的压力是不是平均，还是有一台特别高

yhmo · 2025-04-16T06:27:41Z

yhmo
Apr 16, 2025
Collaborator

日志显示该表的向量维度1024，有2个sealed segment，其中一个segment只有15行数据，另一个的行数没看到，不合理。
用webui观察segment:
http://[proxy_node_address]:9091/webui/data_component

客户端的压测方式可能有问题，nprobe的值不会是1吧，查询向量是随机的还是固定的

9 replies

yhmo Apr 17, 2025
Collaborator

querynode,proxy的日志只显示了从2025/04/17 01:20 到 2025/04/17 06:01:28 大约5个小时的内容。在这5个小时里，只有大约前5分钟出现了一些search failed的报错：

[2025/04/17 01:20:18.130 +00:00] [WARN] [proxy/task_search.go:583] ["search execute failed"] [traceID=e02cb3c848a5d4fdeb792aebec24d454] [nq=1] [error="failed to search/query delegator 2054 for channel by-dev-rootcoord-dml_6_456275818152282729v0: fail to search on QueryNode 2054: worker(2054) query failed: parser searchRequest failed:  => vector dimension mismatch, expected vector size(byte) 4096, actual 20.

这个报错应该是client端的查询输入不正确。在2025/04/17 01:30之后就没有任何search请求的日志。

每个节点内部日志会滚动清理，日志密集的能保留的时间就短。如果你是昨天做的测试，那么这个日志里面就看不到了。

最好是重跑一次压测，跑个半小时左右，然后再搜集日志。另外，看起来你是用java sdk做的压测，你所使用的查询参数是什么？30路压测是30个线程不停地调用search？

jhyever Apr 17, 2025
Author

1、这种java调用的

2、我是30路压测，每次请求是30个并发【备注，不是每秒，是每次】，不停的调用search的

yhmo Apr 17, 2025
Collaborator

String expr = ExprUtils.buildExpr ......

这个expr比较可疑，看起来是个复杂的过滤表达式。对IVF索引来说，过滤表达式直接影响每个请求的latency，过滤量大的latency小，过滤量小的latency大。如果只有极少的数据符合条件，那么querynode的工作量就基本是在expr的解析，以及扫描符合条件的数据上。“扫描符合条件的数据” 这个事情并不像 “计算几万条向量的distance” 那么的容易并行，所以querynode的cpu使用率不会高。

为了验证，你可以把builder.withExpr(expr)这行注释掉，然后再测。

jhyever Apr 17, 2025
Author

去掉也还是一样呢

yhmo Apr 17, 2025
Collaborator

你安装一个标准的监控看看：https://milvus.io/docs/monitor.md
然后在监控页面的QueryNode和Proxy里把跟search相关的指标都截图看一下。我比较怀疑压力没打满。

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

milvus-2.5.1集群150w数据，三台机器，IVF_FLAT索引，30路压测的时候，queryNode节点CPU利用率波动幅度很大 #41341

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 12 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

milvus-2.5.1集群150w数据，三台机器，IVF_FLAT索引，30路压测的时候，queryNode节点CPU利用率波动幅度很大 #41341

jhyever Apr 16, 2025

Replies: 3 comments · 12 replies

jhyever Apr 16, 2025 Author

xiaofan-luan Apr 16, 2025 Maintainer

jhyever Apr 16, 2025 Author

xiaofan-luan Apr 16, 2025 Maintainer

xiaofan-luan Apr 16, 2025 Maintainer

yhmo Apr 16, 2025 Collaborator

yhmo Apr 17, 2025 Collaborator

jhyever Apr 17, 2025 Author

yhmo Apr 17, 2025 Collaborator

jhyever Apr 17, 2025 Author

yhmo Apr 17, 2025 Collaborator

jhyever
Apr 16, 2025

Replies: 3 comments 12 replies

jhyever
Apr 16, 2025
Author

xiaofan-luan
Apr 16, 2025
Maintainer

jhyever Apr 16, 2025
Author

xiaofan-luan Apr 16, 2025
Maintainer

xiaofan-luan Apr 16, 2025
Maintainer

yhmo
Apr 16, 2025
Collaborator

yhmo Apr 17, 2025
Collaborator

jhyever Apr 17, 2025
Author

yhmo Apr 17, 2025
Collaborator

jhyever Apr 17, 2025
Author

yhmo Apr 17, 2025
Collaborator