[Bug]: querynode crash when running cluster in k8 #34983
Assignees
Labels
help wanted
Extra attention is needed
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Comments
msr120295
added
kind/bug
|
/assign @LoveEachDay |
yanliang567
added
triage/accepted
|
WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], |
|
Hello, I want to ask, what is the best version of milvus to run ? i tried image from 2.3.0 to 2.4.6 still error. querynode crashloopback because of s3 |
|
[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]" Did you follow this document to set the right persmisison?
|
|
i use minio for s3. i didnt use external minio. |
|
but did you enable any access control? |
|
I suggest you post out all the changes you've been made on helm chart so we can check it. |
|
i follow this helm https://artifacthub.io/packages/helm/milvus-helm/milvus/4.2.0 this is my helm values, i only change image tag from 2.4.6 to 2.4.6-gpu versioncluster: image: nodeSelector: {} service:
ingress: serviceAccount: metrics: serviceMonitor: livenessProbe: readinessProbe: log: persistence: heaptrack: proxy: tls: rootCoordinator: service: queryCoordinator: service: queryNode: replicas: 1 nodeSelector: {} strategy: {} indexCoordinator: service: indexNode: dataCoordinator: service: dataNode: attu: minio: service: persistence: livenessProbe: readinessProbe: startupProbe: etcd: service: auth: persistence: livenessProbe: readinessProbe: autoCompactionMode: revision extraEnvVars:
pulsar: kafka: |
|
i have added s3 permission and using stand alone minio. outside from milvus helm. but still issue like this
|
|
are u using the right AK/SK or enabled IAM? accessKey: minioadmin |
|
no, i am not using IAM (identity access management) |
|
you can probably test your S3 connectivity with https://github.com/zhagnlu/milvus-store-tool From your log this seems to be clearly a S3 problem |
|
i tried to click your link but it seems 404 not found for milvus store tool |
|
try this |
|
Hi @msr120295 , from the picture, if you want to install milvus cluster with kafka, you can run the cmd And I want to confirm if you're going to use S3 or another oss as your datastore? |
|
yes @locustbaby . i want to use minio as s3 storage. can you provide the correct helm chart values ? |
|
minio: So the helm chart looks as is? |
|
can you share all the step you do to install the cluster? including you you setup your minio? |
|
here is a general guide investigate on this failure This error suggests there's an issue with the authorization parameters in your S3-compatible API request, specifically with the date format in the X-Amz-Credential parameter. Here's how you can address this problem: Check the date format: Ensure you're using the correct date formatting function. Examine the full credential string: |
|
X-Amz-Credential is automatically generated by minio client so please check your system clock and see it's offset is huge |
|
Hi, I have been running into this issue when using docker compose with milvus-standalone. My system clock was off, so I synced it. However the error still presists. $ timedatectl
Local time: Sun 2024-08-18 12:25:17 +03
Universal time: Sun 2024-08-18 09:25:17 UTC
RTC time: Sun 2024-08-18 09:25:17
Time zone: Asia/Riyadh (+03, +0300)
System clock synchronized: yes
NTP service: active
RTC in local TZ: noThe following is my networks:
default:
driver: bridge # Define custom network as bridge
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
restart: on-failure
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3
networks:
- default
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
restart: on-failure
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
- default
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.4.8
restart: on-failure
command: ["milvus", "run", "standalone"]
security_opt:
- seccomp:unconfined
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
networks:
- default
volumes:
milvus_data: |
i think you might need to ask your GCP team for support. |
|
Can you expand on what GCP stands for?I am running milvus locally on my workstation. |
So you deployed minio locally on your work station? |
|
did you enable ntp on all the nodes? |
|
from the S3 SDK document you sould get a request to minio like this: GET -------- HTTP method content-type;host;x-amz-date -------- A list of signed headers Could you use wired shark to capture a pack to S3 see what's inside? |
I am running the everything on a local workstation using docker compose.
The thing is, I get no issues when running the docker images on my machine while it's offline. The issue only shows up when running minio while the machine is connected to the internet. Is there a way to permanently disable minio from connecting to the internet? I don't need to use AWS or any cloud providers for my project. |
|
what's the configuration difference between the two cluster? |
|
The difference is here: Offline networks:
default:
driver: bridge # Define custom network as bridge
offline-network:
driver: bridge # Define custom network as bridge
internal: true # Block internet access
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
restart: on-failure
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3
networks:
- offline-network
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
restart: on-failure
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
- offline-network
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.4.8
restart: on-failure
command: ["milvus", "run", "standalone"]
security_opt:
- seccomp:unconfined
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
networks:
- default
- offline-network
volumes:
milvus_data: |
|
Same error with 2.4.9: Using this helm values |
did you got a chance to check the clock skew between your milvus and your minio? |
|
Hi @xiaofan-luan is this what you mean? I cant run date into the standalone pod: This is currently happening in 10 multiple cluster at the same time, all freshly installed. Worth mentioning that this happens in installation that are located in UAE, an exact same deployment in Europe work just fine. Not sure if thats somehow related. |
|
@xiaofan-luan same problem while using docker compose: |
|
@xiaofan-luan I can confirm what @SalBakraa mentioned, adding but its a bit tricky to port this to kubernetes without using network policies |
|
@grebois Could you provide the output of the |
|
Sure: |
|
@grebois Is the network internal enabled now? |
|
@zwd1208 I am remotely connected to this host, so I cannot really disable the network, but even though, we tried without network once and it worked, the outputs I am sharing are with This is with |
|
Same problem @ 2.4.10 in docker-compose, helm chart and milvus-operator. |
|
Quick Question, We recommend is that you need separate storage for each of your env. Once you enable network, then ntp triggered and change the time to local time so there is a clock skew among different regions. Again this is a rough guess, I think you want to first make your your milvus, etcd and object storage is on same region and same time |
|
@grebois I was unable to reproduce this issue. Could you provide the configurations for deployment using Docker Compose, Helm Chart, and Operator? |
No, I tried deploying it standalone, each with it's "own" minio, as part of the docker-compose.
No changes, all default configuration, I can give you access to one of these machines for you to try it. |
@grebois Alright, you can send the access information to my email at [email protected] |
|
@zwd1208 Sent! |
|
@grebois And the reason is: Milvus uses the AWS SDK to access MinIO. The AWS SDK attempts to access the local address http://169.254.169.254 during initialization to retrieve the region information.
To resolve this issue, you can add the environment variable |
|
@zwd1208 thanks a lot for your outstanding support, the IP is not actually configured, this is all bare metal, it's just the admin panel IP of a backup 5g Router we don't really use, Huawei Hg8245h, just to give a bit more background to other people that might face the same problem, will try setting the MINIO_REGION: us-east-1 and also limit the access to this IP with iptables, great work, highly appreciated. |
|
what a bug! |
Closed
1 task
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Environment
Current Behavior
i tried to run milvus cluster in k8 using helm chart. installed using default helm chart values.
this is k8 pods
but querynode always error and when i see the logs. this is the error
precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000] [2024/07/24 22:34:26.248 +00:00] [WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"] [2024/07/24 22:34:26.248 +00:00] [ERROR] [querynodev2/server.go:369] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:369\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:284\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] [2024/07/24 22:34:26.249 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] [2024/07/24 22:34:26.249 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]i see that the issue is when querynode trying to connect with s3 minio . is there any solution regarding this ? i tried different container image like 2.3.18 / 2.3.16 / 2.4.5 still same. querynode fails
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
[2024/07/24 22:34:26.034 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
2024/07/24 22:34:26 maxprocs: Leaving GOMAXPROCS=20: CPU quota undefined
/ |/ / / /| | / / / / / __/
/ /|/ // // /| |/ / // /\ \
// /////_/__/
Welcome to use Milvus!
Version: v2.4.6
Built: Tue Jul 16 05:16:09 UTC 2024
GitCommit: ce9ab52
GoVersion: go version go1.21.10 linux/amd64
TotalMem: 105404379136
UsedMem: 29528064
open pid file: /run/milvus/querynode.pid
lock pid file: /run/milvus/querynode.pid
[2024/07/24 22:34:26.062 +00:00] [INFO] [roles/roles.go:307] ["starting running Milvus components"]
[2024/07/24 22:34:26.062 +00:00] [INFO] [roles/roles.go:170] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/07/24 22:34:26.075 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/07/24 22:34:26.077 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["my-milvus-etcd:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2024/07/24 22:34:26.077 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3]
[2024/07/24 22:34:26.080 +00:00] [DEBUG] [config/etcd_source.go:91] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[my-milvus-etcd:2379]"]
[2024/07/24 22:34:26.081 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
[2024/07/24 22:34:26.081 +00:00] [INFO] [paramtable/component_param.go:3566] ["DeployModeEnv is not set, use default"] [default=0.5]
[2024/07/24 22:34:26.084 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/07/24 22:34:26.084 +00:00] [INFO] [roles/roles.go:256] [setupPrometheusHTTPServer]
[2024/07/24 22:34:26.084 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/07/24 22:34:26.084 +00:00] [INFO] [http/server.go:152] ["management listen"] [addr=:9091]
[2024/07/24 22:34:26.084 +00:00] [DEBUG] [querynode/service.go:104] [QueryNode] [port=21123]
[2024/07/24 22:34:26.084 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3]
[2024/07/24 22:34:26.085 +00:00] [DEBUG] [querynode/service.go:124] ["QueryNode connect to etcd successfully"]
[2024/07/24 22:34:26.186 +00:00] [DEBUG] [querynode/service.go:134] [QueryNode] [State=Initializing]
[2024/07/24 22:34:26.186 +00:00] [INFO] [querynodev2/server.go:286] ["QueryNode session info"] [metaPath=by-dev/meta]
[2024/07/24 22:34:26.187 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3]
[2024/07/24 22:34:26.190 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2024/07/24 22:34:26.192 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2024/07/24 22:34:26.195 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true]
[2024/07/24 22:34:26.198 +00:00] [DEBUG] [sessionutil/session_util.go:373] ["Session get serverID success"] [key=id] [ServerId=116]
[2024/07/24 22:34:26.198 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querynode] [address=10.1.29.184:21123] [id=116]
[2024/07/24 22:34:26.199 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querynode-116\n"] [filePath=/tmp/milvus/server_id_8]
[2024/07/24 22:34:26.199 +00:00] [INFO] [querynodev2/server.go:160] ["QueryNode init session"] [nodeID=116] ["node address"=10.1.29.184:21123]
[2024/07/24 22:34:26.199 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=false] [mqType=kafka]
[2024/07/24 22:34:26.204 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-bucket] [root=file]
[2024/07/24 22:34:26.205 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=20]
[2024/07/24 22:34:26.205 +00:00] [INFO] [querynodev2/server.go:326] ["queryNode init scheduler"] [policy=fifo]
[2024/07/24 22:34:26.205 +00:00] [INFO] [segments/segment_loader.go:548] ["SegmentLoader created"] [ioPoolSize=160]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240724 22:34:26.205444 47 knowhere_config.cc:107] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384
I20240724 22:34:26.205590 47 knowhere_config.cc:118] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0
I20240724 22:34:26.205642 47 knowhere_config.cc:54] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.3.5-hotfix
I20240724 22:34:26.206149 47 thread_pool.h:172] [KNOWHERE][InitGlobalSearchThreadPool][milvus] Init global search thread pool with size 80
I20240724 22:34:26.206229 47 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
I20240724 22:34:26.206250 47 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX2
I20240724 22:34:26.206295 47 Common.cpp:35] [SERVER][SetIndexSliceSize][milvus] set config index slice size (byte): 16777216
I20240724 22:34:26.206351 47 Common.cpp:41] [SERVER][SetHighPriorityThreadCoreCoefficient][milvus] set high priority thread pool core coefficient: 10
I20240724 22:34:26.206379 47 Common.cpp:48] [SERVER][SetMiddlePriorityThreadCoreCoefficient][milvus] set middle priority thread pool core coefficient: 5
I20240724 22:34:26.206404 47 Common.cpp:55] [SERVER][SetLowPriorityThreadCoreCoefficient][milvus] set low priority thread pool core coefficient: 1
[2024/07/24 22:34:26.206 +00:00] [INFO] [querynodev2/server.go:227] ["set up knowhere build pool size"] [pool_size=10]
I20240724 22:34:26.206501 47 thread_pool.h:152] [KNOWHERE][InitGlobalBuildThreadPool][milvus] Init global build thread pool with size 10
I20240724 22:34:26.206533 47 Common.cpp:62] [SERVER][SetDefaultExecEvalExprBatchSize][milvus] set default expr eval batch size: 8192
I20240724 22:34:26.206629 47 MinioChunkManager.cpp:161] [SERVER][InitSDKAPIDefault][milvus] init aws with log level:fatal
I20240724 22:34:26.243103 47 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
E20240724 22:34:26.246778 47 MinioChunkManager.cpp:237] [SERVER][PreCheck][milvus] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
[2024/07/24 22:34:26.248 +00:00] [WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"]
[2024/07/24 22:34:26.248 +00:00] [ERROR] [querynodev2/server.go:369] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:369\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:284\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]
[2024/07/24 22:34:26.249 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]
[2024/07/24 22:34:26.249 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]
panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
Anything else?
No response
The text was updated successfully, but these errors were encountered: