[Bug]: find no available rootcoord, check rootcoord state #39246
Assignees
Labels
kind/bug
Issues or changes related a bug
triage/needs-information
Indicates an issue needs more information in order to work on it.
Comments
LiSuiTech
added
kind/bug
|
please offer rootcoord log for debug, thanks. check what is the error message of rootcoord to block it from start |
|
rootCoord: |
|
|
I did not see any blocking errors in the pieces of logs above. @LiSuiTech could you please attache the completed milvus logs? For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs. /assign @LiSuiTech |
yanliang567
added
triage/needs-information
|
我现在使用的是 k8s 安装的 这已经是完整的 milvus.log 了， 并且我使用的是 helm chart 方式部署的，然后上面的是报错，最后就是重复的获取 rootcoord 配置信息连接 所以我不知道他它为什么获取不到，我已在 etcd 中看到存在该数据了！ |
|
/assign @liliu-z |
|
don't see a critical reason of why the cluster can not start. The main focus is to read the rootcoord log and understand why it can not start. |
|
/assign @LiSuiTech |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Environment
Current Behavior
root@etcd-0:/# etcdctl get --prefix ""
by-dev/kv/gid/idTimestamp
�l�
by-dev/kv/gid/timestamp
�m�PG
by-dev/kv/querycoord-id-allocator/idTimestamp
�l['�
by-dev/meta/idTimestamp
�l�%
by-dev/meta/queryCoord-ResourceGroup/__default_resource_group
__default_resource_group�="
�=
by-dev/meta/root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection
root
by-dev/meta/root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeListAliases
root
by-dev/meta/root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail
root
by-dev/meta/root-coord/credential/grantee-privileges/public/Collection/.
e03326696e8a3b16
by-dev/meta/root-coord/credential/grantee-privileges/public/Global/.
250dd41b686083b0
by-dev/meta/root-coord/credential/roles/admin
by-dev/meta/root-coord/credential/roles/public
by-dev/meta/root-coord/credential/users/root
{"encrypted_password":"$2a$04$qGsn3BycCS0vYf9i1mjY7u.uqvS3vcDWKcOjXh./MAYWqcRb7.b2S"}
by-dev/meta/root-coord/database/db-info/1
default (鍮��
by-dev/meta/session/datacoord
{"ServerID":18,"ServerName":"datacoord","Address":"127.0.0.1:13333","Exclusive":true,"TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415509,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/session/datanode-18
{"ServerID":18,"ServerName":"datanode","Address":"127.0.0.1:21124","TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415552,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/session/id
19
by-dev/meta/session/indexcoord
{"ServerID":18,"ServerName":"indexcoord","Address":"127.0.0.1:13333","Exclusive":true,"TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415506,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/session/indexnode-18
{"ServerID":18,"ServerName":"indexnode","Address":"127.0.0.1:21121","TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415451,"HostName":"milvus-579df544f9-nbrb5","EnableDisk":true}
by-dev/meta/session/proxy-18
{"ServerID":18,"ServerName":"proxy","Address":"127.0.0.1:19529","TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415543,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/session/querycoord
{"ServerID":18,"ServerName":"querycoord","Address":"127.0.0.1:19531","Exclusive":true,"TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415522,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/session/querynode-18
{"ServerID":18,"ServerName":"querynode","Address":"127.0.0.1:21123","TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{"CurrentIndexVersion":6},"LeaseID":966748226050415454,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/session/rootcoord
{"ServerID":18,"ServerName":"rootcoord","Address":"127.0.0.1:53100","Exclusive":true,"TriggerKill":true,"Version":"2.5.3","IndexEngineVersion":{},"LeaseID":966748226050415495,"HostName":"milvus-579df544f9-nbrb5"}
by-dev/meta/snapshots/root-coord/database/db-info/1_ts455303077165531137
default (鍮��
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
2025/01/14 09:08:24 maxprocs: Updating GOMAXPROCS=4: determined from CPU quota
Set runtime dir at /run/milvus failed, set it to /tmp/milvus directory
/ |/ / / /| | / / / / / __/
/ /|/ // // /| |/ / // /\ \
// /////_/__/
Welcome to use Milvus!
Version: v2.5.3
Built: Mon Jan 13 07:33:44 UTC 2025
GitCommit: ac730be
GoVersion: go version go1.22.0 linux/arm64
TotalMem: 8589934592
UsedMem: 27688960
open pid file: /tmp/milvus/standalone.pid
lock pid file: /tmp/milvus/standalone.pid
[2025/01/14 09:08:24.517 +00:00] [INFO] [roles/roles.go:345] ["starting running Milvus components"]
[2025/01/14 09:08:24.517 +00:00] [INFO] [roles/roles.go:187] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2025/01/14 09:08:24.524 +00:00] [DEBUG] [runtime/asm_arm64.s:1222] ["start refreshing configurations"] [source=FileSource]
[2025/01/14 09:08:24.525 +00:00] [DEBUG] [paramtable/base_table.go:213] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["http://etcd.shangshan-testing.svc.cluster.local:2379"],"KeyPrefix":"by-dev","CertFile":"","KeyFile":"","CaCertFile":"","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2025/01/14 09:08:24.525 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://etcd.shangshan-testing.svc.cluster.local:2379]"] [minVersion=1.3]
[2025/01/14 09:08:24.538 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[http://etcd.shangshan-testing.svc.cluster.local:2379]"]
[2025/01/14 09:08:24.538 +00:00] [DEBUG] [runtime/asm_arm64.s:1222] ["start refreshing configurations"] [source=EtcdSource]
[2025/01/14 09:08:24.542 +00:00] [DEBUG] [runtime/asm_arm64.s:1222] ["start refreshing configurations"] [source=FileSource]
[2025/01/14 09:08:24.542 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2025/01/14 09:08:24.542 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=stdout]
[2025/01/14 09:08:24.593 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/14 09:08:24.593 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/14 09:08:24.593 +00:00] [WARN] [grpcclient/client.go:473] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/14 09:08:24.593 +00:00] [WARN] [grpcclient/client.go:494] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:493\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:128\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:486\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:589\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n | \t/workspace/source/internal/distributed/proxy/service.go:486\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n | \t/workspace/source/internal/distributed/proxy/service.go:412\n | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n | \t/workspace/source/cmd/components/proxy.go:60\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:130\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2025/01/14 09:08:24.595 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/14 09:08:24.595 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/14 09:08:24.595 +00:00] [WARN] [grpcclient/client.go:480] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/14 09:08:24.595 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:493\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:128\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:486\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:589\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n | \t/workspace/source/internal/distributed/proxy/service.go:486\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n | \t/workspace/source/internal/distributed/proxy/service.go:412\n | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n | \t/workspace/source/cmd/components/proxy.go:60\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:130\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2025/01/14 09:08:24.671 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]
Anything else?
No response
The text was updated successfully, but these errors were encountered: