Custom grpc.Resolver not take effects

[bbiao]

Hi, I deploy Etcd in an dynamic environment, which means the machine Etcd used may changed time to time. Usually I use customized grpc.Resolver to find the new Etcd endpoint from a third-party storage. But since I upgrade Etcd client to v3.5, Etcd itself uses a grpc.Resolver to do the endpoint discovery and load-balancing, my own customized grpc.Resolver no longer take effects.

Etcd setup its own grpc.Resolver (

etcd/client/v3/client.go

Line 286 in cca2003

opts := append(dopts, grpc.WithResolvers(c.resolver))

)

// dialWithBalancer dials the client's current load balanced resolver group.  The scheme of the host
// of the provided endpoint determines the scheme used for all endpoints of the client connection.
func (c *Client) dialWithBalancer(dopts ...grpc.DialOption) (*grpc.ClientConn, error) {
	creds := c.credentialsForEndpoint(c.Endpoints()[0])
	opts := append(dopts, grpc.WithResolvers(c.resolver))
	return c.dial(creds, opts...)
}

After that, user's grpc.DialOptions append to it (

etcd/client/v3/client.go

Line 300 in cca2003

opts = append(opts, c.cfg.DialOptions...)

)

// dial configures and dials any grpc balancer target.
func (c *Client) dial(creds grpccredentials.TransportCredentials, dopts ...grpc.DialOption) (*grpc.ClientConn, error) {
	opts, err := c.dialSetupOpts(creds, dopts...)
	if err != nil {
		return nil, fmt.Errorf("failed to configure dialer: %v", err)
	}
	if c.authTokenBundle != nil {
		opts = append(opts, grpc.WithPerRPCCredentials(c.authTokenBundle.PerRPCCredentials()))
	}

	opts = append(opts, c.cfg.DialOptions...)

	dctx := c.ctx
	if c.cfg.DialTimeout > 0 {
		var cancel context.CancelFunc
		dctx, cancel = context.WithTimeout(c.ctx, c.cfg.DialTimeout)
		defer cancel() // TODO: Is this right for cases where grpc.WithBlock() is not set on the dial options?
	}
	target := fmt.Sprintf("%s://%p/%s", resolver.Schema, c, authority(c.endpoints[0]))
	conn, err := grpc.DialContext(dctx, target, opts...)
	if err != nil {
		return nil, err
	}
	return conn, nil
}

In grpc, used the first grpd.Resolver match the scheme: (https://github.com/grpc/grpc-go/blob/5d3d9d7ca53038fa1a7f003561ea3f7c0870c648/clientconn.go#L1763)

// getResolver finds the scheme in the cc's resolvers or the global registry.
// scheme should always be lowercase (typically by virtue of url.Parse()
// performing proper RFC3986 behavior).
func (cc *ClientConn) getResolver(scheme string) resolver.Builder {
	for _, rb := range cc.dopts.resolvers {
		if scheme == rb.Scheme() {
			return rb
		}
	}
	return resolver.Get(scheme)
}

My questions is:

1. Should we just change the order of parameters when call opts = append(opts, c.cfg.DialOptions...) to move the user DialOptions before the etcd default to make some Options override default? In simple, just change that to opts = append(c.cfg.DialOptions, opts...) ？
2. Or keep current code, is there another way to do the dynamic endpoint discovery?

[chaochn47]

Hi @bbiao

Thanks for reaching out!

I think overriding default etcd resolver is not supported in any public documentation and it is subject to be changed in the future. Please see #15145.

For dynamic endpoint discovery, I have two ideas that both v3.4 and v3.5 clientv3 supports.

1. Manually SetEndpoints

   etcd/client/v3/client.go

   Lines 176 to 183 in cca2003

   	// SetEndpoints updates client's endpoints.
   	func (c *Client) SetEndpoints(eps ...string) {
   	c.epMu.Lock()
   	defer c.epMu.Unlock()
   	c.endpoints = eps
   	c.resolver.SetEndpoints(eps)
   	}

2. Sync synchronizes client's endpoints with the known endpoints from the etcd membership.

   etcd/client/v3/client.go

   Lines 186 to 200 in cca2003

   	func (c *Client) Sync(ctx context.Context) error {
   	mresp, err := c.MemberList(ctx)
   	if err != nil {
   	return err
   	}
   	var eps []string
   	for _, m := range mresp.Members {
   	if len(m.Name) != 0 && !m.IsLearner {
   	eps = append(eps, m.ClientURLs...)
   	}
   	}
   	c.SetEndpoints(eps...)
   	c.lg.Debug("set etcd endpoints by autoSync", zap.Strings("endpoints", eps))
   	return nil
   	}

[bbiao]

Thanks!

Before write this discussion, I have fully reading the issue #15145 , but It makes me more confused whether I should use gprc.Resolver since you plan to remove it. -_-#

I will try the SetEndpoints and Sync method you mentioned.

[ahrtr]

I deploy Etcd in an dynamic environment, which means the machine Etcd used may changed time to time

Could you provide more details to help us understand more about your use case?

Usually I use customized grpc.Resolver to find the new Etcd endpoint from a third-party storage. But since I upgrade Etcd client to v3.5, Etcd itself uses a grpc.Resolver to do the endpoint discovery and load-balancing, my own customized grpc.Resolver no longer take effects.

I am not sure I get the point. Do you mean previously when you were running etcd 3.4.x, your customized gRPC resolver was working well (if yes, could you share more details?), but after upgrading to etcd 3.5, then your customized resolver doesn't work any more?

[bbiao]

I deploy Etcd in an dynamic environment, which means the machine Etcd used may changed time to time

Could you provide more details to help us understand more about your use case?

There are two particular concerns in my case:

1. The machine on which the etcd member is deployed may change from one to another, since the machine may require maintenance or have hardware error. For example, the initial deployment has three members, 10.0.0.1 / 10.0.0.2 / 10.0.0.3, and when the machine (10.0.0.3) encounters a hardware error, I need to add a new member on 10.0.0.4 and remove 10.0.0.3 from cluster. For some reason this can happen quite often in my environment. Over time, the initial members may become three different machines altogether.
2. On the client side, I (as the maintainer of the Etcd cluster) do not provide the user with connection details to Etcd (e.g. username/password/endpoint), but just give them a token. The user use my SDK to look up the details from a central store (eg. zookeeper or a config center), and use that information to create a real Etcd client. This is useful when members change from time to time, as the client does not need to know the initial endpoint.

Usually I use customized grpc.Resolver to find the new Etcd endpoint from a third-party storage. But since I upgrade Etcd client to v3.5, Etcd itself uses a grpc.Resolver to do the endpoint discovery and load-balancing, my own customized grpc.Resolver no longer take effects.

I am not sure I get the point. Do you mean previously when you were running etcd 3.4.x, your customized gRPC resolver was working well (if yes, could you share more details?), but after upgrading to etcd 3.5, then your customized resolver doesn't work any more?

When I use etcd client v3's old version (v3.4.13+incompatible, maybe before this commit: #12671), I write code like this:

	dialOptions = append(dialOptions, grp.WithResolver(myResolver))
	return clientv3.New(clientv3.Config{
...
		DialOptions:          dialOptions,
...
	})
}

My resolver will take effects, but after the client v3 is upgraded to the latest version, my resolver will be ignored since the default grpc.Resolver will take control.

etcd/client/v3/client.go

Line 286 in cca2003

opts := append(dopts, grpc.WithResolvers(c.resolver))

My resolver will be appended after the default resolver. Since grpc will use the first resolver it finds that matches the scheme, later resolvers will have no effect.

[ahrtr]

Got it. thx for the feedback. Indeed, the PR #12671 was breaking your use case; sorry for that.

The solution should be simple, just as @chaochn47 's comment above mentioned, you just need to enable the auto sync by setting a duration for AutoSyncInterval, then the etcd clientv3 will do all the rest thing for you.

It's the simplest solution. But it will talk to the etcd cluster periodically and update the endpoints even there is no any change. If you have concern on this, then you can monitor the etcd membership changes yourself, and call (*Client) SetEndpoints when needed.

[bbiao]

Got it. Thanks!

[cfz]

despite the current unstable situation of the grpc-go resolver api, IMHO, it (or its successor) is a great place to extend etcd client to support in house dynamic deployment.

in our organization, we implemented dynamic etcd server address lookup(non DNS compatible) in golang, java and c++. jetcd does not have a equivalent SetEndpoint()/Sync() api as golang implementation. (i believe most of open source c++ clients don't implement them neither).

Then we found out grpc's resolver machinery is the place that we can share the same address lookup/balance logic for 3 languages (or other language in the future)

it would be great if the new design can take custom resolver into account 😄 cc @chaochn47