error reading server preface: http2: frame too large

[yriahi]

Description:

When I clone directly from the Docker container, it works as expected:

export DOLT_REMOTE_PASSWORD=password
dolt clone --user my-user http://localhost:8080/my_database my_database
cloning http://localhost:8080/my_database
cd my_database

Note: cloning from a sidecar container (another container within the same ECS task definition) works well too.

However, when I attempt to clone from my Dolt container running in AWS ECS via an Application Load Balancer (ALB), I encounter the following error:

dolt clone --user my-user http://***.elb.amazonaws.com:8080/my_database
cloning http://***.elb.amazonaws.com:8080/my_database

error: failed to get remote db

cause: could not access dolt url 'http://***.elb.amazonaws.com:8080/my_database':
rpc error: code = Unavailable desc = connection error: 
desc = "error reading server preface: http2: frame too large"

Additional Information:

• Dolt version: 1.37.0. with the following remoteapi config

  remotesapi:
   port: 8080
   read_only: true
  

• ALB configuration: HTTP only (so far for internal testing). idle_timeout increased to 120

Please let me know if you need any additional information to troubleshoot this issue.

Thank you.

[timsehn]

Interesting. @reltuk will look into this.

[reltuk]

@yriahi What's the listener and target rule configuration like for ***.elb.amazonaws.com:8080? Is it pointing at a dolt sql-server with a RemotesAPI exposed with something like --remotesapi-port=...?

RemotesAPI requires HTTP/2, and in plaintext mode it typically requires h2c, but ALB doesn't support h2c, at least at the client-to-LB hop. In this case, I think it might work if you:

1. Make the listener protocol HTTP/2.
2. Make the target group protocol HTTP/2.
3. Change the client URL to use HTTPS, as per ALB HTTP/2 requirements.

Is that something that would be easy for you to try out?

[yriahi]

Hi @reltuk
Thanks for your reply. I made the changes below, and I am getting a different error now - 502 (Bad Gateway). The service is running and I can clone directly from the dolt container while on a utility side car container.

1. I changed the listener to HTTPS
   

2.Ttarget group is on HTTPS, protocol version HTTP2, healthy ( I have 400 as a "good" healthy code - is there a proper remote api status health path?)

3. Added a certificate to the ALB, and attempted a clone on https

export DOLT_REMOTE_PASSWORD=****
dolt clone --user my-user https://my-clone-dolt.com:8080/getting_started
cloning https://my-clone-dolt.com:8080/getting_started
error: failed to get remote db
cause: could not access dolt url 'https://my-clone-dolt.com:8080/getting_started': rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway)

Thanks!

[reltuk]

@yriahi I think the target group protocol should be HTTP, with protocol version will HTTP2.

[reltuk]

And to answer the question about the health check endpoint:

Currently the remotesapi endpoint on sql-server does not expose a health check endpoint. A TCP ping or an HTTP request as something like GET /healthz marking healthy on a 4xx response is probably the right thing for now.

[yriahi]

@reltuk - Thanks for catching that! Target Group protocol changed to HTTP

I got rid of the HTTPS 8080 ALB listener; and swapped to 443 instead for ease of use.

Looking better now with this initial message

...then this error ... :80 i/o timeout

I don't have anything anywhere in Terraform code with port 80; not sure where that is coming from. I will keep digging.

[reltuk]

@yriahi Ahh, really good catch. Ok, so things probably work work end-to-end on the load balancer side.

What's happening here is this:

The remotesapi protocol generates links to itself and embeds them in responses to the dolt client. The dolt client then makes further requests to those links. In this case, the server itself is generating links for http://... instead of https://..., because the server itself is running a non-TLS listener.

The right thing to do is for the remotesapi server to respect X-Forwarded-Proto, and to generate the links with a scheme of https if the client is originally requesting with a scheme of https. I've opened a PR for that here: #7967.

I will keep you posted on when that fix is finalized, reviewed and released.

In the mean time, I think you might be able to make it work with the existing version of dolt if you also create an HTTP listener on port 80 of your ALB. I think it can forward to the same target group (if that's allowed in AWS?) or to a new one that forwards traffic to the same port. I think the traffic can either be forwarded at HTTP/1.1 or HTTP/2, but it should not be HTTPS.

Thanks for your patience and I hope to follow up shortly.

[yriahi]

@reltuk thanks for working on this and submitting the PR.
I'll be creating/testing the non-TLS listener and report back.

[reltuk]

Didn't mean to close this until at least the release goes out. Sorry about that!

[reltuk]

Closing this for now, as the x-forwarded-proto change is the in latest Dolt release, 1.39.4. Please feel free to reopen or comment here if you have any further questions or need continued ideas about iterating on getting ALBs working with older versions of Dolt.