Unable to connect to GCP SQL when using --private-ip flag

[mudasirmirza]

Question

When using Cloud SQL Proxy with the --private-ip flag to connect a MySQL instance hosted in a private IP setup, the connection fails inside a GKE pod. Without the --private-ip flag, the proxy works fine over a public IP, but the private IP configuration fails to function as expected.

The setup uses:

1. A GKE cluster in VPC A with VPC peering to the default VPC (where the Cloud SQL instance is accessible).
2. MySQL is configured to run on port 3306 with Private IP in default VPC
3. The Cloud SQL Proxy is deployed in the same pod as a MySQL client to handle local connections in the GKE cluster running in VPC A
4. Everything is in the same project

Logs when using --private-ip, for some reason, it connects to port 3307

❯ kubectl logs ubuntu-pod-cloud-sql -c cloud-sql-proxy -f
{"severity":"INFO","timestamp":"2024-10-28T11:35:09.344Z","message":"Authorizing with Application Default Credentials"}
{"severity":"INFO","timestamp":"2024-10-28T11:35:09.391Z","message":"[project:region:instance_id] Listening on [::]:3306"}
{"severity":"INFO","timestamp":"2024-10-28T11:35:09.391Z","message":"The proxy has started successfully and is ready for new connections!"}
{"severity":"INFO","timestamp":"2024-10-28T11:35:09.391Z","message":"Enabling pprof endpoints at localhost:9091"}
{"severity":"INFO","timestamp":"2024-10-28T11:36:10.807Z","message":"[project:region:instance_id] accepted connection from 127.0.0.1:60080"}
{"severity":"ERROR","timestamp":"2024-10-28T11:36:40.807Z","message":"[project:region:instance_id] failed to connect to instance: Dial error: failed to dial (connection name = \"project:region:instance_id\"): dial tcp 10.128.192.7:3307: i/o timeout"}

Below is the YAML manifest used to create the POD

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-pod-cloud-sql
  namespace: default
spec:
  containers:
  - name: ubuntu
    image: ubuntu:latest
    command: ["/bin/bash"]
    args: ["-c", "apt-get update && apt-get install -y mysql-client && sleep infinity"]
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi
    volumeMounts:
      - name: service-account
        mountPath: /secrets/
        readOnly: true

  - name: cloud-sql-proxy
    image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.11.4
    args:
      - "--private-ip"
      - "--structured-logs"
      - "--debug"
      - "--port=3306"
      - "--address=0.0.0.0"
      - "project_id:region:instance_id"
      - "--credentials-file=/secrets/key.json"
    securityContext:
      runAsNonRoot: true
    volumeMounts:
      - name: service-account
        mountPath: /secrets/
        readOnly: true

  volumes:
  - name: service-account
    secret:
      secretName: cloud-sql-sa-key

GCP SQL instances is currently configured to use "Public IP" and "Private IP" and there is a list of IPs in the "authorized networks" which has the IP from which the call is being made.

Expected Behavior:

• The Cloud SQL Proxy should allow connections from the pod to the MySQL instance using private IP over port 3306

1. VPC Peering has been established between VPC A and default VPC
2. A valid firewall rule also present on the default VPC to allow IP range of VPC A

After doing all this, I am stuck and unable to make the connection.

Code

No response

Additional Details

No response

[mudasirmirza]

Just to add something I missed. SQL instance only accepts SSL connections

[jackwotherspoon]

Hi @mudasirmirza thanks for the great question! 😄

This appears to be a duplicate question of #1844

TLDR;

• The --private-ip flag is used for Cloud SQL instances with a private IP configured via Private Service Access (PSA).
• PSA VPC peerings are not transitive.
• Cloud SQL instances with a private IP configured via Private Service Connect (PSC) do not run into the issue (--psc flag)

I'll repost the solution of the other issue here for clarity:

Transitive VPC limitation:
One limitation of PSA VPC Network Peering is that VPC peering only broadcasts routes between two VPCs that are directly peered. As a result, VPC Peering isn’t transitive. For example, two networks that are each peered to an intermediary network are unable to communicate directly to one another.

In your instance you have provisioned a Cloud SQL instance with PSA private IP from the default VPC (VPC B in image). Internally, Cloud SQL deploys the instance in its own managed VPC, the “Cloud SQL VPC”, VPC C and automatically peers it with the VPC you configured the instance with (VPC B). Any authorized Google Cloud resources in VPC B may route traffic to the Cloud SQL instances over private IP. However, if you have an additional separate VPC, VPC A, these Google Cloud resources will not be able to directly access your Cloud SQL resources over private IP through VPC B (intermediate VPC). Even if you’ve set up a peering connection between your VPC B and your VPC A ( your inactive peering), resources in VPC A will be unable to connect over private IP due to the nature of transitive VPC peering through private service access.

We have a dedicated docs page "Connector your instance to multiple VPC" that provides a list of solutions around this transitivity issue.

I would recommend configuring your instance to connect via Private Service Connect, this solution mitigates the entire transitive VPC issue and was designed with this exact use-case in mind.

If you have any follow up questions please let me know.

Have a great day 😄

[mudasirmirza]

Hi @jackwotherspoon

Thanks for your quick and detailed response.

The biggest limitation with PSC is that the instance is already present and is being used heavily which means I just can not make that change.

I went through the links that your provided and the only method I think might work is using Cloud Router, but I am unable to find any example of that. It will be great if you can share an example of some scenarios to configure cloud router.

Regards,
Mudasir.

[jackwotherspoon]

I went through the links that your provided and the only method I think might work is using Cloud Router, but I am unable to find any example of that. It will be great if you can share an example of some scenarios to configure cloud router.

@mudasirmirza The Cloud Router section has a "how-to" walkthrough of setting up two HA VPN gateways with Cloud Router, I would give it a look through and see if it meets your needs.

[jackwotherspoon]

@mudasirmirza It is worth pointing out and adding that on Nov. 15th the ability to configure an existing private IP (PSA) instance with a PSC IP will go into Public Preview.

[mudasirmirza]

Hi @jackwotherspoon

This is good to know. I will be trying out the HA VPN setup and see if that works out

[jackwotherspoon]

@mudasirmirza any luck getting a working setup? Let me know if I can assist in any other way 😄

[mudasirmirza]

Hi @jackwotherspoon ,

I tried to setup VPN but to no success. Not sure if it will work with my current setup.

[jackwotherspoon]

on Nov. 15th the ability to configure an existing private IP (PSA) instance with a PSC IP will go into Public Preview.

@mudasirmirza It is quite a complex setup to get working from my own experience. I'll keep you posted with the Public Preview launching next week for enabling PSC on a private IP (PSA) instance. It is definitely the easiest path forward 😄

[mudasirmirza]

Hi @jackwotherspoon,

Please let me know if this feature has been release and if there is any documentation for this. I was trying to check this in the settings of the instance but unable to see anything.