Metabase Cloud SQL Proxy Troubleshooting

This guide helps diagnose and fix Cloud SQL Proxy issues for Metabase on Cloud Run.

Quick Fix Script

Run the automated fix script (project ID is auto-detected based on environment):

# For staging (auto-detects barto-dev)
./scripts/fix-metabase-cloudsql.sh staging

# For production (auto-detects barto-prod)
./scripts/fix-metabase-cloudsql.sh production

# Or specify project explicitly
./scripts/fix-metabase-cloudsql.sh staging barto-dev
./scripts/fix-metabase-cloudsql.sh production barto-prod

Manual gcloud Commands

1. Check if Cloud SQL instances exist

Staging:

# List all SQL instances
gcloud sql instances list --project=barto-dev

# Check Metabase instance (consolidated - hosts both app and metadata databases)
gcloud sql instances describe metabase-db-staging \
  --project=barto-dev \
  --format="table(name,state,connectionName)"

# Verify both databases exist on the instance
gcloud sql databases list --instance=metabase-db-staging --project=barto-dev

Production:

# List all SQL instances
gcloud sql instances list --project=barto-prod

# Check Metabase instance (consolidated - hosts both app and metadata databases)
gcloud sql instances describe metabase-db-production \
  --project=barto-prod \
  --format="table(name,state,connectionName)"

# Verify both databases exist on the instance
gcloud sql databases list --instance=metabase-db-production --project=barto-prod

2. Check current service configuration

# Get current Cloud SQL instances annotation
gcloud run services describe flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format="value(spec.template.metadata.annotations['run.googleapis.com/cloudsql-instances'])"

# Get current VPC connector
gcloud run services describe flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format="value(spec.template.metadata.annotations['run.googleapis.com/vpc-access-connector'])"

# Get full service configuration
gcloud run services describe flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format=yaml

3. Fix Cloud SQL Proxy configuration

Staging:

gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --clear-vpc-connector \
  --add-cloudsql-instances=barto-dev:us-central1:metabase-db-staging,barto-dev:us-central1:flowpos-db

Production:

gcloud run services update flowpos-metabase-production \
  --region=us-central1 \
  --project=barto-prod \
  --clear-vpc-connector \
  --add-cloudsql-instances=barto-prod:us-central1:metabase-db-production,barto-prod:us-central1:flowpos-db-production

4. Verify service account permissions

# Get project number
PROJECT_NUMBER=$(gcloud projects describe barto-dev --format="value(projectNumber)")

# Check if service account has Cloud SQL Client role
gcloud projects get-iam-policy barto-dev \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com AND bindings.role:roles/cloudsql.client" \
  --format="value(bindings.role)"

# Grant Cloud SQL Client role if missing
gcloud projects add-iam-policy-binding barto-dev \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/cloudsql.client"

5. Check latest revision configuration

# Get latest revision name
LATEST_REVISION=$(gcloud run revisions list \
  --service=flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format="value(name)" \
  --limit=1)

# Check revision annotations
gcloud run revisions describe "$LATEST_REVISION" \
  --region=us-central1 \
  --project=barto-dev \
  --format="yaml(spec.template.metadata.annotations)"

6. View Cloud Run logs

# View recent logs
gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=flowpos-metabase-staging" \
  --limit=50 \
  --project=barto-dev \
  --format="table(timestamp,severity,textPayload)"

# Filter for Cloud SQL Proxy logs
gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=flowpos-metabase-staging AND textPayload=~\"cloud.*sql.*proxy\"" \
  --limit=50 \
  --project=barto-dev \
  --format="table(timestamp,severity,textPayload)"

7. Restart the service (to pick up configuration changes)

# Update the service with a new revision (triggers restart)
gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --no-traffic \
  --tag=temp

# Then route traffic back
gcloud run services update-traffic flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --to-latest

Common Issues

Issue: Cloud SQL instances annotation not found

Symptoms:

• Logs show "Connection to 127.0.0.1:5432 refused"
• Revision annotations don't include run.googleapis.com/cloudsql-instances

Fix:

gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --add-cloudsql-instances=barto-dev:us-central1:metabase-db-staging

Issue: VPC connector blocking Cloud SQL Proxy

Symptoms:

• VPC connector annotation is set
• Cloud SQL Proxy cannot connect to Cloud SQL

Fix:

gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --clear-vpc-connector

Issue: Service account missing permissions

Symptoms:

• Cloud SQL Proxy fails to start
• Permission denied errors in logs

Fix:

PROJECT_NUMBER=$(gcloud projects describe barto-dev --format="value(projectNumber)")
gcloud projects add-iam-policy-binding barto-dev \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/cloudsql.client"

Issue: Cloud SQL instances don't exist or are not RUNNABLE

Symptoms:

• gcloud sql instances describe fails
• Instance state is not RUNNABLE

Fix:

1. Verify instances exist: gcloud sql instances list --project=barto-dev
2. Check instance state: gcloud sql instances describe <instance-name> --project=barto-dev
3. If instance doesn't exist, create it or update the connection string in the workflow

Verification Checklist

After fixing, verify:

1. ✅ Cloud SQL instances exist and are RUNNABLE
2. ✅ Service account has roles/cloudsql.client permission
3. ✅ Cloud SQL instances annotation is set correctly
4. ✅ VPC connector is cleared (not set)
5. ✅ Service revision shows correct annotations
6. ✅ Cloud SQL Proxy logs appear in Cloud Run logs

Expected Configuration

For staging:

• Service: flowpos-metabase-staging
• Cloud SQL Instance: barto-dev:us-central1:metabase-db-staging (consolidated - hosts both metabase and flowpos_staging databases)
• VPC Connector: Not set (cleared)
• Service Account: {PROJECT_NUMBER}-compute@developer.gserviceaccount.com with roles/cloudsql.client

For production:

• Service: flowpos-metabase-production
• Cloud SQL Instance: barto-prod:us-central1:metabase-db-production (consolidated - hosts both metabase and flowpos_production databases)
• VPC Connector: Not set (cleared)
• Service Account: {PROJECT_NUMBER}-compute@developer.gserviceaccount.com with roles/cloudsql.client