Skip to main content

Metabase Cloud SQL Proxy Troubleshooting

This guide helps diagnose and fix Cloud SQL Proxy issues for Metabase on Cloud Run.

Quick Fix Script​

Run the automated fix script (project ID is auto-detected based on environment):

# For staging (auto-detects barto-dev)
./scripts/fix-metabase-cloudsql.sh staging

# For production (auto-detects barto-prod)
./scripts/fix-metabase-cloudsql.sh production

# Or specify project explicitly
./scripts/fix-metabase-cloudsql.sh staging barto-dev
./scripts/fix-metabase-cloudsql.sh production barto-prod

Manual gcloud Commands​

1. Check if Cloud SQL instances exist​

Staging:

# List all SQL instances
gcloud sql instances list --project=barto-dev

# Check Metabase instance (consolidated - hosts both app and metadata databases)
gcloud sql instances describe metabase-db-staging \
  --project=barto-dev \
  --format="table(name,state,connectionName)"

# Verify both databases exist on the instance
gcloud sql databases list --instance=metabase-db-staging --project=barto-dev

Production:

# List all SQL instances
gcloud sql instances list --project=barto-prod

# Check Metabase instance (consolidated - hosts both app and metadata databases)
gcloud sql instances describe metabase-db-production \
  --project=barto-prod \
  --format="table(name,state,connectionName)"

# Verify both databases exist on the instance
gcloud sql databases list --instance=metabase-db-production --project=barto-prod

2. Check current service configuration​

# Get current Cloud SQL instances annotation
gcloud run services describe flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format="value(spec.template.metadata.annotations['run.googleapis.com/cloudsql-instances'])"

# Get current VPC connector
gcloud run services describe flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format="value(spec.template.metadata.annotations['run.googleapis.com/vpc-access-connector'])"

# Get full service configuration
gcloud run services describe flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format=yaml

3. Fix Cloud SQL Proxy configuration​

Staging:

gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --clear-vpc-connector \
  --add-cloudsql-instances=barto-dev:us-central1:metabase-db-staging,barto-dev:us-central1:flowpos-db

Production:

gcloud run services update flowpos-metabase-production \
  --region=us-central1 \
  --project=barto-prod \
  --clear-vpc-connector \
  --add-cloudsql-instances=barto-prod:us-central1:metabase-db-production,barto-prod:us-central1:flowpos-db-production

4. Verify service account permissions​

# Get project number
PROJECT_NUMBER=$(gcloud projects describe barto-dev --format="value(projectNumber)")

# Check if service account has Cloud SQL Client role
gcloud projects get-iam-policy barto-dev \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com AND bindings.role:roles/cloudsql.client" \
  --format="value(bindings.role)"

# Grant Cloud SQL Client role if missing
gcloud projects add-iam-policy-binding barto-dev \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/cloudsql.client"

5. Check latest revision configuration​

# Get latest revision name
LATEST_REVISION=$(gcloud run revisions list \
  --service=flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --format="value(name)" \
  --limit=1)

# Check revision annotations
gcloud run revisions describe "$LATEST_REVISION" \
  --region=us-central1 \
  --project=barto-dev \
  --format="yaml(spec.template.metadata.annotations)"

6. View Cloud Run logs​

# View recent logs
gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=flowpos-metabase-staging" \
  --limit=50 \
  --project=barto-dev \
  --format="table(timestamp,severity,textPayload)"

# Filter for Cloud SQL Proxy logs
gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=flowpos-metabase-staging AND textPayload=~\"cloud.*sql.*proxy\"" \
  --limit=50 \
  --project=barto-dev \
  --format="table(timestamp,severity,textPayload)"

7. Restart the service (to pick up configuration changes)​

# Update the service with a new revision (triggers restart)
gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --no-traffic \
  --tag=temp

# Then route traffic back
gcloud run services update-traffic flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --to-latest

Common Issues​

Issue: Cloud SQL instances annotation not found​

Symptoms:

  • Logs show "Connection to 127.0.0.1:5432 refused"
  • Revision annotations don't include run.googleapis.com/cloudsql-instances

Fix:

gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --add-cloudsql-instances=barto-dev:us-central1:metabase-db-staging

Issue: VPC connector blocking Cloud SQL Proxy​

Symptoms:

  • VPC connector annotation is set
  • Cloud SQL Proxy cannot connect to Cloud SQL

Fix:

gcloud run services update flowpos-metabase-staging \
  --region=us-central1 \
  --project=barto-dev \
  --clear-vpc-connector

Issue: Service account missing permissions​

Symptoms:

  • Cloud SQL Proxy fails to start
  • Permission denied errors in logs

Fix:

PROJECT_NUMBER=$(gcloud projects describe barto-dev --format="value(projectNumber)")
gcloud projects add-iam-policy-binding barto-dev \
  --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
  --role="roles/cloudsql.client"

Issue: Cloud SQL instances don't exist or are not RUNNABLE​

Symptoms:

  • gcloud sql instances describe fails
  • Instance state is not RUNNABLE

Fix:

  1. Verify instances exist: gcloud sql instances list --project=barto-dev
  2. Check instance state: gcloud sql instances describe <instance-name> --project=barto-dev
  3. If instance doesn't exist, create it or update the connection string in the workflow

Verification Checklist​

After fixing, verify:

  1. ✅ Cloud SQL instances exist and are RUNNABLE
  2. ✅ Service account has roles/cloudsql.client permission
  3. ✅ Cloud SQL instances annotation is set correctly
  4. ✅ VPC connector is cleared (not set)
  5. ✅ Service revision shows correct annotations
  6. ✅ Cloud SQL Proxy logs appear in Cloud Run logs

Expected Configuration​

For staging:

  • Service: flowpos-metabase-staging
  • Cloud SQL Instance: barto-dev:us-central1:metabase-db-staging (consolidated - hosts both metabase and flowpos_staging databases)
  • VPC Connector: Not set (cleared)
  • Service Account: {PROJECT_NUMBER}-compute@developer.gserviceaccount.com with roles/cloudsql.client

For production:

  • Service: flowpos-metabase-production
  • Cloud SQL Instance: barto-prod:us-central1:metabase-db-production (consolidated - hosts both metabase and flowpos_production databases)
  • VPC Connector: Not set (cleared)
  • Service Account: {PROJECT_NUMBER}-compute@developer.gserviceaccount.com with roles/cloudsql.client