Metabase BI Implementation Plan
Status: 🟢 Deployed (Phase 1, 2, 3 & 4 Complete, Phase 5 Ready) Last Updated: 2025-01-22 Owner: Engineering Team
Overview​
This document tracks the implementation of Metabase Business Intelligence (BI) integration for the FlowPOS multi-tenant POS system across staging and production environments.
Architecture Summary​
- Single Metabase instance per environment (staging/production)
- Two Cloud SQL connections per Metabase instance:
- Metabase Metadata DB: Separate small instance for Metabase internal data
- App Database: Existing POS database for analytics queries
- Multi-tenant isolation: Using locked
business_idparameters in embedded dashboards - Deployment: Cloud Run with autoscaling
- Secrets Management: GitHub Secrets (for GitHub Actions), Doppler (for local development)
Cloud SQL Connection Architecture (socat Approach)​
Why socat? Metabase's PostgreSQL JDBC driver doesn't natively support Cloud SQL Unix sockets. The socat approach bridges this gap by forwarding TCP connections to Unix sockets.
How it works:
-
Cloud Run's Cloud SQL Proxy (via
--add-cloudsql-instances):- Provides Unix sockets at
/cloudsql/PROJECT:REGION:INSTANCE/.s.PGSQL.5432 - No VPC connector needed (Cloud Run handles this internally)
- Provides Unix sockets at
-
socat forwarding (configured in Dockerfile startup script):
- Listens on
127.0.0.1:5432(metadata DB) and127.0.0.1:5433(app DB) - Forwards TCP connections to the corresponding Unix sockets
- Runs as background process before Metabase starts
- Listens on
-
Metabase connection:
- Connects via standard PostgreSQL TCP to
127.0.0.1:5432(metadata) or127.0.0.1:5433(app) - No special JDBC drivers or configuration needed
- Connects via standard PostgreSQL TCP to
Benefits:
- ✅ Works with standard PostgreSQL JDBC driver
- ✅ No VPC connector required
- ✅ Simpler than JDBC Socket Factory approach
- ✅ Automatic socket availability detection
- ✅ Graceful error handling if sockets aren't ready
References:
- Dockerfile:
deploy/gcp/metabase.Dockerfile(startup script with socat) - Workflow:
.github/workflows/deploy-metabase.yml(Cloud SQL instance configuration) - Documentation:
docs/metabase/secrets-socat-approach.md
Environments​
| Environment | GCP Project | Cloud SQL Instance (Consolidated) | Region |
|---|---|---|---|
| Staging | barto-dev | metabase-db-staging ✅ (hosts both app DB flowpos_staging and metadata DB metabase) | us-central1 |
| Production | barto-prod | metabase-db-production ✅ (hosts both app DB flowpos_production and metadata DB metabase) | us-central1 |
Note: Instances have been consolidated to reduce infrastructure costs. Each Metabase instance now hosts both the application database and Metabase metadata database.
Phase 1: Infrastructure Setup​
1.1 Create Metabase Metadata Databases​
Staging Environment (barto-dev)​
-
Create Cloud SQL instance for Metabase metadata:
gcloud sql instances create metabase-db-staging \
--database-version=POSTGRES_15 \
--tier=db-f1-micro \
--region=us-central1 \
--project=barto-dev \
--assign-ipInstance name:
barto-dev:us-central1:metabase-db-staging✅ IP Address:34.136.122.140 -
Create database and user:
CREATE DATABASE metabase;
CREATE USER metabase_user WITH PASSWORD '<strong-password>';
GRANT ALL PRIVILEGES ON DATABASE metabase TO metabase_user; -
Note connection details:
- Connection name:
barto-dev:us-central1:metabase-db-staging✅ - Database name:
metabase✅ - Username:
metabase_user✅ - Password:
pDOiV4mWp0UmC8cmCblyASd7BvQbJY4L(stored insecrets/METABASE_SECRETS_REFERENCE.md)
- Connection name:
Production Environment (barto-prod)​
-
Create Cloud SQL instance for Metabase metadata:
gcloud sql instances create metabase-db-production \
--database-version=POSTGRES_15 \
--tier=db-f1-micro \
--region=us-central1 \
--project=barto-prod \
--assign-ipInstance name:
barto-prod:us-central1:metabase-db-production✅ IP Address:35.224.51.247 -
Create database and user:
CREATE DATABASE metabase;
CREATE USER metabase_user WITH PASSWORD '<strong-password>';
GRANT ALL PRIVILEGES ON DATABASE metabase TO metabase_user; -
Note connection details:
- Connection name:
barto-prod:us-central1:metabase-db-production✅ - Database name:
metabase✅ - Username:
metabase_user✅ - Password:
iNE3VNQz1qYH9OLi64XQwG5K9CVM/fJv(stored insecrets/METABASE_SECRETS_REFERENCE.md)
- Connection name:
Phase 2: Code & Configuration​
2.1 Create Dockerfile​
-
Create
deploy/gcp/metabase.Dockerfile: ✅FROM metabase/metabase:latest
# Install curl for health checks (Metabase image may not have it)
USER root
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
USER metabase
# Metabase runs on port 3000 by default
EXPOSE 3000
# Health check for Cloud Run
# Increased start-period to 120s to allow Metabase full initialization
HEALTHCHECK --interval=30s --timeout=10s --start-period=120s --retries=3 \
CMD curl -f http://localhost:3000/api/health || exit 1 -
Verify Dockerfile builds successfully: ✅
docker build -f deploy/gcp/metabase.Dockerfile -t metabase-test .Status: ✅ Dockerfile tested and verified working
2.2 Create GitHub Actions Workflow​
-
Create
.github/workflows/deploy-metabase.yml: ✅- Use PR-based deployment (triggers on merged PRs) ✅
- Add path filters for Metabase files ✅
- Configure GitHub Secrets integration (updated from Doppler) ✅
- Set up staging/production environment detection ✅
- Configure Cloud SQL instance connections: ✅
- Use
--add-cloudsql-instancesfor both metadata DB and app DB (Cloud Run's built-in Cloud SQL Proxy) - Format:
--add-cloudsql-instances=PROJECT_ID:REGION:INSTANCE_NAME - Use
--clear-vpc-connector(VPC connectors not needed with Cloud Run's Cloud SQL Proxy) - Connection method: socat approach (see Architecture section below)
- Use
- Set Cloud Run resource allocation: ✅
- Memory:
2Gi(Metabase can be resource-intensive) - CPU:
2(for better query performance) - Timeout:
600s(for long-running queries and Metabase initialization) - Min instances:
1(to avoid cold starts) - Max instances:
10(scale based on usage)
- Memory:
- Configure environment variables from GitHub Secrets ✅
- Set port to
3000(Metabase default) ✅ - Add Artifact Registry repository auto-creation ✅
- Add secret validation step ✅
-
Workflow verified and ready: ✅
- Updated to use GitHub Secrets (matching other workflows) ✅
- Tested Dockerfile build ✅
- All deployment steps configured ✅
2.3 Create NestJS Backend Integration​
-
Create Metabase config file: ✅
-
apps/backend/src/config/metabase.config.ts✅ - Read from environment variables: ✅
METABASE_SITE_URLMETABASE_EMBED_SECRET_KEY
-
-
Create Metabase embedding service: ✅
-
apps/backend/src/metabase/metabase-embedding.service.ts✅ - Implement JWT signing logic using
jsonwebtoken✅ - Generate embed URL with locked
business_idparameter ✅ - Add error handling and validation ✅
-
-
Create Metabase controller: ✅
-
apps/backend/src/metabase/metabase.controller.ts✅ - Add authentication guards (
@UseGuards(AuthGuard)) ✅ - Extract
businessIdfrom request using existing pattern: ✅- Check
body.businessId,params.businessId, orquery.businessId - Reference
RolesGuardpattern (seeapps/backend/src/roles/infrastructure/roles.guard.tslines 56-59) - Alternatively, extract from user context if
businessIdis in JWT token
- Check
- Endpoint:
GET /reports/dashboard/:dashboardId/embed-url?businessId=:businessId✅ - Return embed URL as JSON response ✅
-
-
Create Metabase module: ✅
-
apps/backend/src/metabase/metabase.module.ts✅ - Export service and controller ✅
- Import config module ✅
-
-
Install dependencies: ✅
cd apps/backend
pnpm add jsonwebtoken
pnpm add -D @types/jsonwebtoken -
Add module to main app module: ✅
- Import
MetabaseModuleinapps/backend/src/app.module.ts✅
- Import
2.4 Create Frontend Components​
-
Create React component for PWA: ✅
-
apps/frontend-pwa/src/features/reports/MetabaseDashboardFrame.tsx✅ - Automatically gets
businessIdfromuseCurrentBusiness()context (better than planned) ✅ - Fetch embed URL from backend ✅
- Render iframe with proper loading/error states ✅
- Service:
apps/frontend-pwa/src/services/metabaseService.ts✅ - Test component created:
apps/frontend-pwa/src/features/reports/TestDashboard.tsx✅ - Test component integrated into routing (
/reports/TestDashboard) ✅
-
-
Create React component for Web App: ✅
-
apps/web-app/src/features/reports/MetabaseDashboardFrame.tsx✅ - Service hook:
apps/web-app/src/services/client/hooks/use-metabase-service.ts✅
-
-
Create production reports page/route:
- Test component exists and works (
/reports/TestDashboard) ✅ - Create proper reports page (beyond test component):
- Add dedicated reports route/page
- Show available dashboards or dashboard selector
- Integrate
MetabaseDashboardFramecomponent - Add navigation/menu item to access reports
- Improve UX (loading states, error handling, dashboard list)
- Test component exists and works (
Phase 3: Secrets & Configuration​
3.1 Generate JWT Secret​
-
Generate shared secret for JWT signing: ✅
openssl rand -base64 32Secret:
JgFkxtEZcrFLe10MYvewNRN43fkQ8uxOzDpvLUOEsiA=Important: This secret must be identical in both Metabase and backend.
-
Store secret securely: ✅
- Secret stored in
secrets/METABASE_SECRETS_REFERENCE.md✅ - Secret documented in
docs/metabase/github-secrets-reference.md✅ - Will be added to GitHub Secrets (manual configuration required)
- Secret stored in
3.2 Configure GitHub Secrets​
Status: ✅ COMPLETE - All secrets configured in GitHub Actions
Note: Secrets are stored in Doppler for local development, but GitHub Actions uses GitHub Secrets.
Staging Environment (barto-dev)​
-
Configure GitHub Secrets for staging environment: ✅
- Go to: Settings → Secrets and variables → Actions → staging ✅
- Add all required secrets (see
docs/metabase/github-secrets-reference.mdfor complete list and values) ✅ - Required secrets: ✅
METABASE_DB_CONNECTION_URI✅METABASE_EMBED_SECRET_KEY✅METABASE_SITE_URL(will be updated after first deployment) ✅
- Optional secrets configured ✅
-
Also configure backend GitHub Secrets (staging environment): ✅
-
METABASE_SITE_URL(will be updated after first deployment) ✅ -
METABASE_EMBED_SECRET_KEY✅
-
Production Environment (barto-prod)​
-
Configure GitHub Secrets for production environment: ✅
- Go to: Settings → Secrets and variables → Actions → production ✅
- Add all required secrets (see
docs/metabase/github-secrets-reference.mdfor complete list and values) ✅ - Required secrets: ✅
METABASE_DB_CONNECTION_URI✅METABASE_EMBED_SECRET_KEY✅METABASE_SITE_URL(will be updated after first deployment) ✅
- Optional secrets configured ✅
-
Also configure backend GitHub Secrets (production environment): ✅
-
METABASE_SITE_URL(will be updated after first deployment) ✅ -
METABASE_EMBED_SECRET_KEY✅
-
3.3 Verify Secrets​
- Verify all secrets are set in GitHub: ✅
- Check staging environment has all required secrets ✅
- Check production environment has all required secrets ✅
- Reference:
docs/metabase/github-secrets-reference.mdfor complete list ✅
Note: Secrets are also stored in Doppler for local development (see secrets/METABASE_SECRETS_REFERENCE.md)
Phase 4: Deployment​
📚 See detailed guide: docs/metabase/deployment-guide.md
🔧 Helper Scripts:
scripts/verify-metabase-deployment.sh- Verify deployment and get service URLscripts/get-metabase-db-connection.sh- Get database connection details for Metabase UI
4.0 Domain Configuration (Before Deployment)​
Decision: ✅ Option A (Recommended) - Use Cloud Run default URLs initially, update METABASE_SITE_URL after first deployment.
- Decision made: Use Cloud Run default URLs
- Update
METABASE_SITE_URLin GitHub Secrets after first deployment (usescripts/verify-metabase-deployment.sh)
Note: Custom domain mapping (Option B) can be configured later if needed. See docs/metabase/deployment-guide.md for details.
4.1 Deploy to Staging (barto-dev)​
Quick Start:
# 1. Verify GitHub Secrets are configured
# Go to: Settings → Secrets and variables → Actions → staging
# 2. Create and merge PR to develop branch
# (Workflow will auto-deploy on merge)
# 3. Verify deployment
./scripts/verify-metabase-deployment.sh staging
# 4. Update METABASE_SITE_URL in GitHub Secrets (use URL from step 3)
# Go to: Settings → Secrets and variables → Actions → staging
# Update secret: METABASE_SITE_URL = <actual-url>
-
Create PR with Metabase changes: ✅
- Dockerfile ✅
- GitHub Actions workflow ✅
- Backend integration code ✅
- Frontend components ✅
-
Merge PR to
developbranch ✅ -
Verify GitHub Actions workflow runs: ✅
- Check workflow execution in GitHub Actions ✅
- Verify Docker image builds successfully ✅
- Verify Cloud Run deployment succeeds ✅
-
Verify Metabase is accessible: ✅
- Run:
./scripts/verify-metabase-deployment.sh staging✅ - Check Cloud Run service status ✅
- Access Metabase UI at configured URL ✅
- Verify health check endpoint responds ✅
- Run:
-
Update
METABASE_SITE_URLin GitHub Secrets:- Get actual URL from deployment output or verification script ✅
- Update in GitHub Secrets (staging environment)
- Staging URL:
https://flowpos-metabase-staging-3amzmsc2ra-uc.a.run.app
- Staging URL:
- Also update in backend GitHub Secrets (staging environment)
-
Complete Metabase initial setup:
- Create admin user account
- Configure site settings
- Set up email (optional)
4.2 Connect Metabase to App Database (Staging)​
Quick Start:
# Get connection details
./scripts/get-metabase-db-connection.sh staging
-
Access Metabase UI (staging)
-
Add database connection:
- Go to: Settings → Admin → Databases → Add Database
- Database type: PostgreSQL
- Name:
FlowPOS Staging DB - Connection method: Using a connection string (recommended for Cloud SQL)
- Connection string: Use output from
./scripts/get-metabase-db-connection.sh staging- Format:
postgresql://flowpos_staging:<password>@/flowpos_staging?host=/cloudsql/barto-dev:us-central1:flowpos-db
- Format:
- OR Manual configuration (use script output):
- Host:
/cloudsql/barto-dev:us-central1:flowpos-db(Unix socket path) - Port:
5432(or leave empty for Unix socket) - Database name:
flowpos_staging - Username:
flowpos_staging - Password:
[from script output] - Additional JDBC connection string options:
?sslmode=disable
- Host:
- Test connection
- Save connection
-
Verify database sync:
-
Wait for Metabase to complete initial database sync (may take several minutes)
-
Check that tables are visible in Metabase UI
-
Verify
business_idcolumns are present in relevant tables -
Test a simple query:
SELECT COUNT(*) FROM sale WHERE business_id IS NOT NULL;
-
4.3 Deploy to Production (barto-prod)​
Quick Start:
# 1. Verify staging deployment works first!
# 2. Create and merge PR to main branch
# (Workflow will auto-deploy on merge)
# 3. Verify deployment
./scripts/verify-metabase-deployment.sh production
# 4. Update METABASE_SITE_URL in GitHub Secrets
# Go to: Settings → Secrets and variables → Actions → production
# Update secret: METABASE_SITE_URL = <actual-url>
-
Verify staging deployment works correctly first ✅
-
Create PR with Metabase changes ✅
-
Merge PR to
mainbranch ✅ -
Verify GitHub Actions workflow runs: ✅
- Check workflow execution in GitHub Actions ✅
- Verify Docker image builds successfully ✅
- Verify Cloud Run deployment succeeds ✅
-
Verify Metabase is accessible: ✅
- Run:
./scripts/verify-metabase-deployment.sh production✅ - Check Cloud Run service status ✅
- Access Metabase UI at configured URL ✅
- Verify health check endpoint responds ✅
- Run:
-
Update
METABASE_SITE_URLin GitHub Secrets:- Get actual URL from deployment output or verification script ✅
- Update in GitHub Secrets (production environment)
- Production URL:
https://flowpos-metabase-production-mxjtkplbhq-uc.a.run.app
- Production URL:
- Also update in backend GitHub Secrets (production environment)
-
Verify Cloud Run configuration:
- Verify VPC connector is cleared (Cloud Run's Cloud SQL Proxy doesn't need it)
- Verify Cloud SQL connections are configured:
- Metadata DB:
barto-prod:us-central1:metabase-db-production - App DB:
barto-prod:us-central1:flowpos-db-production(configured in workflow) - App DB connection will be added manually via Metabase UI (using socat port 5433)
- Metadata DB:
- Check resource allocation (2Gi memory, 2 CPU, 600s timeout)
- Verify environment variables are set correctly (especially
CLOUD_SQL_INSTANCE_METADATAandCLOUD_SQL_INSTANCE_APP)
-
Complete Metabase initial setup:
- Create admin user account
- Configure site settings:
- Set site name
- Configure timezone
- Disable public sharing (if not already via env var)
- Set up email (optional, for notifications)
4.4 Connect Metabase to App Database (Production)​
Quick Start:
# Get connection details
./scripts/get-metabase-db-connection.sh production
-
Access Metabase UI (production)
-
Add database connection:
- Go to: Settings → Admin → Databases → Add Database
- Database type: PostgreSQL
- Name:
FlowPOS Production DB - Connection method: Using a connection string (recommended for Cloud SQL)
- Connection string: Use output from
./scripts/get-metabase-db-connection.sh production- Format:
postgresql://flowpos_production:<password>@/flowpos_production?host=/cloudsql/barto-prod:us-central1:flowpos-db-production
- Format:
- OR Manual configuration (use script output):
- Host:
/cloudsql/barto-prod:us-central1:flowpos-db-production(Unix socket path) - Port:
5432(or leave empty for Unix socket) - Database name:
flowpos_production - Username:
flowpos_production - Password:
[from script output] - Additional JDBC connection string options:
?sslmode=disable
- Host:
- Test connection
- Save connection
-
Verify database sync:
-
Wait for Metabase to complete initial database sync (may take several minutes)
-
Check that tables are visible in Metabase UI
-
Verify
business_idcolumns are present in relevant tables -
Test a simple query:
SELECT COUNT(*) FROM sale WHERE business_id IS NOT NULL;
-
Phase 5: Dashboard Creation & Configuration​
5.1 Create Sample Dashboards (Staging)​
-
Create test SQL question with
business_idparameter:SELECT
date_trunc('day', s.sale_date) AS day,
SUM(s.total_amount) AS total_sales
FROM sale s
WHERE s.business_id = {{business_id}}
AND s.sale_date BETWEEN {{start_date}} AND {{end_date}}
GROUP BY 1
ORDER BY 1; -
Configure question parameters:
- Define
business_idas UUID parameter - Define
start_dateandend_dateas date parameters - Test with sample
business_id
- Define
-
Create dashboard:
- Add question to dashboard
- Configure dashboard layout
- Set dashboard name:
Sales Dashboard - Staging
-
Configure embedding:
- Go to Dashboard → ⋮ → Embed this dashboard
- Mark
business_idas locked and hidden - Leave
start_dateandend_dateas interactive - Copy dashboard ID (e.g.,
42)
5.2 Test Embedded Dashboard (Staging)​
-
Test backend endpoint: ✅
- Endpoint implemented:
GET /reports/dashboard/:dashboardId/embed-url?businessId=:businessId✅ - JWT signing with locked
business_idparameter ✅
- Endpoint implemented:
-
Test frontend component: ✅
-
MetabaseDashboardFramecomponent created and working ✅ - Test component available at
/reports/TestDashboard✅ - Component automatically gets
businessIdfrom context ✅ - Loading and error states implemented ✅
- Verify dashboard loads in iframe (ready to test once dashboards are created)
- Verify
business_idis locked (user can't change it) - ready to test - Verify date filters work - ready to test
-
-
Verify data isolation:
- Test with different
business_idvalues - Verify each business only sees their own data
- Test that users cannot access other businesses' data
- Test with different
5.3 Create Production Dashboards​
-
Replicate staging dashboards in production:
- Create same SQL questions
- Create same dashboards
- Configure embedding with locked
business_id
-
Document dashboard IDs:
- Staging dashboard IDs:
[list] - Production dashboard IDs:
[list]
- Staging dashboard IDs:
Phase 6: Security Hardening​
6.1 Metabase UI Security​
-
Restrict Metabase UI access:
- Option A: Use Cloud Run IAM authentication
- Option B: IP-restrict ingress
- Option C: Internal domain only
- Recommended: Cloud Run IAM + service account restrictions
-
Configure admin user:
- Use strong password
- Enable 2FA (if available)
- Document admin credentials securely
-
Disable anonymous tracking:
- Set
MB_ANON_TRACKING_ENABLED=falsein Cloud Run env vars
- Set
6.2 Verify Security Configuration​
-
Verify customers cannot access Metabase UI:
- Test that embed URLs work
- Test that direct Metabase URL requires authentication
- Verify
business_idparameter is locked in all dashboards
-
Verify JWT secret matches:
- Check Metabase Cloud Run env var
- Check backend GitHub Secrets config
- Test embed URL generation
6.3 Monitoring & Logging​
-
Set up Cloud Logging alerts:
- Failed authentication attempts
- High error rates (>5% error rate)
- Unusual access patterns
- Database connection failures
-
Monitor Cloud Run metrics:
- CPU usage (alert if >80% for extended periods)
- Memory usage (alert if >85% for extended periods)
- Request latency (alert if p95 >5s)
- Error rates (alert if >1%)
- Instance count (monitor autoscaling behavior)
-
Set up Metabase-specific monitoring:
- Query execution time
- Dashboard load times
- Database sync status
- Connection pool usage
-
Create dashboards:
- Cloud Run service health dashboard
- Metabase usage metrics dashboard
Phase 7: Documentation & Training​
7.1 Documentation​
-
Document dashboard creation process:
- How to create SQL questions with
business_idparameter - How to configure embedding
- How to lock
business_idparameter
- How to create SQL questions with
-
Document API usage:
- Backend endpoint:
GET /reports/dashboard/:dashboardId/embed-url - Required parameters
- Response format
- Backend endpoint:
-
Document frontend integration:
- How to use
MetabaseDashboardFramecomponent - How to pass
businessId
- How to use
7.2 Runbooks​
-
Create troubleshooting guide:
-
Common issues and solutions:
- Metabase won't start: Check Cloud Run logs, verify environment variables, check if socat started successfully
- Database connection fails:
- Verify Cloud SQL instance status (should be RUNNABLE)
- Check if Cloud SQL Proxy is providing Unix sockets (look for
/cloudsql/directory in logs) - Verify socat is running (check for "socat is running" in logs)
- Check service account has
roles/cloudsql.clientpermission - Verify
CLOUD_SQL_INSTANCE_METADATAandCLOUD_SQL_INSTANCE_APPenv vars are set
- Embed URLs not working: Verify JWT secret matches, check dashboard configuration
- High memory usage: Increase Cloud Run memory allocation or optimize queries
- socat connection issues: Check
/tmp/socat-metadata.logand/tmp/socat-app.login container logs
-
How to verify Metabase is running:
# Check Cloud Run service
gcloud run services describe flowpos-metabase-<env> \
--region=us-central1 \
--project=<project-id>
# Check health endpoint
curl https://<metabase-url>/api/health
# Check Cloud Run logs for socat and database connection
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=flowpos-metabase-<env>" \
--limit=50 \
--project=<project-id> \
--format="table(timestamp,severity,textPayload)" | grep -i "socat\|cloud\|sql\|127.0.0.1\|database\|connection" -
How to check database connections:
- Access Metabase UI → Settings → Databases
- Check connection status and last sync time
- Review Cloud SQL logs for connection errors
-
How to regenerate embed URLs:
- Verify JWT secret in both Metabase and backend
- Test embed URL generation via backend endpoint
- Check Metabase embedding settings for dashboard
-
-
Create backup procedures:
-
How to backup Metabase metadata database:
# Export Cloud SQL backup
gcloud sql export sql metabase-db-<env> \
gs://<bucket>/metabase-backup-$(date +%Y%m%d).sql \
--database=metabase \
--project=<project-id> -
How to export dashboard definitions:
- Use Metabase API:
GET /api/dashboard/{id} - Or export via Metabase UI: Dashboard → ⋮ → Download as JSON
- Use Metabase API:
-
How to restore from backup:
- Restore Cloud SQL backup
- Import dashboard definitions via API or UI
- Verify all configurations are restored
-
-
Create deployment runbook:
- Pre-deployment checks
- Deployment steps
- Post-deployment verification
- Rollback procedures
Phase 8: Future Enhancements (Optional)​
8.0 Database Connection Automation (Optional)​
-
Create script to automate database connection setup:
- Use Metabase API to create database connections programmatically
- Script location:
scripts/setup-metabase-db-connection.sh - Accepts environment (staging/production) as parameter
- Reads connection details from GitHub Secrets or Doppler (for local dev)
- Creates connection via Metabase API
-
Alternative: Use Terraform for Metabase configuration:
- Create Terraform module for Metabase setup
- Automate database connection configuration
- Manage dashboards as code (if Metabase supports it)
8.1 Performance Optimization​
-
Enable query caching in Metabase:
- Configure query result caching (Settings → Admin → Caching)
- Set cache TTL based on data freshness requirements
- Monitor cache hit rates
-
Set up result caching for frequently accessed dashboards:
- Enable dashboard caching for popular dashboards
- Configure cache invalidation rules
-
Monitor and adjust Cloud Run resources (CPU/memory):
- Review Cloud Run metrics weekly
- Adjust resources based on actual usage patterns
- Consider scaling up for peak usage times
-
Optimize SQL queries for better performance:
- Add database indexes on frequently queried columns
- Review slow query logs
- Optimize dashboard queries with proper aggregations
8.2 Row-Level Security (RLS)​
- Evaluate need for PostgreSQL RLS
- Create RLS policies for test tables
- Gradually enable RLS on non-critical tables
- Enable RLS on critical tables (sales, inventory, etc.)
8.3 Additional Features​
-
Create more dashboards:
- Inventory dashboard
- Customer analytics dashboard
- Product performance dashboard
- Financial reports dashboard
-
Set up automated reports:
- Scheduled email reports
- Weekly/monthly summaries
Testing Checklist​
Staging Environment​
- Metabase UI is accessible ✅
- Metabase connects to metadata database ✅
- Metabase connects to app database (
flowpos-db) (Phase 4.2) - Can create SQL questions (Phase 5.1)
- Can create dashboards (Phase 5.1)
- Backend endpoint returns valid embed URLs ✅
- Frontend component renders dashboards ✅
- Frontend component automatically gets
businessIdfrom context ✅ - Test component available at
/reports/TestDashboard✅ -
business_idparameter is locked (ready to test once dashboards created) - Data isolation works (different businesses see different data) (ready to test)
- Date filters work correctly (ready to test)
Production Environment​
- Metabase UI is accessible ✅
- Metabase connects to metadata database ✅
- Metabase connects to app database (
flowpos-db-production) (Phase 4.4) - Can create SQL questions (Phase 5.3)
- Can create dashboards (Phase 5.3)
- Backend endpoint returns valid embed URLs ✅
- Frontend component renders dashboards ✅
- Frontend component automatically gets
businessIdfrom context ✅ -
business_idparameter is locked (ready to test once dashboards created) - Data isolation works (different businesses see different data) (ready to test)
- Date filters work correctly (ready to test)
- Security restrictions are in place (Phase 6.1)
- Monitoring is configured (Phase 6.3)
Rollback Plan​
If issues occur during deployment:
-
Metabase deployment fails:
- Check Cloud Run logs
- Verify Cloud SQL connections
- Verify environment variables
- Rollback to previous Cloud Run revision if needed
-
Embedding doesn't work:
- Verify JWT secret matches in both services
- Check backend logs for errors
- Verify dashboard ID is correct
- Test embed URL generation manually
-
Database connection issues:
- Verify Cloud SQL instance is running (state should be RUNNABLE)
- Check IAM permissions (service account needs
roles/cloudsql.client) - Verify Cloud SQL instances are configured in Cloud Run annotations
- Check if socat started successfully (look for "socat is running" in logs)
- Verify Unix sockets are available at
/cloudsql/INSTANCE/.s.PGSQL.5432 - Check environment variables (
CLOUD_SQL_INSTANCE_METADATA,CLOUD_SQL_INSTANCE_APP) - Review socat logs:
/tmp/socat-metadata.logand/tmp/socat-app.log
Notes & Decisions​
Key Decisions​
- Multi-tenant strategy: Using locked
business_idparameters (Option 1) initially - Database separation: Separate Cloud SQL instance for Metabase metadata
- Deployment: Single Metabase instance per environment (not per tenant)
- Security: Embedded dashboards only (no direct Metabase UI access for customers)
- Cloud SQL Connection Method:
- Uses Cloud Run's built-in Cloud SQL Proxy (via
--add-cloudsql-instances) - socat approach: TCP forwarding to Unix sockets (see Architecture section)
- No VPC connector required (Cloud Run handles Cloud SQL Proxy internally)
- Uses Cloud Run's built-in Cloud SQL Proxy (via
- Resource Allocation:
- Memory: 2Gi (to handle complex queries)
- CPU: 2 (for better query performance)
- Timeout: 600s (for long-running queries and Metabase initialization)
- Min instances: 1 (to avoid cold starts)
Important URLs​
Staging:
- Metabase UI:
https://flowpos-metabase-staging-3amzmsc2ra-uc.a.run.app✅ - App Database:
barto-dev:us-central1:flowpos-db - Metadata Database:
barto-dev:us-central1:metabase-db-staging✅ - Cloud Run Service:
flowpos-metabase-staging✅ - Cloud SQL Connection: Via Cloud Run's built-in Cloud SQL Proxy (no VPC connector needed) ✅
- Metadata DB IP:
34.136.122.140(for reference only, not used in connection)
Production:
- Metabase UI:
https://flowpos-metabase-production-mxjtkplbhq-uc.a.run.app✅ - App Database:
barto-prod:us-central1:flowpos-db-production - Metadata Database:
barto-prod:us-central1:metabase-db-production✅ - Cloud Run Service:
flowpos-metabase-production✅ - Cloud SQL Connection: Via Cloud Run's built-in Cloud SQL Proxy (no VPC connector needed) ✅
- Metadata DB IP:
35.224.51.247(for reference only, not used in connection)
Note: ✅ Both deployments complete! Update METABASE_SITE_URL in GitHub Secrets with the actual Cloud Run URLs:
- Staging:
https://flowpos-metabase-staging-3amzmsc2ra-uc.a.run.app - Production:
https://flowpos-metabase-production-mxjtkplbhq-uc.a.run.app
Dashboard IDs​
Staging:
- Sales Dashboard:
[TBD] - Inventory Dashboard:
[TBD] - Customer Analytics:
[TBD]
Production:
- Sales Dashboard:
[TBD] - Inventory Dashboard:
[TBD] - Customer Analytics:
[TBD]
Progress Tracking​
Overall Progress: ~75% (Phase 1, 2, 3 & 4 complete, Phase 5 ready to begin)
- Phase 1: Infrastructure Setup ✅ COMPLETE
- Cloud SQL instances created (staging & production)
- Databases and users created
- JWT secret generated
- Phase 2: Code & Configuration ✅ COMPLETE
- Dockerfile created and tested ✅
- GitHub Actions workflow created and verified ✅
- Workflow updated to use GitHub Secrets (matching other workflows) ✅
- Artifact Registry auto-creation added ✅
- Secret validation step added ✅
- Backend integration complete
- Frontend components created (PWA & Web App) ✅
-
MetabaseDashboardFramecomponent with automaticbusinessIdfrom context ✅ - Service integration complete ✅
- Test component created and integrated (
/reports/TestDashboard) ✅ - Production reports page (beyond test component) - pending
-
- Helper scripts created (verification, DB connection)
- Documentation complete (deployment guide, quick reference, checklist, GitHub Secrets reference)
- Phase 3: Secrets & Configuration ✅ COMPLETE
- JWT secret generated
- All secrets documented:
-
docs/metabase/github-secrets-reference.md(GitHub Secrets) ✅ -
secrets/METABASE_SECRETS_REFERENCE.md(Doppler for local dev) ✅
-
- Setup script created (
scripts/setup-metabase-doppler-secrets.sh) - GitHub Secrets configured ✅
- Configure staging environment secrets ✅
- Configure production environment secrets ✅
- Configure backend secrets (staging & production) ✅
- Phase 4: Deployment ✅ DEPLOYED
- Domain configuration decision made (Cloud Run default URLs)
- Workflow verified and improved (Artifact Registry auto-creation, GitHub Secrets)
- Dockerfile tested and working
- Deployment scripts and documentation ready
- All documentation updated to reference GitHub Secrets ✅
- Staging deployment complete ✅
- Service URL:
https://flowpos-metabase-staging-3amzmsc2ra-uc.a.run.app - Health check verified ✅
- Service URL:
- Production deployment complete ✅
- Service URL:
https://flowpos-metabase-production-mxjtkplbhq-uc.a.run.app - Health check verified ✅
- Service URL:
- Update
METABASE_SITE_URLin GitHub Secrets (staging & production)
- Phase 5: Dashboard Creation
- Phase 6: Security Hardening
- Phase 7: Documentation
- Phase 8: Future Enhancements
Last Review Date: 2025-01-22 Next Review Date: After Metabase initial setup Blockers:
- Done! Deployments complete. ✅
- Update
METABASE_SITE_URLin GitHub Secrets:- Staging:
https://flowpos-metabase-staging-3amzmsc2ra-uc.a.run.app - Production:
https://flowpos-metabase-production-mxjtkplbhq-uc.a.run.app
- Staging:
- Complete Metabase initial setup (create admin user, configure settings)
- Connect Metabase to app databases (staging & production)
Recent Updates (2025-01-22):
- ✅ Updated workflow to use GitHub Secrets instead of Doppler CLI
- ✅ Created comprehensive GitHub Secrets reference documentation
- ✅ Updated all documentation to reference GitHub Secrets
- ✅ Verified Dockerfile build works correctly
- ✅ Added Artifact Registry auto-creation to workflow
- ✅ Added secret validation step to workflow
- ✅ Documented socat approach for Cloud SQL connections
- ✅ Updated timeout from 300s to 600s to match workflow
- ✅ Clarified VPC connector configuration (not needed with Cloud Run's Cloud SQL Proxy)
- ✅ Added troubleshooting steps for socat and database connections
- ✅ Phase 3 & 4 complete - Metabase deployed to staging and production
- ✅ Frontend components verified -
MetabaseDashboardFrameautomatically getsbusinessIdfrom context - ✅ Test component created and integrated at
/reports/TestDashboard