Run database migrations on cloud builds connected to cloud SQL using private IP


Abhilash Cerankara

I'm trying to setup a database migration for a Nodejs application on a cloud build connecting to Cloud sql using a private IP through a Cloud sql proxy. Cloud SQL connections always fail with cloud builds.

Currently, I'm running migrations manually from compute engine.

I followed this SO to set up the build steps. Running node.js database migrations on Google Cloud SQL during Google Cloud Build

cloudbuild.yaml

steps:
  - name: node:12-slim
    args: ["npm", "install"]
    env:
      - "NODE_ENV=${_NODE_ENV}"
  - name: alpine:3.10
    entrypoint: sh
    args:
      - -c
      - "wget -O /workspace/cloud_sql_proxy https://storage.googleapis.com/cloudsql-proxy/v1.16/cloud_sql_proxy.linux.386 &&  chmod +x /workspace/cloud_sql_proxy"
  - name: node:12
    timeout: 100s
    entrypoint: sh
    args:
      - -c
      - "(/workspace/cloud_sql_proxy -dir=/workspace -instances=my-project-id:asia-south1:postgres-master=tcp:5432 & sleep 3) && npm run migrate"
    env:
      - "NODE_ENV=${_NODE_ENV}"
      - "DB_NAME=${_DB_NAME}"
      - "DB_PASS=${_DB_PASS}"
      - "DB_USER=${_DB_USER}"
      - "DB_HOST=${_DB_HOST}"
      - "DB_PORT=${_DB_PORT}"
  - name: "gcr.io/cloud-builders/gcloud"
    entrypoint: "bash"
    args:
      [
        "-c",
        "gcloud secrets versions access latest --secret=backend-api-env > credentials.yaml",
      ]
  - name: "gcr.io/cloud-builders/gcloud"
    args: ["app", "deploy", "--stop-previous-version", "-v", "$SHORT_SHA"]
timeout: "600s"

mistake:

KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
Step #2:     at Client_PG.acquireConnection (/workspace/node_modules/knex/lib/client.js:349:26)

Cloud build roles:

Cloud Build Service Account
Cloud SQL Admin
Compute Network User
Service Account User
Secret Manager Secret Accessor
Serverless VPC Access Admin

The CLOUD SQL ADMIN API is also enabled.

Version:

NPM libs:
  "pg": "8.0.3"
  "knex": "0.21.1"
dwarf

The Cloud SQL Dedicated IP feature uses internal IP addresses hosted in the VPC network and can only be accessed from other resources in the same VPC network.

Since Cloud Build does not support VPC networking, it is not possible to connect to the private IP of a Cloud SQL instance from Cloud Build.

You may want to check out the official Cloud SQL documentation on this topic to choose another option that suits your use case.

Related


Connect to Cloud SQL from Google Functions using private IP

RmR I have a Google Cloud SQL instance and plan to access it from a Google Function using NodeJS. The code is as described in the documentation // mysql const mysql = require('mysql'); const connectionName = process.env.INSTANCE_CONNECTION_NAME || 'project:reg

Connect to Cloud SQL from Google Functions using private IP

RmR I have a Google Cloud SQL instance and plan to access it from a Google Function using NodeJS. The code is as described in the documentation // mysql const mysql = require('mysql'); const connectionName = process.env.INSTANCE_CONNECTION_NAME || 'project:reg

Cloud Sql Proxy Private IP External Application

thousand Simple question: is there a way to connect to a GCP SQL database under a private IP through a cloud sql proxy from an external application? (local development environment) I followed every step in the official tutorial to configure cloud sql proxy wit

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v

Connect Google Cloud Run Service to Google Cloud SQL Database

Giulio I have 2 Google Cloud services: Google Cloud Run Service (Node Js / Strapi) Google Cloud SQL Service (Mysql) I have added the UI's Cloud SQL connection to Google Cloud Run Service and I have the public IP of Google Cloud SQL Service. On top of that, I'v