Database GitOps with Bitbucket Pipelines

Estimated: 40 mins

This is part of our database GitOps series with Bytebase:


This tutorial shows you how to build an database GitOps workflow using Bitbucket Pipelines and Bytebase API. You'll learn to create a streamlined database release workflow where you can:

  • Submit schema migrations through Bitbucket
  • Automatically run SQL reviews on pull requests
  • Auto-create and deploy Bytebase releases when merging to main

While we use Bitbucket Pipelines in this guide, you can apply these concepts to other CI platforms like GitHub Actions, GitLab CI, or Azure DevOps using the Bytebase API.

While we use PostgreSQL with Bitbucket Pipelines in this guide, you can apply these concepts to other SQL or NoSQL databases with any CI platforms like GitHub Actions, GitLab CI, or Azure DevOps using the Bytebase API.

Repository

https://bitbucket.org/p0nyyy/cicd/

Prerequisites

Automatic Rollout across environments

Step 1 - Start Bytebase with ngrok

ngrok is a reverse proxy tunnel, and in our case, we need it for a public network address to allow VCS to call the Bytebase API. ngrok we used here is for demonstration purposes.

ngrok-reverse-proxy

  1. Run Bytebase in Docker with the following command:

    docker run --rm --init \
      --name bytebase \
      --publish 8080:8080 --pull always \
      --volume ~/.bytebase/data:/var/opt/bytebase \
      bytebase/bytebase:3.6.1
  2. Bytebase is running successfully in Docker, and you can visit it via localhost:8080. Register an admin account and it will be granted the workspace admin role automatically.

  3. Login to ngrok Dashboard and complete the Getting Started steps to install and configure. If you want to use the same domain each time you launch ngrok, go to Cloud Edge > Domains, where you'll find the domain <<YOURS>>.ngrok-free.app linked to your account.

  4. Run the ngrok command ngrok http --domain=<<YOURS>>.ngrok-free.app 8080 to start ngrok with your specific domain, and you will see the output displayed below:

    terminal-ngrok

  5. Log in Bytebase and click the gear icon (Settings) on the top right. Click General under Workspace. Paste <<YOURS>>.ngrok-free.app as External URL under Network section and click Update.

    external-url

  6. Now you can access Bytebase via <<YOURS>>.ngrok-free.app.

Step 2 - Create Service Account

  1. Log in as Workspace Admin, and go to IAM & Admin > Users & Groups. Click + Add User, fill in with api-sample, choose the Workspace DBA role sufficient for this tutorial and click Confirm. service-account-create

  2. Find the newly created service account and Copy Service Key. We will use this token to authenticate the API calls. service-account-key

If you have Enterprise Plan, you can create a Custom Role for the service account which require fewer permissions, and assign this role instead of DBA:

  • plans.create
  • plans.get
  • plans.preview
  • releases.check
  • releases.create
  • releases.get
  • rollouts.create
  • rollouts.get
  • rollouts.list
  • sheets.create
  • sheets.get
  • taskRuns.create
  • planCheckRuns.list
  • planCheckRuns.run

Step 3 - Configure SQL Review in Bytebase

Since you will need to run SQL review on your PRs, you need to configure the SQL review in Bytebase.

  1. Go to CI/CD > SQL Review, click Create SQL Review.

  2. Select the Sample Template and click Next. bb-sql-review-sample

  3. Select Prod environment as the attached resources and click Confirm. Now the SQL review is enabled for the Prod environment. bb-sql-review-prod

Note: Usually we enable SQL review for Prod environment as above. In this demo, we would switch to enable it for Test to fit the following GitLab CI workflow.

Step 4 - Copy the Example Repository and Configure Variables

  1. Create a new repository and copy the configuration file bitbucket-pipelines.yml from https://bitbucket.org/p0nyyy/cicd/. It includes sql review and release creation.

  2. Go into bitbucket-pipelines.yml, replace the variable in all 3 steps the values with your own and commit the changes.

    • BYTEBASE_URL: your ngrok url
    • BYTEBASE_SERVICE_ACCOUNT: api-example@service.bytebase.com (the service account you created in the previous step)
    • BYTEBASE_SERVICE_ACCOUNT_SECRET: the password of the service account

Step 5 - Configure in Bitbucket

Before creating the migration files, you need to configure the environment in Bitbucket to match the environment in Bytebase.

  1. Go to Repository settings > Pipelines > Deployments. Rename the environment name to Test-> test and Production-> prod.

    bitb-deployments

  2. Go to Pipelines to enable it for the pull request to the main branch. Without this the pipeline will not be triggered.

    bitb-enable-pipelines

Step 6 - Create the migration files

To create migration files to trigger release creation, the files have to match the following pattern:

  • A migration file should start with digits, which is also its version. e.g. 202505141650_create_table_t1.sql.
  • A migration file may end with ddl or dml to indicate its change type. If it doesn't end with any of the two, its change type is DDL by default.
  1. Within your repository, create the following migration files under migration directory:

    • 202505141650_create_table_t1.sql
    CREATE TABLE t1 (
     id SERIAL PRIMARY KEY,
     name TEXT
    );
  2. Commit to a new branch and create a pull request, the sql-review pipeline will be triggered. There will be a warning in the SQL review result.

    bitb-notnull

  3. According to the SQL review result, you can do some changes to the SQL files and push to the branch. Then you should see the SQL review has passed. There are no warnings in the SQL review result.

    CREATE TABLE t1 (
     id SERIAL PRIMARY KEY,
     name TEXT NOT NULL
    );
  4. When the SQL review is passed, you can merge the pull request. The release pipeline will be triggered to create a release in Bytebase and then roll out automatically on test but waiting for approval on prod.

  5. Click into the pipelines and choose the merge pipeline, you can see release is created in test environment but waiting for approval on prod.

    bitb-waiting-prod

  6. If you expand the log for bytebase-action rollout ..., you can follow the rollout link to Bytebase.

    bb-rollout-test

  7. Click Deploy on deploy-to-prod stage in Bitbucket, the release will be deployed to prod environment.

    bitb-deploy-prod

    bb-rollout-prod

Summary

Now you have learned how to database GitOps with Bitbucket Pipelines. If you want to trigger a release creation with other git providers (e.g. GitHub, GitLab, Azure DevOps), you may customize the workflow file.

Edit this page on GitHub