Database CI/CD and Schema Migration with Redis and GitHub

Apr 14, 2023

Estimated: 30 mins

A series of articles about Database CI/CD and Schema Migration with Redis

Database CI/CD and Schema Migration with Redis
Database CI/CD and Schema Migration with Redis and GitHub (this one)

In the last article Database CI/CD and Schema Migration with Redis, you have tried UI workflow in Bytebase.

This tutorial will bring you to the next level by introducing the GitOps workflow, where you commit the schema change script to the GitHub repository, which will in turn trigger the schema deployment pipeline in Bytebase.

You can use Bytebase free version to finish the tutorial.

Features included

GitOps Workflow

Prerequisites

Before you start this tutorial, make sure:

You have followed our previous UI-based change tutorial Database CI/CD and Schema Migration with Redis.
You have a local running Redis.
You have a GitHub account.
You have a public GitHub repository, e.g redis-test-bb-local.
You have Docker installed locally.
You have a ngrok account.

Step 1 - Run Bytebase in Docker and set the External URL generated by ngrok

ngrok is a reverse proxy tunnel, and in our case, we need it for a public network address in order to receive webhooks from VCS. ngrok we used here is for demonstration purposes. For production use, we recommend using Caddy.

ngrok-reverse-proxy

Run Bytebase in Docker with the following command:

docker run --rm --init \
  --name bytebase \
  --publish 8080:8080 --pull always \
  --volume ~/.bytebase/data:/var/opt/bytebase \
  bytebase/bytebase:2.15.0

Bytebase is running successfully in Docker, and you can visit it via localhost:8080. Register an admin account and it will be granted the workspace admin role automatically.
Login to ngrok Dashboard and complete the Getting Started steps to install and configure. If you want to use the same domain each time you launch ngrok, go to Cloud Edge > Domains, where you'll find the domain <<YOURS>>.ngrok-free.app linked to your account.
Run the ngrok command ngrok http --domain=<<YOURS>>.ngrok-free.app 8080 to start ngrok with your specific domain, and you will see the output displayed below:
Log in Bytebase and click the gear icon (Settings) on the top right. Click General under Workspace. Paste <<YOURS>>.ngrok-free.app as External URL under Network section and click Update.
Now you can access Bytebase via <<YOURS>>.ngrok-free.app.

Step 2 - Find your Redis instance in Bytebase

Visit Bytebase Console through the browser via your ngrok URL. Log in using your account created from the previous tutorial.
If you have followed the last article, you should have a project TestRedis and a database 0.

Step 3 - Connect Bytebase with GitHub.com

Click Settings on the top bar, and then click Workspace > GitOps. Choose GitHub.com and click Next.
Follow the instructions within STEP 2, and in this tutorial, we will use a personal account instead of an organization account. The configuration is similar.
Go to your GitHub account. Click Settings on the dropdown menu.
Click Developer Settings at the bottom of the left side bar. Click OAuth Apps, and click New OAuth App.
Fill Application name and then copy the Homepage and Authorization callback URL in Bytebase and fill them. Click Register application.
After the OAuth application is created successfully. Click Generate a new client secret. Copy Client ID and this newly generated client secret and paste them back in Bytebase.
Click Next. You will be redirected to the confirmation page. Click Confirm and add, and the Git provider is successfully added.

Step 4 - Enable GitOps workflow with Redis

Go to project TestRedis, click GitOps, and choose GitOps Workflow. Click Configure GitOps.
Choose GitHub.com - the provider you just added. It will display all the repositories you can manipulate. Choose redis-test-bb-local.
Keep the default setting, and click Finish.

Step 5 - Change data for Redis by pushing SQL data change files to GitHub

In your GitHub repository redis-test-bb-local, create a folder bytebase, then create a subfolder test, and create an sql file following the pattern {{ENV_ID}}/{{DB_NAME}}##{{VERSION}}##{{TYPE}}##{{DESCRIPTION}}.sql. It is the default configuration for file path template setting under project GitOps.

test/0##2023041410400000##dml##add_country.sql
- test corresponds to {{ENV_ID}}
- 0 corresponds to {{DB_NAME}}
- 2023041410400000 corresponds to {{VERSION}}
- dml corresponds to {{TYPE}}
- add_country corresponds to {{DESCRIPTION}}
Paste the sql script in it.

set country China

Commit and push this file.
Go to Bytebase, and go into project TestRedis. You’ll find there is a new Push Event and a new issue 106 created.
Click issue/106 and go the issue page. Click Resolve issue, and the issue will be Done. You’ll see
- The issue is created via GitHub.com
- The issue is executed without approval because it’s on Test environment where manual approval is skipped by default. The Assignee is Bytebase, because the execution is automatic, and requires no manual approval.
- The SQL is exactly the one we have committed to the GitHub repository.
- The Creator is A, because the GitHub user you use to commit the change has the same email address found in the Bytebase member list.

bb-issue-gitops-change-data-done

Summary and Next

Now you have tried out GitOps workflow, which will store your Redis data in GitHub and trigger the change upon committing the change to the repository, to bring your Redis change workflow to the next level of Database DevOps - Database as Code.

In real world scenario, you might have separate features and main branches corresponding to your dev and production environment, you can check out GitOps with Feature Branch Workflow to learn the setup. Have a try and look forward to your feedback!

Edit this page on GitHub