White Paper: Just-in-Time Database Access Best Practices

Self-host Bytebase

Latest release version: 3.6.1

Self-host is for business with security/compliance requirements. Bytebase offers an air-gapped deployment with a single Go binary.

Prerequisites

Docker

Estimated time: 5 minutes.

  • Docker version must be at least 20.10.24 or pass --security-opt seccomp=unconfined to docker run, otherwise, you will get pthread_create failed error.

  • If accessing from China, pull image registry.cn-shanghai.aliyuncs.com/bytebase/bytebase:3.6.1.

Installation

docker run --rm --init \
  --name bytebase \
  --publish 8080:8080 --pull always \
  --volume ~/.bytebase/data:/var/opt/bytebase \
  bytebase/bytebase:3.6.1

Once you see the Bytebase logo, you can access the console at http://localhost:8080.

ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā•—   ā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—  ā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—
ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā•šā–ˆā–ˆā•— ā–ˆā–ˆā•”ā•ā•šā•ā•ā–ˆā–ˆā•”ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā•ā•ā•
ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā• ā•šā–ˆā–ˆā–ˆā–ˆā•”ā•    ā–ˆā–ˆā•‘   ā–ˆā–ˆā–ˆā–ˆā–ˆā•—  ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā•—
ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—  ā•šā–ˆā–ˆā•”ā•     ā–ˆā–ˆā•‘   ā–ˆā–ˆā•”ā•ā•ā•  ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•‘ā•šā•ā•ā•ā•ā–ˆā–ˆā•‘ā–ˆā–ˆā•”ā•ā•ā•
ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•   ā–ˆā–ˆā•‘      ā–ˆā–ˆā•‘   ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā•‘  ā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—
ā•šā•ā•ā•ā•ā•ā•    ā•šā•ā•      ā•šā•ā•   ā•šā•ā•ā•ā•ā•ā•ā•ā•šā•ā•ā•ā•ā•ā• ā•šā•ā•  ā•šā•ā•ā•šā•ā•ā•ā•ā•ā•ā•ā•šā•ā•ā•ā•ā•ā•ā•

Version 3.6.1 has started on port 8080 šŸš€

Configuration

Use external PostgreSQL to store metadata

By default, Bytebase will use an embedded PostgreSQL database to store metadata. For production usage, it is recommended to use an external PostgreSQL database instead. Check Configure External PostgreSQL for details.

Customize startup options

If you need more control over the server configuration, check other Server Startup Options.

Allow external access via External URL

Check Configure External URL for details.

Enable WebSocket for SQL Editor

SQL Editor autocomplete requires WebSocket. If you access Bytebase via a gateway, you need to enable WebSocket there. Here is an sample NGINX configuration (including the optional HTTPS mentioned below):

http {
    map $http_upgrade $connection_upgrade {
      default upgrade;
      '' close;
    }

    server {
        listen       80;
        listen  [::]:80;
        # Listen HTTPS
        listen       443 ssl;
        listen  [::]:443 ssl;
        server_name  bytebase.example.com;

        # SSL cert and key
        ssl_certificate /path/to/certificate/file;
        ssl_certificate_key /path/to/private/key/file;

       location ~ ^/(v1:adminExecute|lsp) {
            proxy_pass http://bytebase.example.com;
            proxy_http_version 1.1;
            # Enables WebSocket which is required for SQL Editor autocomplete
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            proxy_pass http://bytebase.example.com;
        }

        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
    }
}

Enable HTTPS

Bytebase does not support enabling HTTPS in server configuration. We suggest use NGINX or Caddy as a reverse proxy in front of Bytebase to enable HTTPS.

Troubleshooting

bind: address already in use

If you see bind: address already in use error, it means the port 8080 is already in use on your host. You need to either stop the existing process using the port or configure Bytebase to use a different port via --publish <<YOUR_PORT>>:8080.

Connect database instance on the same host

  • If you run Bytebase inside Docker on Linux, then you need to supply the additional --network host flags in docker run command. This allows Bytebase to connect to database instance running on the same host with localhost.
  • If you run Bytebase inside Docker Desktop on Mac , then you need to use host.docker.internal to connect to database instance running on the same host.

Manifest not found

There may be a few reasons the manifest file is not found:

  • The docker image only supports linux/amd64 and linux/arm64 arch. If it doesn't match your OS arch, you may supply --platform linux/amd64 as a best effort.
  • Your Docker version is too old and doesn't support manifest list. Please install the latest Docker version.

Unable to start using Colima

Due to the vm mechanism of Colima, try to use the --mount option when starting colima as shown below:

mkdir ~/volumes
colima start --mount ~/volumes:w
docker run --init \
  --name bytebase \
  --restart always \
  --publish 80:8080 \
  --volume ~/.bytebase/data:/var/opt/bytebase bytebase/bytebase:3.6.1 \
  --data /var/opt/bytebase \
  --external-url http://bytebase.example.com \
  --port 8080

Kubernetes

Estimated time: 15 minutes.

Deploy to Kubernetes

Make sure to set the replicas to 1, otherwise, it may cause data race issues.

Here is a sample Kubernetes YAML file bb.yaml describing the minimal components and configuration required to run Bytebase in Kubernetes.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: bytebase
  namespace: default
spec:
  # To prevent data races, only request one replica.
  replicas: 1
  selector:
    matchLabels:
      app: bytebase
  template:
    metadata:
      labels:
        app: bytebase
    spec:
      containers:
        - name: bytebase
          image: bytebase/bytebase:3.6.1
          imagePullPolicy: Always
          env:
            - name: PG_URL
              value: 'postgresql://<<user>>:<<secret>>@<<host>>:<<port>>/<<dbname>>'
          args:
            [
              '--data',
              '/var/opt/bytebase',
              '--external-url',
              'http://bytebase.example.com',
              '--port',
              '8080',
            ]
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 300
            periodSeconds: 300
            timeoutSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: bytebase-entrypoint
  namespace: default
spec:
  # Optional
  type: ClusterIP
  selector:
    app: bytebase
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

  1. Start Bytebase with the following command:

    kubectl apply -f bb.yaml

  2. Make sure everything worked by listing your deployments:

    kubectl get statefulsets

    Do the same check for your services:

    kubectl get services

  3. Open a browser and visit http://localhost, you should see Bytebase.

Upgrade

When a new Bytebase release is published, you can change the image version in the yaml file

containers:
  - name: bytebase
    image: bytebase/bytebase:3.6.1

Use Helm Chart

Production Setup External URL

For production setup, you should configure a proper External URL.

Installing the Chart

helm -n <YOUR_NAMESPACE> \
--set "bytebase.option.port"={PORT} \
--set "bytebase.option.external-url"={EXTERNAL_URL} \
--set "bytebase.option.externalPg.url"={PGDSN} \
--set "bytebase.version"={VERSION} \
install <RELEASE_NAME> bytebase-repo/bytebase

For example:

helm -n bytebase \
--set "bytebase.option.port"=443 \
--set "bytebase.option.external-url"="http://bytebase.example.com" \
--set "bytebase.option.externalPg.url"="postgresql://user:secret@foo.ap-east-1.rds.amazonaws.com/postgres" \
--set "bytebase.version"=1.7.0 \
install bytebase-release bytebase-repo/bytebase

Uninstalling the Chart

helm delete --namespace <YOUR_NAMESPACE> <RELEASE_NAME>

Upgrade Bytebase Version/Configuration

Use helm upgrade command to upgrade the bytebase version or configuration.

helm -n <YOUR_NAMESPACE> \
--set "bytebase.option.port"={NEW_PORT} \
--set "bytebase.option.external-url"={NEW_EXTERNAL_URL} \
--set "bytebase.option.externalPg.url"={NEW_PGDSN} \
--set "bytebase.version"={NEW_VERSION} \
upgrade bytebase-release bytebase-repo/bytebase

Deploy with Ingress

We use Ingress-Nginx Controller as ingress controller. You need to config Ingress-Nginx Controller according to your environment.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    # https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#websockets
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
  name: bytebase-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: bytebase-entrypoint
                port:
                  number: 80
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - example.com
      secretName: tls-secret

Note If you use ingress, make sure to use https to access bytebase;

External PostgreSQL

Instead of specify PostgreSQL connection string in helm or Kubernetes yaml file, we allows users to using Kubernetes secrets resources.

Kubernetes

Using the following yaml section to replace the spec.templates.spec.containers.env section:

env:
  - name: PG_URL
    valueFrom:
      secretKeyRef:
        name: secret_name
        key: secrete_key

Helm

Using --set bytebase.option.existingPgURLSecret and --set bytebase.option.existingPgURLSecretKey to specify the secret key and secret name instead of --set "bytebase.option.external-url"={NEW_EXTERNAL_URL}. See more details in Bytebase - Artifact Hub.

Persistent Volume

We don't recommend this. However, if you do not configure External PostgreSQL, then to persist data, you need to use the Persistent Volumes in the cluster. Each cloud provider has its own solution.

For Amazon Elastic Kubernetes Service(EKS)

In AWS EKS, you can use the Amazon EBS CSI driver for persistent volumes. Follow the managing EBS CSI to add it as an Amazon EKS add-on.

Here is a simple example to use an EBS volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bytebase-ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: bytebase-resize-sc
  resources:
    requests:
      storage: 4Gi
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: bytebase-resize-sc
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true

Note Also need to update the statefulset spec of bytebase to replace the emptyDir volume with persistentVolumeClaim:

volumes:
  - name: bytebase-volume
    persistentVolumeClaim:
      claimName: bytebase-ebs-claim

For Google Kubernetes Engine(GKE)

Please follow the Persistent volumes and dynamic provisioning.

AWS Fargate

Create an ECS Cluster

  1. Go to AWS Management Console and navigate to the ECS (Elastic Container Service).
  2. Click Clusters on the left menu and then click Create cluster.
  3. Fill in the cluster name, choose AWS Fargate (serverless) under Infrastructure.
  4. Click Create.

Create an ECS Task Definition

  1. Click Task Definitions on the left menu and then click Create new task definition.
  2. Choose AWS Fargate as the launch type under Infrastructure requirements.
  3. Add a Container, fill in Image URL with bytebase/bytebase:latest and Container port 8080.
  4. Click Create.

Run the Task

  1. Go to the created cluster and click Create under Services.
  2. Under Environment > Compute configuration, choose FARGATE as the launch type.
  3. Under Deployment configuration, Choose Service as the Application type, choose the task definition you created in the previous step, give it a name such as bytebase-service.
  4. Click Create.

Access Bytebase

  1. Go to the service you just created and click Logs tab to see the logs. If you see something like Version 3.6.1(f5891735834ea82fd67bf8181597f86a7f5dab58) has started on port 8080 šŸš€, it means Bytebase is running.
  2. Click Tasks tab to see the task list. Click the task name to see the task details.
  3. Under Cointainer details for bytebase, click Network bindings tab to find the External link.
  4. Click Open address to access Bytebase.

aws-task-network-bindings

Build from Source

Due to the uncontrolled nature of building from source, we only provide technical assistance to the Enterprise customers.

Prerequisites

  1. Install pnpm, Bytebase requires Node.js >=17.0.

  2. Install Go, Bytebase requires Go >= 1.16

  3. It's recommended to run Bytebase application as non-root user for security reason. If you don't have other non-root users on the system, you can follow the following steps to create one, e.g. user bytebase.

    groupadd bytebase && useradd -g bb bytebase
    sudo su bytebase

Download source code from GitHub, then go to the source root directory

If you want to build from a specific release such as 3.6.1, then switch to that release tag.

git checkout tags/3.6.1

Build the source

scripts/build_bytebase.sh [<<out_directory>>]

If out_directoryis not specified, the default directory is ./bytebase-build

Suppose you run scripts/build_bytebase.sh foo After build completes, run:

foo/bytebase --port 8080

(check Server Startup Options for other startup options)

You should see something like this in the console:

ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā•—   ā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—  ā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—
ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā•šā–ˆā–ˆā•— ā–ˆā–ˆā•”ā•ā•šā•ā•ā–ˆā–ˆā•”ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā•ā•ā•
ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā• ā•šā–ˆā–ˆā–ˆā–ˆā•”ā•    ā–ˆā–ˆā•‘   ā–ˆā–ˆā–ˆā–ˆā–ˆā•—  ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā•—
ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—  ā•šā–ˆā–ˆā•”ā•     ā–ˆā–ˆā•‘   ā–ˆā–ˆā•”ā•ā•ā•  ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•‘ā•šā•ā•ā•ā•ā–ˆā–ˆā•‘ā–ˆā–ˆā•”ā•ā•ā•
ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•   ā–ˆā–ˆā•‘      ā–ˆā–ˆā•‘   ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā•‘  ā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—
ā•šā•ā•ā•ā•ā•ā•    ā•šā•ā•      ā•šā•ā•   ā•šā•ā•ā•ā•ā•ā•ā•ā•šā•ā•ā•ā•ā•ā• ā•šā•ā•  ā•šā•ā•ā•šā•ā•ā•ā•ā•ā•ā•ā•šā•ā•ā•ā•ā•ā•ā•

Version 3.6.1 has started on port 8080 šŸš€

Troubleshoot

error: too many open files

Change the open file limit:

ulimit -n 10240

Deploy to PaaS

Edit this page on GitHub