Awesome

Extra MongoDB Tools

This repository provides additional MongoDB Tools with the following functionalities:

• mongo-archive - dump MongoDB backups to disk and upload them to cloud storage.
• mongo-unarchive - download MongoDB dump files from cloud storage and restore them to a live database.

Building Tools

To build the MongoDB Tools, follow these steps:

1. Clone the repository:

   git clone https://github.com/egose/database-tools
   cd database-tools
   

2. Install dependencies and build the Go binaries:

   go mod tidy
   make build
   

This will ensure that all the necessary dependencies are installed and then build the Go binaries in dist directory.

Binary Arguments and Environment Variables

The binaries provided in this repository utilize MongoDB Tools directly, ensuring a familiar interface for users with minimal modifications to the command arguments. The design closely resembles the behavior and command structure of MongoDB's native tools such as mongodump and mongorestore.

mongo-archive

mongo-unarchive

Examples

Dump Database and Upload to Azure Storage

mongo-archive \
--uri="mongodb://<username>:<password>@cluster0.mongodb.net/" \
--db=<dbname> \
--az-account-name=<az_account_name> \
--az-account-key=<az_account_key> \
--az-container-name=<az_container_name>

This example demonstrates how to dump the data from a specified database and upload it to Azure storage. Replace <username>, <password>, <dbname>, <az_account_name>, <az_account_key>, and <az_container_name> with the appropriate values for your setup.

Run Persistent Server for Regular Database Archival

mongo-archive \
--uri="mongodb://<username>:<password>@cluster0.mongodb.net/" \
--db=<dbname> \
--az-account-name=<az_account_name> \
--az-account-key=<az_account_key> \
--az-container-name=<az_container_name> \
--cron \
--cronExpression="* * * * *"

This example demonstrates how to run a persistent server that regularly archives a database. The server will execute the archival process based on the specified cron expression. Replace <username>, <password>, <dbname>, <az_account_name>, <az_account_key>, <az_container_name>, and <cron_expression> with your own values.

Restore the Target Database from Azure Storage

mongo-unarchive \
--uri="mongodb://localhost:27017" \
--db=<dbname> \
--az-account-name=<az_account_name> \
--az-account-key=<az_account_key> \
--az-container-name=<az_container_name>

This example shows how to restore the target database from Azure storage. Replace <dbname>, <az_account_name>, <az_account_key>, and <az_container_name> with your own values. The database will be restored to the MongoDB instance running on localhost:27017.

Restore the Target Database from Azure Storage and Apply Changes

mongo-unarchive \
--uri="mongodb://localhost:27017" \
--db=<dbname> \
--az-account-name=<az_account_name> \
--az-account-key=<az_account_key> \
--az-container-name=<az_container_name> \
--updates-file=/home/nonroot/updates.json

This example demonstrates how to restore the target database from Azure storage and apply changes contained in an updates file. Replace <dbname>, <az_account_name>, <az_account_key>, <az_container_name>, and /home/nonroot/updates.json with your own values. The updates file should contain the necessary instructions to modify the restored database.

An example of updates.json:

[
  {
    "collection": "users",
    "filter": {
      "email": {
        "$exists": true
      }
    },
    "update": [
      {
        "$set": {
          "email": {
            "$replaceOne": {
              "input": "$email",
              "find": "@",
              "replacement": "_"
            }
          }
        }
      }
    ]
  }
]

This JSON file provides an example of updating the users collection in the restored database.

Execute Binary Using Docker Container Image

To execute a binary using a Docker container image, you can use the following command:

docker run --rm \
    -v "$(pwd)/tmp:/tmp" \
    -e MONGOARCHIVE__DUMP_PATH=/tmp/datadump \
    ghcr.io/egose/database-tools:latest \
    mongo-archive \
    --uri="mongodb://<username>:<password>@cluster0.mongodb.net/" \
    --db=<dbname> \
    --az-account-name=<az_account_name> \
    --az-account-key=<az_account_key> \
    --az-container-name=<az_container_name> \
    --keep

Run Kubernetes CronJob with Mounted Volume

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mongo-archive
spec:
  schedule: "0 12 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 3
      template:
        spec:
          restartPolicy: Never
          initContainers:
            - name: backup-permission
              image: alpine:3.18
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args:
                - |
                  rm -rf /tmp/*;
                  adduser -D -u 1000 nonroot;
                  chown nonroot:nonroot /tmp;
              volumeMounts:
                - mountPath: /tmp
                  name: backup-volume
          containers:
            - name: backup-job
              image: ghcr.io/egose/database-tools:0.2.6
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args:
                - |
                  mongo-archive --db=mydb --read-preference=primary --force-table-scan
              env:
                - name: MONGOARCHIVE__URI
                  value: "mongodb+srv://user:password@cluster0.my.mongodb.net"
                - name: MONGOARCHIVE__AZ_ACCOUNT_NAME
                  value: mystorage
                - name: MONGOARCHIVE__AZ_ACCOUNT_KEY
                  value: myaccountkey
                - name: MONGOARCHIVE__AZ_CONTAINER_NAME
                  value: mybackup
              volumeMounts:
                - mountPath: /tmp
                  name: backup-volume
          volumes:
            - name: backup-volume
              persistentVolumeClaim:
                claimName: backup-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: backup-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Backlog

...