Awesome
bash-cloud-backup
bash-cloud-backup is a bash script, which can be used to automate local and cloud backup in Linux/Unix machines.
RELEASE 2.2.0 (3 Feb 2018)
CHANGELOG https://github.com/pontikis/bash-cloud-backup/blob/master/CHANGELOG.md
ATTENTION
Version 2.* is not compatible with previous (deprecated) version 1.*
Version 1 has its own branch (version1).
More at https://github.com/pontikis/bash-cloud-backup/blob/version1/README.md
Features
- bash-cloud-backup keeps rotating compressed tarballs of certain directories/files or databases.
- supported databases
- MySQL (using
mysqldump
) - http://linuxcommand.org/man_pages/mysqldump1.html - Postgresql (using
pg_dump
) - https://www.postgresql.org/docs/current/static/app-pgdump.html
- MySQL (using
- it uses
tar
(for archiving) andgzip
(for compression) or7z
(for compression and AES256 encryption - RECOMMENDED). - backup files are stored in specified directories and (optionally) deleted with rotation (14 days default).
- Amazon S3 sync: After local backup has been completed, the backup directory can be synchronized with Amazon S3, using
aws s3 sync
ors3cmd sync
(optional but recommended). - detailed logs, error reporting, email report
- option to use
nice
andionice
- option to use
trickle
bandwidth shaper - advanced customization using configuration files
(NOTE: 7-zip
does not store the owner/group of the file. On Linux/Unix, in order to backup directories keeping permissions you must use tar
.)
Copyright
Christos Pontikis (http://www.pontikis.gr)
License
MIT (see https://opensource.org/licenses/MIT)
Configuration files
conf.default/global.conf
: global options (sample file)conf.default/backup.conf
: configuration of a backup set (sample file)
Scripts
bash-cloud-backup.sh
: the main script
You may create and use custom scripts (see below - Configuration)
Logs
main log file
bash-cloud-backup is keeping logs (as defined in global.conf
).
logfilepath=/root/backup/log
logfilename=bash-cloud-backup.log
The main log file is logfilepath/logfilename
You should take care for logfile rotation.
nano /etc/logrotate.d/bash-cloud-backup
Add something like
/path/to/backup.log {
weekly
missingok
rotate 14
notifempty
create
}
Leave blank both logfilepath
and logfilename
if you DO NOT WANT to keep the main log file.
temporary log files
Inside tmp_path
bash-cloud-backup is keeping temporary log files, in order to create current session log file. This will be sent by email if you set value to mail_to
parameter.
After current session finished, both the temporary path and its contents are deleted.
- logfile_tmp_header="$tmp_path/header.log"
- logfile_tmp_main="$tmp_path/main.log"
- logfile_tmp_errors="$tmp_path/errors.log"
- logfile_tmp_time_elapsed="$tmp_path/time_elapsed.log"
- logfile_tmp_whole_session="$tmp_path/whole_session.log"
Keep tmp_path
outside backup root.
If you do not set a value, the default value is applied:
tmp_path=/tmp/bash-cloud-backup
External software
-
crudini: (REQUIRED) https://github.com/pixelb/crudini
Installation (for Debian):
apt-get install crudini
-
p7zip: (OPTIONAL but highly recommended) http://p7zip.sourceforge.net
Installation (for Debian):
apt-get install p7zip-full
It is a port of 7za.exe for POSIX systems. 7z is an Excellent archiving software offering high compression ratio and Strong AES-256 encryption. See http://www.7-zip.org.
7z man page http://linux.die.net/man/1/7z
-
trickle bandwidth shaper: (OPTIONAL) https://linux.die.net/man/1/trickle
Installation (for Debian):
apt-get install trickle
-
AWS Command Line Interface: (OPTIONAL) https://aws.amazon.com/cli/
Installation (for Debian):
apt-get install awscli
Configure with
aws configure
-
s3tools: (OPTIONAL) http://s3tools.org/
Installation (for Debian):
apt-get install s3cmd
Configure with
s3cmd --configure
More at http://s3tools.org/s3cmd-howto
NOTE: to select which AWS front-end you will use, set value to parameter amazon_front_end
in global.conf
.
Amazon S3 account
For cloud backup, an Amazon S3 account is needed (http://aws.amazon.com/s3/)
Setup using git (recommended)
installation
cd /path/to/scripts
git clone https://github.com/pontikis/bash-cloud-backup.git
get updates
cd /path/to/scripts/bash-cloud-backup
git fetch
git merge origin
If you are interested on (deprecated) version 1
installation
cd /path/to/scripts
git clone https://github.com/pontikis/bash-cloud-backup.git
cd /path/to/scripts/bash-cloud-backup
git checkout -b version1 origin/version1
get updates
cd /path/to/scripts/bash-cloud-backup
git fetch
git merge origin/version1
Setup by download
If git
is not available, download the source:
https://github.com/pontikis/bash-cloud-backup/archive/master.zip
If you are interested on (deprecated) version 1
https://github.com/pontikis/bash-cloud-backup/archive/version1.zip
Configuration
bash-cloud-backup
uses two configuration files (samples available in /conf.default
folder):
global.conf
which defines global parametersbackup.conf
which defines which files or databases will be backed up (a backup set)
By default, bash-cloud-backup expects these files to be
/etc/bash-cloud-backup/global.conf
/etc/bash-cloud-backup/backup.conf
You may define your own global.conf
and as many backup.conf
you like. So:
Edit global.conf
(global parameters)
cp conf.default/global.conf /etc/bash-cloud-backup/global.conf
nano /etc/bash-cloud-backup/global.conf
For instructions, see sample conf.default/global.conf
https://github.com/pontikis/bash-cloud-backup/blob/master/conf.default/global.conf
ATTENTION: remember to configure properly global.conf
after each update
Edit backup.conf
(create your own backup set)
cp conf.default/backup.conf /etc/bash-cloud-backup/backup.conf
nano /etc/bash-cloud-backup/backup.conf
For instructions, see sample conf.default/backup.conf
https://github.com/pontikis/bash-cloud-backup/blob/master/conf.default/backup.conf
ATTENTION: remember to configure properly backup.conf
after each update
Directories
bash-cloud-backup
will create all directories you define in configuration files (assuming it has the required permissions)
You may add custom commands (optional)
You may create and use
on_backup_started.sh
- before backup startedon_backup_finished.sh
- after backup finished (and before Amazon S3 sync)on_s3_sync_finished.sh
- after Amazon S3 syncon_logfile_created.sh
- after logfile created and main script finished
(these scripts are git ignored)
Security
About MySQL password
DO NOT expose root
password to create backups. Create a 'read only' user for backup purposes.
In most cases the following commands are enough:
GRANT SELECT,RELOAD,FILE,SUPER,LOCK TABLES,SHOW VIEW ON *.*
TO 'bkpadm'@'localhost' IDENTIFIED BY 'bkpadm_password_here'
WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
flush privileges;
Create .my.cnf
file in home folder and add
[client]
password="bkpadm_password_here"
or
[mysqldump]
password="bkpadm_password_here"
(double quotes are permitted in this file and in some cases of special characters in password are necessary)
then
chmod 600 .my.cnf
So YOU DO NOT NEED TO PROVIDE mysql_password
More at http://dev.mysql.com/doc/refman/5.7/en/password-security-user.html
About Postgresql password
DO NOT expose postgres
password to create backups. Create a 'read only' user for backup purposes.
In most cases the following commands are enough:
CREATE USER bkpadm SUPERUSER password 'password';
ALTER USER bkpadm set default_transaction_read_only = on;
In Postgresql you may use .pgpass
file (similar to MySQL .my.cnf
, but more advanced)
More https://www.postgresql.org/docs/9.5/static/libpq-pgpass.html
So YOU DO NOT NEED TO PROVIDE pg_password
However, providing a password in bash-cloud-backup
configuration files is quite secure, as PGPASSWORD
ENVIRONMENTAL VARIABLE is used.
About 7z password
It would be nice if 7z
could use enviromental variables or text fies for password retrieving. Not an easy way to do it. Alternatively, use hidepid
to hide root processes to other users - see below.
How to protect passwords to be exposed in command line
When a 7z (or any other) process is running, all command line arguments (including password) can be exposed to other users using programs like ps
, top
, htop
etc. You may prevent this remounting /proc
with hideid=2
option.
Linux kernel version 3.2+ is required.
You may set this option permanently (using /etc/fstab
)
To find with which options /proc
has been mounted in your system, use
cat /proc/mounts
(in this example /proc
had been mounted with options rw,nosuid,nodev,noexec,relatime
)
With bash-cloud-backup
you may use custom scripts
nano on_backup_started.sh
Set hidepid=2
option
/bin/mount -o remount,rw,nosuid,nodev,noexec,relatime,hidepid=2 /proc
After script finished return to previous status
nano on_s3_sync_finished.sh
Set hidepid=0
option
/bin/mount -o remount,rw,nosuid,nodev,noexec,relatime,hidepid=0 /proc
So, while bash-cloud-backup
is running, nobody can see running procesess (eg mysql, postgres, 7z etc) except their owner of course, usually root
A WORKAROUND FOR OLDER SYSTEMS
In older systems (Linux kernel version < 3.2) you may change mod of ps
, top
, htop
etc
at the beginning of the script
chmod 700 /bin/ps
return to original status at the end of the script.
chmod 755 /bin/ps
Secure files permissions
It is recommended all executable (*.sh) to be mod 700 and text files 600:
chown root:root bash-cloud-backup.sh
chmod 700 bash-cloud-backup.sh
chown root:root /etc/bash-cloud-backup/*.conf
chmod 600 /etc/bash-cloud-backup/*.conf
Run
To perform backup, call (as root
in most cases)
bash-cloud-backup.sh
You may use your own global.conf
and as many backup.conf
you like. In this case, use:
bash-cloud-backup.sh -g /path/to/myglobal.conf -b /path/to/mybackup.conf
Cron automation
su -l root
crontab -e
0 1 * * * /root/scripts/bash-cloud-backup/bash-cloud-backup.sh #Daily Backup
(in this example, every night at 01:00)