Home

Awesome

LFTP MIRROR

It's a python script that allow us to synchronize a directory on a remote server with a local directory via FTP. For this purpose makes use of the great program 'lftp' by Alexander V. Lukyanov, which is indispensable to run this script.

Sometimes we need to keep two directories synced, one on a remote server and another stored locally and only have FTP access to the server, with no possibility of using most appropriate solutions via ssh as rsync. For example, to back up a website in a shared hosting on which we have no more than an FTP account to share files.

The problem then is that do this via FTP is slow, because in some ways forces us to download entire contents of the directory each time or manually control the changes. No addition, unlike rsync, the power to download only that part of the file that has changed. Some FTP clients (mostly graphics) allow us to know which files have changed and download these only, thus accelerating the process of downloading and reducing network traffic. One of these, lftp, it is also one the lightest and fastest out there, one of the few that -by default- incorporates the functionality to synchronize two folders: mirror. This synchronization is bidirectional, so it can be done in both directions, remote to local and local to remote. As it is a command line program and supporting the import of settings via script, is ideal for automating the entire process through a shell script and crontab to make a fully automatic scheduled synchronization.

Since default lftp supports the use of scripts to automate ftp and programming tasks automatically through cron is trivial, which is the need to create a script like this?

Well, this script provides certain advantages over just using lftp:

FILES

The script file

A configuration file's example

The GPLv3 license text

This file in Spanish

This file

PRE-REQUISITES & DEPENDENCIES

For Linux (not tested in Mac):

Obviously, first we need is python. If we are in Linux usually is installed by default.

This script uses several modules included in the python standard library.

The python version required to run the script is 2.7. However, it is possible to run the script in version 2.6 if you install the module argparse, who joined the standard library in version 2.7.

Linux install is usually straightforward, as it comes included in many distributions, for example to install in Debian/Ubuntu:

$ sudo apt-get install python-argparse

lftp

Obviously it is necessary to install this program, which is what makes the dirty work.

Again, Linux install is usually straightforward, as it comes included in many distributions, for example to install in Debian/Ubuntu:

$ sudo apt-get install lftp

INSTRUCTIONS

This is a script designed to work in the command line, given the nature of their function, which is simply to automate a process that once started does not require further intervention on our part.

If you run the program without arguments shows an error message:

$ python lftp_mirror.py

usage: lftp_mirror.py [-h] [-v] {cron,cfg,shell} ...
lftp_mirror.py: error: too few arguments

That would mean that we need to employ at least some of these arguments: cron, cfg or shell or options: -h or -v

These three arguments will be defining the manner of execution of the script, as we mentioned in the introduction.

cron

In this mode runs taking as parameters were included in the script itself. This mode is useful when a synchronization is performed periodically on a single FTP server/directory.

To run it any easier to locate a block like this in the script and modify the entries that are between quotations marks, replacing the default values for the values we need.

#===========================================================================
# SCRIPT PARAMATERS TO EXECUTE THE SCRIPT AS A PROGRAMMED TASK
#===========================================================================

    # ftp user name ('user' by default)
    cron_user = 'user'
    # ftp password, with a minimum security measure, encoded by base64
    # ('password' by default)
    cron_pass = 'cGFzc3dvcmQ='
    # ftp server, name or ip ('localhost' by default)
    cron_site = 'localhost'
    # ftp server port. ('' by default)
    cron_port = ''
    # ftp directory
    cron_remote = 'directory'
    # local directory
    cron_local = '/your/path/to/your/local/directory/'
    # options, same as the shell mode. See shell mode help for more info
    cron_options = ''

#===========================================================================
# END PARAMETERS
#===========================================================================

Finally, only need to add this entry to the crontab (scheduled tasks):

lftp_mirror.py cron

cfg

This mode is similar to the previous, except that the parameters are taken from an external file and allows the execution of several consecutive synchronization operations. This is ideal for running in a single operation, the synchronization of multiple FTP servers/directories. Obviously this way could also be implemented periodically through cron.

To use this mode is only necessary to create a text file with a particular structure. It includes a sample configuration file as a template for us, sample.cfg. This file does not necessarily need this extension, but serves to identify it.

The structure is as follows:

[section]
site = {ftp server URL or IP}
port = (ftp server port)
remote = {remote directory}
local = {local directory}
user = (ftp server username)
password = (user password encoded in base64)
options = (other options)

Where section is an identifier for the synchronization operation. Would normally be the name of the FTP server or directory to synchronize.

Values enclosed in parentheses are optional while enclosed in braces are required. In case you do not specify a username and password, you must add the -a option which specifies that the connection is made with the anonymous user.

You need to create a section like this for every sync operation we want to perform.

A real example can be seen in sample.cfg content:

[debian]
site = ftp.debian.org
port =
remote = /debian/doc
local = debian
user =
password =
options = -aenP --exclude-glob '*.txt' --include-glob 'social*'

[FreBSD]
site = ftp.freebsd.org
port =
remote = /pub/FreeBSD/ERRATA
local = FreeBSD
user =
password =
options = -aenP

In this real example the two operations are executed sequentially, first synchronize with the Debian server and would end with the FreeBSD server.

To run the sync operations in this configuration file would use this command:

python lftp_mirror.py cfg sample.cfg

shell

This mode is used to perform a synchronization operation specifying the arguments directly into the command line. It is useful for occasional synchronization.

For complete instructions, refer to the help built into the script:

$ python lftp_mirror.py shell -h

Although it could be summarized in this command line:

$ python lftp_mirror.py shell site remote local -l user password [options]

Where site is the FTP server, remote the remote directory and local the local directory. user and password are the FTP server's user and password.

The set of arguments and options are detailed below:

Shell mode arguments:

The FTP Server, can be set as an URL or IP address

Remote directory at FTP Server

Local directory

Lftp options available in shell mode:

Show this mode help

The ftp account's username and password

Set user as anonymous

To specify a different port to standard FTP (21)

Establishes a secure connection via SFTP instead of FTP

Delete files in target that are no longer available at source

Download only newer files

Download N files in parallel, using multiple FTP connections simultaneously. N=2 if not provide any value

Reverse mode. Upload files from local to remote

Delete old files before transferring new ones

Descend into subdirectories, before transfer files

Do not creates in destiny empty directories that may exist in origin. Needs the --depth-first option

Don't go to subdirectories

Simulation, don't execute anything. Writes to log

Use cached directory listings

Delete files (not directories) in origin after transfer. CAUTION!

Download only missing files

Download only files already existing at target

Loop until no changes found

Ignore size when deciding whether to download

Ignore time when deciding whether to Download

Don't set file permissions

Don't apply umask to file modes

Don't create symbolic links

Set suid/sgid bits according to remote site

Try to set owner and group on files

Download symbolic links as files

Exclude files that matcht the pattern. Where GP is a glob pattern, for example: *. zip

Include files that matcht the pattern. Where GP is a glob pattern, for example: *. zip

Script options that are not present in lftp:

This option allows you to specify the detailed output synchronization process will not be displayed on the command line and will be added to the registration of activity (in the file and in the mail).

With this option will disable the compressed backups of the local directory.

Disables mail delivery with the activity log.

The mail is sent by default to the local user running the script, using the local mail server. if you want to use a different mail server or send it to other(s) recipient(s), then it is necessary to use the following options:

The mail server that we want to use

The mail server user

The password for that user

The email address to be included as the sender

The email address(es) of who we want to send mail

REPOSITORY

The code is hosted in a Git repository at GitHub, use this to get a clone:

git clone git://github.com/joedicastro/lftp-mirror.git

FEATURES

Every time you complete a execution record is added to the log file that is created in the local root folder and if not indicated otherwise send an email with the same content to the local user. An example of this log can be seen in this e-mail sent after running the sample configuration file, sample.cfg:

From:       youruser@yourcomputer
To:         youruser@yourcomputer
Subject:    FTP Sync - wednesday 08/12/10, 13:52:13
Date:       Wed, 08 Dec 2010 13:52:13 +0100


SCRIPT =====================================================================
lftp_mirror (ver. 0.7)
http://joedicastro.com

Connected to ftp.debian.org as anonymous
Mirror /debian/doc to debian
============================================================================

START TIME =================================================================
                                                wednesday 08/12/10, 13:51:58
============================================================================

CREATED NEW DIRECTORY ______________________________________________________

debian


LFTP OUTPUT ________________________________________________________________

Sending file `debian-manifesto'
Sending file `social-contract.txt'
Replying directory `FAQ'
Making directory `FAQ'
Replying directory `dedication'
Making directory `dedication'
Sending file `FAQ/debian-faq.en.html.tar.gz'
Sending file `FAQ/debian-faq.en.pdf.gz'
Sending file `dedication/dedication-2.2.sigs.tar.gz'
Sending file `FAQ/debian-faq.en.ps.gz'
Sending file `dedication/dedication-5.0.sigs.tar.gz'
Sending file `FAQ/debian-faq.en.txt.gz'



ROTATE COMPRESSED COPIES ___________________________________________________

Created file:

/your/path/debian_08dic2010_13:52_wed.tar.gz


DISK SPACE USED ============================================================
                                                                    1.35 MiB
============================================================================

END TIME ===================================================================
                                                wednesday 08/12/10, 13:52:06
============================================================================





SCRIPT =====================================================================
lftp_mirror (ver. 0.7)
http://joedicastro.com

Connected to ftp.freebsd.org as anonymous
Mirror /pub/FreeBSD/ERRATA to FreeBSD
============================================================================

START TIME =================================================================
                                                wednesday 08/12/10, 13:52:06
============================================================================

CREATED NEW DIRECTORY ______________________________________________________

FreeBSD


LFTP OUTPUT ________________________________________________________________

Replying directory `notices'
Replying directory `patches'
Making directory `patches'
Making directory `notices'
Replying directory `patches/EN-04:01'
Making directory `patches/EN-04:01'
Sending file `notices/FreeBSD-EN-04:01.twe.asc'
Sending file `notices/FreeBSD-EN-05:01.nfs.asc'
Sending file `patches/EN-04:01/twe.patch'
Sending file `notices/FreeBSD-EN-05:02.sk.asc'
Sending file `notices/FreeBSD-EN-05:03.ipi.asc'
Sending file `patches/EN-04:01/twe.patch.asc'
Sending file `notices/FreeBSD-EN-05:04.nfs.asc'
Sending file `notices/FreeBSD-EN-06:01.jail.asc'
Sending file `notices/FreeBSD-EN-06:02.net.asc'



ROTATE COMPRESSED COPIES ___________________________________________________

Created file:

/your/path/FreeBSD_08dic2010_13:52_wed.tar.gz


DISK SPACE USED ============================================================
                                                                   38.18 KiB
============================================================================

END TIME ===================================================================
                                                wednesday 08/12/10, 13:52:13
============================================================================

ALTERNATIVES

If my script do not match what you want, here's a summary of alternatives for UNIX/Linux (which I know). Mine included as reference.

CONTRIBUTION

Contributions and Feedback are most welcome. To contribute to the improvement and development of this scripts, you can send suggestions or bugs via the issues.

LICENSE

The script is distributed under the terms of the GPLv3 license

Apologies for any misspelling or syntax error, English isn't my mother tongue.