Softpanorama May the source be with you, but remember the KISS principle ;-)
	Contents	Bulletin	Scripting in shell and Perl	Network troubleshooting	History	Humor

rsync

Rsync is client server solution for backup up files. The idea was to create more smart clone of the rcp command it went father. Remote machine (the server) should have rsync package installed: one instance of rsync is running as a server (with --daemon option) and the other as a client. Rsync is a mirroring tool that transfers files in only one direction at a time, so the name is somewhat misleading. For bidirectional synchronization see unison.

Rsync can serve as an alternative to rcp, scp, scripted FTP sessions, NFS mounting and FTP NetDrive. If the problem can be solved by mirroring information on a remote server instead of maintaining a single copy this is a tool to use.

It is important to note that rsync can be used locally as very powerful copy utility.

Rsync make sense mostly on large files and/or large directory trees (for example web siites) -- in this case advantage of rsync is that it copies only the diffs of files that have actually changed, compressed and through ssh if you want to for security. Otherwise the tool is an overkill.

Key features:

• Computation of Diffs if the older copy of the file exists on the server - Only actual changed pieces of files are transferred, rather than the whole file. This makes updates faster, especially over slower links like modems. FTP would transfer the entire file, even if only one byte changed.
• Compression of all diffs -- The tiny pieces of diffs are then compressed on the fly, further saving you file transfer time and reducing the load on the network.
• Secure Shell - The data stream from rsync is passed through the ssh protocol to encrypt your session instead of rsh, which is also an option (and required if you don't use ssh - enable it in your /etc/inet.d and restart your inet daemon if you disabled it for security)

Rsync is a standard tool used it to synchronize Website trees from staging to production servers and to backup key areas of the filesystems both automatically through cron and/or via a CGI script.

Often large websites like Softpanorama are first updated on so called staging instance and at night this instance is synchronized with production instance running on a different server using rsync. This mode of operation prevents "small disasters": when something wrong is done to the website tree during the day, immediately detected and corrected. To this end it has an impressive list of features:

• Support for copying links, devices, owners, groups and permissions
• Exclude and exclude-from options similar to GNU tar
• A CVS exclude mode for ignoring the same files that CVS would ignore
• Does not require root privileges
• Pipelining of file transfers to minimize latency costs
• Support for both anonymous or authenticated rsync servers (ideal for mirroring)

Use for local copies of directory trees as "cp on steroids"

This use of rsync does not require present of the server. It can be used as a regular Unix command, similar to cp in Linux world. In addition to copying one tree of directories to another location it had the ability to "flatten" the target. Fro example to copy the content of ISO use

rsync -a /media/ISO /home/iso/RHEL61

Tips:

• The -z option doesn’t make sense for local file transfers, it slows things down.

• To copy some files under your home directory to a USB storage device, you might use

  rsync -a /media/ISO /home/iso/RHEL61

• Directory with trailing slash are treated differently in rsync. In other words /media/ISO; and /media/ISO/ are not the same thing to rsync:
  • Without the final slash, rsync will copy the directory in its entirety.
  • With the trailing slash, it will copy the contents of the directory but won't recreate the directory.

  If you're trying to replicate a directory structure with rsync, you should omit the trailing slash -- for instance, if you're mirroring /var/www on another machine or something like that.

• The archive option (-a), which combines several rsync option: the recursive and copy symlinks options, preserves group and owner, and generally makes rsync suitable for making archive copies. Note that it doesn't preserve hardlinks; if you want to preserve them, you will need to add the hardlinks option (-H).

• Another option you'll probably want to use most of the time is verbose (-v), which tells rsync to report lots of information about what it's doing. You can double and triple up on this option -- so using -v will give you some information, using -vv will give more, and using -vvv will tell you everything that rsync is doing.

• rsync will move hidden files (files whose names begin with a .) without any special options. If you want to exclude hidden files, you can use the option --exclude=".*/". You can also use the --exclude option to prevent copying things like Vim's swap files (.swp) and automatic backups (.bak) created by some programs.

• Be very careful with the delete option. With it, you can whack a bunch of files without meaning to.
• while you're getting used to rsync, it's probably a good idea to use the --dry-run option with your commands to run through the transfer first, without actually copying or synching files.
• If you do start an rsync transfer and realize that you've botched the command in some way that might result in the destruction of data, press Ctrl-c immediately to terminate the transfer. Some files may be gone, but you may be able to save the rest.

Sync with a remote server

To be able to transfer files to the remote server rsync on this server should be running with the option --daemon. If firewall is enabled it should allow rsync port (or ssh port is rsync is via shh)

On Red Hat and derivatives rsync is a standard package with the server part that installs into xinetd. It is disabled by default and you need to enable it by manually editing /etc/xinetd.d/rsync file to change the line disable yes to disable no. The resulting file should look:

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
#       allows crc checksumming etc.
service rsync
{
        disable = no
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

After that you should restart xinetd daemon:

service xinetd restart

To test you can start it from the command line as root or user you want to be able to perform transferes

rsync --verbose  --daemon

Use "rsync --daemon --help" to see the daemon-mode command-line options

After that as on a client side create the directory /tmp/test with a couple of files and issue the command:

[0]root@usrklxbck01:/tmp/test # /usr/bin/rsync -avv  --progress /tmp/test  testserver:/tmp

opening connection using ssh testserver rsync --server -vvlogDtpr . /tmp
root@testserver's password:
building file list ...
3 files to consider
delta-transmission enabled
test/
test/.fte-history
          33 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/3)
test/bash_history
        3926 100%    3.74MB/s    0:00:00 (xfer#2, to-check=0/3)
total: matches=0  hash_hits=0  false_alarms=0 data=3959

sent 4155 bytes  received 70 bytes  497.06 bytes/sec
total size is 3959  speedup is 0.94

Useful rsync command-line switches and options

For a serious discussion of rsync's options, you should turn to the README file that came with the distribution (found under /usr/doc/rsync on Debian and Red Hat) or (preferably) to the manual page.

• By far the most useful option is -a (--archive). This acts like the corresponding option to cp; rsync will:

  • recurse subdirectories (-r);

  • recreate soft links (-l);

  • preserve permissions (-p);

  • preserve timestamps (-t);

  • attempt to copy devices (if invoked as root) (-D);

  • preserve group information (-g) (and userid, if invoked as root) (-o).

  • Note
    • The -H (--hard-links) option is not implied by -a.
• -b (--backup) will force backups to be made; the default extension is a tilde, "~". Overridden with --suffix=SUFFIX.

• -S (--sparse) asks rsync to copy sparse files efficiently.

• -x (--one-file-system): Don't cross filesystems. Useful if you've got -a turned on!

• --daemon run in "daemon" mode, where clients may connect to receive (parts of) pre-defined "modules".

• --config=FILE FILE is used as the config file for a daemon mode instead of /etc/rsyncd.conf.

Example of rsyncd.conf

uid     = root
gid     = root
exclude = tmp/
exclude = var/run


[home]
        path            = /home
        comment         = All home directories
        read only       = true
        hosts allow     = testserver.my.com

This configuration file specifies an rsync daemon that runs as root, and offers /export/home to any client from testserver.my.com

The other options ensure that directories named "tmp" or "/ver/run" are never transferred, and that clients have read-only access to the server.

Rsync on Suse

From SUSE Linux Enterprise Server (SLES 10) Installation and Administration Guide:

rsync is useful when large amounts of data need to be transmitted regularly while not changing too much. This is, for example, often the case when creating backups. Another application concerns staging servers. These are servers that store complete directory trees of Web servers that are regularly mirrored onto a Web server in a DMZ.

40.6.1 Configuration and Operation

rsync can be operated in two different modes. It can be used to archive or copy data. To accomplish this, only a remote shell, like ssh, is required on the target system. However, rsync can also be used as a daemon to provide directories to the network.

The basic mode of operation of rsync does not require any special configuration. rsync directly allows mirroring complete directories onto another system. As an example, the following command creates a backup of the home directory of tux on a backup server named sun:

 rsync -baz -e ssh /home/tux/ tux@sun:backup 

The following command is used to play the directory back:

 rsync -az -e ssh tux@sun:backup /home/tux/ 

Up to this point, the handling does not differ much from that of a regular copying tool, like scp.

rsync should be operated in rsync mode to make all its features fully available. This is done by starting the rsyncd daemon on one of the systems. Configure it in the file /etc/rsyncd.conf. For example, to make the directory /srv/ftp available with rsync, use the following configuration:

gid = nobody
uid = nobody
read only = true
use chroot = no
transfer logging = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log

[FTP]
        path = /srv/ftp
        comment = An Example 

Then start rsyncd with rcrsyncd start. rsyncd can also be started automatically during the boot process. Set this up by activating this service in the runlevel editor provided by YaST or by manually entering the command insserv rsyncd. rsyncd can alternatively be started by xinetd. This is, however, only recommended for servers that rarely use rsyncd.

The example also creates a log file listing all connections. This file is stored in /var/log/rsyncd.log.

It is then possible to test the transfer from a client system. Do this with the following command:

rsync -avz sun::FTP 

This command lists all files present in the directory /srv/ftp of the server. This request is also logged in the log file /var/log/rsyncd.log. To start an actual transfer, provide a target directory. Use . for the current directory. For example:

rsync -avz sun::FTP

By default, no files are deleted while synchronizing with rsync. If this should be forced, the additional option --delete must be stated. To ensure that no newer files are deleted, the option --update can be used instead. Any conflicts that arise must be resolved manually.

40.6.2 For More Information

Important information about rsync is provided in the man pages man rsync and man rsyncd.conf.

A technical reference about the operating principles of rsync is featured in /usr/share/doc/packages/rsync/tech_report.ps

Find latest news about rsync on the project Web site at http://rsync.samba.org/.

Old news ;-)

[Feb 28, 2012] OpenSuse Administration Guide. Chapter 28. File Synchronization

Introduction to rsync¶

rsync is useful when large amounts of data need to be transmitted regularly while not changing too much. This is, for example, often the case when creating backups. Another application concerns staging servers. These are servers that store complete directory trees of Web servers that are regularly mirrored onto a Web server in a DMZ.

28.4.1. Configuration and Operation

rsync can be operated in two different modes. It can be used to archive or copy data. To accomplish this, only a remote shell, like ssh, is required on the target system. However, rsync can also be used as a daemon to provide directories to the network.

The basic mode of operation of rsync does not require any special configuration. rsync directly allows mirroring complete directories onto another system. As an example, the following command creates a backup of the home directory of tux on a backup server named sun:

rsync -baz -e ssh /home/tux/ tux@sun:backup

The following command is used to play the directory back:

rsync -az -e ssh tux@sun:backup /home/tux/

Up to this point, the handling does not differ much from that of a regular copying tool, like scp.

rsync should be operated in “rsync” mode to make all its features fully available. This is done by starting the rsyncd daemon on one of the systems. Configure it in the file /etc/rsyncd.conf. For example, to make the directory /srv/ftp available with rsync, use the following configuration:

gid = nobody
uid = nobody
read only = true
use chroot = no
transfer logging = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log

[FTP]
        path = /srv/ftp
        comment = An Example

Then start rsyncd with rcrsyncd start. rsyncd can also be started automatically during the boot process. Set this up by activating this service in the runlevel editor provided by YaST or by manually entering the command insserv rsyncd. rsyncd can alternatively be started by xinetd. This is, however, only recommended for servers that rarely use rsyncd.

The example also creates a log file listing all connections. This file is stored in /var/log/rsyncd.log.

It is then possible to test the transfer from a client system. Do this with the following command:

rsync -avz sun::FTP

This command lists all files present in the directory /srv/ftp of the server. This request is also logged in the log file /var/log/rsyncd.log. To start an actual transfer, provide a target directory. Use . for the current directory. For example:

rsync -avz sun::FTP .

By default, no files are deleted while synchronizing with rsync. If this should be forced, the additional option --delete must be stated. To ensure that no newer files are deleted, the option --update can be used instead. Any conflicts that arise must be resolved manually.

[Feb 28, 2012] Using rsync and cygwin to Sync Files from a Linux Server to a Windows Notebook PC —

True Blade Systems

Name the script: server-download.sh and put some comments at the start of the script before the actual rsync command:

# server-download.sh
#
# Install on Notebook PCs
#
# rsync tool to download server data
# from [server name]
# to [user name's notebook PC]
#
# uses ssh key pairs to login automatically
#
# last edited: [last edit date] by [author]
#
# download only those files on [server name] in [server target directory]
# that are newer than what is already on the notebook PC

rsync -avzu [user name]@[server name]:[server directory] [notebook PC directory]

Notes:

1. You should add the -n option to rsync the first time this is run to do a "dry run" which does not do any actual downloading. By doing this you will be able to see if the script is going to correctly do what you want. Thus, the rsync command becomes:
   rsync -avzun [remainder as above]

2. You can create additional rsync lines in the script file to download data from more than one directory on the server.

Next you need to create an icon on the desktop that will permit the script to be easily run by the user so that they won't have to enter the script name in a command window to do the file download.

[Feb 28, 2011] Sys Admin v16, i02 Distributing Software Modules Using rsync by Ed Schaefer and John Spurgeon

Distributing software packages to all of our servers is a tedious task. Currently, a release manager makes a connection to each server and transfers files using ftp. This involves entering passwords multiple times, waiting for transfers to complete, changing directories, and keeping files organized. We developed a shell script, distribute_release ( Listing 1 ), that makes the job easier.

Our script has some advantages over the ftp process:

• Directory trees can be used to organize release modules.
• A distribution network defines how files are transferred from server to server.
• When a release module is ready to be distributed, it is replicated to all of the servers in the network using rsync, which helps minimize network traffic.
• Various authentication methods can be used to avoid entering passwords for each server.

We’ll describe the directory structures including creating the distribution network. Then we’ll talk about the scripts. Finally, we’ll discuss an example.

Directory Structures

Each release module is stored in the directory /var/spool/pkg/release/[module]/. A module directory can be flat, or it can contain subdirectories. Hidden directory trees under the ./release/ directory define the distribution network. Therefore, the names of these directories cannot be used as module names.

Transport protocols supported by distribute_release include nfs, rsh, and ssh. If a release module is distributed using nfs, then the directory /var/spool/pkg/release/.nfs/[module]/ contains symbolic links corresponding to the hosts in the server’s distribution network:

/var/spool/pkg/release/.nfs/[module]/[host] -> \
 /net/[host]/var/spool/pkg/release/

When using nfs, rsync functions like an improved copy command, transferring files between the directories /var/spool/pkg/release/[module]/ and /var/spool/pkg/release/.nfs/[module]/[host]/[module].

When using rsh or ssh, the directory structures are similar. With rsh, for example, empty files of the form /var/spool/pkg/release/.rsh/[module]/[host] define the hosts in the distribution network.

The Scripts

Before distribute_release can be called, the directory structures and the distribution network must be created. The script create_distribution ( Listing 2 ) facilitates these tasks.

One argument, the name of a release module, must be passed to create_distribution. When no options are used, the local host functions as a terminal node in the distribution network. In other words, the system may receive updates from another host, but it will not propagate those updates to downstream hosts. Downstream hosts and transport protocols may be specified with the -h and -t options respectively.

When using distribute_release, the name of a release module must be passed to the script. The -q and -v options may be used to control the amount of information displayed to the user. Hosts to be included or excluded from the distribution may be specified using the -i and -e options. The -r option may be used to determine how many times the program will recursively call itself to distribute the module to successive levels in a distribution hierarchy. When using nfs, the recursive calls are made locally. With rsh and ssh, the program calls itself on a remote server.

Distribute_release first gets the argument and any command-line options. Then, for each transport protocol, the script builds a distribution list and executes the appropriate rsync command for each host in the list. If a recursion depth is specified, then another instance of distribute_release is executed in a detached screen session, allowing the parent instance to continue running while the child processes propagate the module to other hosts.

An Example

Our example network (see Figure 1 ) contains five servers -- bambi, pongo, pluto, nemo, and goofy. One of the release modules is named TS1 (located on bambi) and the module is named TS2 (located on pluto). By executing the create_distributions script ( Listing 3 ) on each server, the complete distribution network for both modules is built using the proper create_distribution calls.

Consider the TS1 release module; after the module has been distributed to all of the systems in the network, the directory /var/spool/pkg/release/TS1/ contains the following files and subdirectories:

 ./README
./v1/TS1-v1.pkg
./v2/TS1-v2.pkg
./beta/TS1-v3.pkg

On bambi, the directory /var/spool/pkg/release/.ssh/TS1/ contains a file named pongo. So, executing "distribute_release TS1” on bambi synchronizes the TS1 module with pongo using ssh as the transport protocol. The TS1 module can be distributed from pongo to all servers in the network using the -r option:

distribute_release -r 2 TS1

When using ssh, passwords can be avoided by using public/private key pairs with empty passphrases. When using rsh, you can update /etc/hosts.equiv or the appropriate .rhosts file. Obviously, passwords are not an issue with nfs. Deciding which protocol to use depends on security concerns, potential performance issues, and configuration complexity.

John Spurgeon is a software developer and systems administrator for Intel’s Factory Information Control Systems, IFICS, in Hillsboro, Oregon. He is currently preparing to ride a single-speed bicycle in Race Across America in 2007.

Ed Schaefer is a frequent contributor to Sys Admin. He is a software developer and DBA for Intel’s Factory Information Control Systems, IFICS, in Hillsboro, Oregon. Ed also hosts the monthly Shell Corner column on UnixReview.com. He can be reached at: shellcorner@comcast.net.

[Oct 11, 2010] How to rsync, unison, wget, curl by Xah Lee

2007-03 | Xah Lee Web

One-Way Copying a Directory to Server with rsync

How to copy local directory to a remote machine, in one shot?

For a one-way copying (or updating), use rsync. The remote machine must have rsync installed. Example: rsync -z -a -v --rsh="ssh -l mary" ~/web/ mary@example.org:~/

This will copy the local dir 〔~/web/〕 to the remote dir 〔~/〕 on the machine with domain name “example.org”, using login “mary” thru the ssh protocol. The “-z” is to use compression. The “-a” is for archived mode, basically making the file's meta data (owner/perm/timestamp) same as the local file (when possible) and do recursive (i.e. upload the whole dir). The “-v” is for verbose mode, which basically makes rsync print out which files is being updated. (rsync does not upload files that's already on the destination and identical.)

For example, here's what i use to sync/upload my website on my local machine to my server.

rsync -z -a -v --exclude="*~" --exclude=".DS_Store" --exclude=".bash_history" 
	--exclude="*/_curves_robert_yates/*.png" --exclude="logs/*" --exclude="xlogs/*" 
	--delete --rsh="ssh -l u40651121" ~/web/ u40651121@s168753656.onlinehome.us:~/

I used this command daily. The “--exclude” tells it to disregard any files matching that pattern (i.e. if it matches, don't upload it nor delete it on remote server)

Here's a example of syncing Windows and Mac.

rsync -z -r -v --delete --rsh="ssh -l xah" ~/web/ xah@169.254.125.147:~/web/

Note that “-r” is used intstead of “-a”. The “-r” means recursive, all sub directories and files. Don't use “-a” because that will sync file owner, group, permissions, and others, but because Windows and unix has different permission systems and file systems, so “-a” is usually not what you want. (For a short intro to perm systems on unix and Windows, see: Unix And Windows File Permission Systems)

You can creat a bash alias for the long command e.g. alias l="ls -al";, or use bash's back history by 【Ctrl+r】 then type rsync.

Advanced applications of rsync by Martin Streicher

developerWorks

A quick review

Much like cp, rsync copies files from a source to a destination. Unlike cp, the source and destination of an rsync operation can be local or remote. For instance, the command in Listing 1 copies the directory /tmp/photos and its entire contents verbatim to a home directory.

Listing 1. Copy the contents of a directory verbatim

	
$ 	rsync -n -av /tmp/photos ~
building file list ... done
photos/
photos/Photo 2.jpg
photos/Photo 3.jpg
photos/Photo 6.jpg
photos/Photo 9.jpg

sent 218 bytes  received 56 bytes  548.00 bytes/sec
total size is 375409  speedup is 1370.11

The -v option enables verbose messages. The -a option (where a stands for archive), is a shorthand for -rlptgoD (recurse, copy symbolic links as symbolic links, preserve permissions, preserve file times, preserve group, preserve owner, and preserve devices and special files, respectively). Typically, -a mirrors files; exceptions occur when the destination cannot or does not support the same attributes. For example, copying a directory from UNIX to Windows® does not map perfectly. Some suggestions for unusual cases appear below.

rsync has a lot of options. If you worry that your options or 
	source or destination specifications are incorrect, use -n to perform 
	a dry run. A dry run previews what will happen to each file but does 
	not move a single byte. When you are confident of all the settings, drop the
	-n and proceed. 

Listing 2 provides an example where -n is invaluable. The command in Listing 1 and the following command yield different results.

Listing 2. Copy the contents of a named directory

	
$ rsync -av /tmp/photos/ ~
./
Photo 2.jpg
Photo 3.jpg
Photo 6.jpg
Photo 9.jpg

sent 210 bytes  received 56 bytes  532.00 bytes/sec
total size is 375409  speedup is 1411.31

What is the difference? The difference is the trailing slash on the source argument. If the source has a trailing slash, the contents of the named directory but not the directory itself are copied. A slash on the end of the destination is immaterial.

And Listing 3 provides an example of moving the same directory to another system.

	
$ rsync -av /tmp/photos example.com:album
created directory album
Photo 2.jpg
Photo 3.jpg
Photo 6.jpg
Photo 9.jpg

sent 210 bytes  received 56 bytes  21.28 bytes/sec
total size is 375409  speedup is 1411.31

Assuming that you have the same login name on the remote machine, rsync prompts you with a password and, given the proper credential, creates the directory album and copies the images to that directory. By default, rsync uses Secure Shell (SSH) as its transport mechanism; you can reuse your machine aliases and public keys with rsync.

rsync modes

The examples in Listing 2 and Listing 3 demonstrate two of rsync's four modes. The first example was shell mode, also dubbed local mode. The second sample was remote shell mode and is so named because SSH powers the underlying connection and transfers. rsync has two additional modes. List mode acts like ls: It lists the contents of source, as shown in Listing 4.

	
$ 
drwxr-xr-x         238 2009/08/22 18:49:50 photos
-rw-r--r--        6148 2008/07/03 01:36:18 photos/.DS_Store
-rw-r--r--       71202 2008/06/18 04:51:36 photos/Photo 2.jpg
-rw-r--r--       69632 2008/06/18 04:51:45 photos/Photo 3.jpg
-rw-r--r--       61046 2008/07/14 00:31:17 photos/Photo 6.jpg
-rw-r--r--      167381 2008/07/14 00:31:56 photos/Photo 9.jpg

The fourth mode is server mode. Here, the rsync daemon runs perennially on a machine, accepting requests to transfer files. A transfer can send files to the daemon or request files from it. Server mode is ideal for creating a central backup server or project repository.

To differentiate between remote shell mode and server mode, the latter employs two colons (:) in the source and destination names. Assuming that whatever.example.com exists, the next command copies files from the source to a local destination:

$ rsync -av whatever.example.com::src /tmp

And what exactly is src? It's an rsync module that you define and configure on the daemon's host. A module has a name, a path that contains its files, and some other parameters, such as read only, which protects the contents from modification.

To run an rsync daemon, type:

$ sudo rsync --daemon

Running the rsync daemon as the superuser, root, is not strictly necessary, but the practice protects other files on your machine. Running as root, rsync restricts itself to the module's directory hierarchy (its path) using chroot. After a chroot, all other files and directories seem to vanish. If you choose to run the rsync daemon with your own privileges, choose an unused socket and make sure its modules have sufficient permissions to allow download and/or upload. Listing 5 shows a minimal configuration to share some files in your home directory without the need for sudo. The configuration is stored in file rsyncd.conf.

Listing 5. Simple configuration for sharing files

	
motd file = /home/strike/rsyncd/rsync.motd_file
pid file = /home/strike/rsyncd/rsyncd.pid
port = 7777
use chroot = no

[demo]
path = /home/strike
comment = Martin home directory
list = no

[dropbox]
path = /home/strike/public/dropbox
comment = A place to leave things for Martin
read only = no

[pickup]
path = /home/strike/public/pickup
comment = Get your files here!

The file has two segments. The first segment—here, the first four lines—configures the operation of the rsync daemon. (Other options are available, too.) The first line points to a file with a friendly message to identify your server. The second line points to another file to record the process ID of the server. This is a convenience in the event you must manually kill the rsync daemon:

kill -INT `cat /home/strike/rsyncd/rsyncd.pid`

The two files are in a home directory, because this example does not use superuser privileges to run the software. Similarly, the port chosen for the daemon is above 1000, which users can claim for any application. The fourth line turns off chroot.

The remaining segment is subdivided into small sections, one section per module. Each section, in turn, has a header line and a list of (key-value) pairs to set options for each module. By default, all modules are read only; set read only = no to allow Write operations. Also by default, all modules are listed in the module catalog; set list = no to hide the module.

To start the daemon, run:

$ rsync --daemon --config=rsyncd.conf 

Now, connect to the daemon from another machine, and omit a module name. You should see this:

rsync --port=7777 mymachine.example.com::
Hello! Welcome to Martin's rsync server.

dropbox        	A place to leave things for Martin
pickup         	Get your files here!

If you do not name a module after the colons (::), the daemon responds with a list of available modules. If you name a module but do not name a specific file or directory within the module, the daemon provides a catalog of the module's contents, as shown in Listing 6.

Listing 6. Catalog output of a module's contents

	
rsync --port=7777 mymachine.example.com::pickup
Hello! Welcome to Martin's rsync server.

drwxr-xr-x        4096 2009/08/23 08:56:19 .
-rw-r--r--           0 2009/08/23 08:56:19 article21.html
-rw-r--r--           0 2009/08/23 08:56:19 design.txt
-rw-r--r--           0 2009/08/23 08:56:19 figure1.png

And naming a module and a file copies the file locally, as shown in Listing 7.

Listing 7. Name a module to copy files locally

	
rsync --port=7777 mymachine.example.com::pickup/
Hello! Welcome to Martin's rsync server.

drwxr-xr-x        4096 2009/08/23 08:56:19 .
-rw-r--r--           0 2009/08/23 08:56:19 article21.html
-rw-r--r--           0 2009/08/23 08:56:19 design.txt
-rw-r--r--           0 2009/08/23 08:56:19 figure1.png

You can also perform an upload by reversing the source and destination, then pointing to the module for writes, as shown in Listing 8.

Listing 8. Reverse source and destination directories

	
$ rsync -v --port=7777 application.js mymachine.example.com::dropbox
Hello! Welcome to Martin's rsync server.

application.js

sent 245 bytes  received 38 bytes  113.20 bytes/sec
total size is 164  speedup is 0.58

That's a quick but thorough review. Next, let's see how you can apply rsync to daily tasks. rsync is especially useful for backups. And because it can synchronize a local file with its remote counterpart—and can do that for an entire file system, too—it's ideal for managing large clusters of machines that must be (at least partially) identical.

Back up your data with rsync

Performing backups on a frequent basis is a critical but typically ignored chore. Perhaps it's the demands of running a lengthy backup each day or the need to have large external media to store files; never mind the excuse, copying data somewhere for safekeeping should be an everyday practice.

To make the task painless, use rsync and point to a remote server—perhaps one that your service provider hosts and backs up. Each of your UNIX machines can use the same technique, and it's ideal for keeping the data on your laptop safe.

Establish SSH keys and an rsync daemon on the remote machine, and create a backup module to permit writes. Once established, run rsync to create a daily backup that takes hardly any space, as shown in Listing 9.

Listing 9. Create daily backups

	
#!/bin/sh
# This script based on work by Michael Jakl (jakl.michael AT gmail DOTCOM) and used 
# with express permission.
HOST=mymachine.example.com
SOURCE=$HOME
PATHTOBACKUP=home-backup

date=`date "+%Y-%m-%dT%H:%M:%S"`

rsync -az --link-dest=$PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date

ssh $HOST "rm $PATHTOBACKUP/current && ln -s back-$date $PATHTOBACKUP/current" 

Replace HOST with the name of your backup host and SOURCE with the directory you want to save. Change PATHTOBACKUP to the name of your module. (You can also embed the three final lines of the script in a loop, dynamically change SOURCE, and back up a series of separate directories on the same system.) Here's how the backup works:

• To begin, date is set to the current date and time and yields a string like 2009-08-23T12:32:18, which identifies the backup uniquely.
• The rsync command performs the heavy lifting. -az preserves all file information and compresses the transfers. The magic lies in --link-dest=$PATHTOBACKUP/current, which specifies that if a file has not changed, do not copy it to the new backup. Instead, create a hard link from the new backup to the same file in the existing backup. In other words, the new backup only contains files that have changed; the rest are links.

  More specifically (and expanding all variables), mymachine.example.com::home-backup/current is the current archive. The new archive for /home/strike is targeted to mymachine.example.com::home-backup/back-2009-08-23T12:32:18. If a file in /home/strike has not changed, the file is represented in the new backup by a hard link to the current archive. Otherwise, the new file is copied to the new archive.

  If you touch but a few file or perhaps a handful of directories each day, the additional space required for what is effectively a full backup is paltry. moreover, because each daily backup (except the very first) is so small, you can keep a long history of the files on hand.

• The last step is to alter the organization of the backups on the remote machine to promote the newly created archive to be the current archive, thereby minimizing the differences t command removes the current archive (which is merely a symbolic link) and recreates the same symbolic link pointing to the new archive.

Keep in mind that a hard link to a hard link points to the same file. Hard links are very cheap to create and maintain, so a full backup is simulated using only an incremental scheme.

[Jun 23, 2009] Distributing Software Modules Using rsync by Ed Schaefer and John Spurgeon

Sys Admin v16, i02

Distributing software packages to all of our servers is a tedious task. Currently, a release manager makes a connection to each server and transfers files using ftp. This involves entering passwords multiple times, waiting for transfers to complete, changing directories, and keeping files organized. We developed a shell script, distribute_release ( Listing 1 ), that makes the job easier.

Our script has some advantages over the ftp process:

• Directory trees can be used to organize release modules.
• A distribution network defines how files are transferred from server to server.
• When a release module is ready to be distributed, it is replicated to all of the servers in the network using rsync, which helps minimize network traffic.
• Various authentication methods can be used to avoid entering passwords for each server.

We’ll describe the directory structures including creating the distribution network. Then we’ll talk about the scripts. Finally, we’ll discuss an example.

Directory Structures

Each release module is stored in the directory /var/spool/pkg/release/[module]/. A module directory can be flat, or it can contain subdirectories. Hidden directory trees under the ./release/ directory define the distribution network. Therefore, the names of these directories cannot be used as module names.

Transport protocols supported by distribute_release include nfs, rsh, and ssh. If a release module is distributed using nfs, then the directory /var/spool/pkg/release/.nfs/[module]/ contains symbolic links corresponding to the hosts in the server’s distribution network:

/var/spool/pkg/release/.nfs/[module]/[host] -> \
 /net/[host]/var/spool/pkg/release/

When using nfs, rsync functions like an improved copy command, transferring files between the directories /var/spool/pkg/release/[module]/ and /var/spool/pkg/release/.nfs/[module]/[host]/[module].

When using rsh or ssh, the directory structures are similar. With rsh, for example, empty files of the form /var/spool/pkg/release/.rsh/[module]/[host] define the hosts in the distribution network.

The Scripts

Before distribute_release can be called, the directory structures and the distribution network must be created. The script create_distribution ( Listing 2 ) facilitates these tasks.

One argument, the name of a release module, must be passed to create_distribution. When no options are used, the local host functions as a terminal node in the distribution network. In other words, the system may receive updates from another host, but it will not propagate those updates to downstream hosts. Downstream hosts and transport protocols may be specified with the -h and -t options respectively.

When using distribute_release, the name of a release module must be passed to the script. The -q and -v options may be used to control the amount of information displayed to the user. Hosts to be included or excluded from the distribution may be specified using the -i and -e options. The -r option may be used to determine how many times the program will recursively call itself to distribute the module to successive levels in a distribution hierarchy. When using nfs, the recursive calls are made locally. With rsh and ssh, the program calls itself on a remote server.

Distribute_release first gets the argument and any command-line options. Then, for each transport protocol, the script builds a distribution list and executes the appropriate rsync command for each host in the list. If a recursion depth is specified, then another instance of distribute_release is executed in a detached screen session, allowing the parent instance to continue running while the child processes propagate the module to other hosts.

An Example

Our example network (see Figure 1 ) contains five servers -- bambi, pongo, pluto, nemo, and goofy. One of the release modules is named TS1 (located on bambi) and the module is named TS2 (located on pluto). By executing the create_distributions script ( Listing 3 ) on each server, the complete distribution network for both modules is built using the proper create_distribution calls.

Consider the TS1 release module; after the module has been distributed to all of the systems in the network, the directory /var/spool/pkg/release/TS1/ contains the following files and subdirectories:

 ./README
./v1/TS1-v1.pkg
./v2/TS1-v2.pkg
./beta/TS1-v3.pkg

On bambi, the directory /var/spool/pkg/release/.ssh/TS1/ contains a file named pongo. So, executing "distribute_release TS1” on bambi synchronizes the TS1 module with pongo using ssh as the transport protocol. The TS1 module can be distributed from pongo to all servers in the network using the -r option:

distribute_release -r 2 TS1

When using ssh, passwords can be avoided by using public/private key pairs with empty passphrases. When using rsh, you can update /etc/hosts.equiv or the appropriate .rhosts file. Obviously, passwords are not an issue with nfs. Deciding which protocol to use depends on security concerns, potential performance issues, and configuration complexity.

John Spurgeon is a software developer and systems administrator for Intel’s Factory Information Control Systems, IFICS, in Hillsboro, Oregon. He is currently preparing to ride a single-speed bicycle in Race Across America in 2007.

Ed Schaefer is a frequent contributor to Sys Admin. He is a software developer and DBA for Intel’s Factory Information Control Systems, IFICS, in Hillsboro, Oregon. Ed also hosts the monthly Shell Corner column on UnixReview.com. He can be reached at: shellcorner@comcast.net.

[Apr 14, 2009] The rsync family

Several GUI alternatives are discusses

Apr 14, 2009 | developerworks

If you work with both a laptop and a desktop computer, you know you have to synchronize the machines to keep them up to date. In addition, you probably want to run the synchronization not only at your home but also from a remote site; in my case, whenever I travel with my laptop, I make sure that whatever I do on it gets backed up to my desktop computer. (Losing your laptop and thereby losing all your work isn't nice at all!) Many solutions to this problem exist: This article introduces one such tool—rsync—and mentions several related tools, all of which provide easy synchronization procedures.

RsyncConfig - rsync configuration generator - search.cpan.org

Rsync::Config is a module who can be used to create rsync configuration files. A configuration file (from Rsync::Config point of view) is made by atoms and modules with atoms. A atom is the smallest piece from the configuration file. This module inherits from Rsync::Config::Module .

Rsync tutorial

What is Rsync?

Rsync is a very useful alternative to rcp written by Andrew Tridgell and Paul Mackerras. This tool lets you copy files and directories between a local host and a remote host (source and destination can also be local if you need.) The main advantage of using Rsync instead of rcp, is that rsync can use SSH as a secure channel, send/receive only the bytes inside files that changed since the last replication, and remove files on the destination host if those files were deleted on the source host to keep both hosts in sync. In addition to using rcp/ssh for transport, you can also use Rsync itself, in which case you will connect to TCP port 873.

Whether you rely on SSH or use Rsync explicitely, Rsync still needs to be installed on both hosts. A Win32 port is available if you need, so you can have either one of the host or both be NT hosts. Rsync's web site has some good infos and links. There is also an HOWTO.

Configuring /etc/rsyncd.conf

Being co-written by Andrew Tridgell, author of Samba, it's no surprise that Rsync's configuration file looks just like Samba (and Windows' :-), and that Rsync lets you create projects that look like shared directories under Samba. Accessing remote resources through this indirect channel offers more independence, as it lets you move files on the source Rsync server without changing anything on the destination host.

Any parameters listed before any [module] section are global, default parameters.

Each module is a symbolic name for a directory on the local host. Here's an example:

#/etc/rsyncd.conf

		secrets file = /etc/rsyncd.secrets

		motd file = /etc/rsyncd.motd #Below are actually defaults, but to be on 
		the safe side...

		read only = yes

		list = yes

		uid = nobody

		gid = nobody

		

		[out]

		comment = Great stuff from remote.acme.com

		path = /home/rsync/out

		

		[confidential]

		comment = For your eyes only

		path = /home/rsync/secret-out

		auth users = joe,jane

		hosts allow = *.acme.com

		hosts deny = * 

		list = false 

 

Note: Rsync will not grant access to a protected share if the password file (/etc/rsyncd.secrets, here) is world-readable.

Running RSYNCd

Per the manual page:

The rsync daemon is launched by specifying the --daemon option to rsync. You can launch it either via inetd or as a stand-alone daemon. When run via inetd you should add a line like this to /etc/services:

rsync 873/tcp

... and a single line something like this to /etc/inetd.conf:

rsync stream tcp nowait root /usr/bin/rsync rsyncd --daemon

You will then need to send inetd a HUP signal to tell it to reread its config file. Note that you should not send the rsync server a HUP signal to force it to reread the /etc/rsyncd.conf. The file is re-read on each client connection.

Per the HOWTO:

The rsync daemon is robust, so it is safe to launch it as a stand-alone server. The code that loops waiting for requests is only a few lines long then it forks a new copy. If the forked process dies then it doesn't harm the main daemon.
The big advantage of running as a daemon will come when the planned directory cache system is implemented. The caching system will probably only be enable when running as a daemon. For this reason, busy sites is recommended to run rsync as a daemon. Also, the daemon mode makes it easy to limit the number of concurrent connections.

Since it's not included in the 2.4.3 RPM package, here's the init script to be copied as /etc/rc.d/init.d/rsyncd with symlinks to /etc/rc.d/rc3.d:

#!/bin/sh

		# Rsyncd This shell script takes care of starting and stopping the rsync 
		daemon

		# description: Rsync is an awesome replication tool.

		

		# Source function library.

		. /etc/rc.d/init.d/functions

		

		[ -f /usr/bin/rsync ] || exit 0

		

		case "$1" in

		start)

		action "Starting rsyncd: " /usr/bin/rsync --daemon

		;;

		stop)

		action "Stopping rsyncd: " killall rsync

		;;

		*)

		echo "Usage: rsyncd {start|stop}"

		exit 1

		esac

		exit 0

 

Here's an example under Linux on how to set up a replication through SSH:

rsync -avz -e ssh rsync@remote.acme.com:/home/rsync/out/ /home/rsync/from_remote 
		

An important thing here, is that the presence or absence of a trailing "/" in the source directory determines whether the directory itself is copied, or simply the contents of this source directory.

In other words, the above means that the local host must have a directory available (here, /home/rsync/from_remote to receive the contents of /home/rsync/out sitting on the remote host, otherwise Rsync will happily download all files into the path given as destination without asking for confirmation, and you could end up with a big mess.

On the other hand, rsync -avz -e ssh rsync@remote.acme.com:/home/rsync/out /home/rsync/from_remote means that the an "out" sub-directory is first created under /home/rsync/from_remote on the destination host, and will be populated with the contents of the remote directory ./out. In this case, files will be save on the local host in /home/rsync/from_remote/out, so the former commands looks like a better choice.

Here's how to replicate an Rsync share from a remote host:

rsync -avz rsync@remote.acme.com::out /home/rsync/in 
		

Notice that we do not use a path to give the source resource, but instead just a name ("out"), and that we use :: to separate the server's name and the resource it offers. In the Rsync configuration that we'll see just below, this is shown as a [out] section. This way, admins on remote.acme.com can move files on their server; As long as they remember to update the actual path in the [out] section (eg. PATH=/home/rsync/out to PATH=/home/outgoing), remote Rsync users are not affected.

An Rsync server displays the list of available anonymous shares through rsync remote.acme.com::. Note the ::. For added security, it is possible to prompt for a password when listing private shares, so that only authorized remote users know about the Rsync shares available from your server.

Any NT version?

The NT port only requires the latest and greatest RSYNCx.EXE and Cygnus' CYGWIN1.DLL. The easiest is to keep both in the same directory, but the DLL can be located in any directory found in your PATH environment variable.

Robert Scholte's excellent tutorial on using the NT port of Rsync can be found here.

Instructions on how to install rsync as an NT service are here.

Here's an example based on the sample above:

C:\Rsync>rsync243 -avz joe@linux.acme.com::confidential ./confidential

		Password:

		receiving file list ... done

		./

		./

		wrote 109 bytes read 123 bytes 66.29 bytes/sec

		total size is 0 speedup is 0.00

Useful command-line switches

-v, --verbose increase verbosity

		-q, --quiet decrease verbosity

		-c, --checksum always checksum

		-a, --archive archive mode. It is a quick way of saying you want recursion 
		and want to preserve everything.

		-r, --recursive recurse into directories

		-R, --relative use relative path names

		-u, --update update only (don't overwrite newer files)

		-t, --times preserve times

		-n, --dry-run show what would have been transferred

		-W, --whole-file copy whole files, no incremental checks

		-I, --ignore-times Normally rsync will skip any files that are already the 
		same length and have the same time-stamp. This option turns off this behavior.

		--existing only update files that already exist

		--delete delete files that don't exist on the sending side

		--delete-after delete after transferring, not before

		--force force deletion of directories even if not empty

		-c, --checksum always checksum

		--size-only only use file size when determining if a file should be transferred

		--progress show progress during transfer

		-z, --compress compress file data

		--exclude=PATTERN exclude files matching PATTERN

		--daemon run as a rsync daemon

		--password-file=FILE get password from FILE

Resources

• http://members.ozemail.com.au/~msteveb/rsync/
• http://www.ccp14.ac.uk/ccp14admin/rsync/
• http://optics.ph.unimelb.edu.au/help/rsync/
• http://sunsite.dk/SunSITE/guides/rsync/
• http://sunsite.dk/SunSITE/guides/rsync/rsync-mirroring02.html
• ftp://ftp.minn.net/usr/mmchen/
• http://www.uwsg.indiana.edu/security/rsync.html
• http://filewatcher.org/sec/rsync.html
• http://freeos.com/printer.php?entryID=4042
• A Tutorial on Using rsync
• Rsnapshot (higher-level backup utility based on rsync, improved ease-of-use, allows you to keep multiple snapshots in time of your data, local or remote)

Linux.com :: Back up like an expert with rsync

In the last two months I've been traveling a lot. During the same period my main desktop computer went belly up. I would have been in trouble without rsync at my disposal -- but thanks to my regular use of this utility, my data (or most of it, anyway) was already copied offsite just waiting to be used. It takes a little time to become familiar with rsync, but once you are, you should be able to handle most of your backup needs with just a short script.

What's so great about rsync? First, it's designed to speed up file transfer by copying the differences between two files rather than copying an entire file every time. For example, when I'm writing this article, I can make a copy via rsync now and then another copy later. The second (and third, fourth, fifth, etc.) time I copy the file, rsync copies the differences only. That takes far less time, which is especially important when you're doing something like copying a whole directory offsite for daily backup. The first time may take a long time, but the next will only take a few minutes (assuming you don't change that much in the directory on a daily basis).

Another benefit is that rsync can preserve permissions and ownership information, copy symbolic links, and generally is designed to intelligently handle your files.

You shouldn't need to do anything to get rsync installed -- it should be available on almost any Linux distribution by default. If it's not, you should be able to install it from your distribution's package repositories. You will need rsync on both machines if you're copying data to a remote system, of course.

When you're using it to copy files to another host, the rsync utility typically works over a remote shell, such as Secure Shell (SSH) or Remote Shell (RSH). We'll work with SSH in the following examples, because RSH is not secure and you probably don't want to be copying your data using it. It's also possible to connect to a remote host using an rsync daemon, but since SSH is practically ubiquitous these days, there's no need to bother.

[Aug 26, 2008] Easy local and remote backup of your home network - Lone Wolves - Web, game, and open source development

2006-09-20

I hate making backups by hand. It costs a lot of time and usually I have far better things to do. Long ago (in the Windows 98 era) I made backups to CD only before I needed to reïnstall the OS, which was about once every 18 months, and my code projects maybe twice as often. A lot has changed since those dark times though. My single PC expanded into a network with multiple desktops and a server, I installed a mix of Debian an Ubuntu and ditched Windows, and I have a nice broadband link - just as my friends do. Finally a lazy git like me can set up a decent backup system that takes care of itself, leaving me time to do the "better" things (such as writing about it :-)

There are already quite a few tutorials on the internet explaining various ways to backup your Linux system using built-in commands and a script of some sorts, but I could not find one that suited me so I decided to write another one - one that takes care of backing up my entire network.

[Dec 8, 2006] rsnapshot A Perl-based filesystem snapshot utility.

rsnapshot is a filesystem snapshot utility based on rsync. It makes it easy to make periodic snapshots of local machines, and remote machines over ssh. It uses hard links whenever possible, to greatly reduce the disk space required.

[Dec 8, 2006] Warsync A Perl-based server replication program based on rsync.

Warsync (Wrapper Around Rsync) is a server replication system mainly used to sync servers in LVS clusters. It is based on rsync over ssh and has native support for Debian package synchronization.

[Feb 28, 2000] Everything Linux - A Tutorial on Using Rsync by Michael Holve

November 20, 1999

Rsync is a wonderful little utility that's amazingly easy to set up on your machines. Rather than have a scripted FTP session, or some other form of file transfer script -- rsync copies only the diffs of files that have actually changed, compressed and through ssh if you want to for security. That's a mouthful -- but what it means is:

• Diffs - Only actual changed pieces of files are transferred, rather than the whole file. This makes updates faster, especially over slower links like modems. FTP would transfer the entire file, even if only one byte changed.
• Compression - The tiny pieces of diffs are then compressed on the fly, further saving you file transfer time and reducing the load on the network.
• Secure Shell - The security concious of you out there would like this, and you should all be using it. The stream from rsync is passed through the ssh protocol to encrypt your session instead of rsh, which is also an option (and required if you don't use ssh - enable it in your /etc/inet.d and restart your inet daemon if you disabled it for security).

Rsync is rather versatile as a backup/mirroring tool, offering many features above and beyond the above. I personally use it to synchronize Website trees from staging to production servers and to backup key areas of the filesystems both automatically through cron and by a CGI script. Here are some other key features of rsync:

• Support for copying links, devices, owners, groups and permissions
• Exclude and exclude-from options similar to GNU tar
• A CVS exclude mode for ignoring the same files that CVS would ignore
• Does not require root privileges
• Pipelining of file transfers to minimize latency costs
• Support for anonymous or authenticated rsync servers (ideal for mirroring)

How does it work?

You must set up one machine or another of a pair to be an "rsync server" by running rsync in a daemon mode ("rsync --daemon" at the commandline) and setting up a short, easy configuration file (/etc/rsyncd.conf). Below I'll detail a sample configuration file. The options are readily understood, few in number -- yet quite powerful.

Any number of machines with rsync installed may then synchronize to and/or from the machine running the rsync daemon. You can use this to make backups, mirror filesystems, distribute files or any number of similar operations. Through the use of the "rsync algorithm" which transfers only the diffs between files (similar to a patch file) and then compressing them -- you are left with a very efficient system.

For those of you new to secure shell ("ssh" for short), you should be using it! There's a very useful and quite thourough Getting Started with SSH document available. You may also want to visit the Secure Shell Web Site. Or, just hit the Master FTP Site in Finland and snag it for yourself. It provides a secure, encrypted "pipe" for your network traffic. You should be using it instead of telnet, rsh or rlogin and use the replacement "scp" command instead of "rcp." Setting up a Server

You must set up a configuration file on the machine meant to be the server and run the rsync binary in daemon mode. Even your rsync client machines can run rsync in daemon mode for two-way transfers. You can do this automatically for each connection via the inet daemon or at the commandline in standalone mode to leave it running in the background for often repeated rsyncs. I personally use it in standalone mode, like Apache. I have a crontab entry that synchronizes a Web site directory hourly. Plus there is a CGI script that folks fire off frequently during the day for immediate updating of content. This is a lot of rsync calls! If you start off the rsync daemon through your inet daemon, then you incur much more overhead with each rsync call. You basically restart the rsync daemon for every connection your server machine gets! It's the same reasoning as starting Apache in standalone mode rather than through the inet daemon. It's quicker and more efficient to start rsync in standalone mode if you anticipate a lot of rsync traffic. Otherwise, for the occasional transfer follow the procedure to fire off rsync via the inet daemon. This way the rsync daemon, as small as it is, doesn't sit in memory if you only use it once a day or whatever. Your call.

Below is a sample rsync configuration file. It is placed in your /etc directory as rsyncd.conf.

motd file = /etc/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[simple_path_name]
   path = /rsync_files_here
   comment = My Very Own Rsync Server
   uid = nobody
   gid = nobody
   read only = no
   list = yes
   auth users = username
   secrets file = /etc/rsyncd.scrt

Various options that you would modify right from the start are the areas in italics in the sample above. I'll start at the top, line by line, and go through what you should pay attention to. What the sample above does is setup a single "path" for rsync transfers to that machine.

Starting at the top are four lines specifying files and their paths for rsync running in daemon mode. The first is a "message of the day" (motd) file like you would use for an FTP server. This file's contents get displayed when clients connect to this machine. Use it as a welcome, warning or simply identification. The next line specifies a log file to send diagnostic and norml run-time messages to. The PID file contains the "process ID" (PID) number of the running rsync daemon. A lock file is used to ensure that things run smoothly. These options are global to the rsync daemon.

The next block of lines is specific to a "path" that rsync uses. The options contained therein have effect only within the block (they're local, not global options). Start with the "path" name. It's somewhat confusing that rsync uses the term "path" -- as it's not necessarily a full pathname. It serves as an "rsync area nickname" of sorts. It's a short, easy to remember (and type!) name that you assign to a try filesystem path with all the options you specify. Here are the things you need to set up first and foremost:

• path - this is the actual filesystem path to where the files are rsync'ed from and/or to.
• comment - a short, descriptive explanation of what and where the path points to for listings.
• auth users - you really should put this in to restrict access to only a pre-defined user that you specify in the following secrets file - does not have to be a valid system user.
• secrets file - the file containing plaintext key/value pairs of usernames and passwords.

One thing you should seriously consider is the "hosts allow" and "hosts deny" options for your path. Enter the IPs or hostnames that you wish to specifically allow or deny! If you don't do this, or at least use the "auth users" option, then basically that area of your filesystem is wide open to the world by anyone using rsync! Something I seriously think you should avoid...

Check the rsyncd.conf man page with "man rsyncd.conf" and read it very carefully where security options are concerned. You don't want just anyone to come in and rsync up an empty directory with the "--delete" option, now do you?

The other options are all explained in the man page for rsyncd.conf. Basically, the above options specify that the files are chmod'ed to uid/gid, the filesystem path is read/write and that the rsync path shows up in rsync listings. The rsync secrets file I keep in /etc/ along with the configuration and motd files, and I prefix them with "rsyncd." to keep them together.

Using Rsync Itself

Now on to actually using, or initiating an rsync transfer with rsync itself. It's the same binary as the daemon, just without the "--daemon" flag. It's simplicity is a virtue. I'll start with a commandline that I use in a script to synchronize a Web tree below.

rsync --verbose  --progress --stats --compress --rsh=/usr/local/bin/ssh \
      --recursive --times --perms --links --delete \
      --exclude "*bak" --exclude "*~" \
      /www/* webserver:simple_path_name

Let's go through it one line at a time. The first line calls rsync itself and specifies the options "verbose," progress" and "stats" so that you can see what's going on this first time around. The "compress" and "rsh" options specify that you want your stream compressed and to send it through ssh (remember from above?) for security's sake.

The next line specifies how rsync itself operates on your files. You're telling rsync here to go through your source pathname recursively with "recursive" and to preserve the file timestamps and permissions with "times" and "perms." Copy symbolic links with "links" and delete things from the remote rsync server that are also deleted locally with "delete."

Now we have a line where there's quite a bit of power and flexibility. You can specify GNU tar-like include and exclude patterns here. In this example, I'm telling rsync to ignore some backup files that are common in this Web tree ("*.bak" and "*~" files). You can put whatever you want to match here, suited to your specific needs. You can leave this line out and rsync will copy all your files as they are locally to the remote machine. Depends on what you want.

Finally, the line that specifies the source pathname, the remote rsync machine and rsync "path." The first part "/www/*" specifies where on my local filesytem I want rsync to grab the files from for transmission to the remote rsync server. The next word, "webserver" should be the DNS name or IP address of your rsync server. It can be "w.x.y.z" or "rsync.mydomain.com" or even just "webserver" if you have a nickname defined in your /etc/hosts file, as I do here. The single colon specifies that you want the whole mess sent through your ssh tunnel, as opposed to the regular rsh tunnel. This is an important point to pay attention to! If you use two colons, then despite the specification of ssh on the commandline previously, you'll still go through rsh. Ooops. The last "www" in that line is the rsync "path" that you set up on the server as in the sample above.

Yes, that's it! If you run the above command on your local rsync client, then you will transfer the entire "/www/*" tree to the remote "webserver" machine except backup files, preserving file timestamps and permissions -- compressed and secure -- with visual feedback on what's happening.

Note that in the above example, I used GNU style long options so that you can see what the commandline is all about. You can also use abbreviations, single letters -- to do the same thing. Try running rsync with the "--help" option alone and you can see what syntax and options are available.

Rsync on the Net

You can find the rsync distribution at the rsync home page or hit the rsync FTP site.

There are also various pages of information on rsync out there, many of which reside on the rsync Web site. Below are three documents that you should also read thouroughly before using rsync so that you understand it well:

• Rsync man page
• Rsyncd.conf man page
• Rsync algorithm technical report

Recommended Links

rsync documentation

• An html version of the rsync man page.
• An html version of the rsyncd.conf man page.
• The FAQ (frequently asked questions list).
• A nice tutorial on setting up ssh to avoid password prompts also covers how you can restrict the commands allowed and setup a cron job to run rsync.
• Karsten Thygesen has written a doc on how to setup anonymous rsync servers.
• Jim Davis has written a doc on how to setup cfengine to use rsync.
• Anthony Wesley has written a doc on how to build rsync for Windows95.
• Mike McHenry has written up come info on how to get rsync working under NT.
• Michael Holve has written a very useful rsync tutorial.
• Daniel Teklu has a detailed HOWTO on rsync and stunnel.
• A html version of the original rsync technical report.
• A copy of Andrew Tridgell's PhD thesis (which includes three chapters on rsync).
• A nice page on how rsync works.
• A copy of John Langford's thesis on Multiround rsync, which is not used in rsync, but is interesting none-the-less.
• The rsync TODO.
• A DevShed tutorial on rsync.
• Notes on rsync and Debian mirrors.

Advanced applications of rsync by Martin Streicher

Mirror Your Web Site With rsync HowtoForge - Linux Howtos and Tutorials Version 1.0 Author: Falko Timme <ft [at] falkotimme [dot] com. Last edited 04/20/2006

rsync Tips & Tricks

Everything Solaris

rsync home page

rsync FTP site.

Rsync man page

Rsyncd.conf man page

Rsync algorithm technical report

wincent.com Remote backups via rsync between RHEL 5.3 and Mac OS X 10.5.7 Leopard

Etc

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg  : C++ Humor : ARE YOU A BBS ADDICT? : Object oriented programmers of all nations :  C Humor : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor :  PseudoScience Related Humor : Networking Humor  : Shell Humor: Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : The Most Comprehensive Collection of Editor-related Humor : Microsoft plans to buy Catholic Church : Education Humor : IBM Humor :  Assembler-related Humor :  VIM Humor Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor : Best Russian Programmer Humor : Russian Musical Humor : The Perl Purity Test :  Politically Incorrect Humor : GPL-related Humor : OFM Humor : IDS Humor : Real Programmers Humor : Scripting Humor : Web Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor :

The Last but not Least

Copyright © 1996-2013 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine. This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: January 12, 2013