rsync

Example: --no-devices, --no-perms have different defaults in various circumstances
Example: --no-whole-file, --no-blocking-io, --no-dirs.

The order of the options is important. The result of
--no--recursive --archive is that --recursive would end up being turned on, the opposite of
--archive --no--recursive.

rsync --list-only uname@Real-World-Systems.com:*.txt # list all txt files on that host's root directory
-rw-r----- 160 2017/05/18 20:31:30 apacheNotes.txt
-rw-r--r-- 1490 2018/10/22 16:57:32 pepper-181022.txt
-rw-r----- 3679 2017/12/29 19:04:47 smacproPart4.txt

rsync --list-only /Volumes/big/*.MOV # list all the MOV files on that volume/dirctory
-rw-r--r-- 308842544 2018/05/16 07:56:53 CamperGrandTour_2064.MOV
-rw-r--r-- 12014106 2018/06/23 15:40:36 DoorClosingWithGap188.MOV
-rw-r--r-- 185538306 2017/02/18 13:29:27 Hummers_0655.MOV
-rw-r--r-- 19434966 2019/01/19 09:28:38 IMG_0332.MOV

Does not preserve hardlinks, ---hard-links.

If -archive follows --compress it will reenable the "--no-" options

The side-effects of the --files-from option are NOT positional, as it affects the default state of several options and slightly changes the meaning of --archive.

Following only seen with --verbose --verbose
xxx is over max-size
[sender] hiding file email/Trash because of pattern Trash*.

Filenames are displayed in --out-format of %n%L by default, ( name and if it is a link, where it points).
At the first level verbosity no mention is made if a file had its attributes changed.
More than two are used for debugging

.d        var/
.f        var/logPrune.sh 

Each file, dir, etc. that gets updated in a significant way (a transferred file, a recreated symlink/device, or a touched directory) will be output.
(e.g. With --itemize-changes the logging of names includes any item changed.

The out-formated string is output prior to a file's transfer.
If a transfer-statistic option is requested, logging is done at the end of the file's transfer. When this late logging is in effect and --progress is specified, the name of the file being transferred is output prior to its progress information followed by the out-format output.

rsync --archive --verbose --rsync-path="rsync --log-file=/tmp/rlog" src/ dest/

Number of files: 88417
Number of files transferred: 0
Total file size: 5008416760 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2710302
File list generation time: 1.156 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 20
Total bytes received: 2710322

sent 20 bytes  received 2710322 bytes  285299.16 bytes/sec
total size is 5008416760  speedup is 1847.89

782448  63%  110.64kB/s    0:00:04

Statistics can be misleading with the delta-transfer algorithm. For example, if the file is a log previously sent the reported rate will drop dramatically when the receiver gets to the new data, and the transfer will take much longer to finish than predicted.

When a file transfer finishes, the progress line is replaced with a summary line like :

1238099 100%  146.38kB/s    0:00:08  (xfer#5, to-check=169/396)

Does not affect the copying of symlinks or other special files.

transfer handling

This does not affect updating a file's attributes (e.g. permissions, ownership, etc.) when the file does not need to be transferred, nor does it affect the updating of any non-regular files.
Implies --inplace

partdir will be in the destination directory.
If partdir is not an absolute path, an exclude rule is added at the end of all existing excludes preventing sending partial-dir files on the sending side, and prevent untimely deletion of partial-dir items on the receiving side.
An example: the above --partial-dir option would add the equivalent of -f '-p .rsync-partial/' at the end of any other filter rules.

When supplying exclude rules, you may need to add your own exclude/hide/protect rule for the partial-dir because
(1) the auto-added rule may be ineffective at the end of your other rules, or
(2) you may wish to override rsync's exclude choice. For instance, make rsync clean-up any left-over partial-dirs that may be lying around, specify --delete-after and add a "risk" filter rule, eg. -f 'R .rsync-partial/'. (Avoid using --delete-before or --delete-du ing unless you don't need rsync to use any of the left-over partial-dir data during the current run.)

IMPORTANT: the --partial-dir should not be writable by other users or it is a security risk.
AVOID "/tmp".

You can also set the partial-dir value the RSYNC_PARTIAL_DIR environment variable. Setting this in the environment does not force --partial to be enabled, but rather it affects where partial files go when --partial is specified. For instance, instead of using --partial-dir=.rsync-tmp along with --progress, Set RSYNC_PARTIAL_DIR=.rsync-tmp environment variable and then use -P to turn on the use of the .rsync-tmp dir for partial transfers.
The only times that --partial does not look for this environment value are (1) when --inplace was specified (since --inplace conflicts with --partial-dir), and (2) when --delay-updates was specified (see below).

For the purposes of the daemon-config's "refuse options" setting, --partial-dir does NOT imply --partial. This is so that a refusal of the --partial option can be used to disallow the overwriting of destination files with a partial transfer, while still allowing the safer idiom provided by --partial-dir. If --whole-file is specified (or implied), partial-dir files are removed.

Requires enough free disk space on the receiving side to hold an additional copy of all the updated files.

Do not use an absolute path to --partial-dir unless
(1) there is no chance of any of the files in the transfer having the same name (since all the updated files will be put into a single directory if the path is absolute)
and
(2) there are no mount points in the hierarchy (since the delayed updates will fail if they can't be renamed into place).

See also the "atomic-rsync" perl script in the "support" subdir for an update algorithm that is even more atomic (it uses --link-dest and a parallel hierarchy of files).

Effects:
1. in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash),
2. file data will inconsistent during the transfer,
3. file data may be left in an inconsistent state after the transfer if the transfer is interrupted or fails,
4. a file that does not have write permissions can not be updated, and
5. the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file
   Combine with --backup, since rsync will use the backup file as the basis file for the transfer to prevent this.

WARNING: Do not use this to update files being accessed by others.

Useful for transferring large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.

Implies --partial (since an interrupted transfer does not delete the file),
Conflicts with --partial-dir and --delay-updates.

rsync -av /foo/bar/baz.c remote:/tmp/ 

 rsync -avR /foo/bar/baz.c remote:/tmp/ 

These extra path elements are called "implied directories" (i.e. the "foo" and the "foo/bar" directories in the above example). rsync sends these implied directories as real directories in the file list, even if a path element is really a symlink on the sending side. This prevents unexpected behaviors when copying the full path of a file that you didn't realize had a symlink in its path.
To duplicate a server-side symlink, include both the symlink via its path, and referent directory via its real path.

To limit the amount of path information that is sent as implied directories for each path, insert a dot and a slash into the source path.

                 rsync -avR /foo/./bar/baz.c remote:/tmp/

For instance, if a command-line arg or a files-from entry told rsync to transfer the file "path/foo/file", the directories "path" and "path/foo" are implied when --relative is used. If "path/foo" is a symlink to "bar" on the destination system, the receiving rsync would ordinarily delete "path/foo", recreate it as a directory, and receive the file into the new directory. With --no-implied-dirs, the receiving rsync updates "path/foo/file" using the existing path elements, which means that the file ends up being created in "path/bar".
Another way to accomplish this link preservation is to use the --keep-dirlinks option (which will also affect symlinks to directories in the rest of the transfer).

without --backup-dir,

1. --omit-dir-times will be implied, and
2. if --delete is also in effect (without --delete-excluded), rsync will add a "protect" filter-rule for the backup suffix to the end of all existing excludes (e.g. -f "P *~").

--dirs is implied by --files-from or --list-only (including an implied --list-only usage) if wasn't specified (so that directories are seen in the listing).
Specify --no-dirs to turn this off.

Link handling

Without this , if the sending side has replaced a directory with a symlink to a directory, the receiving side will delete anything that is in the way of the new symlink, including a directory hierarchy (as long as --force or --delete is in effect).

See --keep-dirlinks for an analogous option for the receiving side.

For example, transfer a directory "foo" that contains a file "file", but "foo" is a symlink to directory "bar" on the receiver.
Without --keep-dirlinks, the receiver deletes symlink "foo", recreates it as a directory, and receives the file into the new directory.
With --keep-dirlinks, the receiver keeps the symlink and "file" ends up in "bar". caution: when using --keep-dirlinks, you must trust all the symlinks in the copy!
If it is possible for an untrusted user to create their own symlink to any directory, the user could then (on a subsequent copy) replace the symlink with a real directory and affect the content of whatever directory the symlink references.
For backup copies, use a bind mount instead of a symlink to modify your receiving hierarchy.

See also --copy-dirlinks for an analogous option for the sending side.

When updating a non-empty destination, ensures that files that are hard-linked together on the source are hard-linked together on the destination.
Does NOT break existing hard links on the destination that do not exist between the source files.
If one or more extra-linked files have content changes, they will become unlinked when updated (if not using --inplace ).

Rsync can only detect hard links between files that are in the transfer set. If rsync updates a file that has extra hard-link connections to files outside the transfer, that linkage will be broken.
Using --inplace to avoid this breakage, there may be unintended changes due to lingering hard links (see --inplace ).

If incremental recursion is active rsync may transfer a missing hard-linked file before it finds that another link for that contents exists elsewhere in the hierarchy.
This does not affect the accuracy of the transfer, just its efficiency, To avoid this disable incremental recursion with --no-inc-recursive

Attributes, Meta-data, i.e. file permissions, Access Control Lists, owner, group …

With --no--perms:

An item that should only apply to a directory is prefixed with D', or to a file if prefixed with 'F'.
For example:

--chmod=Dg+s,ug+w,Fo-w,+X

See --perms and --executability for how the resulting permission value can be applied to the files in the transfer.

only if the receiver is super-user (see --super and --fake-super ).
Caution: The preservation of ownership will associate matching names but if it falls back to using the ID number (see --numeric-ids it will surely be incorrect!!.

The preservation of group information will associate matching names by default, but may fall back to using the ID number

 rsync -av --rsync-path="rsync --fake-super" /src/ host:/dest/

overridden by both --super and --no-super.

See fake super in rsyncd.conf.

If repeated, rsync omits all mount-point directories.
Otherwise, it includes an empty directory at each mount-point it encounters (using the attributes of the mounted directory because those of the underlying mount-point directory are inaccessible).

If collapse symlinks (via --copy-links or --copy-unsafe-links), a symlink to a directory on another device is treated like a mount-point. Symlinks to non-directories are unaffected by this option.

This is a transfer rule, not an exclude, it doesn't affect the data that goes into the file-lists, and thus it doesn't affect deletions. It just limits the files that the receiver requests to be transferred.

--max-delete=0 generates warning about files in the destination without removing them.

Deletions will also occur when --dirs is enabled for directories whose contents are being copied.

Use --dry-run option to see what files are going to be deleted.

--delete-before, --delete-after, --prune-empty-dirs, and --delay-updates.
The default with --delete is --delete-during (use --del --delete-during to request this deletion mode explicitly).
--delete-delay is a better choice than using --delete-after.
Incremental recursion can be disabled using --no-inc-recursive.

If the sending side detects I/O errors, deletion of any files will be disabled,
Especially useful with NFS errors on the sending side)

May be combined with one of the --delete-WHEN options as well as --delete-excluded.
Default is --delete-during .
See --delete-delay and --delete-after.

Helpful when the destination filesystem is low on space.
Introduces a delay before the start of the transfer which might cause the transfer to timeout
Forces non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at once
(see --recursive).

• KB (1,000) : K KiB kilibyte (1024):
• MB ( 1,000,000) :M MiB megibyte (1024*1024)
• GB (1,000 MB) :G GiB gibibyte (1024*1024*1024)
• +1 or -1 to offset by one byte.

Examples: --max-size=1.5mb-1 is 1,499,999 bytes, and
--max-size=2g+1 is 2,147,483,649 bytes.

With [user@]host::module/path, the remote shell will be used to run an rsync daemon on the remote host, and all data will be transmitted through that remote shell connection, not a socket connection to a running rsync daemon on the remote host.
See using rsync-daemon features via a remote-shell connection

Command-line arguments are permitted in rmsh provided that is presented to rsync as a single argument.
Use spaces (not tabs or other whitespace) to separate rmsh into the command and arguments.
Use apostrophe and/or quotes to preserve spaces in an argument (not backslashes).
Doubling an apostrophe inside a apostrophe'd string results in apostrophe ( likewise for quotes ) (attention to which quotes the shell is parsing and which quotes rsync is parsing).

Examples:

 -e 'ssh -p 2234'
  -e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"' 

RSYNC_RSH environment variable specifies remote shell program, which accepts the same range of values as -e.
See --blocking-io which is affected by this option.

                  rsync -avR --rsync-path="cd /a/b && rsync" host:c/d /e/  

Filters out the .rsync-filter files themselves .

1. The exclude list is initialized with :

   RCS   SCCS   CVS   CVS.adm   RCSLOG   cvslog.*   tags   TAGS    .make.state   .nse_depinfo   *~   #*   .#*   ,*   _$*   *$   *.old   *.bak    *.BAK   *.orig   *.rej   .del-*   *.a   *.olb   *.o   *.obj   *.so   *.exe    *.Z   *.elc   *.ln   core   .svn/   .git/   .bzr/

2. Files in the CVSIGNORE environment variable (delimited by whitespace).
3. Files listed in $HOME/.cvsignore
4. Files in a directory containing a .cvsignore
5. Files matching a pattern in .cvsignore These patterns are split on whitespace. See CVS

if used with --iconv, the args related to the remote side will also be translated to the remote character-set. The translation happens before wild-cards are expanded. See --files-from .

Useful when there is not enough free space to hold a copy of the largest file in the destination filesystem. This requires rsync to it copy the file into place when the file is transfered.

Combined with --delay-updates , which will ensure that all copied files get put into subdirectories in the destination hierarchy, awaiting the end of the transfer.
If there is not enough room to duplicate all the arriving files on the destination filesystem, use --partial-dir with a relative path; because it is OK to stash off a copy of a single file in a subdir in the destination hierarchy, rsync will use the partial-dir as a staging area to bring over the copied file, and then rename it into place from there. (Specifying a --partial-dir with an absolute path does not have this side-effect.)

Use of --delete might get rid of any potential fuzzy-match files, so either use --delete-after or specify some filename exclusions to prevent this.

multiple --compare-dest directories may be provided, which will cause rsync to search the list in the order specified for an exact match. If a match is found that differs only in attributes, a local copy is made and the attributes updated. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

If DIR is a relative path, it is relative to the destination directory. See also --copy-dest and --link-dest.

Multiple --copy-dest directories may be provided, which will cause rsync to search the list in the order specified for an unchanged file. If a match is not found, a basis file from one of the DIRs will be selected to try to speed up the transfer.

If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --link-dest.

rsync -av --link-dest=$PWD/prior_dir host:src_dir/ new_dir/

multiple --link-dest directories may be provided, which will cause rsync to search the list in the order specified for an exact match. If a match is found that differs only in attributes, a local copy is made and the attributes updated. If a match is not found, a basis file from one of the DIR will be selected to try to speed up the transfer.

works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists), and as malleable (so it might change the attributes of a destination file, which affects all the hardlinked versions).

combine this option with --ignore-times, rsync will not link any files together because it only links identical files together as a substitute for transferring the file, never as an additional check after the file is updated.

If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --copy-dest.

Simple character-class matching is supported: each must consist of a list of letters inside the square brackets (e.g. no special classes, such as "[:alpha:]", are supported).

asterisk (*) and question-mark (?) have no special meaning.

Example specifies 6 suffixes to skip (since 1 of the 5 rules matches 2 suffixes):

--skip-compress=gz/jpg/mp[34]/7z/bz2

A copy from a daemon rsync will adds skipped suffixes to its list of non-compressing files (and its list may be configured to a different default).

For shell transfers use "Public key authentication".

By default username and groupname are used to establish ownership.
If a user or group on the source system does not match the destination, the numeric ID from the source system is used.

If on the source system, user jsmith is user ID 507 and on destination is user ID 509 and bsmith is user ID 507, files on the the destination will belong to bsmith !

See the comments on the "use chroot" setting in the rsyncd.conf manpage for information on how the chroot setting affects rsync's ability to look up the names of the users and groups and what you can do about it.
The special user and groups ID 0 are never mapped names even if the --numeric-ids option is not specified.
Only applicable when transfering files with various owners.

Output a literal backslash (\) and a hash (#), and 3 octal digits. For example, a newline would output as "\#012". A literal backslash that is in a filename is not escaped unless it is followed by a hash and 3 digits (0-9).

the use of transfer rules, such as the --min-size option, does not affect what goes into the file list, and thus does not leave directories empty, even if none of the files in a directory match the transfer rule.

Because the file-list is actually being pruned, also affects what directories get deleted when a delete is active. However, keep in mind that excluded files and directories can prevent existing items from being deleted due to an exclude both hiding source files and protecting destination files. See the perishable filter-rule option for how to avoid this.

prevent the pruning of certain empty directories from the file-list by using a global "protect" filter.
For instance, this option would ensure that the directory "emptydir" was kept in the file-list:

--filter 'protect emptydir/'

Here's an example that copies all .pdf files in a hierarchy, only creating the necessary destination directories to hold the .pdf files, and ensures that any superfluous files and directories in the destination are removed (note the hide filter of non-directories being used instead of an exclude): rsync -avm --del --include='*.pdf' -f 'hide,! */' src/ dest TO retain superfluous destination files, the "--include='*/' --exclude='*'"

Write the batch directly to portable media: if this media fills before the end of the transfer, apply that partial transfer to the destination and repeat the process to get the rest of the changes (this will use a partially updated destination system while the multi-update cycle is happening).

you only save bandwidth when pushing changes to a remote system because this allows the batched data to be diverted from the sender into the batch file without having to flow over the wire to the receiver (when pulling, the sender is remote, and thus can't write the batch).

For a list of charset names local iconv library supports, run "iconv --list".

The following list contain all the coded character …
  , 8859_1, 8859_2, 8859_3, 8859_4, 8859_5, 8859_6, 8859_7, 8859_8, 8859_9, 10646-1:1993, 10646-1:1993/UCS4,
  ANSI_X3.4-1968, ANSI_X3.4-1986, ANSI_X3.4, ANSI_X3.110-1983, ANSI_X3.110, ARABIC, ARABIC7, ARMSCII-8, ASCII, 
…
 US-ASCII, US, UTF-7, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE, UTF7, UTF8, …

With --protect-args filenames for transfer on the command-line are translated. See --files-from .

No conversion of names in filters (including include/exclude files) is done!.
Specify extra include/exclude rules if there are filename differences on the two sides.

With --iconv the daemon uses the charset specified in its "charset" configuration

some debug messages will only be output when --msgs2stderr is specified, especially those pertaining to I/O and buffer debugging. This option was added to 3.1.0, so an older rsync on the server side might reject your attempts at fine-grained control (if one or more flags needed to be send to the server and the server was too old to understand them). See also the "max verbosity" caveat above when dealing with a daemon.

If standard input is a socket then rsync will assume that it is being run via inetd, otherwise
it will detach from the current terminal and become a background daemon, reading rsyncd.conf on each connect made by a client. config

"exported" directories (modules) must be specified in rsyncd.conf

 echo "++++++++++++++++++" >> rsync.err  # initalize rsync.err so tail is ok
 tail -f rsync.err &     # continue to show errors as rsync runs                         †
 rsync --progress --archive --hard-links --compress --update     --stats               \
                  --log-file=`date +rsync_%y%m%d.log` --log-format='%t %o %B %l %M %f' \
       uname@webSite.com:public_html backupDir   2>> rsync.err 

Expect to see

uname@webSite.com password:

Use ^Z to push the job into the backround.

The progress of rsync will be displayed.

2011/10/26 21:14:30 recv rwx------ 20480 2011/10/26-21:14:10 jupiter/downloads
2011/10/26 21:14:30 recv rw------- 4520453 2011/09/23-13:00:56 jupiter/downloads/ACPIspec40a.pdf
     4520453 100%    2.70MB/s    0:00:01 (xfer#1, to-check=47541/88416)
2011/10/26 21:14:31 recv rw-r--r-- 1364852 2011/10/18-10:18:34 jupiter/downloads/nuketheappstore.zip
     1364852 100%    1.42MB/s    0:00:00 (xfer#2, to-check=47246/88416)
…

Number of files: 88416
Number of files transferred: 2
Total file size: 5004066643 bytes
Total transferred file size: 5885305 bytes
Literal data: 5885305 bytes
Matched data: 0 bytes
File list size: 2710269
File list generation time: 1.120 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 70
Total bytes received: 7497967

sent 70 bytes  received 7497967 bytes  714098.76 bytes/sec
total size is 5004066643  speedup is 667.38

 fg to bring the tail process to the foreground.

 then ^C to interrupt the tail.