2-Way Sync with RSYNC

I have two MacBook Pros.  One is a 32GB 15″ and the other is a 16GB 15″.  The 16GB stays at home connected to a 27″ monitor although I am buying a 49″ Curved monitor this month.  However I want to synchronize my Microsoft Word and Excel templates between the machines.  Additionally, I use OmniGraffle for my network diagraming.  I have stencils I use regularly and want to synchronize them between the machines.

Challenge: Sometimes I may be on my home MBP and create a Microsoft Word Template and want it on my “travel” MBP.  However sometime, I may create a Microsoft Word Template on my Travel MBP and want it on my Home MBP.

I will also cover some common uses for rsync

The Basics

Most macOS based devices should have rsync included with them.  If you are running a Linux distro, you will need to use yum (rpm based) or apt-get (Debian based) depending on your platform. And if you are running Windows you will need cygwin  or the Ubuntu integration (which will need the apt-get package installed).

At the end of this article I have the options for rsync

In this example, I will use 3 folders on my MacBook:

folder1 = Will represent my home MacBook Pro connected to monitor and keyboard.
folder2 = Will represent my travel MacBook Pro. The one I use when not at home
folder3 = Will represent my Dropbox folder that is synchronized to both MacBooks.

In this test, I will create some files in folder1​ and different files in folder2 .  I will then synchronize both folder1 and folder2 to folder3 and in the end, I should have duplicate files in folder1, folder2 and folder3

Here we go

The Setup

Making Directories / Folder

First, we are going to create the three folders on my ~/Desktop folder

mkdir folder1 folder2 folder3

Creating Files in folder1

We are going to use touch to create some empty files for time purposes.

touch file1 file2 file3 file4

Creating Files in folder2

We are going to use touch to create some empty files for time purposes.

touch file5 file6 file7 file8

Syncing to the “Dropbox” folder

Remember that folder3 represents the Dropbox folder that is synchronized between both computers.

From “home” computer to “Dropbox”

rsync -auv /Users/mannyfernandez/Desktop/folder1/ /Users/mannyfernandez/Desktop/folder3

You can see that we are using the -auv switches

-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
-u, --update skip files that are newer on the receiver
-v, --verbose increase verbosity

As you can see, we copied 5 files.  The .DS_Store file is a macOS file that in most cases you do not want. More on that later.

From “travel” computer to “Dropbox”

rsync -auv /Users/mannyfernandez/Desktop/folder2/ /Users/mannyfernandez/Desktop/folder3

Again we are using the same command but referencing a different folder

Validating “Dropbox”

We can see the aggregate of both folders in “Dropbox”

Syncing “Dropbox” to the “Computers”

So now that “Dropbox” has your files from folder1 and folder2 we are going to modify the command to synchronize from folder3 or Dropbox to each computer.

rsync -auv /Users/mannyfernandez/Desktop/folder3/ /Users/mannyfernandez/Desktop/folder1

Here you see the source being folder3 and the destination folder1

The output shows file5, file6, file7 and file8 have been copied.

An ls shows files1-8 in the folder

rsync -auv /Users/mannyfernandez/Desktop/folder3/ /Users/mannyfernandez/Desktop/folder2

An ls shows files1-8 in the folder

Automating the process

I created two scripts (one on each computer) to automatically run and copy the data.

sync1.sh

#!/bin/bash
rsync -auv /Users/mannyfernandez/Desktop/folder1/ /Users/mannyfernandez/Desktop/folder3
rsync -auv /Users/mannyfernandez/Desktop/folder3/ /Users/mannyfernandez/Desktop/folder1

synch2.sh

#!/bin/bash
rsync -auv /Users/mannyfernandez/Desktop/folder2/ /Users/mannyfernandez/Desktop/folder3
rsync -auv /Users/mannyfernandez/Desktop/folder3/ /Users/mannyfernandez/Desktop/folder2

To automate this on macOS and Linux, you can use the crontab -e command.  I like to add the text below with # before each line so you can see it in the cron tab.

* * * * *  command to execute
│ │ │ │ │
│ │ │ │ └─── day of week (0 - 6) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
│ │ │ └──────── month (1 - 12)
│ │ └───────────── day of month (1 - 31)
│ └────────────────── hour (0 - 23)
└─────────────────────── min (0 - 59)

To run my example, you would add the following to the crown -e

60 * * * * ~/Desktop/synch1.sh

Making Scripts Executable

To make the scripts executable, you are going to need to change access permissions.

chmod +x %script-name%

.DS_Store

.DS_Store is a file that stores custom attributes of its containing folder, such as the position of icons or the choice of a background image.  You may not want these option on the other computer.  You can include the –exclude command to exclude it from being synchronized.

RSYNC Option

OPTIONS SUMMARY
Here is a short summary of the options available in rsync.

-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don’t send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place
--append append data onto shorter files
--append-verify --append w/old data in file checksum
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the tree
-k, --copy-dirlinks transform symlink to dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
-E, --executability preserve executability
--chmod=CHMOD affect file and/or directory permissions
-A, --acls preserve ACLs (implies -p)
-X, --xattrs preserve extended attributes
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve modification times
-O, --omit-dir-times omit directories from --times
--super receiver attempts super-user activities
--fake-super store/recover privileged attrs using xattrs
-S, --sparse handle sparse files efficiently
-n, --dry-run perform a trial run with no changes made
-W, --whole-file copy files whole (w/o delta-xfer algorithm)
-x, --one-file-system don’t cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that exist on receiver
--remove-source-files sender removes synchronized files (non-dir)
--del an alias for --delete-during
--delete delete extraneous files from dest dirs
--delete-before receiver deletes before transfer (default)
--delete-during receiver deletes during xfer, not before
--delete-delay find deletions during, delete after
--delete-after receiver deletes after transfer, not before
--delete-excluded also delete excluded files from dest dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of dirs even if not empty
--max-delete=NUM don’t delete more than NUM files
--max-size=SIZE don’t transfer any file larger than SIZE
--min-size=SIZE don’t transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at end
-m, --prune-empty-dirs prune empty directory chains from file-list
--numeric-ids don’t map uid/gid values by user/group name
--timeout=SECONDS set I/O timeout in seconds
--contimeout=SECONDS set daemon connection timeout in seconds
-I, --ignore-times don’t skip files that match size and time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare received files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
--skip-compress=LIST skip compressing files with suffix in LIST
-C, --cvs-exclude auto-ignore files in the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter=’dir-merge /.rsync-filter’
repeated: --filter=’- .rsync-filter’
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don’t exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *from/filter files are delimited by 0s
-s, --protect-args no space-splitting; wildcard chars only
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we’re doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read daemon-access password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating dest
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
--iconv=CONVERT_SPEC request charset conversion of filenames
--checksum-seed=NUM set block/file checksum seed (advanced)
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (see below for -h comment)

Rsync can also be run as a daemon, in which case the following options are accepted:
Rsync can also be run as a daemon, in which case the following options are accepted:

--daemon run as an rsync daemon
--address=ADDRESS bind to the specified address
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--config=FILE specify alternate rsyncd.conf file
--no-detach do not detach from the parent
--port=PORT listen on alternate port number
--log-file=FILE override the "log file" setting
--log-file-format=FMT override the "log format" setting
--sockopts=OPTIONS specify custom TCP options
-v, --verbose increase verbosity
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
-h, --help show this help (if used after --daemon)