Return to sparse-block.diff CVS log

Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / rsync / patches

rsync 3.2.3

This patch adds the --sparse-block option.  Andrea Righi writes:

  In some filesystems, typically optimized for large I/O throughputs (like
  IBM GPFS, IBM SAN FS, or distributed filesystems in general) a lot of
  lseek() operations can strongly impact on performances. In this cases it
  can be helpful to enlarge the block size used to handle sparse files
  directly from a command line parameter.

  For example, using a sparse write size of 32KB, I've been able to
  increase the transfer rate of an order of magnitude copying the output
  files of scientific applications from GPFS to GPFS or GPFS to SAN FS.

  -Andrea

To use this patch, run these commands for a successful build:

    patch -p1 <patches/sparse-block.diff
    ./configure                               (optional if already run)
    make

based-on: e94bad1c156fc3910f24e2b3b71a81b0b0bdeb70
diff --git a/fileio.c b/fileio.c
--- a/fileio.c
+++ b/fileio.c
@@ -34,6 +34,7 @@
 #define ALIGNED_LENGTH(len) ((((len) - 1) | (ALIGN_BOUNDARY-1)) + 1)
 
 extern int sparse_files;
+extern int sparse_files_block_size;
 
 OFF_T preallocated_len = 0;
 
@@ -149,7 +150,7 @@ int write_file(int f, int use_seek, OFF_T offset, const char *buf, int len)
 	while (len > 0) {
 		int r1;
 		if (sparse_files > 0) {
-			int len1 = MIN(len, SPARSE_WRITE_SIZE);
+			int len1 = MIN(len, sparse_files_block_size);
 			r1 = write_sparse(f, use_seek, offset, buf, len1);
 			offset += r1;
 		} else {
diff --git a/options.c b/options.c
--- a/options.c
+++ b/options.c
@@ -76,6 +76,7 @@ int remove_source_files = 0;
 int one_file_system = 0;
 int protocol_version = PROTOCOL_VERSION;
 int sparse_files = 0;
+long sparse_files_block_size = SPARSE_WRITE_SIZE;
 int preallocate_files = 0;
 int do_compression = 0;
 int do_compression_level = CLVL_NOT_SPECIFIED;
@@ -692,6 +693,7 @@ static struct poptOption long_options[] = {
   {"sparse",          'S', POPT_ARG_VAL,    &sparse_files, 1, 0, 0 },
   {"no-sparse",        0,  POPT_ARG_VAL,    &sparse_files, 0, 0, 0 },
   {"no-S",             0,  POPT_ARG_VAL,    &sparse_files, 0, 0, 0 },
+  {"sparse-block",     0,  POPT_ARG_LONG,   &sparse_files_block_size, 0, 0, 0 },
   {"preallocate",      0,  POPT_ARG_NONE,   &preallocate_files, 0, 0, 0},
   {"inplace",          0,  POPT_ARG_VAL,    &inplace, 1, 0, 0 },
   {"no-inplace",       0,  POPT_ARG_VAL,    &inplace, 0, 0, 0 },
@@ -2684,6 +2686,12 @@ void server_options(char **args, int *argc_p)
 		args[ac++] = arg;
 	}
 
+	if (sparse_files_block_size) {
+		if (asprintf(&arg, "--sparse-block=%lu", sparse_files_block_size) < 0)
+			goto oom;
+		args[ac++] = arg;
+	}
+
 	if (io_timeout) {
 		if (asprintf(&arg, "--timeout=%d", io_timeout) < 0)
 			goto oom;
diff --git a/rsync.1.md b/rsync.1.md
--- a/rsync.1.md
+++ b/rsync.1.md
@@ -379,6 +379,7 @@ detailed description below for a complete description.
 --super                  receiver attempts super-user activities
 --fake-super             store/recover privileged attrs using xattrs
 --sparse, -S             turn sequences of nulls into sparse blocks
+--sparse-block=SIZE      set block size used to handle sparse files
 --preallocate            allocate dest files before writing them
 --write-devices          write to devices as files (implies --inplace)
 --dry-run, -n            perform a trial run with no changes made
@@ -1477,6 +1478,18 @@ your home directory (remove the '=' for that).
     opposed to allocated sequences of null bytes) if the kernel version and
     filesystem type support creating holes in the allocated data.
 
+0.  `--sparse-block=SIZE`
+
+    Change the block size used to handle sparse files to SIZE bytes.  This
+    option only has an effect if the `--sparse` (`-S`) option was also
+    specified.  The default block size used by rsync to detect a file hole is
+    1024 bytes; when the receiver writes data to the destination file and
+    option `--sparse` is used, rsync checks every 1024-bytes chunk to detect if
+    they are actually filled with data or not.  With certain filesystems,
+    optimized to receive data streams for example, enlarging this block size
+    can strongly increase performance.  The option can be used to tune this
+    block size.
+
 0.  `--dry-run`, `-n`
 
     This makes rsync perform a trial run that doesn't make any changes (and
diff -Nurp a/rsync.1 b/rsync.1
--- a/rsync.1
+++ b/rsync.1
@@ -455,6 +455,7 @@ detailed description below for a complet
 --super                  receiver attempts super-user activities
 --fake-super             store/recover privileged attrs using xattrs
 --sparse, -S             turn sequences of nulls into sparse blocks
+--sparse-block=SIZE      set block size used to handle sparse files
 --preallocate            allocate dest files before writing them
 --write-devices          write to devices as files (implies --inplace)
 --dry-run, -n            perform a trial run with no changes made
@@ -1541,6 +1542,16 @@ NTFS, etc.), this option may have no pos
 If combined with \fB\-\-sparse\fP, the file will only have sparse blocks (as
 opposed to allocated sequences of null bytes) if the kernel version and
 filesystem type support creating holes in the allocated data.
+.IP "\fB\-\-sparse-block=SIZE\fP"
+Change the block size used to handle sparse files to SIZE bytes.  This
+option only has an effect if the \fB\-\-sparse\fP (\fB\-S\fP) option was also
+specified.  The default block size used by rsync to detect a file hole is
+1024 bytes; when the receiver writes data to the destination file and
+option \fB\-\-sparse\fP is used, rsync checks every 1024-bytes chunk to detect if
+they are actually filled with data or not.  With certain filesystems,
+optimized to receive data streams for example, enlarging this block size
+can strongly increase performance.  The option can be used to tune this
+block size.
 .IP "\fB\-\-dry-run\fP, \fB\-n\fP"
 This makes rsync perform a trial run that doesn't make any changes (and
 produces mostly the same output as a real run).  It is most commonly used
diff -Nurp a/rsync.1.html b/rsync.1.html
--- a/rsync.1.html
+++ b/rsync.1.html
@@ -370,6 +370,7 @@ detailed description below for a complet
 --super                  receiver attempts super-user activities
 --fake-super             store/recover privileged attrs using xattrs
 --sparse, -S             turn sequences of nulls into sparse blocks
+--sparse-block=SIZE      set block size used to handle sparse files
 --preallocate            allocate dest files before writing them
 --write-devices          write to devices as files (implies --inplace)
 --dry-run, -n            perform a trial run with no changes made
@@ -1420,6 +1421,18 @@ opposed to allocated sequences of null b
 filesystem type support creating holes in the allocated data.</p>
 </dd>
 
+<dt><code>--sparse-block=SIZE</code></dt><dd>
+<p>Change the block size used to handle sparse files to SIZE bytes.  This
+option only has an effect if the <code>--sparse</code> (<code>-S</code>) option was also
+specified.  The default block size used by rsync to detect a file hole is
+1024 bytes; when the receiver writes data to the destination file and
+option <code>--sparse</code> is used, rsync checks every 1024-bytes chunk to detect if
+they are actually filled with data or not.  With certain filesystems,
+optimized to receive data streams for example, enlarging this block size
+can strongly increase performance.  The option can be used to tune this
+block size.</p>
+</dd>
+
 <dt><code>--dry-run</code>, <code>-n</code></dt><dd>
 <p>This makes rsync perform a trial run that doesn't make any changes (and
 produces mostly the same output as a real run).  It is most commonly used