1: This patch adds the --sparse-block option. Andrea Righi writes:
2:
3: In some filesystems, typically optimized for large I/O throughputs (like
4: IBM GPFS, IBM SAN FS, or distributed filesystems in general) a lot of
5: lseek() operations can strongly impact on performances. In this cases it
6: can be helpful to enlarge the block size used to handle sparse files
7: directly from a command line parameter.
8:
9: For example, using a sparse write size of 32KB, I've been able to
10: increase the transfer rate of an order of magnitude copying the output
11: files of scientific applications from GPFS to GPFS or GPFS to SAN FS.
12:
13: -Andrea
14:
15: To use this patch, run these commands for a successful build:
16:
17: patch -p1 <patches/sparse-block.diff
18: ./configure (optional if already run)
19: make
20:
21: based-on: e94bad1c156fc3910f24e2b3b71a81b0b0bdeb70
22: diff --git a/fileio.c b/fileio.c
23: --- a/fileio.c
24: +++ b/fileio.c
25: @@ -34,6 +34,7 @@
26: #define ALIGNED_LENGTH(len) ((((len) - 1) | (ALIGN_BOUNDARY-1)) + 1)
27:
28: extern int sparse_files;
29: +extern int sparse_files_block_size;
30:
31: OFF_T preallocated_len = 0;
32:
33: @@ -149,7 +150,7 @@ int write_file(int f, int use_seek, OFF_T offset, const char *buf, int len)
34: while (len > 0) {
35: int r1;
36: if (sparse_files > 0) {
37: - int len1 = MIN(len, SPARSE_WRITE_SIZE);
38: + int len1 = MIN(len, sparse_files_block_size);
39: r1 = write_sparse(f, use_seek, offset, buf, len1);
40: offset += r1;
41: } else {
42: diff --git a/options.c b/options.c
43: --- a/options.c
44: +++ b/options.c
45: @@ -76,6 +76,7 @@ int remove_source_files = 0;
46: int one_file_system = 0;
47: int protocol_version = PROTOCOL_VERSION;
48: int sparse_files = 0;
49: +long sparse_files_block_size = SPARSE_WRITE_SIZE;
50: int preallocate_files = 0;
51: int do_compression = 0;
52: int do_compression_level = CLVL_NOT_SPECIFIED;
53: @@ -692,6 +693,7 @@ static struct poptOption long_options[] = {
54: {"sparse", 'S', POPT_ARG_VAL, &sparse_files, 1, 0, 0 },
55: {"no-sparse", 0, POPT_ARG_VAL, &sparse_files, 0, 0, 0 },
56: {"no-S", 0, POPT_ARG_VAL, &sparse_files, 0, 0, 0 },
57: + {"sparse-block", 0, POPT_ARG_LONG, &sparse_files_block_size, 0, 0, 0 },
58: {"preallocate", 0, POPT_ARG_NONE, &preallocate_files, 0, 0, 0},
59: {"inplace", 0, POPT_ARG_VAL, &inplace, 1, 0, 0 },
60: {"no-inplace", 0, POPT_ARG_VAL, &inplace, 0, 0, 0 },
61: @@ -2684,6 +2686,12 @@ void server_options(char **args, int *argc_p)
62: args[ac++] = arg;
63: }
64:
65: + if (sparse_files_block_size) {
66: + if (asprintf(&arg, "--sparse-block=%lu", sparse_files_block_size) < 0)
67: + goto oom;
68: + args[ac++] = arg;
69: + }
70: +
71: if (io_timeout) {
72: if (asprintf(&arg, "--timeout=%d", io_timeout) < 0)
73: goto oom;
74: diff --git a/rsync.1.md b/rsync.1.md
75: --- a/rsync.1.md
76: +++ b/rsync.1.md
77: @@ -379,6 +379,7 @@ detailed description below for a complete description.
78: --super receiver attempts super-user activities
79: --fake-super store/recover privileged attrs using xattrs
80: --sparse, -S turn sequences of nulls into sparse blocks
81: +--sparse-block=SIZE set block size used to handle sparse files
82: --preallocate allocate dest files before writing them
83: --write-devices write to devices as files (implies --inplace)
84: --dry-run, -n perform a trial run with no changes made
85: @@ -1477,6 +1478,18 @@ your home directory (remove the '=' for that).
86: opposed to allocated sequences of null bytes) if the kernel version and
87: filesystem type support creating holes in the allocated data.
88:
89: +0. `--sparse-block=SIZE`
90: +
91: + Change the block size used to handle sparse files to SIZE bytes. This
92: + option only has an effect if the `--sparse` (`-S`) option was also
93: + specified. The default block size used by rsync to detect a file hole is
94: + 1024 bytes; when the receiver writes data to the destination file and
95: + option `--sparse` is used, rsync checks every 1024-bytes chunk to detect if
96: + they are actually filled with data or not. With certain filesystems,
97: + optimized to receive data streams for example, enlarging this block size
98: + can strongly increase performance. The option can be used to tune this
99: + block size.
100: +
101: 0. `--dry-run`, `-n`
102:
103: This makes rsync perform a trial run that doesn't make any changes (and
104: diff -Nurp a/rsync.1 b/rsync.1
105: --- a/rsync.1
106: +++ b/rsync.1
107: @@ -455,6 +455,7 @@ detailed description below for a complet
108: --super receiver attempts super-user activities
109: --fake-super store/recover privileged attrs using xattrs
110: --sparse, -S turn sequences of nulls into sparse blocks
111: +--sparse-block=SIZE set block size used to handle sparse files
112: --preallocate allocate dest files before writing them
113: --write-devices write to devices as files (implies --inplace)
114: --dry-run, -n perform a trial run with no changes made
115: @@ -1541,6 +1542,16 @@ NTFS, etc.), this option may have no pos
116: If combined with \fB\-\-sparse\fP, the file will only have sparse blocks (as
117: opposed to allocated sequences of null bytes) if the kernel version and
118: filesystem type support creating holes in the allocated data.
119: +.IP "\fB\-\-sparse-block=SIZE\fP"
120: +Change the block size used to handle sparse files to SIZE bytes. This
121: +option only has an effect if the \fB\-\-sparse\fP (\fB\-S\fP) option was also
122: +specified. The default block size used by rsync to detect a file hole is
123: +1024 bytes; when the receiver writes data to the destination file and
124: +option \fB\-\-sparse\fP is used, rsync checks every 1024-bytes chunk to detect if
125: +they are actually filled with data or not. With certain filesystems,
126: +optimized to receive data streams for example, enlarging this block size
127: +can strongly increase performance. The option can be used to tune this
128: +block size.
129: .IP "\fB\-\-dry-run\fP, \fB\-n\fP"
130: This makes rsync perform a trial run that doesn't make any changes (and
131: produces mostly the same output as a real run). It is most commonly used
132: diff -Nurp a/rsync.1.html b/rsync.1.html
133: --- a/rsync.1.html
134: +++ b/rsync.1.html
135: @@ -370,6 +370,7 @@ detailed description below for a complet
136: --super receiver attempts super-user activities
137: --fake-super store/recover privileged attrs using xattrs
138: --sparse, -S turn sequences of nulls into sparse blocks
139: +--sparse-block=SIZE set block size used to handle sparse files
140: --preallocate allocate dest files before writing them
141: --write-devices write to devices as files (implies --inplace)
142: --dry-run, -n perform a trial run with no changes made
143: @@ -1420,6 +1421,18 @@ opposed to allocated sequences of null b
144: filesystem type support creating holes in the allocated data.</p>
145: </dd>
146:
147: +<dt><code>--sparse-block=SIZE</code></dt><dd>
148: +<p>Change the block size used to handle sparse files to SIZE bytes. This
149: +option only has an effect if the <code>--sparse</code> (<code>-S</code>) option was also
150: +specified. The default block size used by rsync to detect a file hole is
151: +1024 bytes; when the receiver writes data to the destination file and
152: +option <code>--sparse</code> is used, rsync checks every 1024-bytes chunk to detect if
153: +they are actually filled with data or not. With certain filesystems,
154: +optimized to receive data streams for example, enlarging this block size
155: +can strongly increase performance. The option can be used to tune this
156: +block size.</p>
157: +</dd>
158: +
159: <dt><code>--dry-run</code>, <code>-n</code></dt><dd>
160: <p>This makes rsync perform a trial run that doesn't make any changes (and
161: produces mostly the same output as a real run). It is most commonly used
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to sparse-block.diff CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / rsync / patches