The FAT sector that was written 256 times

On a FAT32 volume mounted -o sync, asking Linux to reserve 16 MiB for each of sixteen files used to take twelve and a half minutes. With three small patches merged for Linux 7.4, the same request takes three and a half seconds. The disk did not get faster, and no instruction got cheaper. The kernel just stopped writing the same 512-byte sector to the disk 256 times in a row.

That is the whole story, and it is worth telling slowly, because the patch series is a near-perfect specimen of a kind of optimisation that turns up everywhere: in database drivers, in network clients, in your own code. A fixed cost is paid once per item when it could be paid once per batch. The interesting parts are not the four lines that make the headline. They are how you would find a cost like this from a benchmark table, why the obvious fix was impossible until something else changed, and how the change was cut into pieces small enough to review.

What you need to follow this

You should be comfortable reading C and know roughly what a system call is. You do not need to have read kernel code before. Every kernel term is defined before it is used, and the four short theory sections below cover the background. If you already know how FAT works, what FALLOC_FL_KEEP_SIZE means and what a buffer head is, skip to The old loop.

The series is by Matheus Alves de Almeida. It was acked by OGAWA Hirofumi, who maintains the FAT driver, and committed by Andrew Morton. Linus Torvalds merged it on 27 October 2026, after the 7.3 release, so 7.4 will be the first release to carry it. All the code quoted here comes from fs/fat/ in Linus’s tree, either side of those commits. All the benchmark numbers come from the author’s commit message; I explain at the end what I could and could not check myself.

Theory: clusters, and the table that chains them

A FAT volume divides its data area into clusters, which are fixed-size allocation units. The cluster size is chosen when the volume is formatted. It is always a whole number of sectors, from one 512-byte sector up to tens of kilobytes. A file always occupies whole clusters, and its clusters do not have to be next to each other on the disk.

So something has to record which clusters belong to which file, and in what order. That is the File Allocation Table, the FAT that gives the filesystem its name. It is an array with one entry per cluster. On FAT32 each entry is four bytes, and its value means one of three things:

• 0: this cluster is free;
• a cluster number: this cluster belongs to a file, and the file continues in that cluster;
• an end-of-chain marker (0x0FFFFFFF, as Linux writes it): this is the file’s last cluster.

A file’s directory entry stores the number of its first cluster. Everything after that is a linked list written into the table: entry 5 says “next is 6”, entry 6 says “next is 7”, and so on until the end-of-chain marker.

Two facts about this layout drive everything that follows.

The first is the arithmetic of sectors. Disks read and write whole sectors, not bytes. A 512-byte sector of the table holds 512 / 4 = 128 entries. If a file’s clusters are allocated in order, which they usually are on a fresh volume, its chain runs along consecutive entries. So the next 128 clusters of a growing file all have their entries in the same sector of the table.

The second is that growing a file by one cluster means touching two entries: the new cluster’s entry becomes end-of-chain, and the old last cluster’s entry stops being end-of-chain and points at the new one. Hold on to that “two”.

Theory: two copies, and a hint about free space

FAT volumes normally carry two copies of the table, one after the other. The second copy is a mirror: whenever the driver changes a sector of the first copy, it copies the whole sector into the second. In Linux this is fat_mirror_bhs() in fs/fat/fatent.c. The mirror exists so that a disk error in one copy does not lose the chain of every file. It also means that every change to the table is written twice.

FAT32 also keeps a small FSInfo sector with two hints: how many clusters are free, and where the last allocation happened. Linux keeps these in memory as sbi->free_clusters and sbi->prev_free. The allocator starts searching just after prev_free, which is why a growing file tends to get consecutive clusters.

Theory: fallocate and FALLOC_FL_KEEP_SIZE

fallocate(fd, mode, offset, len) asks the filesystem to reserve disk space for a range of a file without writing any data there. A video recorder that is about to write a long clip, a download manager that knows the final size, or a database growing its log can reserve space up front. The file then ends up in fewer pieces, and the program learns about a full disk before it is halfway through, not after.

There are several modes. The one that matters here is FALLOC_FL_KEEP_SIZE: reserve the space, but do not change the file’s length as stat reports it. The clusters are allocated and chained, and they sit beyond the end of the file, ready for the writes that will extend it.

The FAT driver supports exactly two modes. Without flags, it extends the file and zeroes the new space through an ordinary expanding truncate. With FALLOC_FL_KEEP_SIZE, it allocates clusters without zeroing them. Everything else returns EOPNOTSUPP. The patch series changes only the second path:

/* fs/fat/file.c, fat_fallocate(), before the series */
if (mode & FALLOC_FL_KEEP_SIZE) {
	ondisksize = inode->i_blocks << 9;
	if ((offset + len) <= ondisksize)
		goto error;

	/* First compute the number of clusters to be allocated */
	mm_bytes = offset + len - ondisksize;
	nr_cluster = (mm_bytes + (sbi->cluster_size - 1)) >>
		sbi->cluster_bits;

	/* Start the allocation.We are not zeroing out the clusters */
	while (nr_cluster-- > 0) {
		err = fat_add_cluster(inode);
		if (err)
			goto error;
	}
}

Read the loop at the bottom. To reserve nr_cluster clusters, it calls a function that adds one cluster, nr_cluster times. A 16 MiB reservation with 512-byte clusters is 32,768 calls.

Theory: buffer heads, and what -o sync costs

The kernel does not write metadata straight to the disk. It reads a disk block into memory, changes it there, and marks it dirty, meaning it differs from what is on disk. Writeback threads flush dirty blocks some time later. The kernel’s handle for one cached disk block is a struct buffer_head, or “bh”. The FAT driver works on its table in units of buffer heads, one per sector.

Mounting with -o sync changes the contract. Every metadata change must reach the disk before the system call returns. In the FAT driver, that means calls like fat_sync_bhs(), which submits the dirty buffers and then waits for each one:

/* fs/fat/misc.c */
int fat_sync_bhs(struct buffer_head **bhs, int nr_bhs)
{
	int i, err = 0;

	for (i = 0; i < nr_bhs; i++)
		write_dirty_buffer(bhs[i], 0);

	for (i = 0; i < nr_bhs; i++) {
		wait_on_buffer(bhs[i]);
		if (!err && buffer_write_io_error(bhs[i]))
			err = -EIO;
	}
	return err;
}

On a sync mount, the mirror copy is written the same way, but one buffer at a time: fat_mirror_bhs() calls sync_dirty_buffer() on each copied sector when the superblock has SB_SYNCHRONOUS set.

A synchronous write costs a round trip to the device: submit, wait for the device to say it is durable, continue. Whatever else the code does, a loop that does one of those per item will be paced by that round trip.

Removable media such as SD cards and USB sticks are the usual place to meet both FAT and sync mounts. Some desktop setups and embedded systems mount removable media synchronously so that pulling the card out loses less.

The old loop

Now trace one call of the loop, fat_add_cluster(inode), on a sync mount. Before the series it was two steps:

/* fs/fat/inode.c, before the series */
int fat_add_cluster(struct inode *inode)
{
	int err, cluster;

	err = fat_alloc_clusters(inode, &cluster, 1);
	if (err)
		return err;
	/* FIXME: this cluster should be added after data of this
	 * cluster is writed */
	err = fat_chain_add(inode, cluster, 1);
	if (err)
		fat_free_clusters(inode, cluster);
	return err;
}

Step one, fat_alloc_clusters(inode, &cluster, 1), finds a free entry and marks it end-of-chain. When it finishes, it writes out the sector it changed: fat_sync_bhs() on the first copy, because inode_needs_sync() is true on a sync mount, and then fat_mirror_bhs(), which copies the sector into the second table and syncs that too. That is two synchronous sector writes.

Step two, fat_chain_add(inode, cluster, 1), links the new cluster onto the file. It finds the file’s current last cluster and rewrites that entry to point at the new one, through fat_ent_write(..., wait) with wait = inode_needs_sync(inode). That function syncs the sector and then mirrors it, synchronously. That is two more synchronous sector writes. (A file’s very first cluster has no previous entry to link, so it skips this step and records the start cluster in the inode instead.)

Four synchronous writes per cluster. Now put that next to the arithmetic of sectors. The new cluster’s entry and the previous cluster’s entry are almost always in the same 512-byte sector, and the next 128 clusters all land in that sector too. So while a file grows through one sector’s worth of clusters, that sector is written to the first table about 256 times (128 end-of-chain marks plus 128 links) and to the second table another 256 times. The contents change by four bytes each time.

Step through it and watch the counters. In “one call per cluster” mode, each square costs four writes. In “one call for all” mode, which is what the patched code does, nothing goes to disk until the call ends, and then the sector goes out once to each table.

The figure starts a fresh file, so its first cluster costs two writes, not four, and the first table’s counter stops at 255. In the middle of a long file, the first cluster of each new sector also rewrites the last entry of the previous sector, and the count per sector is exactly 256.

Reading the benchmark like an engineer

The commit message for the last patch carries a table of twelve benchmarks. Each one opens count files on a fresh FAT32 volume and times this loop:

start = now_ns();
for (unsigned int i = 0; i < count; i++)
     fallocate(fds[i], FALLOC_FL_KEEP_SIZE, 0, bytes);
elapsed = now_ns() - start;

The volume ran under QEMU/KVM on a virtio-blk disk with cache=none, which stops the host’s page cache from absorbing writes, so a synchronous write really waits for the host’s disk. Each figure is the mean of 10 runs, except one row with 3. Here is the table as published, regrouped:

The summaries going around picked out “265×”. That is a real row, but it is the best one, and the rows around it vary in three things at once: cluster size, file count and file length. A table like this is hard to read as it stands. The first thing to do with it is to stop looking at improvement factors.

Step one: divide by the unit of work

Every row does a different amount of work, so the raw times cannot be compared. The question is what the unit of work is. The code answers it: the loop runs once per cluster. So divide each time by the number of clusters allocated, which is files × length ÷ cluster size. This short script does exactly that, using only numbers from the commit message:

# per_cluster.py: normalise the benchmark table by clusters allocated
rows = [
    # cluster bytes, files, bytes per file, sync?, before s, after s
    (512,  16,  256 << 20, False, 2.328,     0.175),
    (512,  16,   16 << 20, False, 0.149659,  0.014036),
    (512,  16,   16 << 20, True,  747.993,   3.445),
    (512,  16,    1 << 20, True,  46.63,     0.333),
    (512,  128,   1 << 20, False, 0.090873,  0.004686),
    (512,  128,  16 << 10, False, 0.001287,  0.000241),
    (512,  128, 128 << 10, True,  47.284,    0.350),
    (512,  128,   4 << 10, True,  1.98,      0.144),
    (4096, 16,  256 << 20, False, 0.295392,  0.024733),
    (4096, 16,   16 << 20, False, 0.020509,  0.001404),
    (4096, 16,   16 << 20, True,  92.489,    0.348),
    (4096, 16,    1 << 20, True,  5.837,     0.048),
]

def size(n):
    return f"{n >> 20} MiB" if n >= 1 << 20 else f"{n >> 10} KiB"

def per(t, n):
    s = t / n
    if s >= 1e-3: return f"{s*1e3:.3f} ms"
    if s >= 1e-6: return f"{s*1e6:.2f} us"
    return f"{s*1e9:.0f} ns"

print(f"{'clus':>5} {'files':>5} {'length':>8} {'sync':>4} {'clusters':>10}"
      f" {'before/clus':>12} {'after/clus':>11}")
for cb, files, length, sync, before, after in rows:
    n = files * length // cb
    print(f"{cb:>5} {files:>5} {size(length):>8} {'yes' if sync else 'no':>4}"
          f" {n:>10,} {per(before, n):>12} {per(after, n):>11}")

$ python3 per_cluster.py
 clus files   length sync   clusters  before/clus  after/clus
  512    16  256 MiB   no  8,388,608       278 ns       21 ns
  512    16   16 MiB   no    524,288       285 ns       27 ns
  512    16   16 MiB  yes    524,288     1.427 ms     6.57 us
  512    16    1 MiB  yes     32,768     1.423 ms    10.16 us
  512   128    1 MiB   no    262,144       347 ns       18 ns
  512   128   16 KiB   no      4,096       314 ns       59 ns
  512   128  128 KiB  yes     32,768     1.443 ms    10.68 us
  512   128    4 KiB  yes      1,024     1.934 ms   140.62 us
 4096    16  256 MiB   no  1,048,576       282 ns       24 ns
 4096    16   16 MiB   no     65,536       313 ns       21 ns
 4096    16   16 MiB  yes     65,536     1.411 ms     5.31 us
 4096    16    1 MiB  yes      4,096     1.425 ms    11.72 us

Look at the “before” column. On a sync mount, five of the six rows cost between 1.411 and 1.443 ms per cluster. That holds for 512-byte and 4 KiB clusters, for 16 files and 128, for 1 MiB and 16 MiB. On a default mount, every row costs between 278 and 347 ns per cluster. The total times span more than six orders of magnitude, and once you divide by clusters, the old code is two flat lines.

A flat line is a diagnosis. If the time per cluster does not depend on the cluster’s size, the cost is not in moving bytes. It is a fixed toll paid once per allocation call. When the toll is fixed and paid per item, making each payment a little cheaper helps a little. Paying it fewer times is what helps a lot.

Step two: fit the toll

You can make “fixed toll per cluster” precise. Suppose each run costs $a$ per cluster plus $b$ per fallocate call:

Fitting that by least squares to the six sync-mount rows, before and after the patch, takes a few more lines appended to the same script:

# Least squares for T = a*clusters + b*calls (one fallocate call per file).
def fit(points):
    nn = sum(n * n for n, c, t in points); nc = sum(n * c for n, c, t in points)
    cc = sum(c * c for n, c, t in points)
    nt = sum(n * t for n, c, t in points); ct = sum(c * t for n, c, t in points)
    det = nn * cc - nc * nc
    return (nt * cc - ct * nc) / det, (ct * nn - nt * nc) / det

for label, col in (("before", 4), ("after", 5)):
    points = [(r[1] * r[2] // r[0], r[1], r[col]) for r in rows if r[3]]
    a, b = fit(points)
    print(f"\nsync {label}: {a*1e6:.2f} us per cluster + {b*1e3:.3f} ms per call")
    for n, c, t in points:
        print(f"  {n:>8,} clusters {c:>4} calls"
              f"  measured {t:8.3f} s  model {a*n + b*c:8.3f} s")

Run with those lines added, the script prints the table above and then:

sync before: 1426.31 us per cluster + 3.609 ms per call
   524,288 clusters   16 calls  measured  747.993 s  model  747.857 s
    32,768 clusters   16 calls  measured   46.630 s  model   46.795 s
    32,768 clusters  128 calls  measured   47.284 s  model   47.199 s
     1,024 clusters  128 calls  measured    1.980 s  model    1.923 s
    65,536 clusters   16 calls  measured   92.489 s  model   93.533 s
     4,096 clusters   16 calls  measured    5.837 s  model    5.900 s

sync after: 6.53 us per cluster + 1.075 ms per call
   524,288 clusters   16 calls  measured    3.445 s  model    3.439 s
    32,768 clusters   16 calls  measured    0.333 s  model    0.231 s
    32,768 clusters  128 calls  measured    0.350 s  model    0.352 s
     1,024 clusters  128 calls  measured    0.144 s  model    0.144 s
    65,536 clusters   16 calls  measured    0.348 s  model    0.445 s
     4,096 clusters   16 calls  measured    0.048 s  model    0.044 s

Look at the “before” half first. Two parameters predict six measurements, ranging from two seconds to twelve minutes, each within 3%. The per-call term also explains the odd row out, the 4 KiB files: with only 8 clusters per file, the 3.6 ms per call is a large part of each file’s cost, so its per-cluster figure is higher.

Now connect the fit to the code. The trace above found four synchronous sector writes per cluster. If those writes are the toll, each one costs about 1.43 ms ÷ 4 ≈ 0.36 ms. That is a plausible round trip for a durable write from a VM to a host disk with caching off. That last step is inference: nobody timed an individual write. But it is consistent, and it tells you what to predict for the fix.

Step three: predict the fix before you read its numbers

If writes are the toll, the speedup should be roughly the ratio of write counts. The old code does 4 writes per cluster. Code that writes each table sector once per copy does 2 writes per 128 clusters. The ratio is

The two long sync rows, where per-call overhead is spread thinnest, measured 217× and 265×. A back-of-the-envelope model that ignores everything except counting writes lands between them. That agreement is the best evidence available here that write count really is the mechanism.

The “after” column is not as tidy. The same two-term fit misses some after-patch rows by about 30%, because the new code’s cost has a different structure. It scales with table sectors and flush batches, not clusters, and six rows cannot pin that down. What the column does show clearly is a floor of roughly a millisecond per fallocate call on a sync mount. That floor is why the gain shrinks as files get smaller: 217× for 16 MiB files, 13.7× for 4 KiB ones. Batching only helps as much as there is to batch. A call that allocates 8 clusters has little toll to share out.

The default-mount rows improve about 13×, from roughly 280 ns to 22 ns per cluster. No disk waits are involved there, so this is CPU work saved per call: taking the FAT lock, searching from the free-space hint, walking to the end of the file’s chain, dirtying and mirroring buffers. I have not profiled it, so I will not divide the 280 ns among those.

Why it wasn’t a four-line change

If the fix is “allocate them all at once”, why was the loop there at all? Because the allocator could not do it. Before the series, fat_alloc_clusters() began with this:

/* fs/fat/fatent.c, fat_alloc_clusters(), before the series */
struct buffer_head *bhs[MAX_BUF_PER_PAGE];
int i, count, err, nr_bhs, idx_clus;

BUG_ON(nr_cluster > (MAX_BUF_PER_PAGE / 2));	/* fixed limit */

BUG_ON crashes the kernel if its condition is true. MAX_BUF_PER_PAGE is PAGE_SIZE / 512, which is 8 on the usual 4 KiB pages. The allocator collects every table sector it changes into the fixed array bhs[], so that it can sync and mirror them all at the end. An entry can straddle two sectors (FAT12’s entries are a byte and a half wide), so each cluster may need two slots, hence the division by 2. The allocator could hand out at most four clusters per call. That was fine for every caller it had: fat_add_cluster() asked for one, and the directory code asks for at most two. A fallocate that wanted 32,768 had no way to say so.

So the series has three patches, and only the last one is the headline.

Patch 1: bound the batch by memory, not by the request

fat: remove cluster allocation size limit keeps the fixed 8-slot array. When the array is about to overflow, it flushes what it has collected and keeps going:

/* fs/fat/fatent.c, inside the allocation loop, after patch 1 */
if (nr_bhs + fatent.nr_bhs > MAX_BUF_PER_PAGE) {
	err = fat_flush_bhs(sb, bhs, &nr_bhs,
			    inode_needs_sync(inode));
	if (err)
		goto out;
}
fat_collect_bhs(bhs, &nr_bhs, &fatent);

fat_flush_bhs() is new, but the logic in it is not. fat_free_clusters(), the function that frees a chain, had always done exactly this, because a chain being freed can be any length. The patch lifts that code into a helper and calls it from both places:

static int fat_flush_bhs(struct super_block *sb, struct buffer_head **bhs,
			 int *nr_bhs, bool sync)
{
	int err, i;

	if (sync) {
		err = fat_sync_bhs(bhs, *nr_bhs);
		if (err)
			return err;
	}
	err = fat_mirror_bhs(sb, bhs, *nr_bhs);
	if (err)
		return err;
	for (i = 0; i < *nr_bhs; i++)
		brelse(bhs[i]);
	*nr_bhs = 0;

	return 0;
}

This is the general pattern of bounded batching. The batch size is set by the memory you are willing to hold, not by how much the caller asked for. You flush when it is full, flush the rest at the end, and any request size works. Since fat_collect_bhs() skips a sector that is already in the array, and 128 consecutive entries share one sector, eight slots cover about a thousand clusters between flushes.

Notice what patch 1 does not change: every caller still asks for one or two clusters, so the flush inside the loop never fires for them. The patch is behaviour-preserving on its own. It removes a limit nobody was hitting yet.

Patch 2: an interface that does not need the whole answer

The next obstacle is the shape of fat_alloc_clusters(). It returns the allocated cluster numbers in an array the caller provides, int *cluster, one slot per cluster. That suits the directory code, which needs every number. A 16 MiB reservation with 512-byte clusters would need 32,768 of them, 128 KiB. Kernel stacks are 16 KiB on x86-64, so that array would have to be allocated separately just to be thrown away. The list is already in the table, because the allocator chains each new cluster to the previous one as it goes. The caller only needs the head.

fat: support adding multiple clusters gives the allocator a flag and two front doors:

static int __fat_alloc_clusters(struct inode *inode, int *cluster, int nr_cluster,
				bool return_all);

int fat_alloc_clusters(struct inode *inode, int *cluster, int nr_cluster)
{
	return __fat_alloc_clusters(inode, cluster, nr_cluster, true);
}

int fat_alloc_chain(struct inode *inode, int *cluster, int nr_cluster)
{
	return __fat_alloc_clusters(inode, cluster, nr_cluster, false);
}

and inside the loop records only what was asked for:

if (return_all)
	cluster[idx_clus] = entry;
else if (!idx_clus)
	*cluster = entry;

fat_add_cluster(inode) becomes fat_add_clusters(inode, nr_cluster). It allocates a chain, then links it onto the file with one fat_chain_add(). That function already took a count, because the directory code links up to two clusters at a time, so it needed no change. Every existing caller of the old function is updated to pass 1. Again, nothing behaves differently yet.

Patch 3: the four lines

Only now is fat: allocate multiple clusters in fallocate a four-line change:

-		/* Start the allocation.We are not zeroing out the clusters */
-		while (nr_cluster-- > 0) {
-			err = fat_add_clusters(inode, 1);
-			if (err)
-				goto error;
-		}
+		/* Allocate the clusters. We are not zeroing them out */
+		err = fat_add_clusters(inode, nr_cluster);
+		if (err)
+			goto error;

How the series is cut

This is the part worth copying. The first two patches each change capability without changing behaviour, so a reviewer can check each one by asking a single question: is this a faithful refactor? The last patch changes behaviour and is small enough to read in one breath. If the speedup had uncovered a bug, git bisect would land on a four-line commit, not a hundred-line one. The whole series adds 57 lines and removes 26 across four files, and none of it is clever.

What did not change

Being precise about scope matters as much as the speedup.

• Ordinary writes are untouched. When a program simply write()s past the end of a file, the driver allocates clusters in __fat_get_block(), and that still calls fat_add_clusters(inode, 1). The comment above it, “TODO: multiple cluster allocation would be desirable.”, is still there. If your recorder writes without preallocating, this series does nothing for it.
• Plain fallocate is untouched. Without FALLOC_FL_KEEP_SIZE, FAT goes through fat_cont_expand(), which zeroes the new space, a different path.
• Reading is untouched. Nothing here changes how fast data moves, only how fast space is reserved.

So who benefits? Programs that call fallocate with FALLOC_FL_KEEP_SIZE on FAT, and the more so on sync mounts. To find out whether yours does, ask it rather than guessing:

$ strace -f -e trace=fallocate your-recorder ...

What batching costs

Batching is rarely free, and two consequences are visible in the code. I found both by reading the source, not by testing them.

The FAT lock is held across disk waits. The allocator takes lock_fat(), a per-filesystem mutex, before it searches. Before the series, the one-cluster call released it before syncing: the out: label calls unlock_fat() first, then fat_sync_bhs() and fat_mirror_bhs(). After patch 1, the flush inside the loop syncs while the lock is held. On a sync mount, a large reservation now holds the lock through every flush, and any other allocation on that volume waits for the whole call. Before, it could slip in between clusters. That trade suits a camera writing one file. It might matter on a volume with several writers, and nobody has measured that yet as far as I know.

Failure is now all-or-nothing. Before, if the volume ran out of space halfway through, the loop stopped with the clusters it had already chained still attached to the file, beyond its end, because FALLOC_FL_KEEP_SIZE had been asked for. After, __fat_alloc_clusters() first checks the free-cluster count. If the search still fails partway, it frees the partial chain with fat_free_clusters() before returning -ENOSPC. For a caller this is the better behaviour, but it is a change.

The pattern, beyond FAT

Strip away the filesystem and the shape is this: a fixed cost per call, a loop of calls, and an interface that only takes one item. You have seen it before:

• the ORM that issues one query per row (the “N+1 query” problem);
• the logger that calls write() once per line on an unbuffered file descriptor;
• the importer that commits after every INSERT, paying an fsync per row;
• the HTTP client that fetches 500 resources over 500 fresh connections.

The method that finds them is the method above, and it does not need kernel knowledge:

1. Normalise by the unit of work. Divide time by items, not bytes, and see whether the line is flat. Flat means a toll.
2. Find the toll in the code. Look for what happens once per call: a lock, a system call, a durable write, a network round trip. Count them.
3. Predict, then compare. The ratio of tolls before and after is your expected speedup. If the measurement disagrees, your model is missing something, so go and find it before trusting either.
4. Change the interface to take a count. Faster single-item calls are a dead end when the caller has a thousand items in hand.
5. Bound the batch by memory, flush when full. Never let the request size set the buffer size.
6. Cut the change so only the last step changes behaviour. Each earlier step should be a refactor a reviewer can verify on its own.
7. Measure in the configuration where the toll is real. cache=none and -o sync are what made the effect visible. With the host cache absorbing writes, it would have looked like a 13× CPU improvement.

Try it yourself

This program does what the commit message’s benchmark does. It creates COUNT files in a directory, times one FALLOC_FL_KEEP_SIZE reservation per file, and reports the time per cluster. It reads the cluster size from fstatvfs(), which FAT fills with the cluster size.

/* fatbench.c: time fallocate(FALLOC_FL_KEEP_SIZE) over COUNT files. */
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/statvfs.h>
#include <time.h>
#include <unistd.h>

static double now(void)
{
	struct timespec ts;

	clock_gettime(CLOCK_MONOTONIC, &ts);
	return ts.tv_sec + ts.tv_nsec / 1e9;
}

int main(int argc, char **argv)
{
	if (argc != 4) {
		fprintf(stderr, "usage: %s DIR COUNT BYTES\n", argv[0]);
		return 2;
	}
	unsigned count = strtoul(argv[2], NULL, 0);
	long long bytes = strtoll(argv[3], NULL, 0);
	int *fds = calloc(count, sizeof(*fds));
	char path[4096];

	for (unsigned i = 0; i < count; i++) {
		snprintf(path, sizeof(path), "%s/f%u", argv[1], i);
		unlink(path);	/* start from an empty file every run */
		fds[i] = open(path, O_CREAT | O_WRONLY, 0644);
		if (fds[i] < 0) {
			perror(path);
			return 1;
		}
	}

	struct statvfs sv;
	if (fstatvfs(fds[0], &sv)) {
		perror("fstatvfs");
		return 1;
	}

	double start = now();
	for (unsigned i = 0; i < count; i++) {
		if (fallocate(fds[i], FALLOC_FL_KEEP_SIZE, 0, bytes)) {
			perror("fallocate");
			return 1;
		}
	}
	double elapsed = now() - start;

	unsigned long long clusters =
		(unsigned long long)count * ((bytes + sv.f_bsize - 1) / sv.f_bsize);
	printf("%u x %lld bytes, %lu-byte clusters: %.3f s, %llu clusters, %.2f us/cluster\n",
	       count, bytes, sv.f_bsize, elapsed, clusters, elapsed / clusters * 1e6);

	for (unsigned i = 0; i < count; i++)
		close(fds[i]);
	free(fds);
	return 0;
}

To run it against a FAT32 image with 512-byte clusters, mounted synchronously, use a loop device with direct I/O so that the host’s page cache does not hide the writes. That is the closest local equivalent to the patch author’s cache=none:

truncate -s 2G fat.img
mkfs.fat -F 32 -S 512 -s 1 fat.img
dev=$(sudo losetup --find --show --direct-io=on fat.img)
sudo mkdir -p /mnt/fat
sudo mount -o sync,uid=$(id -u),gid=$(id -g) "$dev" /mnt/fat
gcc -O2 -Wall -o fatbench fatbench.c
./fatbench /mnt/fat 16 $((1 << 20))

Start with 1 MiB files. On a kernel without the series and a disk like the author’s, the model above predicts about 47 seconds for that line, and the 16 MiB version runs for over twelve minutes. Run it on a 7.3 kernel and on 7.4 and compare. The absolute numbers depend entirely on your device. The shape should not: a flat time per cluster before, and a large drop after.

To count the writes as well as time them, read the device’s I/O counters either side of a run. The fifth field of /sys/block/<dev>/stat is the number of write requests completed:

stat=/sys/block/${dev#/dev/}/stat
w0=$(awk '{print $5}' $stat)
./fatbench /mnt/fat 16 $((1 << 20))
w1=$(awk '{print $5}' $stat)
echo "$((w1 - w0)) writes"

What it printed here

I ran it on Ubuntu’s 7.0.0-42 kernel, which is before the series. The image was a file on ext4 on a Samsung NVMe SSD, attached as a loop device with direct I/O and mounted -o sync. No VM was involved, so these figures are not comparable with the author’s, only with each other:

$ ./fatbench /mnt/fat 128 $((4 << 10))
128 x 4096 bytes, 512-byte clusters: 0.600 s, 1024 clusters, 586.33 us/cluster
$ ./fatbench /mnt/fat 16 $((1 << 20))
16 x 1048576 bytes, 512-byte clusters: 17.580 s, 32768 clusters, 536.50 us/cluster
$ ./fatbench /mnt/fat 128 $((128 << 10))
128 x 131072 bytes, 512-byte clusters: 12.135 s, 32768 clusters, 370.33 us/cluster
$ ./fatbench /mnt/fat 16 $((16 << 20))
16 x 16777216 bytes, 512-byte clusters: 258.620 s, 524288 clusters, 493.28 us/cluster

The 1 MiB line, run four times, took between 14.5 and 18.0 s, so take ±20% as the noise on any one line. Within that noise, the time per cluster is flat across a 512-fold range of work, as it was on the author’s machine.

The write counters settle the mechanism. The 16 × 1 MiB run made 131,184 writes for 32,768 clusters, and the 16 × 16 MiB run made 2,097,780 for 524,288. That is 4.00 writes per cluster both times, the four synchronous sector writes the code trace predicted, plus about a hundred for directory entries. Each write costs this disk about 0.12 ms, a third of what the author’s VM paid, which is why the old code runs at about 0.5 ms per cluster here, not 1.4.

Then I ran the same 1 MiB line on the same sync mount, but with the loop device’s direct I/O switched off. Time per cluster dropped to 14 µs, still flat. The writes were all still issued, but they landed in the host’s page cache. That is the trap in point 7 above, measured: without --direct-io=on you will still see the flat line, but at a thirtieth of its real height.

A note on the evidence

Observed by me. The code. I read fs/fat/fatent.c, inode.c, misc.c, file.c and dir.c in Linus Torvalds’s tree at the merged commits and at their parent. Every code excerpt here is copied from those files or from the patches’ diffs. The commit hashes, authorship, the Acked-by: from the FAT maintainer, Andrew Morton’s commit date (21 October) and Linus’s merge (27 October, after the v7.3 tag of 18 October) all come from the repository’s history. The per_cluster.py output is what the script printed when I extracted its two blocks from this post’s own markdown and ran them together, so the copy you read is the copy that ran.

Measured by someone else. Every timing is from the commit message of 8d40831419c4, on the author’s VM. I have no variance, only means, and one row rests on 3 samples. Twelve rows on one machine are good evidence for the mechanism and weak evidence for the size of the speedup on your hardware.

Measured by me. On a 7.0 kernel only, so the “before” side only: fatbench.c as printed here, compiled with -Wall -Wextra without a warning, and the times and write counts in What it printed here. The four writes per cluster are now a measurement, from the device’s own counters, not just a reading of the code. I have not run a kernel with the series, so the “after” column is still the author’s alone.

Reconstructed. The 0.36 ms per write on the author’s VM is the measured four writes divided into the fitted cost; nobody timed a write there. The 256× prediction ignores flush batching, per-call overhead and the different waiting patterns of the two tables (the first table’s sectors are submitted together and then waited on, the mirror’s one at a time). That it lands between the two measured values is encouraging, not proof. The two trade-offs in What batching costs also come from reading the code. Neither has been tested.

How the setup was run. truncate, mkfs.fat and gcc ran exactly as printed. I attached and mounted the image through udisks (udisksctl loop-setup, then udisksctl mount -o sync) and switched on direct I/O with sudo losetup --direct-io=on, rather than running the losetup --find and mount lines verbatim. The result is the same: a direct-I/O loop device mounted sync. One trap if you go that way: a desktop may automount a new loop device without sync before you get to it, so check findmnt before you trust a number.

Corrected. Early summaries of this patch led with “265×” without saying it was the best of twelve rows, and some compared it with rows that used a different cluster size. The typical sync gain for files of a megabyte or more is 120–265×, and the default-mount gain is 10–20×.