The struct that wanted a 32-byte boundary

A struct with opinions

It looks like a bag of doubles. Four grid constants, a couple of strides, an int flag. Nothing about it says this must begin at an address divisible by 32, and yet that is exactly what it says.

typedef struct {
    double dx2;        /* dx^2 */
    double dy2;        /* dy^2 */
    double inv_dz2;    /* 1/dz^2 (0.0 for 2D) */
    double inv_factor; /* 1 / (2 * (1/dx^2 + 1/dy^2 + inv_dz2)) */
    size_t stride_z;   /* nx*ny for 3D, 0 for 2D */
    size_t k_start;    /* first interior k index */
    size_t k_end;      /* one-past-last interior k index */
    size_t nz;         /* grid points in z */
    __m256d dx2_inv_vec;
    __m256d dy2_inv_vec;
    __m256d dz2_inv_vec;
    __m256d neg_inv_factor_vec;
    int initialized;
} jacobi_avx2_context_t;

This is the per-solver state for an AVX2 Jacobi smoother in a CFD library — a Poisson solver that gets hammered every pressure step, so the constants are computed once at init and pre-broadcast into vector registers rather than re-splatted a few million times inside the loop. Perfectly ordinary optimisation.

It was allocated the perfectly ordinary way:

jacobi_avx2_context_t* ctx =
    (jacobi_avx2_context_t*)cfd_calloc(1, sizeof(jacobi_avx2_context_t));

That line is a bug, and the interesting part is not that it is a bug. It is that the bug had been shipping and passing.

One red square

The CI matrix is green except for one square. Windows, MSVC, AVX2 on: green. Linux, GCC, AVX2 off: green. Linux, GCC, AVX2 on — -DCFD_ENABLE_AVX2=ON — five suites segfaulting:

PoissonJacobiSimdTest
PoissonRedBlackSimdTest
SolverProjectionTest
SolverProjectionSimdTest
LinearSolverTest

Same commit. Same source. One compiler flag between passing and a signal.

The shape of the failure is the first useful thing. It is not “AVX2 is broken,” because the Windows AVX2 build is green. It is not “Linux is broken,” because the non-AVX2 Linux build is green. It is the intersection — and an intersection failure is nearly always a difference in what the two sides assumed rather than what either of them did.

The second useful thing is that a segfault in numerical code is a slightly odd animal. Bad numerics give you NaN, a divergent residual, an assertion. They do not usually give you SIGSEGV. A segfault means an address, and every address in that solver comes from either the grid arrays or the context struct.

What alignment is, in one minute

If you already think in cache lines, skip ahead. Everyone else: this is the whole concept, and you need it to read the rest.

A value is aligned to N bytes when its address is divisible by N. That is the entire definition. 0x7f40 is 32-byte aligned because 0x7f40 % 32 == 0; 0x7f50 is not, because it leaves a remainder of 16.

Hardware cares because memory does not arrive one byte at a time. It moves in fixed-size blocks, and a value that sits entirely inside one block is fetched in one go, while a value straddling a boundary needs two accesses and some glue. So every type carries a natural alignment — usually its own size. A double is 8 bytes and wants 8-byte alignment. An int wants 4.

You have never had to think about this, because for ordinary types the compiler handles it invisibly: it orders and pads struct members so each lands on its natural boundary, and the allocator hands back addresses that work for anything you might reasonably store. That quiet competence is the reason alignment feels like someone else’s problem.

You can ask a type what it wants. C11 gives you _Alignof:

_Alignof(char)    /* 1  */
_Alignof(double)  /* 8  */
_Alignof(__m256d) /* 32 */

That last line is where this post lives. Because there is a category of type whose alignment is stricter than anything the allocator promised to provide — and when you meet one, the quiet competence stops, silently, with no diagnostic and no warning. The rest of this is what that looks like when it happens to you, and who exactly was supposed to be responsible.

The red herring

The obvious suspect is the inner loop, because that is where the vector instructions are:

__m256d p_xp = _mm256_loadu_pd(&p_old[idx + 1]);   /* x+1 */
__m256d p_xm = _mm256_loadu_pd(&p_old[idx - 1]);   /* x-1 */
__m256d p_yp = _mm256_loadu_pd(&p_old[idx + nx]);  /* y+1 */

Stencil access. idx + 1 and idx - 1 are neighbour offsets, so consecutive iterations read at every possible misalignment relative to a 32-byte boundary — one of these pointers is essentially guaranteed to be unaligned. Which is precisely why they are loadu, the unaligned form, and why they have always been correct.

That is the trap. You spend an hour staring at the loop because the loop is where the SIMD is, and the loop is innocent. The vector data was never the problem. The problem is the vector members, and they are touched in a function that contains no loop at all.

What __m256d actually declares

__m256d is not a typedef for “four doubles.” On GCC and Clang it is declared roughly as:

typedef double __m256d __attribute__((__vector_size__(32), __aligned__(32)));

That __aligned__(32) is the whole story. Alignment propagates outward: a struct’s alignment is the maximum alignment of its members, and its size is padded up to a multiple of that. So the innocent-looking bag of doubles above is a 224-byte type with a 32-byte alignment requirement, and its members land here:

sizeof  = 224
alignof = 32

offset dx2_inv_vec        = 64
offset dy2_inv_vec        = 96
offset dz2_inv_vec        = 128
offset neg_inv_factor_vec = 160
offset initialized        = 192

The eight scalars fill bytes 0–63 exactly, so the four vector members sit at 64, 96, 128 and 160 — every one of them a multiple of 32 measured from the start of the struct. Which is the catch. Those offsets are only real addresses if the struct itself starts on a 32-byte boundary. If it starts 16 bytes off, then every one of those members is 16 bytes off, and the type’s central promise is false for the entire object.

What malloc actually promises

Here is the sentence that the bug lives inside. C11 §7.22.3:

The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object with a fundamental alignment requirement.

A fundamental alignment is one less than or equal to _Alignof(max_align_t). On x86-64 glibc, that is 16. Anything stricter — 32, 64 — is an extended alignment, and the standard allocation functions say nothing about it whatsoever.

So cfd_calloc was never obliged to return a 32-byte-aligned pointer. Not on Linux, not on Windows, not on a good day. This is worth being precise about, because it changes what kind of bug this is: nothing regressed, no guarantee broke, no allocator changed its behaviour. The guarantee was never there. The code had been asking for something it had no grounds to expect and getting it often enough not to notice.

That distinction matters for how you go looking for the rest of them. “What changed?” is the wrong question. The right one is “where else am I assuming a promise nobody made?”

Reproducing it in one file

You do not have to take any of that on faith, and you should not. The bug reduces to a single file, and the reduction is more useful than the original because it removes the luck.

The trick is to stop waiting for the allocator to hand you a bad address and just construct one. Round a pointer up to a 32-byte boundary, then deliberately add 16 — which is exactly the freedom §7.22.3 gives calloc, made repeatable. Here is the whole thing; save it as align_demo.c:

/* Reduced from lib/src/solvers/linear/avx2/linear_solver_jacobi_avx2.c
 *
 * Build:  clang -mavx2 -O2 -o align_demo align_demo.c
 * Run:    ./align_demo good     -> exits 0
 *         ./align_demo bad      -> crashes
 */
#include <immintrin.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct {
    double  dx2, dy2, inv_dz2, inv_factor;
    size_t  stride_z, k_start, k_end, nz;
    __m256d dx2_inv_vec;
    __m256d dy2_inv_vec;
    __m256d dz2_inv_vec;
    __m256d neg_inv_factor_vec;
    int     initialized;
} jacobi_ctx_t;

/* The init path, verbatim in shape: compute scalars, broadcast, store. */
static void ctx_init(jacobi_ctx_t* ctx, double dx, double dy, double dz)
{
    ctx->dx2     = dx * dx;
    ctx->dy2     = dy * dy;
    ctx->inv_dz2 = (dz > 0.0) ? 1.0 / (dz * dz) : 0.0;

    double factor   = 2.0 * (1.0 / ctx->dx2 + 1.0 / ctx->dy2 + ctx->inv_dz2);
    ctx->inv_factor = 1.0 / factor;

    ctx->dx2_inv_vec        = _mm256_set1_pd(1.0 / ctx->dx2);
    ctx->dy2_inv_vec        = _mm256_set1_pd(1.0 / ctx->dy2);
    ctx->dz2_inv_vec        = _mm256_set1_pd(ctx->inv_dz2);
    ctx->neg_inv_factor_vec = _mm256_set1_pd(-ctx->inv_factor);
    ctx->initialized        = 1;
}

int main(int argc, char** argv)
{
    int bad = (argc > 1 && strcmp(argv[1], "bad") == 0);

    /* Read the spacings at runtime so nothing is constant-folded --
       in the real solver they come from the grid. */
    double dx = (argc > 2) ? atof(argv[2]) : 0.1;
    double dy = (argc > 3) ? atof(argv[3]) : 0.1;
    double dz = (argc > 4) ? atof(argv[4]) : 0.1;

    printf("sizeof(jacobi_ctx_t)  = %zu\n", sizeof(jacobi_ctx_t));
    printf("_Alignof(jacobi_ctx_t)= %zu\n", _Alignof(jacobi_ctx_t));
    printf("_Alignof(max_align_t) = %zu\n", _Alignof(max_align_t));

    /* Take a 32-byte-aligned address, then optionally break it by 16 --
       exactly the freedom C11 7.22.3 gives calloc, made deterministic. */
    unsigned char* raw = (unsigned char*)malloc(sizeof(jacobi_ctx_t) + 64);
    uintptr_t      a   = ((uintptr_t)raw + 31) & ~(uintptr_t)31;
    jacobi_ctx_t*  ctx = (jacobi_ctx_t*)(a + (bad ? 16 : 0));

    printf("ctx = %p  (ctx %% 32 = %llu)\n",
           (void*)ctx, (unsigned long long)((uintptr_t)ctx % 32));
    printf("&ctx->dx2_inv_vec = %p\n", (void*)&ctx->dx2_inv_vec);
    fflush(stdout);

    printf("calling ctx_init...\n");
    fflush(stdout);
    ctx_init(ctx, dx, dy, dz);

    printf("survived: inv_factor = %g\n", ctx->inv_factor);
    free(raw);
    return 0;
}

Three things in there are load-bearing. ctx_init is the real one, copied over unchanged in shape — compute the scalars, _mm256_set1_pd the four constants, set the flag. The spacings come from argv rather than being literals, because with constants the compiler folds the whole struct into a rodata blob and you get different code than the solver produces. And the three lines that build ctx are the entire experiment: round up to a boundary, then add 16 or don’t.

Nothing else is doing any work. There is no solve loop, no grid, no library. Run it both ways.

Aligned:

$ ./align_demo good
sizeof(jacobi_ctx_t)  = 224
_Alignof(jacobi_ctx_t)= 32
_Alignof(max_align_t) = 8
ctx = 0000012286866E80  (ctx % 32 = 0)
&ctx->dx2_inv_vec = 0000012286866EC0
calling ctx_init...
survived: inv_factor = 0.00166667

Sixteen bytes off:

$ ./align_demo bad
sizeof(jacobi_ctx_t)  = 224
_Alignof(jacobi_ctx_t)= 32
_Alignof(max_align_t) = 8
ctx = 0000018863526E90  (ctx % 32 = 16)
&ctx->dx2_inv_vec = 0000018863526ED0
calling ctx_init...
Segmentation fault

That third line is worth a pause, because it is smaller than the 16 I quoted earlier. It is genuinely 8 here: this is the Microsoft ABI, where max_align_t is 8-byte aligned, and 16 was the glibc figure. The heap still hands back 16-byte-aligned blocks in practice — the survey further down measures it — but on Windows the standard’s guarantee is weaker still. Which only sharpens the point: 32 was never covered anywhere, by any margin, on either platform.

Now look closely at where those addresses came from, because this is the part that reframes the whole bug: that is Windows. Both runs. The same 0xC0000005 the CI never saw, on the platform whose green square started this.

Which means the platform difference was never about the instruction at all. VMOVAPD faults on Windows exactly as it faults on Linux; the CPU does not know which kernel booted it. Windows was green for one reason only — its heap happened to hand back an address that worked. Take that luck away and Windows crashes on the first try, every time.

So point a debugger at it:

(e74c.d5e4): Access violation - code c0000005 (first chance)
rcx=0000028575e5fa90
align_demo_ni!ctx_init+0x5e:
00007ff7`ebd6773e c5fd295940      vmovapd ymmword ptr [rcx+40h],ymm3

There is the whole bug on four lines. The faulting instruction is vmovapd, an aligned store. Its destination is [rcx+40h] — offset 64, which the table above says is dx2_inv_vec. And the base pointer ends in 0x90, so rcx % 32 is 16.

That last number is the entire diagnosis. Not a corrupted struct, not a bad index, not a numerical blow-up: a pointer that is sixteen bytes short of where its type needed it to be.

The instruction that faults

Now the part the original write-up got slightly wrong, and the part I find genuinely interesting.

First, terminology, because the crash has two names depending on who is telling you. The CPU raises a #GP — a general-protection fault — when an instruction that requires alignment gets an address that lacks it. The operating system then translates that into something you recognise: SIGSEGV on Linux, and the 0xC0000005 access violation you just saw on Windows. Same hardware event, two vocabularies. (The original engineering note guessed “SIGBUS or SIGSEGV”; on x86-64 a #GP is delivered as SIGSEGV. SIGBUS is a different fault.)

The intuitive story is that misaligned vector memory faults, so the crash must be in the code doing vector memory operations — the loop. But VEX-encoded AVX instructions with a memory operand have no alignment requirement. vmulpd ymm0, ymm0, [rdi + 64] will happily multiply against a memory operand sitting at any address at all. That is a deliberate design change from the legacy SSE encodings, where mulpd xmm0, [mem] did fault on a misaligned operand. Under AVX, only the explicitly-aligned moves — VMOVAPD, VMOVAPS, VMOVDQA — still enforce it.

So the loop, which reads the vector members purely as multiplicands, was never going to fault either. Both of the places you would look are clean.

Here is the actual codegen, cross-compiled to the Linux ABI at -O2 -mavx2. This is init:

vbroadcastsd  ymm3, xmm1
vmovapd  ymmword ptr [rdi + 64], ymm3
vpermpd  ymm1, ymm1, 85                  # ymm1 = ymm1[1,1,1,1]
vmovapd  ymmword ptr [rdi + 96], ymm1
vbroadcastsd  ymm0, xmm0
vxorpd  xmm1, xmm2, xmmword ptr [rip + .LCPI0_2]
vmovapd  ymmword ptr [rdi + 128], ymm0
vbroadcastsd  ymm0, xmm1
vmovapd  ymmword ptr [rdi + 160], ymm0
mov  dword ptr [rdi + 192], 1

Four aligned stores, at 64, 96, 128 and 160 — the four vector members, in order. The crash is the first of them, in the constructor, before a single grid point has been touched.

Two lines in there are worth decoding, since they are the ones that look like noise. vpermpd ymm1, ymm1, 85 carries Clang’s own comment: 85 is 0b01010101, four 2-bit selectors all naming lane 1, so it broadcasts one double across the register — that is _mm256_set1_pd for a value that already happened to be in lane 1. And vxorpd against a constant from .LCPI0_2 is a sign-bit flip: it is the minus in -ctx->inv_factor.

And now the loop, in full — this is the entire vectorised body, not an excerpt:

vmovupd  ymm0, ymmword ptr [rbx + 8*r14 + 16]
vmovupd  ymm1, ymmword ptr [r9 + 8*r14]
vmovupd  ymm2, ymmword ptr [r11 + 8*r14]
vaddpd   ymm0, ymm0, ymmword ptr [rbx + 8*r14]
vmulpd   ymm0, ymm0, ymmword ptr [rdi + 64]
vaddpd   ymm1, ymm1, ymmword ptr [rdx + 8*r14 + 8]
vmovupd  ymm3, ymmword ptr [rcx + 8*r14]
vmulpd   ymm1, ymm1, ymmword ptr [rdi + 96]
vaddpd   ymm0, ymm0, ymm1
vaddpd   ymm1, ymm2, ymmword ptr [r10 + 8*r14]
vmulpd   ymm1, ymm1, ymmword ptr [rdi + 128]
vaddpd   ymm0, ymm0, ymm1
vsubpd   ymm0, ymm3, ymm0
vmulpd   ymm0, ymm0, ymmword ptr [rdi + 160]
vmovupd  ymmword ptr [rsi + 8*r14], ymm0

Fifteen instructions. Nine of them touch memory. Every single load and store is the u form, and every arithmetic memory operand is VEX-encoded. Not one of them requires alignment. The hot loop reads all four misaligned members, millions of times, and is fine.

This explains the thing that made the failure feel so unrelated to the code: the backtrace points at solver setup, in a function whose entire job is assigning constants. It looks like the struct is corrupt. The struct is fine. The address is wrong, and it has been wrong since the moment calloc returned.

Why Windows was green

Not because MSVC is more forgiving. MSVC gives __m256d the same 32-byte alignment and emits the same aligned stores — the reproducer above proves it, since that crash was a Windows binary.

So the question is narrower than it first looks. Not “why does Windows tolerate this” — it doesn’t — but “why did the Windows heap keep handing back good addresses?”

I measured it. A hundred thousand malloc(224) calls on this machine:

100000 allocations of 224 bytes
  8-byte aligned : 100000  (100.0%)
 16-byte aligned : 100000  (100.0%)
 32-byte aligned :  49999  ( 50.0%)

Half. Exactly half, and the mechanism is visible if you print consecutive addresses:

000001FF90772880  %32= 0  delta=240
000001FF90772970  %32=16  delta=240
000001FF90772A60  %32= 0  delta=240
000001FF90772B50  %32=16  delta=240

Same-size blocks come back 240 bytes apart — 224 of payload plus 16 of bookkeeping. And 240 is 16 more than a multiple of 32, so the alignment alternates: good, bad, good, bad, forever. Any given allocation is a coin flip, but it is not a random one. It is a parity bit, decided by how many blocks of that size were handed out before it.

Which is why the suite passed reproducibly. In a deterministic test binary, the solver’s context is the same allocation in the same sequence on every run, so it lands on the same side of the parity every time. The tests were not getting lucky over and over. They were getting the same answer to the same question, and nobody had noticed the question was being asked.

That is the genuinely uncomfortable half of this bug. A test that fails is doing its job. A test that passes on a parity bit is worse than a failing test, because you believe it. The Linux build was not less correct than the Windows one. It was more honest.

Why NEON never needed this

The same library has NEON implementations of the same solvers, with the same shape of context struct, and they allocate with plain cfd_calloc to this day:

jacobi_neon_context_t* ctx =
    (jacobi_neon_context_t*)cfd_calloc(1, sizeof(jacobi_neon_context_t));

That is not an oversight waiting to bite. It is correct, and the reason is the whole rule in one comparison. float64x2_t is 16 bytes with a 16-byte alignment requirement. On aarch64, _Alignof(max_align_t) is 16. So a NEON context’s alignment is a fundamental alignment, and §7.22.3 obliges calloc to satisfy it. The promise that AVX2 needed and never had is one NEON gets for free.

Which means the rule people take away from bugs like this — “structs with SIMD types need aligned allocation” — is the wrong rule, and following it would have you churning correct NEON code. The actual rule has nothing to do with SIMD:

If a type’s alignment exceeds _Alignof(max_align_t), malloc and calloc do not cover it.

SIMD types are just the most common way to acquire an extended alignment by accident. Cache-line padding — _Alignas(64) on a struct to stop false sharing between threads — is the other one, and it fails identically and for the same reason.

The fix, and the half that is easy to forget

/* Use aligned allocation for struct containing __m256d members */
jacobi_avx2_context_t* ctx =
    (jacobi_avx2_context_t*)cfd_aligned_calloc(1, sizeof(jacobi_avx2_context_t));

That is the visible half. Here is the half that will hurt you:

static void jacobi_avx2_destroy(poisson_solver_t* solver) {
    if (solver && solver->context) {
        cfd_aligned_free(solver->context);   /* NOT cfd_free */
        solver->context = NULL;
    }
}

Because there is no one portable aligned allocator to call. The menu, in full:

The aligned_alloc size rule sounds like a trap and here is not: a type with 32-byte alignment always has a sizeof that is a multiple of 32, because the compiler pads it to one. Our struct is 224 bytes, which is 7 × 32. Any correctly-aligned type satisfies that constraint for free.

The library splits on the platform, which is why the pairing matters:

void* cfd_aligned_malloc(size_t size) {
#ifndef _WIN32
    if (posix_memalign(&ptr, alignment, size) != 0) { /* ... */ }
#else
    ptr = _aligned_malloc(size, 32);
#endif
    return ptr;
}

void cfd_aligned_free(void* ptr) {
    if (ptr != NULL) {
#ifndef _WIN32
        free(ptr);
#else
        _aligned_free(ptr);
#endif
    }
}

Look at the asymmetry: on POSIX, cfd_aligned_free is free, so getting this wrong on Linux does nothing at all. On Windows it corrupts the heap — silently, at some unrelated later allocation, with no stack pointing anywhere near the solver.

So the natural way to half-finish this fix is to change the allocator, watch Linux go green, and ship. You will have converted a loud, reproducible, immediate crash on Linux into a quiet delayed heap corruption on Windows. The two lines are one change and have to move together.

Catching it in thirty seconds

Three tools, cheapest first. All of them beat reading the loop for an hour.

The debugger, one expression. When you have a SIGSEGV on a vector instruction, you do not need a theory. You need the base register modulo the alignment:

(gdb) p $rdi % 32
$1 = 16

Non-zero and the argument is over. This is the single highest-value habit in the post: when a crash lands on a vmov* and the operand is a struct member, check the base pointer’s remainder before you check anything else.

The sanitizer. UBSan has an alignment check that fires at the misaligned access rather than waiting for the hardware:

clang -mavx2 -O1 -g -fsanitize=alignment -fsanitize-trap=alignment demo.c

On the good path the program exits 0; on the bad path it traps immediately. It is strictly better than waiting for #GP, because it fires even on builds where the compiler happened to choose unaligned moves and the bug would otherwise stay hidden — which is precisely the Windows situation. Turning this on in one CI job would have caught the bug on every platform rather than the one that got unlucky.

The compile-time guard. Best of all, because it cannot be forgotten. Put this next to any context struct that is still allocated with the plain allocator:

_Static_assert(_Alignof(jacobi_neon_context_t) <= _Alignof(max_align_t),
               "extended alignment: use cfd_aligned_calloc/cfd_aligned_free");

Today it passes — that is the point of the NEON section above. The day somebody widens a member, or ports the struct to AVX2, or adds _Alignas(64) for cache-line padding, the build stops:

error: static assertion failed due to requirement
       '_Alignof(avx2_ctx_t) <= _Alignof(max_align_t)':
       extended alignment: use cfd_aligned_calloc/cfd_aligned_free
note: expression evaluates to '32 <= 16'

32 <= 16, in a compiler error, at the exact line responsible. That is the same fact the CPU was trying to communicate with a segfault on one platform, delivered years earlier and to the right person.

Three fixes that do not work

_Alignas(32) on the struct. It compiles, it is not wrong, and it fixes nothing. The type already had 32-byte alignment — that is what caused this. Restating it does not create an obligation on malloc, which is still permitted to return a 16-byte-aligned block for a type with an extended alignment. You have documented the requirement without satisfying it.

Switching more intrinsics to the u forms. There is nothing left to switch. The data accesses were already loadu/storeu, and the faulting stores are compiler-generated from plain struct assignment, not from an intrinsic whose spelling you get to choose. You can force unaligned access to the members — memcpy into them, or #pragma pack to strip the struct’s alignment — and both work, in the sense that the crash stops. What they do is delete the requirement rather than meet it: you give up aligned access to those members everywhere, permanently, to avoid calling a different allocator once.

Over-aligning everything. Routing all of cfd_calloc through the aligned path makes the symptom go away and taxes every small allocation in the library to fix four files. Alignment is a property of a type; the fix belongs where the type is, not in the allocator everyone shares.

What actually caught it

A dedicated CI job:

  # SIMD (AVX2) build - Test SIMD optimizations (AVX2 REQUIRED)
  simd-test-linux:
    name: SIMD AVX2 (Linux GCC)

Not the main Linux job, which was green throughout — because without -DCFD_ENABLE_AVX2=ON the AVX2 translation units are excluded by their compile-time guards and never compiled at all. The default matrix square was not testing this code weakly. It was not testing it.

Which generalises past SIMD, and is the bit I would actually put on a wall: a build flag that changes which files compile is a separate build, and needs its own square in the matrix. Feature flags that select a code path get exercised incidentally by everything around them. Flags that select translation units get exercised by nothing. A green matrix tells you about the configurations in it, and the number of configurations in it is usually smaller than the number you ship.

The fix has since become the house style across the AVX2 solvers — Jacobi, Red-Black SOR, plain SOR, CG, and the shared SIMD templates behind BiCGSTAB and GMRES all allocate their contexts aligned now. The NEON ones still do not, and still should not.

The takeaway

Alignment is the rare requirement that is invisible at both ends. The type declares it and you cannot see the declaration, because it is inside a compiler-provided typedef you never open. The allocator declines to honour it and you cannot see the refusal, because a refusal looks exactly like a success and hands you a valid pointer either way. The two facts only meet at a vmovapd you did not write, in a function you were not looking at, on the one platform whose heap parity did not happen to fall the right way.

The wider version, the one that outlives this bug: when a type carries a requirement, find out who is responsible for meeting it, and check that they know. __m256d requires 32 bytes. calloc promises 16. Both were doing precisely their jobs. Nothing in C makes the two of them talk — so you have to, in a _Static_assert, on a line somebody will read.

A note on the evidence

The bug, the platform split, the five failing suites and the fix are from a real CFD library and its CI. The code shown is current source, which means it has moved on since the crash: the struct has picked up a dz2_inv_vec member from a later 3D extension, so it is 224 bytes and four vectors today, where at the time of the fault it was smaller and had three. The alignment argument is unchanged by that, and the offsets, sizes and assembly quoted here are all from the struct as it stands now.

Everything in Reproducing it in one file, Why Windows was green and Catching it in thirty seconds was run on my machine while writing this, and the output is pasted rather than described. The reproducer is printed in full and is exactly what produced the output shown; it is a reduction — the real ctx_init body and struct, with the solve loop trimmed and the spacings taken from argv so nothing constant-folds. It is not the shipping solver, and it is not the original CI crash: it is a demonstration that reproduces the same fault deterministically, which the original could not do.

The debugger session is Windows cdb against that reduction. The register is rcx rather than the rdi in the Linux listings because the two ABIs pass the first pointer argument in different registers; same pointer, same struct, different calling convention. I built it with -fno-inline so the frame is readable — at full -O2 the compiler inlines ctx_init into main and the fault lands on the same store with a different addressing mode.

The assembly listings are Clang 21 cross-compiled with --target=x86_64-unknown-linux-gnu -mavx2 -O2, verbatim and in order, including Clang’s own # comments. I no longer have the GCC build that produced the original SIGSEGV, so: the instruction selection follows from the type’s declared alignment and the VEX encoding rules rather than from anything Clang-specific, and GCC gives __m256d the same 32-byte alignment, but I did not personally watch GCC emit it. The #GP-to-SIGSEGV mapping on Linux is the documented behaviour rather than something I observed here; the Windows 0xC0000005 half I did observe.

The alignment survey is 100,000 allocations in one process on one Windows machine, so the 50% and the 240-byte spacing are that allocator on that day, not a law. The parity argument holds wherever same-size blocks are evenly spaced by something that is not a multiple of 32, which is common but not guaranteed.

UBSan’s full runtime does not link on this LLVM install — the standalone library is missing Windows imports — so I ran -fsanitize-trap=alignment, which traps without a message. The readable runtime error: ... which requires 32 byte alignment diagnostic that -fsanitize=undefined prints on Linux is the documented behaviour; I did not exercise it here.

One thing I would still change and have not: cfd_aligned_calloc computes count * size without an overflow check, which real calloc is required to detect. Every call site in the library passes a literal 1, so nothing is reachable today. It is still a worse function than the one it is imitating.