Reverse Engineering HxStore.hxd, the New Outlook Message Store

New Outlook keeps a local cache of your mail in a single file called HxStore.hxd. On my machine it's 1.1 GB, sitting in the profile directory, and searching it through the Outlook UI is slow enough that I went looking for a way to query it directly. Grep finds a string in that file in under a second, so the data is clearly there in some readable form.

There's no documentation for it. Searching for prior work mostly turns up other people asking the same question, quite a few of them in computer forensics forums: an examiner has a disk image, there's an HxStore.hxd in the Outlook profile, and the usual answer is to give up and pull the mailbox from Exchange instead. That doesn't help if the server is gone. The nearest published research is Chivers on Windows Mail's store.vol, which is a genuinely different format (ESE database with MAPI property tags) and doesn't apply here. I checked before starting: zero ESE magic bytes, zero MAPI tags in the whole file.

This post covers how the format works and how I got there. The parser is hxprobe and the full format documentation is in SPEC.md, including the parts I'm still unsure about.

What the file looks like

The first eight bytes are ASCII Nostromo, followed by a version byte, 'i' (0x69) on the macOS build. Two measurements decide whether the file is worth attacking at all. Entropy sits between 6.3 and 6.9 across the whole file, where encrypted data would be close to 8.0, so it's compressed rather than encrypted. And about 30% of the bytes are printable, evenly distributed, so there is plaintext in there.

There are also 6,922 occurrences of the UTF-16LE string IPM.Note, which is the MAPI item class for a mail message. That's roughly the number of messages in my mailbox, so records are findable even before understanding the container.

One thing to know before touching it: Outlook holds HxStore.lock and rewrites the file while it's running. I watched a store shrink from 1,153,433,600 to 1,151,718,400 bytes mid-session. Everything below is done against a copy.

Getting the compression wrong

My first approach was to find an IPM.Note anchor, scan backwards for something that looked like the start of a compressed block, and try to decode it as LZ4. This produces output, which is the problem.

LZ4 block format has no checksum and no framing. Every byte is a valid token, so decoding from the wrong offset doesn't fail, it just produces plausible garbage. I was getting message bodies that started correctly and then dissolved into mojibake, senders like (AIM, and subjects that turned out to be fragments of a different message. To tell good decodes from bad ones I wrote scoring functions: longest run of a repeated byte, ratio of printable characters, whether the output contained common English words. They picked the least-bad candidate reasonably well, but that isn't the same as being correct, and you can't write a specification on top of a heuristic.

The better option is that Outlook ships the code that writes the file.

lipo -thin arm64 HxCore -output HxCore.arm64
otool -arch arm64 -tv HxCore.arm64 > HxCore.asm

The C++ symbols survive in HxCore.framework, which narrows the search a lot. Hx::Storage::KeyValueStore, KeyValueBlock, KeyDirectory, StoreFile, BufferReader, Transaction, StoreVersionMismatch.

The block header writer is at 0xf44670. It stores a magic value, then computes crc32(block+8, len-8) into offset +0x04, then crc32(block+4, 0x1c) into offset +0x00. The validator at 0xf4552c reloads [block+0x14] as the length and re-checks the second CRC before the payload is touched at all. That gives the whole container:

+0x00  u32  crc32(block[0x04 .. 0x20])          header checksum
+0x04  u32  crc32(block[0x08 .. 0x28 + len])    payload checksum
+0x08  u64  magic  0x5d0245643b706a05
+0x10  u32  type            (8 dominates, 16 is rare)
+0x14  u32  payload_len     compressed bytes, from +0x28
+0x18  u32  inflated_len    exact decompressed size
+0x1c  u32  4               constant in every block observed
+0x28  ...  LZ4-compressed payload

The checksums are standard CRC-32 with the zlib polynomial 0xEDB88320; HxCore calls libz's _crc32 from 35 different sites. The asymmetry between the two is easy to get wrong and fails silently if you do: the header CRC covers [0x04, 0x20) and excludes itself, while the payload CRC starts at 0x08, which is the magic, not the payload.

With both checksums plus the declared inflated_len, the parser stops needing heuristics. Scan the file for the 8-byte magic, subtract 8, verify both CRCs, then decompress and require the output length to match exactly. A false positive can't survive all three checks, so a wrong offset fails loudly instead of producing readable nonsense.

On a 56 MB store, 13,103 of 13,116 blocks (99.90%) verify and decompress exactly, giving 221 MB of decompressed data. The thirteen failures aren't lost data: eleven have corrupt header CRCs and sit in stale regions, and three have valid headers wrapping payloads that aren't LZ4 at all. That's expected in an append-oriented store that reuses space.

The codec itself is unmodified LZ4 block format. Token byte, literal length as a 255-continuation varint, literals, 16-bit little-endian back-reference distance, match length with a minimum of 4. Overlapping copies are legal and encode run-length expansion, so matches have to be copied one byte at a time rather than as a block move. The +4 minimum and the varint scheme match the length decoder in HxCore at x86-64 0x136c9de.

Records, and why fixed offsets don't work

Inside a decompressed block, messages are anchored on the UTF-16LE IPM.Note string, with NUL-terminated UTF-16LE runs around it holding the sender address, display name, Message-ID, subject and body preview. A single block usually holds several records back to back.

The obvious approach is to map each field to a fixed displacement from the anchor. That works just well enough to mislead. Measured across 16,721 records, the sender address turns up at -74, at -82, and at +1174. Only two displacements are stable at all: 0, which is the anchor by definition, and +18, which holds the Message-ID in about 60% of records.

The reason is that the fields are variable length and NUL-terminated, so every position depends on the length of everything before it. There is no fixed layout to find. What is stable is the order of the fields, and the order is documented, by Microsoft, in the same profile directory.

The Osa logs

Outlook 15 Profiles/Main Profile/Osa/OutlookServiceApiLogs_*/ contains about 41,800 gzipped XML request and response logs from the live sync protocol. They're plain gzip, so gzcat reads them directly.

Message content is redacted, which initially looks like a dead end:

<Subject>pii:140D5125557024F9</Subject>
<Preview>pii:F9F27BA10DE18B7</Preview>

The field names, their order, and their types are all in the clear though, and these are the same objects that HxStore persists. A GetMessage response opens its MessageHeader like this:

ConversationId · ConversationIndex_Substrate · ImmConversationId · MessageId ·
ImmId · Topic · NormalizedSubject · Preview · LastDeliveryTime ·
LastModifiedTime · ReceivedOrRenewTime · SentTime · ScheduleStatus ·
SenderDisplayNamesCollection{Name,Address} · From{…} · ChangeKey ·
ItemClass_Substrate · InternetMessageId · SortTime · MailboxGuid · …

That resolved two things I had already misdiagnosed. A long base64 string kept appearing in my subject column and I'd assumed it was a decode error; it's AntispamSafeLinksMsgData_Substrate, real SafeLinks scan metadata that legitimately lives in the record. I'd also noticed the subject appeared to be stored twice in a row and written it off as a quirk of the writer, but Topic and NormalizedSubject are two separate protocol fields that happen to sit adjacent, the first with reply prefixes stripped and the second with them intact.

Instead of indexing by byte offset, the parser now walks the string runs in sequence and assigns each slot a name from that schema, checking the value against the type the schema says it should hold. A slot that fails its check is left empty rather than filled with whatever happened to be sitting there.

Timestamps

I could not find a date field for a long time. I scanned for Windows FILETIME values (8 bytes, 100-nanosecond units since 1601) in every plausible range around every anchor and found nothing, so I concluded the timestamps lived in some binary property section I hadn't mapped and shipped a database with no date column.

The Osa logs settle it in one line, because they emit the same field in both raw and rendered form in a single response:

<ReceivedOrRenewTime d="c">639201014590000000</ReceivedOrRenewTime>
<LastDeliveryTime>2026-07-19T23:44:19.000Z</LastDeliveryTime>

639201014590000000 divided by 10^7 is the number of seconds since year 1, and lands exactly on that rendered timestamp. These are .NET ticks, 100-nanosecond units with an epoch of 0001-01-01, not FILETIME. Decoding one as FILETIME puts you in the year 3626, which is exactly why scanning FILETIME ranges turned up nothing.

unix = (ticks - 621355968000000000) / 10^7

Finding the values after that is easy, but choosing between them is not. Scanning each record's span for 8-byte values inside a 2015 to 2027 tick window finds a timestamp in 100% of records, and a single record holds between one and twelve distinct ticks: sent, delivered, last modified, and various sync stamps. Fixed displacements only caught about a quarter of records, for the same variable-length reason as every other field, and ordinal position ("always the third tick") isn't stable either.

What is reliable is ordering. A message is sent before it is delivered, modified or synced, so the earliest tick in the record span is the send time. Scoring the two obvious candidates over the whole store, on the basis that a real send time is close to unique per message while a sync stamp collapses onto the handful of moments Outlook last ran:

That 4.43% is visible directly in the output. Before I fixed this, eight unrelated newsletters all carried the exact second at which I'd taken the snapshot. Taking the earliest tick gives 100% date coverage, no dates in the future, and a distribution weighted towards recent mail.

Subjects, and testing your own assumptions

The subject is identified by the Topic / NormalizedSubject pair being stored adjacent and nearly identical. Requiring an exact match recovered 51% of subjects, and I wrote the remainder off as not being stored.

That was wrong, and what caught it was writing a command that tries to prove the opposite: for every record where no subject was recovered, dump every string in its span so I can see what was skipped. It reported that 99.8% of them contained an unclaimed candidate, which sounds catastrophic until you look at what the candidates were. Mostly Anonymous, Tech Lead, Slack: display names and enum values, not subjects. Filtering to runs that read like actual subject text brought it to 30%, and that part was real.

Three things were causing it. Topic strips reply prefixes and NormalizedSubject doesn't, so the pair reads Re: Quarterly Report against Quarterly Report and an exact comparison never matches. Topic also frequently isn't a standalone string at all: it gets concatenated onto the end of the packed recipient-name collection with no separator, so the record holds Alice Turner␄Bob NakamuraQuarterly Report, and matching requires comparing against the tail of that run rather than the whole thing. And some records store the subject exactly once with no sibling, which I had been refusing on principle. A lone run is safe to take if it sits where the schema puts the subject and reads as several words, but only with that position bound; without it the same fallback returns body previews and fields belonging to the neighbouring record.

Together those took it from 51% to 85.7%. Re-running the same test now leaves 23 records holding a candidate, and they're all things that genuinely aren't subjects: a fragment starting mid-word, an auto-reply body, and packed quick-reply lists like Got it, thanks! + Yes, will do. + Anonymous. The rest of the records have no subject text anywhere in their span, so the subject for those is presumably stored once per conversation somewhere I haven't mapped yet.

Results

hxprobe db snapshot.hxd mail.db

1.2 seconds and 67 MB peak RSS against the 56 MB store, producing 7,193 messages. The file is memory-mapped throughout and never read into an owned buffer, which matters at this size; an early Python version of this called read() on the whole file and took the machine down with it.

That last row looks like a parser failure and isn't. HxStore is a cache rather than an archive: it keeps the complete body for a minority of messages and a preview of roughly 255 characters for everything else, fetching the real body from the server when you open the message. The cap is sharp. Bucketing the longest field in each record puts 89% of them between 200 and 250 characters, with a maximum of 292. The first time I saw short bodies I assumed something was truncating them.

Searching is then a normal FTS5 query:

SELECT sent_utc, sender, subject FROM messages
WHERE id IN (SELECT rowid FROM messages_fts WHERE messages_fts MATCH 'invoice')
ORDER BY sent_unix DESC;

Windows

Windows Mail and the Windows build of Outlook use the same Hx engine and the same store format. I've only tested against macOS, which writes version byte 'i', but HxCore carries NostromoH, NostromoH9 and NostromoI as string literals and Windows Mail samples in the published literature show 'h', so the container should be identical.

hxprobe accepts 'i' and 'h' and warns on anything else rather than guessing. Since every block carries two CRC-32s, a real format difference would fail validation rather than silently producing wrong data. I don't have a Windows store to test against, so if you do, I'd like to hear how it goes.

Still unmapped

• The u64 at +0x20 in the block header. It's covered by the header CRC, so it's meaningful, but I don't know what it holds.
• Block type 8 versus 16.
• The authoritative recipient table. Addresses are recovered from inside the record span, so ordering and the To/Cc/Bcc distinction aren't available.
• Folder membership, read state and flags. The Osa logs name the fields (FocusedClassification, HasAttachment, IsRead), which at least gives a target list.
• How pages are allocated and reclaimed. Page size is 4096 and the region offsets in the header are known, but the allocation strategy isn't.

The specification marks every claim with how it was established: read out of the disassembly, measured against a live store, or inferred and not yet confirmed.

This is independent interoperability research, done on my own machine against my own mailbox, by observing a file format and log files that are readable on disk. It reproduces no Microsoft code.