How to recover from error during drat list?

[chmac]

Here's the output of drat list.

drat list output

$ sudo drat list /dev/disk2s3 0 /
Password:
Opening file at `/dev/disk2s3` in read-only mode ... OK.
Simulating a mount of the APFS container.
Validating checksum of block 0x0 ... OK.
Locating the checkpoint descriptor area:
- Its length is 284 blocks.
- It is contiguous.
- The address of its first block is 0x1.
Loading the checkpoint descriptor area into memory ... OK.
Locating the most recent well-formed container superblock in the checkpoint descriptor area:
- It lies at index 69 within the checkpoint descriptor area.
- The corresponding checkpoint starts at index 68 within the checkpoint descriptor area, and spans 2 blocks.

Loading the corresponding checkpoint ... OK.
- There are 8 checkpoint-mappings in this checkpoint.

Reading the Ephemeral objects used by this checkpoint ... OK.
Validating the Ephemeral objects ... OK.
The container superblock states that the container object map has Physical OID 0x2ec48d.
Loading the container object map ... OK.
Validating the container object map ... OK.
Reading the root node of the container object map B-tree ... OK.
Validating the root node of the container object map B-tree ... OK.
The container superblock lists 1 APFS volumes, whose superblocks have the following Virtual OIDs:
- 0x413

Reading the APFS volume superblocks ... OK.
Validating the APFS volume superblocks ... OK.

 Volume list
================
 0: xxxxxxx
The volume object map has Physical OID 0x2ec487.
Reading the volume object map ... OK.
Validating the volume object map ... OK.
Reading the root node of the volume object map B-tree ... OK.
Validating the root node of the volume object map B-tree ... OK.
The file-system tree root for this volume has Virtual OID 0x414.
Looking up this Virtual OID in the volume object map ... corresponding block address is 0x1651f.
Reading ... validating ... FAILED.
Going back to look at the previous checkpoint instead.

I also got this during my first aid attempt:

Checking the extent ref tree.
error: (oid 0x2ec287) apfs_extentref: bt: invalid o_type (0x40000003, expected 0x40000002)
Extent ref tree is invalid.

Then I read this answer by @jivanpal. My guess is that I need to switch a single bit on disk, so that the o_type goes from 0x40000003 to 0x40000002. But I have no idea how to figure out which bit that is!

Also, Disk Utility refuses to make a copy of this container. It says "operation not permitted" every time, so I can't take a backup. I have a 2.8T partition with just 140G of data inside it, and I don't have a spare disk handy to image the entire partition unfortunately. :-(

[chmac]

Adding a bit more context, I tried to read block 0x1651f, but I also got an error.

Reading block 0x1651f ... validating ... FAILED.
The specified block may contain file-system data or be free space.

drat read /dev/disk2s3 0x1651f output

$ sudo drat read /dev/disk2s3 0x1651f
Opening file at `/dev/disk2s3` in read-only mode ... OK.

Reading block 0x1651f ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0x1651f:
--------------------------------------------------------------------------------
Stored checksum:    0x64fcd478c89f7be0
OID:                0x7b770ea1e39e98a9
XID:                0xeb5daa83da6f51de
Storage type:       Physical
Type flags:         No-header
Type:               Unknown value (0x359)
Subtype:            Unknown value (0xa798)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.

[jivanpal]

Regarding the extent ref tree, it would seem the B-tree root node is corrupted. It could indeed be a single flipped bit as you said, but we'd need to investigate further to be sure. However, the extent ref tree is not usually important, and Drat doesn't currently use it in any case, so I wouldn't bother diving into that unless it becomes necessary for some reason.

Regarding the FS root tree, and block 0x1651f: Is the drive/volume encrypted? If so, then what you're seeing is expected, since it's ciphertext. Unfortunately, we don't yet support encryption. If it is not encrypted, then I would guess that the block now stores file data, and the object map that resolved Virtual OID 0x414 to that block address is simply invalid/outdated. Analysis of the data available in the apparent block header (apparent checksum/OID/XID), as in the drat read output you provided, seems to support this hunch; the file command tells me: /dev/fd/11: fifo (named pipe).

[chmac]

@jivanpal Yes, the drive is encrypted. The 3rd partition contains a "container".

Ahhh, drat doesn't support encryption at all? Somehow I thought that I had unlocked / decrypted the container and only the volume itself was failing. But now you mention it, of course I haven't supplied the encryption password anywhere to drat, so it makes total sense that drat can't see the unencrypted data.

Okay, well, it seems like I'll have to recommend the iBoySoft to the person who's disk it is. Thanks for the quick reply. 👍

[jivanpal]

If you mean that full-disk encryption is being used on the drive, then this isn't a problem (assuming you have decrypted the whole drive, which appears to be the case, else we wouldn't have any info at all). I'm referring to APFS software-level encryption, which is what FileVault employs on non-T2 Macs that are using APFS. This is sometimes referred to as "FileVault 3". So if you formatted the partition as "APFS (Encrypted)" before installing macOS, or enabled FileVault on the system after the fact, then the answer to my question is "yes".

When APFS encryption is enabled, once you get inside the volume, all the structures therein are encrypted. This is because encryption is done on a per-volume basis, so you don't need an encryption key until you actually get to looking at the FS root tree and file data. If the FS root tree were not encrypted, then you'd be able to see file metadata (file/directory names, timestamps, etc.).

Encryption support in Drat is on the roadmap, but it's near the bottom of the list, so I don't expect it will be implemented before the end of the year.

I've not used iBoySoft's tool, but I can recommend Disk Drill. However, if you have APFS encryption enabled, then YMMV.

[chmac]

@jivanpal Oh, I think I'm confused. There are 3 partitions:

• time machine backup, encrypted, APFS
• FAT partition, unencrypted
• APFS partition, encrypted, contains only 1 volume

When I plugged in the drive, it asked me for the password for the 1st and 3rd partitions, and I entered the passphrase for partition 3. But I can see the names of the volumes for both in Disk Utility.

I think given that my volume is encrypted, I won't be able to get it to work with drat. Or am I misunderstanding something?

PS: Thanks for the Disk Drill recommendation, and all the quick answers, I really appreciate your openness to helping folks here and on StackOverflow.

[jivanpal]

Yeah, that's correct, if the APFS partition is encrypted (which will be with APFS encryption, a.k.a. FileVault), then Drat currently will not be able to help you.