Available implementation

[Schamper]

Hey @huven, I stumbled upon your Reddit post and subsequently this GitHub repository while writing my blog post on ASIF. Just wanted to point you in this direction:

https://github.com/fox-it/dissect.hypervisor/blob/0c8976613a369923e69022304b2f0ed587e997e2/dissect/hypervisor/disk/asif.py
https://github.com/fox-it/dissect.hypervisor/blob/0c8976613a369923e69022304b2f0ed587e997e2/dissect/hypervisor/disk/c_asif.py

Hope you find it interesting, and maybe you could help improve this implementation with your findings!

[huven]

Thanks, that will save me a lot of time!! I spent most of yesterday evening figuring out the directory structure, I see a lot of similarities with my notes 😄

Will have some time at the end of the week to do a more thorough compare, and update this repo.

[huven]

A quick first scan shows a lot of similarities indeed. Also some differences, like I found some chunks (every second table entry?) holding a bitmap of used sectors instead of disk data, probably similar to what sparsebundle is doing.
Maybe that is already covered by your code, need to spend more time analyzing that later this week, I'm not very familiar with Python. Will post here when I know for sure 😎

Will keep this repo focused on documenting the format specification, which I think is useful as the license of this repo is more suitable for documentation.

Is your blog post already published?

[Schamper]

Also some differences, like I found some chunks (every second table entry?) holding a bitmap of used sectors instead of disk data, probably similar to what sparsebundle is doing.
Maybe that is already covered by your code

It's been a few months since I looked actively at the format so I don't really remember, but a quick peek in my IDA database does reveal there being bitmap table entries. I probably saw it, but didn't really care for it for my read-only implementation. In the end I only spent a handful of hours working on it, so it's not going to be a 100% complete reference implementation, but a good start nonetheless.

Will keep this repo focused on documenting the format specification, which I think is useful as the license of this repo is more suitable for documentation.

Sounds good!

Is your blog post already published?

I started writing it yesterday, so no not yet, but I'll keep you in the loop!

[huven]

Yeah, the bitmaps are for later, in the end I do want to write to the image, but reading first 😄

I do think there is one more level of structure inside the Table: it looks like the entries there are grouped with size 2048 (or better: 4 * block size, i.e. your num_reserved_table_entries).

After one group follows one table entry referring to the bitmap chunk describing usage of the preceding 2048 chunks. So that makes the total group size 2049 explaining why you have a lot of (num_reserved_table_entries + 1) in the disassembly.

This also shows in your _read function, where the data_idx takes a +1 correction each time it crosses a 2048 boundary, skipping the table entry pointing to the bitmap chunk.

If the above is true, I'd suggest renaming num_reserved_table_entries to data_chunk_group_size or something alike, after which the formulas are probably easier to read, as the table-size needs to be a multiple of (data_chunk_group_size+1).

_num_table_entries -> usable table entries
_num_reserved_directory_entries -> number of data chunk groups
_num_usable_entries -> number of data chunk entries
etc.

[Schamper]

I'm working on my blog post again and wanted to let you know I took another look. Your insights into the bitmaps definitely helped me further understand it. In fact, looking back into my IDA database, my original variable names are very much oriented towards "groups" of data entries 😅. I think I abandoned it in favor of some plausible sounding LLM-generated names. Should've persisted in my artisan approach after all.

I cleaned up some of the math and variable names in this PR: fox-it/dissect.hypervisor#86.
I also renamed some of the flags. Looking back, I think I looked at the wrong strings to base those flag names on.

I still didn't implement the bitmap since I still don't have much use for it, but noticed (you probably saw this already) that macOS seems to mark bitmap entries as in-use in groups of 32 blocks (i.e. if you write to the first block, the first 32 blocks will be marked as allocated).

I hope to wrap up the blog soon and will leave a comment here when it's up!

[huven]

Thanks, looking forward to your blog article!

Will compare your latest update with the current document here, to see if I can improve it in any way.

Haven't gotten around implementing the bitmap myself either, will give that a try soon as that will surface any incompatibilities quickly.

[Schamper]

You can find the blog here: https://schamper.dev/dissecting-apples-sparse-image-format-asif/

Thanks for your insights!

I think implementing the bitmaps should be fairly easy. Maybe I'll add it at a later point too 😄.

[huven]

Congrats on completing the blog! 🥳 I'll definitely give that a good read.

Made two additions to the document yesterday, one inspired by your code (how to use the first two flag-bits, although 0b10 is still a mystery to me), and added a remark to ignore directory entries with value 0.

You might consider adding that to the code, when reading sectors from a table that does not exist (0 valued in the directory) it looks like the current code tries to interpret chunk 0 as table. And while an app is not supposed to read initialized sectors, it does happen e.g. when probing what is stored on the image.

[Schamper]

Ah good point. I completely forgot! I definitely thought about it, but forget to actually implement it haha. Will add it soon.

[huven]

..., but noticed (you probably saw this already) that macOS seems to mark bitmap entries as in-use in groups of 32 blocks (i.e. if you write to the first block, the first 32 blocks will be marked as allocated).

I'm not really sure about that, here's an experiment:

IMG=/private/tmp/asif-one-sector.asif

rm -f "$IMG"
diskutil image create blank --format ASIF --size 64m -fs None "$IMG"

DEV=$(diskutil image attach --noMount "$IMG" | awk '/^\/dev\// {print $1; exit}')
RDEV="/dev/r$(basename "$DEV")"

python3 - <<'PY' >/private/tmp/sector-a.bin
import sys
sys.stdout.buffer.write(b"A" + b"\x00" * 511)
PY

dd if=/private/tmp/sector-a.bin of="$RDEV" bs=512 count=1 seek=0 conv=notrunc
hdiutil eject "$DEV"

After which hexdump shows:

[...]
00400000  c0 00 00 00 00 00 00 05  00 00 00 00 00 00 00 00  |................|
00400010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00404000  00 00 00 00 00 00 00 06  00 00 00 00 00 00 00 00  |................|
00404010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00500000  41 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |A...............|
00500010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00600000  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00600010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00700000

Looks like just one sector is marked in the bitmap (chunk 6) here. I noticed the same 32-bytes remark in your blog article, so maybe that needs a small amendment.

[Schamper]

I wrote to /dev/disk# with just a fh.write("chunk 0, block 0"), perhaps the difference between disk and rdisk. Anyway, I amended the post. Thanks!

[huven]

I wrote to /dev/disk# with just a fh.write("chunk 0, block 0"), perhaps the difference between disk and rdisk. Anyway, I amended the post. Thanks!

Yup, the 'r' is most likely what caused the difference.

[huven]

One more interesting observation: your article mentions the 4 PiB limit, though that seems determined by the 1MB chunk size (default).

A very rough estimate shows maximum addressable size to be ~ (chunk size)^3/256, so setting chunk size to 2^31 that would give 32 YiB (10^24)

There's another limit that hits earlier, which is max sector count: setting sector size to largest power of two (2^15), multiplying that with (2^64-1) gives 512 ZiB (10^21).

Not 100% sure if another limit would hit earlier, but it looks like 4 PiB is a bit conservative 😄

[Schamper]

I moreso meant it for that specific disk and the default settings, not the technical limits of the format. I added a few words to avoid confusion on that part!