Problem with zip64 on native Xiaomi archiver utility

[dmkl]

We are streaming lots of archives with zstream. It works very well except the native Xiaomi archiver utility, that fails to unzip a zip64 archive. The first idea was that this is a bug in that utility, but it handles zip64 archives from another streaming solution mod_zip with no issues.

Another hint that something might be wrong with zip64 format is that zipdetails mac/linux util fails to display zstreams zip64 details. Here is an example script and files.

Zstream.zip([
  Zstream.entry("test/10.txt", File.stream!("10.txt", [], 512), coder: Zstream.Coder.Stored)
])
|> Stream.into(File.stream!("/tmp/test.zip"))
|> Stream.run

Zstream.zip([
  Zstream.entry("test/10.txt", File.stream!("10.txt", [], 512), coder: Zstream.Coder.Stored)
], zip64: true)
|> Stream.into(File.stream!("/tmp/test_zip64.zip"))
|> Stream.run

$ zipdetails /tmp/test.zip

0000 LOCAL HEADER #1       04034B50
0004 Extract Zip Spec      14 '2.0'
0005 Extract OS            00 'MS-DOS'
0006 General Purpose Flag  0808
     [Bit  3]              1 'Streamed'
     [Bit 11]              1 'Language Encoding'
0008 Compression Method    0000 'Stored'
000A Last Mod Time         56385EDD 'Tue Jan 24 11:54:58 2023'
000E CRC                   00000000
0012 Compressed Length     00000000
0016 Uncompressed Length   00000000
001A Filename Length       000B
001C Extra Length          0000
001E Filename              'test/10.txt'
0029 PAYLOAD               123456789.

0033 STREAMING DATA HEADER 08074B50
0037 CRC                   E0117757
003B Compressed Length     0000000A
003F Uncompressed Length   0000000A

0043 CENTRAL HEADER #1     02014B50
0047 Created Zip Spec      34 '5.2'
0048 Created OS            00 'MS-DOS'
0049 Extract Zip Spec      14 '2.0'
004A Extract OS            00 'MS-DOS'
004B General Purpose Flag  0808
     [Bit  3]              1 'Streamed'
     [Bit 11]              1 'Language Encoding'
004D Compression Method    0000 'Stored'
004F Last Mod Time         56385EDD 'Tue Jan 24 11:54:58 2023'
0053 CRC                   E0117757
0057 Compressed Length     0000000A
005B Uncompressed Length   0000000A
005F Filename Length       000B
0061 Extra Length          0000
0063 Comment Length        0000
0065 Disk Start            0000
0067 Int File Attributes   0000
     [Bit 0]               0 'Binary Data'
0069 Ext File Attributes   81A40000
006D Local Header Offset   00000000
0071 Filename              'test/10.txt'

007C END CENTRAL HEADER    06054B50
0080 Number of this disk   0000
0082 Central Dir Disk no   0000
0084 Entries in this disk  0001
0086 Total Entries         0001
0088 Size of Central Dir   00000039
008C Offset to Central Dir 00000043
0090 Comment Length        0012
0092 Comment               'Created by Zstream'
Done

$ zipdetails /tmp/test_zip64.zip
0000 PREFIX DATA
Done

10.txt
test.zip
test_zip64.zip

[ananthakumaran]

Please confirm if the mod_zip generated files are actually in zip64 format. From a quick look at the code, mod_zip only uses zip64 if the file size is > 4GB. If you have zip64 file encoded using the data descriptor method (stream) that works in Xiaomi, please upload it here, then it would be easier to debug the issue.

[dmkl]

Yes, we are talking about 4Gb+ files that mod_zip streams as zip64. I've attached smaller files just as an example, because the problem is reproducible with them both with the native Xiaomi-phones archiver utility and with zipdetails.

[ananthakumaran]

I meant, I want a file that works in Xiaomi, I am assuming the above one is not working in Xiaomi?

[dmkl]

Sorry for the confusion. All mod_zip files are working on Xiaomi, test.zip works on Xiaomi as well, test_zip64.zip doesn't work on Xiaomi.

Broken zstreams zip64 files (both under and over 4Gb) either do not unpack on Xiaomi or extract files with 0 bytes content like on the screenshot below.

[ananthakumaran]

All mod_zip files are working on Xiaomi, test.zip works on Xiaomi as well, test_zip64.zip doesn't work on Xiaomi.

I want a zip64 file (with data descriptor encoding) that works on Xiaomi. The test.zip is non zip64 file generated by zstream.

[ananthakumaran]

I built the mod_zip locally and modified it to use zip64 for all file sizes. Since it knows the file size, it is setting the correct size in the local header itself

0031 Extra ID #0002        0001 'ZIP64'
0033   Length              0010
0035   Uncompressed Size   000000000000000A
003D   Compressed Size     000000000000000A

This is not possible in zstream case, as it works on stream abstraction and doesn't know the file size. Full output below

0000 LOCAL HEADER #1       04034B50
0004 Extract Zip Spec      2D '4.5'
0005 Extract OS            00 'MS-DOS'
0006 General Purpose Flag  0808
     [Bit  3]              1 'Streamed'
     [Bit 11]              1 'Language Encoding'
0008 Compression Method    0000 'Stored'
000A Last Mod Time         5638780B 'Tue Jan 24 20:30:22 2023'
000E CRC                   00000000
0012 Compressed Length     FFFFFFFF
0016 Uncompressed Length   FFFFFFFF
001A Filename Length       0006
001C Extra Length          0021
001E Filename              '10.txt'
0024 Extra ID #0001        5455 'UT: Extended Timestamp'
0026   Length              0009
0028   Flags               '03 mod access'
0029   Mod Time            63CFF287 'Tue Jan 24 20:30:23 2023'
002D   Access Time         63CFF287 'Tue Jan 24 20:30:23 2023'
0031 Extra ID #0002        0001 'ZIP64'
0033   Length              0010
0035   Uncompressed Size   000000000000000A
003D   Compressed Size     000000000000000A
0045 PAYLOAD               123456789.

004F STREAMING DATA HEADER 08074B50
0053 CRC                   E0117757
0057 Compressed Length     000000000000000A
005F Uncompressed Length   000000000000000A

0067 CENTRAL HEADER #1     02014B50
006B Created Zip Spec      2D '4.5'
006C Created OS            00 'MS-DOS'
006D Extract Zip Spec      2D '4.5'
006E Extract OS            00 'MS-DOS'
006F General Purpose Flag  0808
     [Bit  3]              1 'Streamed'
     [Bit 11]              1 'Language Encoding'
0071 Compression Method    0000 'Stored'
0073 Last Mod Time         5638780B 'Tue Jan 24 20:30:22 2023'
0077 CRC                   E0117757
007B Compressed Length     FFFFFFFF
007F Uncompressed Length   FFFFFFFF
0083 Filename Length       0006
0085 Extra Length          001D
0087 Comment Length        0000
0089 Disk Start            0000
008B Int File Attributes   0000
     [Bit 0]               0 'Binary Data'
008D Ext File Attributes   081A4000
0091 Local Header Offset   00000000
0095 Filename              '10.txt'
009B Extra ID #0001        5455 'UT: Extended Timestamp'
009D   Length              0005
009F   Flags               '03 mod access'
00A0   Mod Time            63CFF287 'Tue Jan 24 20:30:23 2023'
00A4 Extra ID #0002        0001 'ZIP64'
00A6   Length              0010
00A8   Uncompressed Size   000000000000000A
00B0   Compressed Size     000000000000000A

00B8 ZIP64 END CENTRAL DIR 06064B50
     RECORD
00BC Size of record        000000000000002C
00C4 Created Zip Spec      2D '4.5'
00C5 Created OS            00 'MS-DOS'
00C6 Extract Zip Spec      2D '4.5'
00C7 Extract OS            00 'MS-DOS'
00C8 Number of this disk   00000000
00CC Central Dir Disk no   00000000
00D0 Entries in this disk  0000000000000001
00D8 Total Entries         0000000000000001
00E0 Size of Central Dir   0000000000000051
00E8 Offset to Central dir 0000000000000067

00F0 ZIP64 END CENTRAL DIR 07064B50
     LOCATOR
00F4 Central Dir Disk no   00000000
00F8 Offset to Central dir 00000000000000B8
0100 Total no of Disks     00000001

0104 END CENTRAL HEADER    06054B50
0108 Number of this disk   0000
010A Central Dir Disk no   0000
010C Entries in this disk  0001
010E Total Entries         0001
0110 Size of Central Dir   00000051
0114 Offset to Central Dir 00000067
0118 Comment Length        0000
Done

[dmkl]

Here are some more examples:

Working file from mod_zip https://dmkl.s3.eu-west-1.amazonaws.com/zip/4gb_mod_zip.zip (works on Xiaomi)

Broken file with the same content from zstream https://dmkl.s3.eu-west-1.amazonaws.com/zip/4gb_zstream.zip

Is there a way to extend zstream in the similar manner to fix the issue if we know file sizes upfront?

[ananthakumaran]

test_zip64_1.zip

test_zip64_2.zip

test_zip64_3.zip

could you check if any of the following variation works?

Is there a way to extend zstream in the similar manner to fix the issue if we know file sizes upfront?

if crc and size available, this is possible, but might require a bit of work on zstream side

[dmkl]

None of the three work on Xiaomi :\

[dmkl]

We have the CRC and the size for all files. It would be fantastic to support this case!

[ananthakumaran]

if CRC and file size are available, and the coder is Zstream.Coder.Stored, then yes, it should be possible to drop the data descriptor and put all the details in the local header itself. I will see if I can get something out this weekend

[dmkl]

Thanks a lot! Looking forward to it. Please let me know if I can help with anything.

[ananthakumaran]

could you try #19