javax.imageio.IIOException: destination width * height * samplesPerPixel > Integer.MAX_VALUE

[bezverec]

Při importu velkých TIF map do ProArcu jsem narazil na limit Java ImageIO.

Dostanu chybu

javax.imageio.IIOException: destination width * height * samplesPerPixel > Integer.MAX_VALUE

Příklad problematického souboru:

41174 × 29077 = 1 197 216 398 pixelů
41174 × 29077 × 3 = 3 591 649 194 vzorků

Je to na části skenovaná mapa formátu A0+, spojená Photoshopem do bigTIFF (6.84 GB), ale downsamplovaná z 16bit do 8bit RGB TIFu (3.6 GB).

Šlo mi to obejít generováním datastreamů FULL, PREVIEW a THUMBNAIL přes externí konvertor. VIPS.

Importní profil jsem použil:

# Import profile for very large map TIFFs.
# It forces external TIFF -> JPEG conversion through libvips to avoid Java ImageIO limits.

import.requiredDatastreamId=NDK_ARCHIVAL, NDK_USER
import.skippedDatastreamId=

import.tiff_to_jpg_convertor.processor=tiffToJpg
processor.tiffToJpg.type=gm
processor.tiffToJpg.exec=/opt/proarc-tools/tiff-to-jpg.sh
processor.tiffToJpg.arg=--proarc-placeholder
processor.tiffToJpg.timeout=3600000

import.ndk_archival.processor=grkMasterMap
processor.grkMasterMap.type=kdu_compress
processor.grkMasterMap.exec=/opt/proarc-tools/grk-master.sh
processor.grkMasterMap.arg=--proarc-placeholder
processor.grkMasterMap.environment.GRK_THREADS=1
processor.grkMasterMap.timeout=3600000

import.ndk_user.processor=grkUserMap
processor.grkUserMap.type=kdu_compress
processor.grkUserMap.exec=/opt/proarc-tools/grk-user.sh
processor.grkUserMap.arg=--proarc-placeholder
processor.grkUserMap.environment.GRK_THREADS=1
processor.grkUserMap.timeout=3600000

import.text_ocr.file.suffix=.ocr.txt
import.alto.file.suffix=.ocr.xml

import.image.preview.maxHeight=1000
import.image.thumbnail.maxWidth=120
import.image.thumbnail.maxHeight=128

Skript:

tiff-to-jpg.sh

Účel:

• převádí TIFF na JPEG pro FULL
• při argumentu -resize generuje zmenšený JPEG pro PREVIEW a THUMBNAIL
• obchází Java ImageIO limit u obrovských TIFFů
• používá vips thumbnail nebo vips copy

Celé znění:

#!/bin/sh
set -eu

resize=
positionals=

while [ "$#" -gt 0 ]; do
    case "$1" in
        --proarc-placeholder)
            ;;
        -resize)
            shift
            resize="${1:-}"
            ;;
        *)
            positionals="${positionals}
$1"
            ;;
    esac
    shift || true
done

input=$(printf '%s\n' "$positionals" | sed '/^$/d' | tail -n 2 | head -n 1)
output=$(printf '%s\n' "$positionals" | sed '/^$/d' | tail -n 1)

if [ -z "$input" ] || [ -z "$output" ]; then
    echo "Usage: tiff-to-jpg.sh [-resize pixels] input.tif output.jpg" >&2
    exit 2
fi

export VIPS_DISC_THRESHOLD="${VIPS_DISC_THRESHOLD:-100m}"
export VIPS_CONCURRENCY="${VIPS_CONCURRENCY:-2}"

if [ -n "$resize" ]; then
    resize=$(printf '%s' "$resize" | sed 's/x.*$//')
    exec vips thumbnail "$input" "$output[Q=90,strip,optimize_coding]" "$resize" --size down
fi

exec vips copy "$input" "$output[Q=90,strip,optimize_coding]"

Chtěl jsem se zeptat, jestli to jde vyřešit jinak. Tohle jsem zkoušel na testovací instanci 5.0.3, "chyba" je ale stejná / podobná pro jádro 4.x.x.
Nezkoušel jsem ještě tak velkou mapu exportovat a nahrát do Krameria, takže vlastně nevím, jestli má smysl to řešit, nebo je to otázka spíše akademická. Pokud s tím má někdo zkušenosti, neváhejte se mi ozvat. Takových map budu mít více, tak si nejsem jistý, jak k tomu přistoupit.

Předpokládám, že se jedná o limit Javy, ale nevím to :)

Validace JHOVE:

<?xml version="1.0" encoding="UTF-8"?>
<jhove xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schema.openpreservation.org/ois/xml/ns/jhove" xsi:schemaLocation="http://schema.openpreservation.org/ois/xml/ns/jhove https://schema.openpreservation.org/ois/xml/xsd/jhove/1.10/jhove.xsd" name="Jhove" release="1.34.0" date="2025-07-02">
 <date>2026-05-13T12:19:06+02:00</date>
 <repInfo uri="C:\temp\jhove\to-validate\mapa2_no_xmp_clean.tif">
  <reportingModule release="1.9.5" date="2024-08-22">TIFF-hul</reportingModule>
  <lastModified>2026-05-04T13:02:41+02:00</lastModified>
  <size>3591649652</size>
  <format>TIFF</format>
  <version>5.0</version>
  <status>Well-Formed and valid</status>
  <sigMatch>
  <module>TIFF-hul</module>
  </sigMatch>
  <mimeType>image/tiff</mimeType>
  <profiles>
   <profile>Baseline RGB (Class R)</profile>
   <profile>DLF Benchmark for Faithful Digital Reproductions of Monographs and Serials: color</profile>
  </profiles>
  <properties>
   <property>
    <name>TIFFMetadata</name>
    <values arity="Array" type="Property">
    <property>
     <name>ByteOrder</name>
     <values arity="Scalar" type="String">
      <value>little-endian</value>
     </values>
    </property>
    <property>
     <name>IFDs</name>
     <values arity="List" type="Property">
     <property>
      <name>Number</name>
      <values arity="Scalar" type="Integer">
       <value>1</value>
      </values>
     </property>
     <property>
      <name>IFD</name>
      <values arity="Array" type="Property">
      <property>
       <name>Offset</name>
       <values arity="Scalar" type="Long">
        <value>3591649202</value>
       </values>
      </property>
      <property>
       <name>Type</name>
       <values arity="Scalar" type="String">
        <value>TIFF</value>
       </values>
      </property>
      <property>
       <name>Entries</name>
       <values arity="List" type="Property">
       <property>
        <name>NisoImageMetadata</name>
        <values arity="Scalar" type="NISOImageMetadata">
         <value>
       <mix:mix xmlns:mix="http://www.loc.gov/mix/v20" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/mix/v20 http://www.loc.gov/standards/mix/mix20/mix20.xsd">
        <mix:BasicDigitalObjectInformation>
         <mix:ObjectIdentifier>
          <mix:objectIdentifierType>JHOVE</mix:objectIdentifierType>
         </mix:ObjectIdentifier>
         <mix:FormatDesignation>
          <mix:formatName>image/tiff</mix:formatName>
         </mix:FormatDesignation>
         <mix:byteOrder>little endian</mix:byteOrder>
         <mix:Compression>
          <mix:compressionScheme>Uncompressed</mix:compressionScheme>
         </mix:Compression>
        </mix:BasicDigitalObjectInformation>
        <mix:BasicImageInformation>
         <mix:BasicImageCharacteristics>
          <mix:imageWidth>41174</mix:imageWidth>
          <mix:imageHeight>29077</mix:imageHeight>
          <mix:PhotometricInterpretation>
           <mix:colorSpace>RGB</mix:colorSpace>
           <mix:ReferenceBlackWhite>
            <mix:Component>
             <mix:componentPhotometricInterpretation>R</mix:componentPhotometricInterpretation>
             <mix:footroom>
              <mix:numerator>0</mix:numerator>
             </mix:footroom>
             <mix:headroom>
              <mix:numerator>255</mix:numerator>
             </mix:headroom>
             </mix:Component>
            <mix:Component>
             <mix:componentPhotometricInterpretation>G</mix:componentPhotometricInterpretation>
             <mix:footroom>
              <mix:numerator>0</mix:numerator>
             </mix:footroom>
             <mix:headroom>
              <mix:numerator>255</mix:numerator>
             </mix:headroom>
             </mix:Component>
            <mix:Component>
             <mix:componentPhotometricInterpretation>B</mix:componentPhotometricInterpretation>
             <mix:footroom>
              <mix:numerator>0</mix:numerator>
             </mix:footroom>
             <mix:headroom>
              <mix:numerator>255</mix:numerator>
             </mix:headroom>
             </mix:Component>
            </mix:ReferenceBlackWhite>
          </mix:PhotometricInterpretation>
         </mix:BasicImageCharacteristics>
        </mix:BasicImageInformation>
        <mix:ImageCaptureMetadata>
         <mix:ScannerCapture>
          <mix:ScanningSystemSoftware>
           <mix:scanningSoftwareName>IrfanView</mix:scanningSoftwareName>
          </mix:ScanningSystemSoftware>
         </mix:ScannerCapture>
         <mix:orientation>normal*</mix:orientation>
        </mix:ImageCaptureMetadata>
        <mix:ImageAssessmentMetadata>
         <mix:SpatialMetrics>
          <mix:samplingFrequencyUnit>in.</mix:samplingFrequencyUnit>
          <mix:xSamplingFrequency>
           <mix:numerator>600</mix:numerator>
          </mix:xSamplingFrequency>
          <mix:ySamplingFrequency>
           <mix:numerator>600</mix:numerator>
          </mix:ySamplingFrequency>
         </mix:SpatialMetrics>
         <mix:ImageColorEncoding>
          <mix:BitsPerSample>
           <mix:bitsPerSampleValue>8</mix:bitsPerSampleValue>
           <mix:bitsPerSampleValue>8</mix:bitsPerSampleValue>
           <mix:bitsPerSampleValue>8</mix:bitsPerSampleValue>
           <mix:bitsPerSampleUnit>integer</mix:bitsPerSampleUnit>
          </mix:BitsPerSample>
          <mix:samplesPerPixel>3</mix:samplesPerPixel>
         </mix:ImageColorEncoding>
        </mix:ImageAssessmentMetadata>
       </mix:mix>
         </value>
        </values>
       </property>
       <property>
        <name>NewSubfileType</name>
        <values arity="Scalar" type="Long">
         <value>0</value>
        </values>
       </property>
       <property>
        <name>SampleFormat</name>
        <values arity="Array" type="Integer">
         <value>1</value>
         <value>1</value>
         <value>1</value>
        </values>
       </property>
       <property>
        <name>MinSampleValue</name>
        <values arity="Array" type="Integer">
         <value>0</value>
         <value>0</value>
         <value>0</value>
        </values>
       </property>
       <property>
        <name>MaxSampleValue</name>
        <values arity="Array" type="Integer">
         <value>255</value>
         <value>255</value>
         <value>255</value>
        </values>
       </property>
       <property>
        <name>TransferRange</name>
        <values arity="Array" type="Integer">
         <value>0</value>
         <value>255</value>
         <value>0</value>
         <value>255</value>
         <value>0</value>
         <value>255</value>
        </values>
       </property>
       <property>
        <name>Threshholding</name>
        <values arity="Scalar" type="Integer">
         <value>1</value>
        </values>
       </property>
       <property>
        <name>StripOffsets</name>
        <values arity="Array" type="Long">
         <value>8</value>
         <value>126486536</value>
         <value>252973064</value>
         <value>379459592</value>
         <value>505946120</value>
         <value>632432648</value>
         <value>758919176</value>
         <value>885405704</value>
         <value>1011892232</value>
         <value>1138378760</value>
         <value>1264865288</value>
         <value>1391351816</value>
         <value>1517838344</value>
         <value>1644324872</value>
         <value>1770811400</value>
         <value>1897297928</value>
         <value>2023784456</value>
         <value>2150270984</value>
         <value>2276757512</value>
         <value>2403244040</value>
         <value>2529730568</value>
         <value>2656217096</value>
         <value>2782703624</value>
         <value>2909190152</value>
         <value>3035676680</value>
         <value>3162163208</value>
         <value>3288649736</value>
         <value>3415136264</value>
         <value>3541622792</value>
        </values>
       </property>
       <property>
        <name>RowsPerStrip</name>
        <values arity="Scalar" type="Long">
         <value>1024</value>
        </values>
       </property>
       <property>
        <name>StripByteCounts</name>
        <values arity="Array" type="Long">
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>126486528</value>
         <value>50026410</value>
        </values>
       </property>
       <property>
        <name>PlanarConfiguration</name>
        <values arity="Scalar" type="Integer">
         <value>1</value>
        </values>
       </property>
       <property>
        <name>TIFFITProperties</name>
        <values arity="List" type="Property">
        <property>
         <name>BackgroundColorIndicator</name>
         <values arity="Scalar" type="String">
          <value>background not defined</value>
         </values>
        </property>
        <property>
         <name>ImageColorIndicator</name>
         <values arity="Scalar" type="String">
          <value>image not defined</value>
         </values>
        </property>
        <property>
         <name>TransparencyIndicator</name>
         <values arity="Scalar" type="String">
          <value>no transparency</value>
         </values>
        </property>
        <property>
         <name>PixelIntensityRange</name>
         <values arity="Array" type="Integer">
          <value>0</value>
          <value>255</value>
         </values>
        </property>
        <property>
         <name>RasterPadding</name>
         <values arity="Scalar" type="String">
          <value>1 byte</value>
         </values>
        </property>
        <property>
         <name>BitsPerRunLength</name>
         <values arity="Scalar" type="Integer">
          <value>8</value>
         </values>
        </property>
        <property>
         <name>BitsPerExtendedRunLength</name>
         <values arity="Scalar" type="Integer">
          <value>16</value>
         </values>
        </property>
        </values>
       </property>
       </values>
      </property>
      </values>
     </property>
     </values>
    </property>
    </values>
   </property>
  </properties>
 </repInfo>
</jhove>

exiftool:

PS D:\exiftool-13.55_64> .\exiftool.exe -G1 -a -s -u -ee .\mapa2_no_xmp_clean.tif
[ExifTool]      ExifToolVersion                 : 13.55
[System]        FileName                        : mapa2_no_xmp_clean.tif
[System]        Directory                       : .
[System]        FileSize                        : 3.6 GB
[System]        FileModifyDate                  : 2026:05:04 13:02:41+02:00
[System]        FileAccessDate                  : 2026:05:13 12:36:41+02:00
[System]        FileCreateDate                  : 2026:05:13 12:35:55+02:00
[System]        FilePermissions                 : -rw-rw-rw-
[File]          FileType                        : TIFF
[File]          FileTypeExtension               : tif
[File]          MIMEType                        : image/tiff
[File]          ExifByteOrder                   : Little-endian (Intel, II)
[IFD0]          ImageWidth                      : 41174
[IFD0]          ImageHeight                     : 29077
[IFD0]          BitsPerSample                   : 8 8 8
[IFD0]          Compression                     : Uncompressed
[IFD0]          PhotometricInterpretation       : RGB
[IFD0]          StripOffsets                    : (Binary data 302 bytes, use -b option to extract)
[IFD0]          Orientation                     : Horizontal (normal)
[IFD0]          SamplesPerPixel                 : 3
[IFD0]          RowsPerStrip                    : 1024
[IFD0]          StripByteCounts                 : (Binary data 288 bytes, use -b option to extract)
[IFD0]          XResolution                     : 600
[IFD0]          YResolution                     : 600
[IFD0]          PlanarConfiguration             : Chunky
[IFD0]          ResolutionUnit                  : inches
[IFD0]          Software                        : IrfanView
[Composite]     ImageSize                       : 41174x29077
[Composite]     Megapixels                      : 1197.2

[SykoraLukas]

Ano, zdá se, že se jedná o limit JAVY. Mohu ale raději poprosit o zaslání souboru na email lukas.sykora@inovatika.cz, abych to prověřil a případně zjistil, jestli má nějaké řešení bez dalších programů?

[bezverec]

zkusil jsem to tedy dokončit včetně nahrání do Krameria (import NDK PSP), zde je, myslím si, úspěšný výsledek: https://kramerius.vkol.cz/uuid/uuid:af1432c5-73af-4fcd-b151-891fe8def3eb

[bezverec]

Jen dodám, že jsem pro úspěšný export do NDK PSP upravoval importní profil i skript, což není důležité, ale abych byl transparentní, přidávám... A myslím, že některý z náhledů i tak byl pro načtení v ProArcu moc velký, musel bych ještě změnit trochu VIPS parametry.

Jinak mi tam samozřejmě chybí zadní strana (nešla mi spojit) a z pohledu prezerevace si nechávám hlavně bigTIFF a původní naskenované části.

Aby mi nespadla konverze TIFF -> JP2, omezil jsem Grok na 1 vlákno.

Netuším, jak funguje Java ImageIO, takže to jsem se řešit nepokoušel. Není to asi úplně nejčastější use case (gigapixelové skeny), ale pokud by to libovolným způsobem šlo vyřešit, určitě jsem pro :)

importní profil

# Import profile for very large map TIFFs.
# It forces external TIFF -> JPEG conversion through libvips to avoid Java ImageIO limits.

# The tiff-to-jpg wrapper creates empty OCR/ALTO placeholders before ProArc
# checks OCR siblings, so PSP export has mandatory TXTGRP/ALTOGRP streams.
import.requiredDatastreamId=ALTO, NDK_ARCHIVAL, NDK_USER, TEXT_OCR
import.skippedDatastreamId=

import.tiff_to_jpg_convertor.processor=tiffToJpg
processor.tiffToJpg.type=gm
processor.tiffToJpg.exec=/opt/proarc-tools/tiff-to-jpg.sh
processor.tiffToJpg.arg=--proarc-placeholder
processor.tiffToJpg.environment.PROARC_CREATE_EMPTY_OCR=1
processor.tiffToJpg.timeout=3600000

import.ndk_archival.processor=grkMasterMap
processor.grkMasterMap.type=kdu_compress
processor.grkMasterMap.exec=/opt/proarc-tools/grk-master.sh
processor.grkMasterMap.arg=--proarc-placeholder
processor.grkMasterMap.environment.GRK_THREADS=1
processor.grkMasterMap.timeout=3600000

import.ndk_user.processor=grkUserMap
processor.grkUserMap.type=kdu_compress
processor.grkUserMap.exec=/opt/proarc-tools/grk-user.sh
processor.grkUserMap.arg=--proarc-placeholder
processor.grkUserMap.environment.GRK_THREADS=1
processor.grkUserMap.timeout=3600000

import.text_ocr.file.suffix=.ocr.txt
import.alto.file.suffix=.ocr.xml

import.image.preview.maxHeight=1000
import.image.thumbnail.maxWidth=120
import.image.thumbnail.maxHeight=128

skript

#!/bin/sh
set -eu

resize=
positionals=

while [ "$#" -gt 0 ]; do
    case "$1" in
        --proarc-placeholder)
            ;;
        -resize)
            shift
            resize="${1:-}"
            ;;
        *)
            positionals="${positionals}
$1"
            ;;
    esac
    shift || true
done

input=$(printf '%s\n' "$positionals" | sed '/^$/d' | tail -n 2 | head -n 1)
output=$(printf '%s\n' "$positionals" | sed '/^$/d' | tail -n 1)

if [ -z "$input" ] || [ -z "$output" ]; then
    echo "Usage: tiff-to-jpg.sh [-resize pixels] input.tif output.jpg" >&2
    exit 2
fi

create_empty_ocr() {
    if [ "${PROARC_CREATE_EMPTY_OCR:-0}" != "1" ]; then
        return 0
    fi

    base="${input%.*}"
    txt="${base}.ocr.txt"
    alto="${base}.ocr.xml"

    if [ -s "$txt" ] && [ -s "$alto" ]; then
        return 0
    fi

    width=$(vipsheader -f width "$input")
    height=$(vipsheader -f height "$input")
    file_name=$(basename "$input" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g; s/"/\&quot;/g')
    page_id=$(basename "$base" | sed 's/[^A-Za-z0-9_-]/_/g')
    processing_time=$(date -u '+%Y-%m-%dT%H:%M:%SZ')

    if [ ! -s "$txt" ]; then
        printf '\n' > "$txt"
    fi

    if [ ! -s "$alto" ]; then
        cat > "$alto" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<alto xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.loc.gov/standards/alto/ns-v2#">
  <Description>
    <MeasurementUnit>pixel</MeasurementUnit>
    <sourceImageInformation>
      <fileName>${file_name}</fileName>
    </sourceImageInformation>
    <OCRProcessing ID="IdOcr">
      <ocrProcessingStep>
        <processingDateTime>${processing_time}</processingDateTime>
        <processingSoftware>
          <softwareCreator>ProArc</softwareCreator>
          <softwareName>Empty OCR placeholder</softwareName>
          <softwareVersion>1.0</softwareVersion>
        </processingSoftware>
      </ocrProcessingStep>
    </OCRProcessing>
  </Description>
  <Layout>
    <Page ID="id_${page_id}" PHYSICAL_IMG_NR="1" HEIGHT="${height}" WIDTH="${width}">
      <PrintSpace HEIGHT="${height}" WIDTH="${width}" VPOS="0" HPOS="0"/>
    </Page>
  </Layout>
</alto>
EOF
    fi

    echo "tiff-to-jpg: empty OCR/ALTO placeholders ready txt=$txt alto=$alto"
}

export VIPS_DISC_THRESHOLD="${VIPS_DISC_THRESHOLD:-100m}"
export VIPS_CONCURRENCY="${VIPS_CONCURRENCY:-2}"

create_empty_ocr

if [ -n "$resize" ]; then
    resize=$(printf '%s' "$resize" | sed 's/x.*$//')
    exec vips thumbnail "$input" "$output[Q=90,strip,optimize_coding]" "$resize" --size down
fi

exec vips copy "$input" "$output[Q=90,strip,optimize_coding]"

[SykoraLukas]

Doplněna podpora pro velké tify (zkoušeno na 2 obrázcích - 3,3 GB A 5,2 GB). Oba už procházejí. Od verze 5.0.5

[bezverec]

Díky. Teď už mi to funguje i bez workaroundů a sloppy fixů. :)

---

Padal mi naopak Grok, ale to není problém Java Image IO. Nicméně to jsem vyřešil přiřazením swapu.

sudo btrfs filesystem mkswapfile --size 64g --uuid clear /swapfile-proarc
sudo swapon /swapfile-proarc
echo '/swapfile-proarc none swap defaults,pri=10 0 0' | sudo tee -a /etc/fstab
free -h
swapon --show