Undesired texts are removed when applying redaction

[hieu-lam]

Hi,

I have a problem with Pymupdf redaction. The issue is basically some of the text around the annotated part is also deleted when apply redaction. You can see the example below:

After redaction, it will become:

I follow the instruction to use fitz.TOOLS.set_small_glyph_heights(on=True) and get the following result:

This time the annotated part has been smaller, but still the result is not as I expected:

I get the quads annotations from page.search_for() and add the redaction with page.add_redact_annot().

The code is as below:

page = doc.loadPage(0)
annot  = page.firstAnnot 
while annot:
    page.add_redact_annot(annot.rect, text=None, fontname=None, fontsize=11, align=0, fill=(1, 1, 1), text_color=(0, 0, 0))
    annot = annot.next 
page.apply_redactions()

Is there anyway to improve this?

And by the way, some of us use Adobe Reader tool to add the annotation manually. Those annotations could not be "squeeze", so the surrounding text is removed unwantedly.

The example below is an annotation made by Adobe Reader Comment tool.

On the Pymupdf it says: "Text removal is done by character: A character is removed if its bbox has a non-empty overlap with a redaction rectangle (changed in MuPDF v1.17)". I guess this is the main issue here. Is there anyway get around this with a fix?

Thanks for your help.

[JorjMcKie]

This is a very common thing:
The redact logic removes on a by-character level. Every character whose glyph bbox has a non-empty intersection with a redaction rectangle will be removed. The standard glyph bbox is controlled by the font's line height.
So depending on how the PDF was made - which line height was used - there may be non-empty overlaps between the lines, which leads to your problem.
But this is not an error - it works as designed.

What can you do?

1. Reduce the height of the redaction rect by brute force. E.g. build a sub-rect which has the same horizontal middle line, but only 80% of the total height of the original rectangle ... 🤔
2. Set a global parameter fitz.TOOLS.set_small_glyph_heights(True) - before you calculate the redaction rectangle. You probably do this via page.search_for(...), or by looking at the spans of text extraction, etc. All these methods respect that global parameter and deliver glyph heights which fit the actual glyph, i.e. no head room above that, etc.

Please consult the documentation here: https://pymupdf.readthedocs.io/en/latest/textpage.html#span-dictionary for a detailed description.

[JorjMcKie]

Oops, sorry I responded without reading your full post!
You already followed point 2. above.
The problem is, that character removal happens inside MuPDF code, which I can obviously not influence. That code of course knows nothing about my global parameter.
So I am afraid you must use option 1 above - as ugly as this may be.

[JorjMcKie]

Example calculation:

delta = fitz.Point(0, rect.height) * 0.2
new_rect = fitz.Rect(rect.tl + delta, rect.br - delta)

Then new_rect will have the same horizontal middle line and width as rect, but only 60% of rect's height.

[JorjMcKie]

An interesting question might be, how to detect this type of situation beforehand - so the remedy only needs to be applied in that case.

One way is to check whether line bboxes overlap. Maybe there is even a constant percentage of those overlaps wrt line height.
When detected, compute a line bbox with an "optimized" reduced height - even without setting the global glyph height reduction - and take each annot rectangle's intersection with that reduced line bbox for the redaction annotation.