Address inability to OCR outside Latin/CJK#427
Open
JorjMcKie wants to merge 1 commit into
Open
Conversation
Out OCR support is restricted to (extended) Latin and CJK - for a number of reasons. This change tries to ensure that no OCR is ever recommended nor tried for pages that contain Unicodes outside these ranges. * If analyze_page detects "unsafe" characters, it will not recommend OCR * If an OCR plugin (RapidOCR and friends) either detects legible but unsafe Unicodes or recognizes unsafe Unicodes, the page will not be OCR'd.
| BLOCK_VECTOR = mupdf.FZ_STEXT_BLOCK_VECTOR | ||
| # Thresholds | ||
| BAD_CHAR_THRESHOLD = 0.05 # >=5% bad chars suggests OCR | ||
| BAD_CHAR_THRESHOLD = 0.10 # >=10% bad chars suggests OCR |
Collaborator
There was a problem hiding this comment.
This change seems unrelated to what this commit is about. Could we move it into a separate commit/pr?
|
|
||
| # --- final OCR decision --- | ||
| if unsafe_text_spans: | ||
| # If there are unsafe text spans, we recommend OCR. |
Collaborator
There was a problem hiding this comment.
Should this comment say "we do not recommend OCR"?
Comment on lines
-324
to
+343
| or chars_bad / chars_total > BAD_CHAR_THRESHOLD | ||
| or bad_areas / txt_area > BAD_CHAR_THRESHOLD | ||
| or (chars_bad / chars_total > BAD_CHAR_THRESHOLD | ||
| and bad_areas / txt_area > BAD_CHAR_THRESHOLD) |
Collaborator
There was a problem hiding this comment.
Can this be moved to a separate commit/pr?
Comment on lines
+123
to
+128
| if any(contains_unsafe(text) for _, text, _ in result): | ||
| # If any recognized text contains unsafe characters, we skip OCR | ||
| # entirely. This is because we have no way yet to make correct | ||
| # insertions of such text into the page. | ||
| return | ||
|
|
Collaborator
There was a problem hiding this comment.
Is this checking for bad characters in the output of the ocr engine? If so the comment seems misleading?
Is there a still a risk of crashing at this stage?
Comment on lines
+123
to
+128
| if any(contains_unsafe(text) for _, text, _ in result): | ||
| # If any recognized text contains unsafe characters, we skip OCR | ||
| # entirely. This is because we have no way yet to make correct | ||
| # insertions of such text into the page. | ||
| return | ||
|
|
Collaborator
There was a problem hiding this comment.
Similar question to above, about why we are worrying about bad characters in the output from the ocr engine.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Out OCR support is restricted to (extended) Latin and CJK - for a number of reasons. This change tries to ensure that no OCR is ever recommended nor tried for pages that contain Unicodes outside these ranges.