DECIMER Detection Threshold Modification Issue

### Issue Type

Performance

### Source

GitHub (source)

### DECIMER Image Transformer Version

v2.x (Mask R-CNN based) 

### OS Platform and Distribution

_No response_

### Python version

3.9, TensorFlow 2.x 

### Current Behaviour?


# Technical Report: DECIMER Detection Threshold Modification Issue

**Date:** October 28, 2025  
**System:** DECIMER-Image_Transformer v2.x (Mask R-CNN based)  
**Python:** 3.9, TensorFlow 2.x  
**Environment:** structure_pipeline conda env  
**Issue:** Unable to lower detection threshold below inherent model limit  
**Objective:** Increase recall from 12/13 to 13/13 structures on test image

---

## Problem Statement

We are attempting to increase DECIMER's recall for chemical structure detection on textbook pages. Despite successfully modifying multiple configuration parameters at both file-level and runtime, the detection count remains fixed at 12 structures. We seek guidance on accessing lower-level inference parameters or alternative approaches.

**Test Case:**
- **Image:** Klein Organic Chemistry textbook page (page_0501.jpg)
- **Ground Truth:** 13 chemical structures (manually verified)
- **DECIMER Output:** Consistently 12 structures (missing 1 small structure)
- **Lowest Score Detected:** 0.520 (well above our attempted threshold of 0.3)

---

## System Configuration

### Installation
```bash
pip install decimer-segmentation
# Version: Latest available on PyPI
# Dependencies: tensorflow, keras, opencv-python
```

### Model Location
```
/home/gert/miniconda3/envs/structure_pipeline/lib/python3.9/site-packages/
├── decimer_segmentation/
│   ├── __init__.py (loads pre-trained model)
│   ├── mrcnn/
│   │   ├── config.py (InferenceConfig class)
│   │   └── model.py (Mask R-CNN implementation)
│   └── [pre-trained weights embedded]
```

### Config File Path
```
/home/gert/miniconda3/envs/structure_pipeline/lib/python3.9/site-packages/
  decimer_segmentation/mrcnn/config.py
```

---

## Approaches Attempted

### Attempt 1: Direct Config File Modification

**Method:**
```python
# Edited config.py line 176
DETECTION_MIN_CONFIDENCE = 0.3  # Originally 0.7
```

**Result:** ❌ No change  
**Python Restart:** Yes (new process started)  
**Structures Detected:** 12 (unchanged)

---

### Attempt 2: Runtime Config Manipulation

**Method:**
```python
import decimer_segmentation

model = decimer_segmentation.model
config = model.config

# Verify access
print(config.DETECTION_MIN_CONFIDENCE)  # Outputs: 0.5 (from file edit)

# Modify at runtime
config.DETECTION_MIN_CONFIDENCE = 0.3
print(config.DETECTION_MIN_CONFIDENCE)  # Outputs: 0.3 (confirmed changed)

# Run detection
results = get_mrcnn_results(image)
```

**Result:** ❌ No change  
**Config Value Verified:** 0.3 (successfully modified)  
**Structures Detected:** 12 (unchanged)

**Conclusion:** Config object modified, but changes not reflected in inference.

---

### Attempt 3: Module Reload with Config Pre-modification

**Method:**
```python
# Step 1: Modify config file
with open(config_path, 'w') as f:
    f.write('DETECTION_MIN_CONFIDENCE = 0.3')

# Step 2: Remove all loaded modules
for mod in list(sys.modules.keys()):
    if 'decimer' in mod.lower() or 'mrcnn' in mod.lower():
        del sys.modules[mod]

# Step 3: Import fresh
from decimer_segmentation import get_mrcnn_results
```

**Result:** ❌ No change  
**Verification:** Module successfully reloaded (import timestamp changed)  
**Structures Detected:** 12 (unchanged)

---

### Attempt 4: RPN Parameter Adjustment

**Method:**
```python
config.RPN_NMS_THRESHOLD = 0.5        # From 0.7
config.PRE_NMS_LIMIT = 10000          # From 6000
config.POST_NMS_ROIS_INFERENCE = 3000 # From 1000
config.DETECTION_MIN_CONFIDENCE = 0.3 # From 0.7
```

**Available Parameters Modified:**
```
DETECTION_MIN_CONFIDENCE     0.7 → 0.3
RPN_NMS_THRESHOLD           0.7 → 0.5
PRE_NMS_LIMIT               6000 → 10000
POST_NMS_ROIS_INFERENCE     1000 → 3000
DETECTION_NMS_THRESHOLD     0.3 (unchanged)
```

**Result:** ❌ No change  
**Structures Detected:** 12 (unchanged)

---

## Detailed Findings

### Detection Score Distribution

All attempts produced identical score distribution:

| Structure | Score | Notes |
|-----------|-------|-------|
| 1-6 | 1.000 | High-confidence structures |
| 7 | 0.998 | |
| 8 | 0.998 | |
| 9 | 0.996 | |
| 10 | 0.993 | |
| 11 | 0.948 | |
| 12 | 0.520 | Lowest score detected |
| **13** | **Not detected** | **Target structure missing** |

**Critical Observation:** 
- Lowest detected score is 0.520 (significantly above 0.3 threshold)
- No structures detected in the 0.3-0.5 range
- Suggests filtering occurs **before** `get_mrcnn_results()` returns

### Code Path Analysis
```python
# User calls:
get_mrcnn_results(image)
  ↓
# Which executes:
results = model.detect([image], verbose=1)
  ↓
# model.detect() is a TensorFlow/Keras graph operation
# Threshold filtering happens HERE (not accessible from Python)
  ↓
# Returns filtered results
scores = results[0]["scores"]  # Already filtered
bboxes = results[0]["rois"]    # Already filtered
```

**Hypothesis:** The `model.detect()` method applies thresholds during graph execution, making post-loading config changes ineffective.

---

## Available Config Attributes (InferenceConfig)

Successfully accessed attributes (via `dir(config)`):
```python
# Detection parameters
DETECTION_MIN_CONFIDENCE     = 0.3  (modified, not applied)
DETECTION_NMS_THRESHOLD      = 0.3  (modified, not applied)
DETECTION_MAX_INSTANCES      = 100

# RPN parameters
RPN_NMS_THRESHOLD           = 0.5  (modified, not applied)
PRE_NMS_LIMIT               = 10000 (modified, not applied)
POST_NMS_ROIS_INFERENCE     = 3000  (modified, not applied)
POST_NMS_ROIS_TRAINING      = 2000  (read-only, not used in inference)

# Other parameters
IMAGE_MIN_DIM, IMAGE_MAX_DIM, etc.
```

**Note:** All threshold modifications were verified (printed values confirmed), but did not affect detection output.

---

## Theories on Root Cause

### Theory 1: TensorFlow Graph Compilation ⭐ Most Likely

The Mask R-CNN model is a compiled TensorFlow graph. Threshold values may be:
- **Hard-coded in the graph** during initial model creation
- **Compiled into TensorFlow operations** (tf.where, tf.gather based on threshold)
- **Frozen during SavedModel export**

**Evidence:**
- Config changes verified but not applied
- Identical behavior across all approaches
- `model.detect()` is a tf.function or Keras call

**Solution if True:**
Requires rebuilding the model graph or accessing intermediate layer outputs.

### Theory 2: RPN Objectness Filtering

The Region Proposal Network generates proposals with objectness scores. If RPN was trained conservatively or has an implicit threshold, proposals for the 13th structure may never reach the detection head.

**Evidence:**
- Score gap: 0.520 (detected) → nothing between 0.3-0.5 → 0.0 (not detected)
- Suggests binary decision at RPN stage

**Solution if True:**
Modify RPN layer directly or retrain with softer objectness criterion.

### Theory 3: Training Data Bias

The pre-trained model may have learned to assign low confidence to structures that:
- Are very small (<50px width)
- Have faint lines
- Overlap with text
- Are at page edges

**Evidence:**
- The missing 13th structure is small and near text
- Model consistently scores it below detection threshold

**Solution if True:**
Fine-tune on textbook pages with small/challenging structures.

---

## Questions for DECIMER Developers / Community

### Critical Questions

1. **Is DETECTION_MIN_CONFIDENCE applied during TensorFlow graph execution?**
   - If yes, how can we modify it post-model-loading?
   - Does it require recompiling the model graph?

2. **Can we access raw RPN proposals before filtering?**
   - E.g., `model.rpn.predict()` to get all proposals with objectness scores
   - Would allow manual filtering with custom thresholds

3. **Is there an RPN objectness threshold separate from DETECTION_MIN_CONFIDENCE?**
   - Our modifications to `RPN_NMS_THRESHOLD` had no effect
   - Is there a hidden `RPN_OBJECTNESS_THRESHOLD` parameter?

4. **Does `model.detect()` accept runtime threshold parameters?**
```python
   # Something like:
   results = model.detect([image], 
                         min_confidence=0.3,  # Runtime override
                         max_instances=150)
```

5. **Can we access intermediate layer outputs?**
```python
   # Get detection head scores BEFORE confidence filtering
   intermediate_model = Model(inputs=model.input,
                             outputs=model.get_layer('mrcnn_class').output)
   all_scores = intermediate_model.predict(image)
```

### Setup Information

**DECIMER Installation:**
```bash
pip install decimer-segmentation
# No custom compilation flags
# Using pre-built PyPI package
```

**TensorFlow Version:**
```bash
python -c "import tensorflow as tf; print(tf.__version__)"
# Output: 2.x (with CUDA disabled warnings ignored)
```

**Model Loading:**
```python
# Automatic on import
from decimer_segmentation import get_mrcnn_results
# Model loaded from package's embedded weights
```

---

## Attempted Workarounds (Did Not Work)

### ❌ Modify config.py before import
**Result:** No effect (model loads with cached weights)

### ❌ Runtime config.DETECTION_MIN_CONFIDENCE = 0.3
**Result:** Config changed, output unchanged

### ❌ Delete sys.modules and reimport
**Result:** Module reloaded, output unchanged

### ❌ Increase PRE_NMS_LIMIT, POST_NMS_ROIS_INFERENCE
**Result:** Config changed, output unchanged

### ❌ Lower RPN_NMS_THRESHOLD to 0.5
**Result:** Config changed, output unchanged

---

## Desired Outcome

**Goal:** Detect the 13th structure (currently missed) which is:
- Small (~100x50 pixels)
- Near text region
- Likely assigned score between 0.3-0.5 by the model

**Acceptable Solutions:**

1. ✅ **Modify existing model** to output lower-confidence detections
2. ✅ **Access RPN proposals** directly for manual filtering  
3. ✅ **Rebuild model graph** with new thresholds (if necessary)
4. ✅ **Fine-tune model** on similar structures (longer-term)

---

## Files Available for Sharing

If helpful for debugging, we can provide:

1. **Test image:** `page_0501.jpg` (2890x3698px, chemistry textbook page)
2. **Ground truth annotations:** 13 structure bounding boxes
3. **Current detection output:** JSON with 12 structures + scores
4. **Modified config.py:** Our edited version with DETECTION_MIN_CONFIDENCE=0.3
5. **Full Python environment:** conda environment export

---

## System Details
```
OS: Ubuntu 24.04
Python: 3.9.x
Conda Environment: structure_pipeline
CUDA: Available but disabled (CPU inference acceptable)
RAM: Sufficient (16GB+)
Storage: SSD

Relevant Packages:
- decimer-segmentation (latest PyPI)
- tensorflow 2.x
- opencv-python
- pillow
- numpy
```

---

## Request for Assistance

We have exhausted standard configuration modification approaches. We seek guidance on:

1. How to access lower-level inference parameters in DECIMER
2. Whether post-loading threshold modification is possible
3. Alternative methods to increase recall without retraining
4. Contact information for DECIMER maintainers if this requires deeper investigation

**This is not a bug report** - we understand the model may be working as designed. We are seeking advice on accessing advanced configuration options or modifying inference behavior for our specific use case (high-recall chemical structure extraction from textbook pages).

---

## Appendix: Reproducible Test Case
```python
import sys
sys.path.insert(0, '/path/to/DECIMER-Image_Transformer')

from PIL import Image
import numpy as np
from decimer_segmentation import get_mrcnn_results
import decimer_segmentation

# Modify config
model = decimer_segmentation.model
config = model.config
config.DETECTION_MIN_CONFIDENCE = 0.3

# Verify change
print(f"Config value: {config.DETECTION_MIN_CONFIDENCE}")  # Prints: 0.3

# Run detection
img = np.array(Image.open('page_0501.jpg'))
masks, bboxes, scores = get_mrcnn_results(img)

# Check results
print(f"Structures detected: {len(bboxes)}")  # Always prints: 12
print(f"Lowest score: {min(scores)}")         # Always prints: 0.520

# Expected: More detections (13+) with scores down to 0.3
# Actual: 12 detections, lowest score 0.520
```

---

**Thank you for any guidance you can provide!**

---

## Contact

For follow-up questions or to provide the test image/environment, please contact via:
- GitHub Issue: [Would open on DECIMER repository]
- Email: [Contact information if provided]

---

**Generated:** October 28, 2025  
**Report Version:** 2.0 (Final Investigation Results)
EOF

echo ""
echo "✅ Verslag opgedateer: DECIMER_THRESHOLD_INVESTIGATION_REPORT.md"
echo ""
echo "📧 Hierdie verslag is gereed om te deel met:"
echo "   - DECIMER GitHub: https://github.com/Kohulan/DECIMER-Image_Transformer/issues"
echo "   - Mask R-CNN experts"
echo "   - TensorFlow community forums"
echo ""
echo "📎 Sluit in:"
echo "   - Die verslag (DECIMER_THRESHOLD_INVESTIGATION_REPORT.md)"
echo "   - Die toets beeld (page_0501.jpg)"
echo "   - Jou conda environment details"

### Which images caused the issue? (This is mandatory for images related issues)

_No response_

### Standalone code to reproduce the issue

```shell
I want to change the settings so that it extracts structures more aggressively.  There are 13 structures, it only finds 12?
```

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Structure	Score	Notes
1-6	1.000	High-confidence structures
7	0.998
8	0.998
9	0.996
10	0.993
11	0.948
12	0.520	Lowest score detected
13	Not detected	Target structure missing

DECIMER Detection Threshold Modification Issue #131

Description

Issue Type

Source

DECIMER Image Transformer Version

OS Platform and Distribution

Python version

Current Behaviour?

Technical Report: DECIMER Detection Threshold Modification Issue

Problem Statement

System Configuration

Installation

Model Location

Config File Path

Approaches Attempted

Attempt 1: Direct Config File Modification

Attempt 2: Runtime Config Manipulation

Attempt 3: Module Reload with Config Pre-modification

Attempt 4: RPN Parameter Adjustment

Detailed Findings

Detection Score Distribution

Code Path Analysis

Available Config Attributes (InferenceConfig)

Theories on Root Cause

Theory 1: TensorFlow Graph Compilation ⭐ Most Likely

Theory 2: RPN Objectness Filtering

Theory 3: Training Data Bias

Questions for DECIMER Developers / Community

Critical Questions

Setup Information

Attempted Workarounds (Did Not Work)

❌ Modify config.py before import

❌ Runtime config.DETECTION_MIN_CONFIDENCE = 0.3

❌ Delete sys.modules and reimport

❌ Increase PRE_NMS_LIMIT, POST_NMS_ROIS_INFERENCE

❌ Lower RPN_NMS_THRESHOLD to 0.5

Desired Outcome

Files Available for Sharing

System Details

Request for Assistance

Appendix: Reproducible Test Case

Contact

Which images caused the issue? (This is mandatory for images related issues)

Standalone code to reproduce the issue

Relevant log output

Code of Conduct

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions