SAR2Optical

Overview

This is a Pix2Pix CGAN implementation for translating Synthetic Aperture Radar (SAR) images to optical images.

The demo: yuulind/sar2rgb

Table of Contents

• Dataset
• Results & Model Files
• Loss
• Installation
• Usage
• Models
• License
• Acknowledgments

Dataset

For this project, paired SAR and optical (RGB) images from the Sentinel‑1 and Sentinel‑2 satellites are used to train the models. The dataset source is Sentinel-1&2 Image Pairs, Michael Schmitt, Technical University of Munich (TUM). The dataset is available on Kaggle at Sentinel-1&2 Image Pairs (SAR & Optical), uploaded and curated by Paritosh Tiwari (@requiemonk).

The dataset is divided into three splits: training, validation, and testing. We randomly sampled 1,600 image pairs for validation, and another 1,600 pairs were allocated for test. The remaining 12,800 image pairs were used for training. All splits have similar category distributions. The IDs for each split can be found in the split.json.

1. Download dataset by running the command below.

   python utils/data_downloader.py

   • More information on How to set up your API keys for Kaggle API

2. Use a predefined split via split.json

   • You simply need to modify two key in the config.yaml file

   ...
   
   # Dataset parameters
   dataset:
      ...
      
      split_mode: "split"
    
      ...
      
      split_file: "./data/split.json"
    
      ...
   
   ...
   

Please ensure your dataset is organized in the following directory structure:

/SAR2Optical/data
└───v_2
    ├───agri
    │   ├───s1
    │   └───s2
    ├───barrenland
    │   ├───s1
    │   └───s2
    ├───grassland
    │   ├───s1
    │   └───s2
    └───urban
        ├───s1
        └───s2

If you have used the data_downloader.py, it will be downloaded automatically in this structure.

Results & Model Files

• BASELINE FID SCORE is between 30 and 40
• Baseline score is calculated by randomly sampling two sets from real images and comparing the them.
• Trained model files are uploaded to huggingface hub and kaggle.

Number	Name	Model Type	Description	Link to Model File
1	pix2pix_gen_180.pth	Generator	Pix2Pix generator with transpose convolution, 180 epochs. FID score is 185.04	pix2pix_gen_180.pth
2	pix2pix_gen_265.pth	Generator	Pix2Pix generator with transpose convolution, 265 epochs. FID score is 189.81	pix2pix_gen_265.pth
3	pix2pix_gen_295.pth	Generator	Pix2Pix generator with transpose convolution, 295 epochs. FID score is 187.73	pix2pix_gen_295.pth
4	pix2pix_disc_180.pth	Discriminator	Pix2Pix discriminator from epoch 180, with transpose convolution generator.	pix2pix_disc_180.pth
5	pix2pix_disc_265.pth	Discriminator	Pix2Pix discriminator from epoch 265, with transpose convolution generator.	pix2pix_disc_265.pth
6	pix2pix_disc_295.pth	Discriminator	Pix2Pix discriminator from epoch 295, with transpose convolution generator.	pix2pix_disc_295.pth

Loss Graphs

• Discriminator vs Epoch
• Generator Overall Loss vs Epoch
• Generator GAN Loss vs Epoch
• Generator L1 Loss vs Epoch

Installation

1. Install uv (if you haven't already (Seriously, it's such a nice tool (just use uv))):

   curl -LsSf https://astral.sh/uv/install.sh | sh
   # Or on Windows: powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

   For more information on uv please check Astral.uv

2. Clone the repository:

   https://github.com/yuuIind/SAR2Optical.git
   cd SAR2Optical

3. Install dependencies using uv: uv will automatically create a virtual environment (.venv) and install the packages. You can customize the installation based on your hardware and what you plan to do:

   For Inference only (Default):

   uv sync

   (Note: This automatically installs the CPU version of PyTorch).

   • CUDA version: Add cuda version if your hardware supports:

     • --extra cpu: Installs the pytorch for CPU (this is the default mode)
     • --extra cu128: Installs the pytorch with CUDA 12.8
     • --extra cu130: Installs the pytorch with CUDA 13.0

     Example (Installing for cuda 12.8):

     uv sync --extra cu128

   • Specific Workflows: Add groups based on what you want to do:

     • --group data: Installs the Kaggle API for downloading the Sentinel dataset (utils/data_downloader.py).
     • --group train: Installs comet-ml, scipy, and tqdm for training models and tracking experiments (train.py).
     • --group export: Installs onnx and onnxruntime for exporting the model and running ONNX inference (torch2onnx.py).

     Example (Installing just train and data dependencies):

     uv sync --group train --group data

   For full Setup (Training, Data downloading, and ONNX Export):

   uv sync --all-groups

   Explicit Hardware Selection (Optional): If you want to explicitly force a CPU or CUDA 12.8 installation, append the --extra flag:

   uv sync --all-groups --extra cpu
   # OR
   uv sync --all-groups --extra cu128

4. Activate the environment:

   source .venv/bin/activate  # On Windows use `.venv\Scripts\activate`

   (Alternatively, you can prefix any command with uv run, e.g., uv run python train.py, without needing to activate the environment).

Usage

Prepare data

Before training, you need to prepare the dataset. Check Dataset section for more information on how to find the dataset. (Requires the data dependency group: uv sync --group data)

Train

To train the model, run the following command: (Requires the train dependency group: uv sync --group train)

python train.py

Evaluate

To evaluate the model’s performance, you can use metrics such as PSNR and SSIM. Example command:

python test.py

Inference

To infer on a single image, please run the following command after changing config accordingly. Example command:

   inference:
      image_path: "./data/imgs/sample.jpg"  # path to single image for inference
      output_path: "./output/sample_output.jpg"  # directory to save output images
      gen_checkpoint: "./models/checkpoints/pix2pix_gen_X.pth"  # path to generator checkpoint
      device: "cpu"  # or "cuda" or "cuda:0" for specific GPU

python inference.py

ONNX Export

You can export to ONNX with torch2onnx.py. (Requires the export dependency group: uv sync --group export)

Change the config accordingly

export:
   gen_checkpoint: "path/to/model"  # path to generator checkpoint for export
   export_path: "output.onnx"  # path to save exported model
   is_dynamic: true  # whether to export with dynamic axes
   input_shape: [1, 3, 256, 256]  # input shape for the model if not using dynamic axes
   onnx:
      opset_version: 22  # ONNX opset version for export

Run the command

python torch2onnx.py

You can perform inference with the converted onnx file using the following command:

python onnx_inference.py --model sar2rgb.onnx --input data/examples/ROIs1868_summer_s1_59_p10.png --output output.jpg

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgments

1. Sentinel-1&2 Image Pairs, Michael Schmitt, Technical University of Munich (TUM)

@misc{1436631,
	author = {Schmitt, Michael},
	title = {{SEN1-2}},
	year = {2018},
	type = {Dataset},
	abstract = {SEN1-2 is a dataset consisting of 282,384 pairs of corresponding
synthetic aperture radar and optical image patches acquired by the Sentinel-1 and Sentinel-2 remote sensing satellites, respectively.},
	keywords = {Remote sensing, deep learning, data fusion, synthetic aperture radar imagery, optical imagery},
	doi = {},
	note = {},
}

2. Image-to-Image Translation Using Conditional Adversarial Networks:

@article{pix2pix2017,
  title={Image-to-Image Translation with Conditional Adversarial Networks},
  author={Isola, Phillip and Zhu, Jun-Yan and Zhou, Tinghui and Efros, Alexei A},
  journal={CVPR},
  year={2017}
}

3. Pix2Pix Official Lua Implementation

4. Pix2Pix Official Pytorch Implementation

@inproceedings{CycleGAN2017,
  title={Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks},
  author={Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A},
  booktitle={Computer Vision (ICCV), 2017 IEEE International Conference on},
  year={2017}
}


@inproceedings{isola2017image,
  title={Image-to-Image Translation with Conditional Adversarial Networks},
  author={Isola, Phillip and Zhu, Jun-Yan and Zhou, Tinghui and Efros, Alexei A},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on},
  year={2017}
}