GeSeNet

GeSeNet: A General Semantic-guided Network with Couple Mask Ensemble for Medical Image Fusion [Paper] [Google Scholar]

in IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS)
by Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov

Requirements

• python 3.7
• torch 1.7.0
• torchvision 0.8.0
• opencv 4.5
• numpy 1.21.6
• pillow 9.4.0

Dataset setting

We give 5 test image pairs as examples in three modalities (i.e., MRI-CT, MRI-PET, MRI-SPECT), respectively.

Moreover, you can set your own test datasets of different modalities under ./test_images/..., like:

test_images
├── MRI_CT
|   ├── CT
|   |   ├── 1.png
|   |   ├── 2.png
|   |   └── ...
|   ├── MRI
|   |   ├── 1.png
|   |   ├── 2.png
|   |   └── ...

The datasets in our paper are all from: [Harvard medical images]

Test

The pre-trained model has given in ./model/GeSeNet.pth. Please run test.py to get fused results, and you can check them in:

results
├── MRI_CT
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── MRI_PET
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── MRI_SPECT
|   ├── 1.png
|   ├── 2.png
|   └── ...

Experimental results

The qualitative comparison results of our GeSeNet with nine state-of-the-art methods on MRI-CT, MRI-PET and MRI-SPECT image pairs.

Please refer to the paper for more experimental results and details.

Citation

@article{li2023gesenet,
  title={GeSeNet: A General Semantic-guided Network with Couple Mask Ensemble for Medical Image Fusion},
  author={Li, Jiawei and Liu, Jinyuan and Zhou, Shihua and Zhang, Qiang and Kasabov, Nikola},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  year={2023},
  publisher={IEEE}
}

Realted works

• Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. Learning a Coordinated Network for Detail-refinement Multi-exposure Image Fusion. IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT), 2022, 33(2): 713-727. [Paper]
• Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. Infrared and visible image fusion based on residual dense network and gradient loss. Infrared Physics & Technology, 2023, 128: 104486. [Paper]
• Jia Lei, Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. GALFusion: Multi-exposure Image Fusion via a Global-local Aggregation Learning Network. IEEE Transactions on Instrumentation and Measurement (IEEE TIM), 2023, 72: 1-15. [Paper] [Code]

Contact

If you have any questions, please create an issue or email to me (Jiawei Li).