Merge back releases/1.1.0 (#79)

CHANGELOG.md

@@ -1,15 +1,38 @@
 # Change Log
 
-## [1.1.0 Unreleased]
+## [1.2.0 Unreleased]
 
 ### Summary
 
-* Support OpenVINO IR (.xml) / ONNX (.onnx) model file for `Explainer` model
-* Enable AISE: Adaptive Input Sampling for Explanation of Black-box Models
-* Upgrade OpenVINO to 2024.3.0
+*
+
+### What's Changed
+
+*
+
+### Known Issues
+
+* Runtime error from ONNX / OpenVINO IR models while conversion or inference for XAI in https://github.com/openvinotoolkit/openvino_xai/issues/29
+* Models not supported by white box XAI methods in https://github.com/openvinotoolkit/openvino_xai/issues/30
+
+### New Contributors
+
+* N/A
+
+---
+
+## [1.1.0]
+
+### Summary
+
+* Support PyTorch models with `insert_xai()` API for saliency map generation on PyTorch / ONNX runtime
+* Support OpenVINO IR (.xml) / ONNX (.onnx) model files for `Explainer`
+* Enable AISE method: Adaptive Input Sampling for Explanation of Black-box Models
+* Add Pointing Game, Insertion-Deletion AUC and ADCC quality metrics for saliency maps
+* Upgrade OpenVINO to 2024.4.0
 * Add saliency map visualization with explanation.plot()
 * Enable flexible naming for saved saliency maps and include confidence scores
-* Add Pointing Game, Insertion-Deletion AUC and ADCC quality metrics for saliency maps
+* Add XAI method documentation
 
 ### What's Changed
 
@@ -26,6 +49,11 @@
 * Add [Insertion-Deletion AUC](https://arxiv.org/abs/1806.07421) saliency map quality metric by @GalyaZalesskaya in https://github.com/openvinotoolkit/openvino_xai/pull/56
 * Add [ADCC](https://arxiv.org/abs/2104.10252) saliency map quality metric by @GalyaZalesskaya in https://github.com/openvinotoolkit/openvino_xai/pull/57
 * Enable AISE for detection: Adaptive Input Sampling for Explanation of Black-box Models by @negvet in https://github.com/openvinotoolkit/openvino_xai/pull/55
+* Enable prediction attribute for methods that is used for enhancing saliency map overlay by @negvet in https://github.com/openvinotoolkit/openvino_xai/pull/62
+* Add documentation per-method, including summary and usage guide by @negvet in https://github.com/openvinotoolkit/openvino_xai/pull/63
+* Support Pytorch models for `insert_xai` API by @goodsong81 in https://github.com/openvinotoolkit/openvino_xai/pull/61
+* Auto-detect feature layer for Pytorch models by @goodsong81 in https://github.com/openvinotoolkit/openvino_xai/pull/64
+* Upgrade OpenVINO to 2024.4.0 by @goodsong81 in https://github.com/openvinotoolkit/openvino_xai/pull/72
 
 ### Known Issues
 

README.md

@@ -8,10 +8,10 @@
 [Install](#installation) •
 [Quick Start](#quick-start) •
 [License](#license) •
-[Documentation](https://openvinotoolkit.github.io/openvino_xai/releases/1.0.0)
+[Documentation](https://openvinotoolkit.github.io/openvino_xai/releases/1.1.0)
 
 ![Python](https://img.shields.io/badge/python-3.10%2B-green)
-[![OpenVINO](https://img.shields.io/badge/openvino-2024.2-purple)](https://pypi.org/project/openvino/)
+[![OpenVINO](https://img.shields.io/badge/openvino-2024.4-purple)](https://pypi.org/project/openvino/)
 [![codecov](https://codecov.io/gh/openvinotoolkit/openvino_xai/graph/badge.svg?token=NR0Z0CWDK9)](https://codecov.io/gh/openvinotoolkit/openvino_xai)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 [![PyPI](https://img.shields.io/pypi/v/openvino_xai)](https://pypi.org/project/openvino_xai)
@@ -24,9 +24,9 @@
 ![OpenVINO XAI Concept](docs/source/_static/ovxai-concept.svg)
 
 **OpenVINO™ Explainable AI (XAI) Toolkit** provides a suite of XAI algorithms for visual explanation of
-[**OpenVINO™**](https://github.com/openvinotoolkit/openvino) Intermediate Representation (IR) models.
+[**OpenVINO™**](https://github.com/openvinotoolkit/openvino) as well as [**PyTorch**](https://pytorch.org) and [**ONNX**](https://onnx.ai/) models.
 
-Given **OpenVINO** models and input images, **OpenVINO XAI** generates **saliency maps**
+Given AI models and input images, **OpenVINO XAI** generates **saliency maps**
 which highlights regions of the interest in the inputs from the models' perspective
 to help users understand the reason why the complex AI models output such responses.
 
@@ -51,14 +51,15 @@ for i, image in enumerate(images):
 
 ## Features
 
-### What's new in v1.0.0
+### What's new in v1.1.0
 
-* Support generation of classification and detection per-class and per-image saliency maps
-* Enable White-Box ([ReciproCAM](https://arxiv.org/abs/2209.14074)) and Black-Box ([RISE](https://arxiv.org/abs/1806.07421v3)) eXplainable AI algorithms
-* Support CNNs and Transformer-based architectures (validation on diverse set of [timm](https://github.com/huggingface/pytorch-image-models) models)
-* Enable `Explainer` (stateful object) as the main interface for XAI algorithms
-* Support `AUTO` mode by default to detect the best XAI method for given models
-* Expose `insert_xai` functional API to support XAI head insertion for OpenVINO IR models
+* Support PyTorch models with `insert_xai()` API for saliency map generation on PyTorch / ONNX runtime
+* Support OpenVINO IR (.xml) / ONNX (.onnx) model files for `Explainer`
+* Enable AISE method: Adaptive Input Sampling for Explanation of Black-box Models
+* Add Pointing Game, Insertion-Deletion AUC and ADCC quality metrics for saliency maps
+* Upgrade OpenVINO to 2024.4.0
+* Add saliency map visualization with explanation.plot()
+* Enable flexible naming for saved saliency maps and include confidence scores
 
 Please refer to the [change logs](CHANGELOG.md) for the full release history.
 
@@ -67,15 +68,17 @@ Please refer to the [change logs](CHANGELOG.md) for the full release history.
 At the moment, *Image Classification* and *Object Detection* tasks are supported for the *Computer Vision* domain.
 *Black-Box* (model agnostic but slow) methods and *White-Box* (model specific but fast) methods are supported:
 
-| Domain          | Task                 | Type      | Algorithm           | Links |
-|-----------------|----------------------|-----------|---------------------|-------|
-| Computer Vision | Image Classification | White-Box | ReciproCAM          | [arxiv](https://arxiv.org/abs/2209.14074) / [src](openvino_xai/methods/white_box/recipro_cam.py) |
-|                 |                      |           | VITReciproCAM       | [arxiv](https://arxiv.org/abs/2310.02588) / [src](openvino_xai/methods/white_box/recipro_cam.py) |
-|                 |                      |           | ActivationMap       | experimental / [src](openvino_xai/methods/white_box/activation_map.py)                           |
-|                 |                      | Black-Box | AISEClassification  | [src](openvino_xai/methods/black_box/aise.py)                                                    |
-|                 |                      |           | RISE                | [arxiv](https://arxiv.org/abs/1806.07421v3) / [src](openvino_xai/methods/black_box/rise.py)      |
-|                 | Object Detection     | White-Box | ClassProbabilityMap | experimental / [src](openvino_xai/methods/white_box/det_class_probability_map.py)                |
-|                 |                      | Black-Box | AISEDetection       | [src](openvino_xai/methods/black_box/aise.py)                                                    |
+| Domain          | Task                 | Type      | Algorithm              | Links |
+|-----------------|----------------------|-----------|------------------------|-------|
+| Computer Vision | Image Classification | White-Box | ReciproCAM             | [paper](https://openaccess.thecvf.com/content/CVPR2024W/XAI4CV/papers/Byun_ReciproCAM_Lightweight_Gradient-free_Class_Activation_Map_for_Post-hoc_Explanations_CVPRW_2024_paper.pdf) / [src](openvino_xai/me4thods/white_box/recipro_cam.py) |
+|                 |                      |           | VITReciproCAM          | [paper](https://arxiv.org/abs/2310.02588) / [src](openvino_xai/methods/white_box/recipro_cam.py) |
+|                 |                      |           | ActivationMap          | experimental / [src](openvino_xai/methods/white_box/activation_map.py)                           |
+|                 |                      | Black-Box | AISEClassification     | [src](openvino_xai/methods/black_box/aise/classification.py)                                     |
+|                 |                      |           | RISE                   | [paper](https://arxiv.org/abs/1806.07421v3) / [src](openvino_xai/methods/black_box/rise.py)      |
+|                 | Object Detection     | White-Box | DetClassProbabilityMap | experimental / [src](openvino_xai/methods/white_box/det_class_probability_map.py)                |
+|                 |                      | Black-Box | AISEDetection          | [src](openvino_xai/methods/black_box/aise/detection.py)                                          |
+
+See more method comparison at the [User Guide](docs/source/user-guide.md).
 
 ### Supported explainable models
 
@@ -86,10 +89,6 @@ Please refer to the following known issues for unsupported models and reasons.
 * [Runtime error from ONNX / OpenVINO IR models while conversion or inference for XAI (#29)](https://github.com/openvinotoolkit/openvino_xai/issues/29)
 * [Models not supported by white box XAI methods (#30)](https://github.com/openvinotoolkit/openvino_xai/issues/30)
 
-> **_WARNING:_**  OpenVINO XAI is fully validated on OpenVINO 2024.2.0. Following issue might be observed if older version of OpenVINO is used.
-> * [OpenVINO IR branch insertion not working for models converted directly from torch models with OVC (#26)](https://github.com/openvinotoolkit/openvino_xai/issues/26)
-> A simple workaround is to convert Torch models to ONNX models and then convert to OpenVINO models to feed to OpenVINO XAI. Please refer to [the code example](openvino_xai/utils/model_export.py).
-
 > **_NOTE:_**  GenAI / LLMs would be also supported incrementally in the upcoming releases.
 
 ---
@@ -133,6 +132,18 @@ pip install -e .[dev]
 ```
 </details>
 
+<details>
+<summary>(Optional) Enable PyTorch support</summary>
+
+You can enjoy the PyTorch XAI feature if the PyTorch is installed along with the OpenVINO XAI.
+
+```bash
+# Install PyTorch (CPU version as example)
+pip3 install torch --index-url https://download.pytorch.org/whl/cpu
+```
+Please refer to the [PyTorch Installation Guide](https://pytorch.org/get-started/locally/) for other options.
+</details>
+
 <details>
 <summary>Verify installation</summary>
 
@@ -151,7 +162,7 @@ pre-commit run --all-files
 
 ### Hello, OpenVINO XAI
 
-Let's imagine the case that our OpenVINO IR model is up and running on a inference pipeline.
+Let's imagine the case that our OpenVINO model is up and running on a inference pipeline.
 While watching the outputs, we may want to analyze the model's behavior for debugging or understanding purposes.
 
 By using the **OpenVINO XAI** `Explainer`, we can visualize why the model gives such responses.
@@ -194,6 +205,28 @@ Original image | Explained image
 
 We can see that model is focusing on the body or skin area of the animals to tell if this image contains actual cheetahs.
 
+### Insert XAI head to your models
+
+Using the `insert_xai` API, we can insert XAI head to existing OpenVINO or PyTorch models directly and get additional "saliency_map" output in the same inference pipeline.
+
+```python
+import torch
+import timm
+
+# Get a PyTorch model from TIMM
+torch_model: torch.nn.Module = timm.create_model("resnet18.a1_in1k", in_chans=3, pretrained=True)
+
+# Insert XAI head
+model_xai: torch.nn.Module = xai.insert_xai(torch_model, xai.Task.CLASSIFICATION)
+
+# Torch XAI model inference
+model_xai.eval()
+with torch.no_grad():
+    outputs = model_xai(torch.from_numpy(image_norm))
+    logits = outputs["prediction"]  # BxC
+    saliency_maps = outputs["saliency_map"]  # BxCxHxW: per-class saliency map
+```
+
 ### More advanced use-cases
 
 Users could tweak the basic use-case according to their purpose, which include but not limited to:
@@ -203,17 +236,19 @@ Users could tweak the basic use-case according to their purpose, which include b
 * Customize output image visualization options
 * Explain multiple class targets, passing them as label indices or as actual label names
 * Call explainer multiple times to explain multiple images or to use different targets
-* Using `insert_xai` API, insert XAI head to your OpenVINO IR model and get additional saliency map output in the same inference pipeline
+* Insert XAI head to your PyTorch models and export to ONNX format to generate saliency maps on ONNX Runtime
+  (Refer to the [full example script](./examples/run_torch_onnx.py))
 
 Please find more options and scenarios in the following links:
 
 * [OpenVINO XAI User Guide](docs/source/user-guide.md)
 * [OpenVINO Notebook - XAI Basic](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-1-basic/explainable-ai-1-basic.ipynb)
 * [OpenVINO Notebook - XAI Deep Dive](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-2-deep-dive/explainable-ai-2-deep-dive.ipynb)
+* [OpenVINO Notebook - Saliency Map Interpretation](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/explainable-ai-3-map-interpretation/explainable-ai-3-map-interpretation.ipynb)
 
 ### Playing with the examples
 
-Please look around the runnable [example scripts](./examples) and play with them to get used to the `Explainer` APIs.
+Please look around the runnable [example scripts](./examples) and play with them to get used to the `Explainer` and `insert_xai` APIs.
 
 ```bash
 # Prepare models by running tests (need "pip install openvino_xai[dev]" extra option)
@@ -224,6 +259,9 @@ pytest tests/test_classification.py
 # All outputs will be stored in the corresponding output directory
 python examples/run_classification.py .data/otx_models/mlc_mobilenetv3_large_voc.xml \
 tests/assets/cheetah_person.jpg --output output
+
+# Run PyTorch and ONNX support example
+python examples/run_torch_onnx.py
 ```
 
 ---