Skip to content

Commit

Permalink
Merge branch 'master' of https://github.com/mahmoudnafifi/HistoGAN
Browse files Browse the repository at this point in the history
  • Loading branch information
@mahmoudnafifi
mahmoudnafifi committed Aug 10, 2021
2 parents 4211159 + 629c77a commit dc543f8
Showing 1 changed file with 21 additions and 10 deletions.
31 changes: 21 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,14 +59,15 @@ pip install torch-optimizer
pip install retry
```

You may face some problems in installing `dlib` on Windows via `pip`. It is required only for the face pre-processing option (see below for more details). In order to install `dlib` for Windows, please follow this [link](https://stackoverflow.com/questions/41912372/dlib-installation-on-windows-10). If couldn't install `dlib`, you can comment this [line]() and do not use the `--face_extraction` option for ReHistoGAN.
You may face some problems in installing `dlib` on Windows via `pip`. It is required only for the face pre-processing option (see below for more details). In order to install `dlib` for Windows, please follow this [link](https://stackoverflow.com/questions/41912372/dlib-installation-on-windows-10). If couldn't install `dlib`, you can comment this [line](https://github.com/mahmoudnafifi/HistoGAN/blob/1fc4a0e0f3908ca67ea53c7b0996c28b41414e0d/utils/face_preprocessing.py#L2) and do not use the `--face_extraction` option for ReHistoGAN.

* * *

### Histogram loss
We provide a [Colab notebook example](https://colab.research.google.com/drive/1dAF1_oAQ1c8OMLqlYA5V878pmpcnQ6_9?usp=sharing) code to compute our histogram loss. This histogram loss is differentiable and can be easily integrated into any deep learning optimization.

In the [Colab](https://colab.research.google.com/drive/1dAF1_oAQ1c8OMLqlYA5V878pmpcnQ6_9?usp=sharing) tutorial, we provide different versions of the histogram class to compute histogram loss for different color spaces: uv-RGB, rg-chroma, and CIE Lab. For CIE Lab, input images are supposed to be already in CIE LAB space before computing the loss.
In the [Colab](https://colab.research.google.com/drive/1dAF1_oAQ1c8OMLqlYA5V878pmpcnQ6_9?usp=sharing) tutorial, we provide different versions of the histogram class to compute histogram loss for different color spaces: uv-RGB, rg-chroma, and CIE Lab. For CIE Lab, input images are supposed to be already in CIE LAB space before computing the loss. <!--The code is also provided in `./histogram_classes`. In HistoGAN and ReHistoGAN, we trained using uv-RGB histogram features. To use rg-chroma or CIE Lab, you simply replace the code of `RGBuvHistBlock` in `./histoGAN/histoGAN.py` with the code of the target histogram in `./hisotgram_classes`. Note that for the CIE LAB histograms, you need to first convert loaded images into the CIE LAB space in the `Dataset` class in both histoGAN and ReHistoGAN codes. That also requires converting the generated images back to sRGB space before saving them.!-->


If you faced issues with memory, please check this [issue](https://github.com/mahmoudnafifi/HistoGAN/issues/8) for potential solutions.

Expand All @@ -83,23 +84,25 @@ To train a histoGAN model on a dataset located at `./datasets/faces/` for exampl

`XX` should be replaced with the number of iterations.

During training, you can watch example samples generated by the generator in the results directory (specified by `--results_dir`). Each column in the generated sample images shares the same training histogram feature.
During training, you can watch example samples generated by the generator network in the results directory (specified by `--results_dir`). Each column in the generated sample images shares the same training histogram feature. Shown below is the training progress of a HistoGAN trained on the [FFHQ](https://github.com/NVlabs/ffhq-dataset) dataset using `--network_capacity 16` and `--image_size 256`.

<p align="center">
<img width = 25% src="https://user-images.githubusercontent.com/37669469/128811430-ce7c58d1-d439-4b05-a774-488f4d2e7bfd.gif">
</p>

There is no clear criterion to stop training, so watching generated samples will help to detect when the generator network starts diverging. You may need to increase the number of training steps (specified by `--num_train_steps`) if the generator didn't diverge by the end of training.
There is no clear criterion to stop training, so watching generated samples will help to detect when the generator network starts diverging. Also reporting the FID score after each checkpoint may help.

You may need to increase the number of training steps (specified by `--num_train_steps`), if the generator didn't diverge by the end of training. If the network starts generating degraded results after a short period of training, you may either want to reduce the network capacity (specified by `--network_capacity`) or use data augmentation by using `--aug_prob` value that is higher than 0.

If the network starts generating degraded results after a short period of training, you may either want to reduce the network capacity (specified by `--network_capacity`) or use data augmentation by using `--aug_prob` value that is higher than 0.
Shown below is the training progress of HistoGAN when trained on portrait images from the [WikiArt](https://github.com/cs-chan/ArtGAN/tree/master/WikiArt%20Dataset) dataset with and without augmentation applied. As shown, the generator starts to generate degraded images after a short training period, while it keeps generating reasonable results when augmentation is applied.

Shown below is the training progress of HistoGAN when trained on portrait images from the [WikiArt](https://github.com/cs-chan/ArtGAN/tree/master/WikiArt%20Dataset) dataset with and without augmentation applied. As shown, the generator began to generate degraded images after a short training period, while it keeps generating reasonable results when augmentation is applied.
Note that the shown samples below were generated after training on cropped face images from the original portrait set. Specifically, we extracted ~7,000 faces using [`face_preprocessing.py`](https://github.com/mahmoudnafifi/HistoGAN/blob/master/utils/face_preprocessing.py), which applies the FFHQ pre-processing to extract faces. You can download our processed portrait set from [here](https://ln4.sync.com/dl/d7addacf0/b978wvm4-9dndxvh6-hc4ss39y-5hpck6si). Do not forget to also give the credit to the [WikiArt](https://github.com/cs-chan/ArtGAN/tree/master/WikiArt%20Dataset) dataset if you used our processed set.

<p align="center">
<img width = 50% src="https://user-images.githubusercontent.com/37669469/128811547-8f33511f-fe29-492d-bb64-ea9c4ceeabe1.gif">
</p>

Note that the shown samples above were generated after training on cropped face images from the original portrait set. Specifically, we extracted ~7,000 faces using [`face_preprocessing.py`](), which applies the FFHQ pre-processing to extract faces. You can download our processed portarit set from [here](https://ln4.sync.com/dl/d7addacf0/b978wvm4-9dndxvh6-hc4ss39y-5hpck6si). Do not forget to also give the credit to the [WikiArt](https://github.com/cs-chan/ArtGAN/tree/master/WikiArt%20Dataset) dataset if you used our processed set.


#### Testing
Here is an example of how to generate new samples of a trained histoGAN models named *Faces_histoGAN*:
Expand Down Expand Up @@ -133,7 +136,7 @@ Additional useful parameters are given below.
* `--image_size`: Output image size (should be a power of 2).
* `--batch_size` and `--gradient_accumulate_every`: To control the size of mini-batch and the accumulation in computing the gradient.
* `--network_capacity`: To control network capacity.
* `--attn-layers`: To add a self-attention to the designated layer(s) of the discriminator (and the corresponding layer(s) of the generator). For example, if you would like to add a self-attention layer after the output of the 1st and 2nd layers, use `--attn-layers 1,2`. In our training, we did not use any attention layers, but it could improve the results if added.
* `--attn_layers`: To add a self-attention to the designated layer(s) of the discriminator (and the corresponding layer(s) of the generator). For example, if you would like to add a self-attention layer after the output of the 1st and 2nd layers, use `--attn_layers 1,2`. In our training, we did not use any attention layers, but it could improve the results if added.
* `--results_dir`: Results directory (for testing and evaluation during training).
* `--target_hist`: Target histogram (image, npy file of target histogram, or directory of either images or histogram files). To generate a histogram of images, check `create_hist_sample.py`.
* `--generate`: Set to True for testing.
Expand All @@ -157,16 +160,24 @@ Additional useful parameters are given below.
Very soon!

#### Trained models
As mentioned in the paper, we trained HistoGAN on several datasets. Our pre-trained models were trained using `--network_capacity = 16` and `--image_size = 256` due to hardware limitations. **Better results can be achieved by increasing the network capacity and using attention layers (--attn-layers).** Here are examples of our trained models (note: these models include both generator and discriminator nets):
As mentioned in the paper, we trained HistoGAN on several datasets. Our pre-trained models were trained using `--network_capacity = 16` and `--image_size = 256` due to hardware limitations. **Better results can be achieved by increasing the network capacity and using attention layers (`--attn_layers`).** Here are examples of our trained models (note: these models include both generator and discriminator nets):

* [Faces](https://ln3.sync.com/dl/ef2cce1a0/c7frehvr-kaexbw44-tvupizhg-xyqjvp98) | [Google Drive mirror](https://drive.google.com/file/d/1jkJBzXsakEtuVEwQqTCBb_R_Kefa2XJ1/view?usp=sharing)
* [Faces_20](https://ln4.sync.com/dl/49c229e50/fr2stver-g36e3km7-qj3vwwtf-qwqa5ufb)
* [Cars](https://ln3.sync.com/dl/5c2bc1a60/y3sx9dnq-m5gcspa5-6zz9d4fd-7j3yemca)
* [Flowers](https://ln3.sync.com/dl/e869e6d50/pnqiaccc-vtszcvs4-tt7x2y9v-gubvyymm)
* [Anime](https://ln3.sync.com/dl/13bacb830/u4hm9bcw-hbnfeg8j-pmrwsdjp-iwar8ee8)
* [Landscape](https://ln3.sync.com/dl/dfdbb5600/r2gzhwtk-5guyb8s8-j6a99c72-excf5kjy)
* [PortraitFaces](https://ln4.sync.com/dl/f3b08fed0/zdqb9qxe-4wrasq8r-7ihvp73m-zv82vat5)
* [PortraitFaces_aug](https://ln4.sync.com/dl/f061af1b0/t7ukam9v-nk6napgw-krpn2wa9-h2agy8in)
* [PortraitFaces_20_aug](https://ln4.sync.com/dl/f8e934db0/9bddn4cf-fnhtvkwi-jg4zmpb4-dq8ypv4r)


Note that for model names that include `_20`, use `--network_capacity 20` in testing. If the model name includes `_aug`, make sure to set `--aug_prob` to any value higher than zero. Below are examples of generated samples from each model. Each shown group of generated images share the same histogram feature.


![pre-trained](https://user-images.githubusercontent.com/37669469/128905351-83b145a9-d045-47b0-835d-05974ef4106d.gif)

Note that for model names that include `_20`, use `--network_capacity 20` in testing. If the model name includes `_aug_`, make sure to set `--aug_prob` to any value higher than zero. Below are examples of generated samples from each model. Each shown group of generated images share the same histogram feature.

* * *

Expand Down

0 comments on commit dc543f8

Please sign in to comment.