-
Notifications
You must be signed in to change notification settings - Fork 82
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ninja: build stopped: subcommand failed. #25
Comments
Same issue as above |
This appears to be a gcc version non-adaptation problem in Linux. The same problem happened when running stylegan2. The main problem is that upfirdn2d does not compile. My solution is to write a python version of upfirdn2d.py: |
I think you should check your GCC version, with reference to NVlabs/stylegan3#2 |
Is this something to do with the cuda version? The cuda version I am using is 10.2.
starting in debug mode
Traceback (most recent call last):
File "/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1673, in _run_ninja_build
env=env)
File "/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "train_ddgan.py", line 601, in
init_processes(0, size, train, args)
File "train_ddgan.py", line 471, in init_processes
fn(rank, gpu, args)
File "train_ddgan.py", line 192, in train
from score_sde.models.discriminator import Discriminator_small, Discriminator_large
File "/data/songwei/YING/denoising-diffusion-gan/score_sde/models/discriminator.py", line 11, in
from . import up_or_down_sampling
File "/data/songwei/YING/denoising-diffusion-gan/score_sde/models/up_or_down_sampling.py", line 15, in
from score_sde.op import upfirdn2d
File "/data/songwei/YING/denoising-diffusion-gan/score_sde/op/init.py", line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File "/data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_act.py", line 23, in
os.path.join(module_path, "fused_bias_act_kernel.cu"),
File "/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1302, in jit_compile
is_standalone=is_standalone)
File "/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1407, in write_ninja_file_and_build_library
error_prefix=f"Error building extension '{name}'")
File "/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1683, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused': [1/3] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output fused_bias_act_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/TH -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/THC -isystem /data/songwei/anacondaV100/envs/ddpmgan/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output fused_bias_act_kernel.cuda.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/TH -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/THC -isystem /data/songwei/anacondaV100/envs/ddpmgan/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c++14 -c /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
nvcc fatal : Unknown option '-generate-dependencies-with-compile'
[2/3] c++ -MMD -MF fused_bias_act.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/TH -isystem /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/THC -isystem /data/songwei/anacondaV100/envs/ddpmgan/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp -o fused_bias_act.o
In file included from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/c10/core/DeviceType.h:8:0,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/c10/core/Device.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/c10/core/Allocator.h:6,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/ATen.h:7,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/extension.h:4,
from /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:8:
/data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp: In function ‘at::Tensor fused_bias_act(const at::Tensor&, const at::Tensor&, const at::Tensor&, int, int, float, float)’:
/data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:14:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
^
/data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:20:5: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(input);
^
In file included from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/Tensor.h:3:0,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/Context.h:4,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/ATen.h:9,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/extension.h:4,
from /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:8:
/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
DeprecatedTypeProperties & type() const {
^~~~
In file included from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/c10/core/DeviceType.h:8:0,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/c10/core/Device.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/c10/core/Allocator.h:6,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/ATen.h:7,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/extension.h:4,
from /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:8:
/data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:14:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
^
/data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:21:5: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(bias);
^
In file included from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/Tensor.h:3:0,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/Context.h:4,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/ATen.h:9,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/torch/extension.h:4,
from /data/songwei/YING/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:8:
/data/songwei/anacondaV100/envs/ddpmgan/lib/python3.6/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
DeprecatedTypeProperties & type() const {
^~~~
ninja: build stopped: subcommand failed.
The text was updated successfully, but these errors were encountered: