Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building the C library and executable on Windows fails #21

Open
acarta opened this issue May 5, 2022 · 8 comments
Open

Building the C library and executable on Windows fails #21

acarta opened this issue May 5, 2022 · 8 comments

Comments

@acarta
Copy link

acarta commented May 5, 2022

The Windows build of CG fails, both when using the included makefiles and when running build.jl directly from Julia.

Trying to use the top-level makefile (libcg/makefile) with mingw32-make gives the following output:

PS C:\Users\Administrator\repos\libcg> C:\mingw64\bin\mingw32-make.exe
C:/mingw64/bin/mingw32-make.exe -C CG
mingw32-make[1]: Entering directory 'C:/Users/Administrator/repos/libcg/CG'
process_begin: CreateProcess(NULL, sed -n "s/^version *= *\"\(.*\)\"/\1/p" C:/Users/Administrator/repos/libcg/CG/Project.toml, ...) failed.
makefile:9: pipe: Bad file descriptor
process_begin: CreateProcess(NULL, uname, ...) failed.
makefile:10: pipe: Bad file descriptor
File not found - *.jl
process_begin: CreateProcess(NULL, basename cg-, ...) failed.
makefile:19: pipe: Bad file descriptor
julia --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate()'
julia --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build -e 'using Pkg; Pkg.instantiate()'
julia --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build C:/Users/Administrator/repos/libcg/CG/build/build.jl cg-/cg
✔ [02m:17s] PackageCompiler: compiling base system image (incremental=false)
✔ [02m:30s] PackageCompiler: compiling incremental system image
Precompiling project...
  6 dependencies successfully precompiled in 5 seconds
[ Info: PackageCompiler: Executing C:\Users\Administrator\repos\libcg\CG\build\generate_precompile.jl => C:\Users\ADMINI~1\AppData\Local\Temp\2\jl_packagecompiler_OV2PkH\jl_A8E1.tmp
[ Info: PackageCompiler: Done
✔ [01m:08s] PackageCompiler: compiling incremental system image
cp README.md C:/Users/Administrator/repos/libcg/CG/build/INSTALL.txt C:/Users/Administrator/repos/libcg/CG/build/install.sh cg-
process_begin: CreateProcess(NULL, cp README.md C:/Users/Administrator/repos/libcg/CG/build/INSTALL.txt C:/Users/Administrator/repos/libcg/CG/build/install.sh cg-, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make[1]: *** [makefile:41: build] Error 2
mingw32-make[1]: Leaving directory 'C:/Users/Administrator/repos/libcg/CG'
mingw32-make: *** [makefile:29: C:/Users/Administrator/repos/libcg/target/lib/libcg.dll] Error 2
PS C:\Users\Administrator\repos\libcg>

For completeness, I also tried to build only CG using the top-level makefile which gave the following output:

PS C:\Users\Administrator\repos\libcg> C:\mingw64\bin\mingw32-make.exe CG
C:/mingw64/bin/mingw32-make.exe -C CG
mingw32-make[1]: Entering directory 'C:/Users/Administrator/repos/libcg/CG'
process_begin: CreateProcess(NULL, sed -n "s/^version *= *\"\(.*\)\"/\1/p" C:/Users/Administrator/repos/libcg/CG/Project.toml, ...) failed.
makefile:9: pipe: Bad file descriptor
process_begin: CreateProcess(NULL, uname, ...) failed.
makefile:10: pipe: Bad file descriptor
File not found - *.jl
process_begin: CreateProcess(NULL, basename cg-, ...) failed.
makefile:19: pipe: Bad file descriptor
julia --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate()'
julia --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build -e 'using Pkg; Pkg.instantiate()'
julia --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build C:/Users/Administrator/repos/libcg/CG/build/build.jl cg-/cg
✔ [02m:17s] PackageCompiler: compiling base system image (incremental=false)
✔ [02m:30s] PackageCompiler: compiling incremental system image
Precompiling project...
  6 dependencies successfully precompiled in 5 seconds
[ Info: PackageCompiler: Executing C:\Users\Administrator\repos\libcg\CG\build\generate_precompile.jl => C:\Users\ADMINI~1\AppData\Local\Temp\2\jl_packagecompiler_qWsRqf\jl_6757.tmp
[ Info: PackageCompiler: Done
✔ [01m:09s] PackageCompiler: compiling incremental system image
cp README.md C:/Users/Administrator/repos/libcg/CG/build/INSTALL.txt C:/Users/Administrator/repos/libcg/CG/build/install.sh cg-
process_begin: CreateProcess(NULL, cp README.md C:/Users/Administrator/repos/libcg/CG/build/INSTALL.txt C:/Users/Administrator/repos/libcg/CG/build/install.sh cg-, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make[1]: *** [makefile:41: build] Error 2
mingw32-make[1]: Leaving directory 'C:/Users/Administrator/repos/libcg/CG'
mingw32-make: *** [makefile:29: C:/Users/Administrator/repos/libcg/target/lib/libcg.dll] Error 2

I was able to compile via the following

using PackageCompiler, TOML
build_dir = "build"
target_dir = joinpath("..", "..", "target")
project_toml = realpath("Project.toml")

PackageCompiler.create_library(".", target_dir;
                            lib_name="cg",
                            precompile_execution_file=[joinpath(build_dir, "generate_precompile.jl")],
                            precompile_statements_file=[joinpath(build_dir, "additional_precompile.jl")],
                            incremental=false,
                            filter_stdlibs=true,
                            header_files = [joinpath(build_dir, "cg.h")],
                            force=true,
)

However, I am unable to compile the executable in libcg/main-c using either libcg/main-c/makefile or direct invocation of gcc.

For the makefile, the output is

PS C:\Users\Administrator\repos\libcg\main-c> C:\mingw64\bin\mingw32-make.exe
mkdir -p C:/Users/Administrator/repos/libcg/main-c/../target/bin 2>&1 > /dev/null
The system cannot find the path specified.

For using gcc directly, compiling the main.o object file seems to work; running the following throws no errors, and a main.o appears within libcg/main-c:

PS C:\Users\Administrator\repos\libcg\main-c> C:\mingw64\bin\gcc.exe  main.c -c -o main.o -O2 -fPIE -IC:\Users\Administrator\AppData\Local\Programs\Julia-1.7.2\include\julia -I../target/include

However, the linking fails with the following output:

PS C:\Users\Administrator\repos\libcg\main-c> C:\mingw64\bin\gcc.exe -o main-c.exe main.o -lm -L..\target\bin\ -ljulia
c:/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: main.o:main.c:(.text.startup+0x1d): undefined reference to `init_julia'
c:/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: main.o:main.c:(.text.startup+0x7e): undefined reference to `julia_cg'
c:/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: main.o:main.c:(.text.startup+0x8d): undefined reference to `shutdown_julia'
collect2.exe: error: ld returned 1 exit status

Version Info

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS =
@theweaklink
Copy link

theweaklink commented May 6, 2022

I am getting the same problem building the main-c example with MSVC and Julia 1.6.6 (LTS). Building lib CG is ok but the linking to it fails on Windows.
(it doesn't help to solve the problem but at least I can reproduce it with a different compiler and Julia version - I'll let you know if I find out how to solve it)

@theweaklink
Copy link

theweaklink commented May 6, 2022

No luck either through the generated .lib from the following .def file using lib.exe nor dlltool.exe

LIBRARY "cg.dll"
EXPORTS
get_sysimage_path
init_julia
jl_RTLD_DEFAULT_handle_pointer DATA
jl_dispatch_fvars_idxs DATA
jl_dispatch_fvars_offsets DATA
jl_dispatch_reloc_slots DATA
jl_dispatch_target_ids DATA
jl_get_ptls_states_slot DATA
jl_sysimg_fvars_base
jl_sysimg_fvars_offsets DATA
jl_sysimg_gvars_base DATA
jl_sysimg_gvars_offsets DATA
jl_system_image_data DATA
jl_system_image_size DATA
jl_tls_offset DATA
julia_cg
set_depot_load_path
setup_args
shutdown_julia

Still getting unresolved symbols when linking main-c

@simonbyrne
Copy link
Owner

I'm not that familiar with building software on Windows. @kmsquire may have some advice, but my only suggestion would be to try to build it in Cygwin

@acarta
Copy link
Author

acarta commented May 12, 2022

I saw that some similar issues were addressed/discussed in PR #18; within the comments @kmsquire mentions that libcg was working in C at that point.

Looking through the PackageCompiler release history, it looks like that would have been ~v1.3.0 of PackageCompiler. I tried reverting back to that version (using the current LTS version of Julia, v1.6.6), but running libcg/CG/makefile gives the following output:

PS C:\Users\Administrator\repos\libcg\CG> C:\mingw64\bin\mingw32-make.exe
process_begin: CreateProcess(NULL, sed -n "s/^version *= *\"\(.*\)\"/\1/p" C:/Users/Administrator/repos/libcg/CG/Project.toml, ...) failed.
makefile:9: pipe: Bad file descriptor
process_begin: CreateProcess(NULL, uname, ...) failed.
makefile:10: pipe: Bad file descriptor
File not found - *.jl
process_begin: CreateProcess(NULL, basename cg-, ...) failed.
makefile:19: pipe: Bad file descriptor
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate()'
  Installing known registries into `C:\Users\Administrator\.julia`
     Cloning registry from "https://github.com/JuliaRegistries/General.git"
       Added registry `General` to `C:\Users\Administrator\.julia\registries\General`
    Updating registry at `C:\Users\Administrator\.julia\registries\General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Updating `C:\Users\Administrator\repos\libcg\CG\Project.toml`
  [42fd0dbc] + IterativeSolvers v0.9.2
    Updating `C:\Users\Administrator\repos\libcg\CG\Manifest.toml`
  [42fd0dbc] + IterativeSolvers v0.9.2
  [3cdcf5f2] + RecipesBase v1.2.1
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [2f01184e] + SparseArrays
  [4ec0a83e] + Unicode
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build -e 'using Pkg; Pkg.instantiate()'
┌ Warning: The active manifest file has dependencies that were resolved with a different julia version (1.7.2). Unexpected behavior may occur.
└ @ C:\Users\Administrator\repos\libcg\CG\build\Manifest.toml:0
Precompiling project...
  4 dependencies successfully precompiled in 4 seconds
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build C:/Users/Administrator/repos/libcg/CG/build/build.jl cg-/cg
  0 dependencies successfully precompiled in 1 seconds
  3 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
  0 dependencies successfully precompiled in 0 seconds
  3 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c: In function 'set_depot_path':
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c:61:5: warning: 'strncpy' specified bound depends on the length of the source argument [-Wstringop-overflow=]
     strncpy(dir, root_dir, root_dir_len);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c:47:24: note: length computed here
     int root_dir_len = strlen(root_dir);
                        ^~~~~~~~~~~~~~~~
[ Info: ===== Start precompile execution =====
[ Info: ===== End precompile execution =====
[ Info: PackageCompiler: creating system image object file, this might take a while...
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate()'
Precompiling project...
  3 dependencies successfully precompiled in 4 seconds
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build -e 'using Pkg; Pkg.instantiate()'
┌ Warning: The active manifest file has dependencies that were resolved with a different julia version (1.7.2). Unexpected behavior may occur.
└ @ C:\Users\Administrator\repos\libcg\CG\build\Manifest.toml:0
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build C:/Users/Administrator/repos/libcg/CG/build/build.jl cg-/cg
  0 dependencies successfully precompiled in 1 seconds
  3 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
  0 dependencies successfully precompiled in 0 seconds
  3 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c: In function 'set_depot_path':
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c:61:5: warning: 'strncpy' specified bound depends on the length of the source argument [-Wstringop-overflow=]
     strncpy(dir, root_dir, root_dir_len);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c:47:24: note: length computed here
     int root_dir_len = strlen(root_dir);
                        ^~~~~~~~~~~~~~~~
[ Info: ===== Start precompile execution =====
[ Info: ===== End precompile execution =====
[ Info: PackageCompiler: creating system image object file, this might take a while...
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate()'
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build -e 'using Pkg; Pkg.instantiate()'
┌ Warning: The active manifest file has dependencies that were resolved with a different julia version (1.7.2). Unexpected behavior may occur.
└ @ C:\Users\Administrator\repos\libcg\CG\build\Manifest.toml:0
C:/Users/Administrator/AppData/Local/Programs/Julia-1.6.6/bin/julia.exe --startup-file=no --project=C:/Users/Administrator/repos/libcg/CG/build C:/Users/Administrator/repos/libcg/CG/build/build.jl cg-/cg
  0 dependencies successfully precompiled in 1 seconds
  3 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
  0 dependencies successfully precompiled in 0 seconds
  3 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c: In function 'set_depot_path':
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c:61:5: warning: 'strncpy' specified bound depends on the length of the source argument [-Wstringop-overflow=]
     strncpy(dir, root_dir, root_dir_len);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\Users\Administrator\.julia\packages\PackageCompiler\drmGx\src\julia_init.c:47:24: note: length computed here
     int root_dir_len = strlen(root_dir);
                        ^~~~~~~~~~~~~~~~
[ Info: ===== Start precompile execution =====
[ Info: ===== End precompile execution =====
[ Info: PackageCompiler: creating system image object file, this might take a while...
cp README.md C:/Users/Administrator/repos/libcg/CG/build/INSTALL.txt C:/Users/Administrator/repos/libcg/CG/build/install.sh cg-
process_begin: CreateProcess(NULL, cp README.md C:/Users/Administrator/repos/libcg/CG/build/INSTALL.txt C:/Users/Administrator/repos/libcg/CG/build/install.sh cg-, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make: *** [makefile:41: build] Error 2

@simonbyrne
Copy link
Owner

We have CI set up on Windows: I just reran it on Julia 1.7, and it passed:
https://github.com/simonbyrne/libcg/runs/6410992790?check_suite_focus=true
However as I said above, I think it executes under cygwin.

process_begin: CreateProcess(NULL, sed -n "s/^version *= *\"\(.*\)\"/\1/p" C:/Users/Administrator/repos/libcg/CG/Project.toml, ...) failed.
makefile:9: pipe: Bad file descriptor

suggests that it doesn't have sed installed?

@acarta
Copy link
Author

acarta commented May 13, 2022

Indeed, sed and the other Unix-related tools are not installed. In that case, it makes sense that the makefile fails.

However, as I described above (and as @theweaklink corroborates) even bypassing the makefile and invoking either GCC or MSVC directly also fails during the compilation of the executable.

Is using Cygwin a hard requirement?

@simonbyrne
Copy link
Owner

Unfortunately I don't have enough experience on windows to say. Maybe @staticfloat might be able to help?

@matthijscox
Copy link

To bypass this issue I wrote my own, much simpler, Makefile examples without any Unix shell scripting. It's in this EmbedJuliaInC repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants