Build the PDF docs on Buildkite

[DilumAluthge]

Ref #198

[DilumAluthge]

@mortenpi After I run make.jl, where does it save the PDF files?

[mortenpi]

After I run make.jl, where does it save the PDF files?

I think the make.jl copies it into JULIA_DOCS_TMP.

However, I am not sure that it makes sense to borrow the make.jl script from the other repo at all. Couldn't we just call

make -C $(JULIA_SOURCE)/doc pdf texplatform=docker JULIA_EXECUTABLE=$(julia_exec)

in the YAML and then pass the PDF on to the deploy_docs pipeline?

The latter would also have to be modified a little of course. The PDFs get deployed onto the assets branch of the repo, and we also overwrite the history on every push.

Also, cc @fredrikekre since he's more familiar with the PDF deployment.

[DilumAluthge]

@staticfloat It's failing to load the rootfs. Maybe it's running out of disk space?

[DilumAluthge]

What are the possible values for texplatform? Docker isn't available here, but we do have LaTeX already installed in this rootfs image.

[mortenpi]

I guess as you figured out, the texplatform=.. part can be omitted, in which case it uses the "native" backend (i.e. local installation). Do we have the full distro on rootfs? And it also requires some custom fonts and Python's Pygments (for highlighting) to be installed.

Here is the Dockerfile that sets up the Docker image: https://github.com/JuliaDocs/DocumenterLaTeX.jl/blob/master/docker/Dockerfile

[DilumAluthge]

We should have everything we need in the existing rootfs image, but we can make a new one if need be.

Right now the build is failing, but I'm not sure what the cause is.

[mortenpi]

What's the working directory here? I see on 477 that make: Entering directory '/cache/build/default-amdci5-1/julialang/julia-buildkite/doc which is different from 573 where SystemError: opening file "/cache/build/default-amdci5-1/julialang/julia-buildkite/pdf/build/doc/UnicodeData.txt".

[DilumAluthge]

Oh maybe it's because I set the BUILDROOT environment variable. Let me comment that out.

[mortenpi]

Alright, it looks like it complains about Pygments being missing:

! Package minted Error: You must have `pygmentize' installed to use this package.

sudo apt install python-pygments ought to be enough for this one. But after that we'll probably still be missing some fonts.

[DilumAluthge]

We'll likely need to make a separate rootfs image just for this job.

JuliaCI/rootfs-images#224

[mortenpi]

Hmm.. is make missing from the new rootfs image? And Documenter also needs git, which I think might also be missing?

[DilumAluthge]

JuliaCI/rootfs-images#225

[DilumAluthge]

Okay, it looks like the PDF was built successfully. And it built in 27 minutes, which is faster than GitHub Actions (probably because our CI machines are newer).

@mortenpi Would you mind looking at the PDF (which was uploaded as a Buildkite artifact) and making sure it looks correct?

[DilumAluthge]

@staticfloat In this PR, we build Julia in the build_pdf_docs job. I had trouble getting the "use a pre-built Julia" to work.

In a follow-up PR, we can switch the build_pdf_docs job to using a pre-built Julia, but for this PR, it might be easier to leave it as-is so that we can get this PR merged relatively quickly.

[staticfloat]

It should be identical to the doctest builder. Also, why aren't we just adding this on to the end of the doctest builder? Because we need pdflatex? Why not try switching the doctest builder to the new latex rootfs you have here and building the HTML and PDF docs at once?

[DilumAluthge]

Also, why aren't we just adding this on to the end of the doctest builder?

This was intentional. It seems to me that the PDF build currently breaks a lot. And I didn't want a broken PDF build to prevent the regular (i.e. HTML) docs from being deployed. So I figured we'd keep them separate, at least initially, so that we can monitor the stability of the PDF build for a while.

[mortenpi]

The PDF looks fine to me! I think we just now need a variant of the deploy_docs.yml job to actually push the PDF to the assets branch?

This was intentional. It seems to me that the PDF build currently breaks a lot. And I didn't want a broken PDF build to prevent the regular (i.e. HTML) docs from being deployed.

I think this is a fair concern. I suppose they could be in the same builder if it is somehow possible to split things into independent-ish steps on Buildkite?

[staticfloat]

This is looking good to me, next up is actual uploading.

[DilumAluthge]

The core functionality is here. We use a pre-built Julia, build the PDF, and upload it as a Buildkite artifact.

I'm going to merge this now so that we can start running it on Base Julia (and thus catch any regressions). The signed pipeline to upload the PDF to the correct GitHub repo can be done in a follow-up PR.