stack haddock --haddock-for-hackage

[pbrisbin]

I've spent the better part of two days trying to manually upload haddocks to Hackage (they won't build there due to a system library dependency). I've arrived at a state where:

• stack haddock can build complete, locally browse-able documentation successfully, but it cannot build in a way suitable for publishing to Hackage (if this is wrong, please-please correct me!)
• cabal haddock can build documentation meant for Hackage, but it cannot build my project's dependencies documentation (due to a quickjump-related bug) so what it publishes is missing effectively all links

Therefore, I would love to see stack haddock reach feature-parity with cabal haddock --haddock-for-hackage, so I could solve the first point and stop fighting with the second.

---

As far as I can tell, --haddock-for-hackage does a number of things. Some of them I could accomplish externally with the docs after stack haddock produces them, but others I cannot.

The things cabal haddock --haddock-for-hackage does, that I can't seem to make happen by any means are:

• The HTML has the correct "preamble" (e.g. Contents link goes back to the main page)
• Links to dependencies go to the correct on-Hackage locations (/package/$pkg-$version)

The things cabal haddock --haddock-for-hackage does, that I could make happen externally, but ideally wouldn't have to are:

• A correctly-structured, and correctly compressed <package>-<version>-docs.tar.gz file is created

  • It must have a single, top-level directory <package>-<version>-docs that contains only the package's docs
  • It must be (if using GNU tar) compressed with --format=ustar

And lastly, cabal upload --publish --documentation <archive> handles the authorized PUT to /package/<package>-<version>/docs to actually publish them, which is another thing I could handle externally, but would be ideal for stack upload to know how to do.

As mentioned, I've been fighting with this, so I have some details that might be useful for investigating this:

The --haddock-for-hackage option implies at least:

• --haddock-hyperlinked-source
• --haddock-html-location='/package/$pkg-$version
• --haddock-content-location='/package/$pkg-$version
• --haddock-quickjump
• --haddock-preamble=?I'm not sure?

It's hard to exactly trace through the Cabal code, but the --x-location options are key, and are particularly hard (impossible?) for me to accomplish any way besides using cabal haddock, as it seems to end up interacting with an isolated package registry in order to locate the dependencies in order to construct by-dependency options to pass to haddock telling it where each thing is. In other words, stack haddock kind of has to do it for me.

[mpilgrem]

@pbrisbin, thanks for reporting. I'll look into this, but it is part of Stack that I have not previously investigated, so please bear with me.

[mpilgrem]

@pbrisbin, while looking into the background to this, I identified: https://hackage.haskell.org/package/standalone-haddock. Beyond its description, I've not looked at it further. Mentioning it in case it is of immediate assistance.

[pbrisbin]

That does look super useful, thanks!

[mpilgrem]

Notes for myself:

First, looking at what Cabal (the library) does, Cabal.Simple.Haddock.haddock includes:

flags = case haddockTarget of
          ForDevelopment -> flags'
          ForHackage -> flags'
            { haddockHoogle       = Flag True
            , haddockHtml         = Flag True
            , haddockHtmlLocation = Flag (pkg_url ++ "/docs")
            , haddockContents     = Flag (toPathTemplate pkg_url)
            , haddockLinkedSource = Flag True
            , haddockQuickJump    = Flag True
            }
        pkg_url       = "/package/$pkg-$version"

        htmlTemplate  = fmap toPathTemplate . flagToMaybe . haddockHtmlLocation
                        $ flags

Also:

fromFlags :: PathTemplateEnv -> HaddockFlags -> HaddockArgs
fromFlags env flags =
    mempty {
      ...
      argLinkSource = if fromFlag (haddockLinkedSource flags)
                               then Flag ("src/%{MODULE/./-}.html"
                                         ,"src/%{MODULE/./-}.html#%{NAME}"
                                         ,"src/%{MODULE/./-}.html#line-%{LINE}")
                               else NoFlag,
      argLinkedSource = haddockLinkedSource flags,
      argQuickJump = haddockQuickJump flags,
      ...
      argContents = fmap (fromPathTemplate . substPathTemplate env)
                    (haddockContents flags),
      ...
      argOutput =
          Flag $ case [ Html | Flag True <- [haddockHtml flags] ] ++
                      [ Hoogle | Flag True <- [haddockHoogle flags] ]
                 of [] -> [ Html ]
                    os -> os,
      ...
    }

Also:

getInterfaces :: Verbosity
              -> LocalBuildInfo
              -> ComponentLocalBuildInfo
              -> Maybe PathTemplate -- ^ template for HTML location
              -> IO HaddockArgs
getInterfaces verbosity lbi clbi htmlTemplate = do
    (packageFlags, warnings) <- haddockPackageFlags verbosity lbi clbi htmlTemplate
    traverse_ (warn (verboseUnmarkOutput verbosity)) warnings
    return $ mempty {
                 argInterfaces = packageFlags
               }

Also (Stack does not support Haddock before version 2.17.2):

renderPureArgs :: Version -> Compiler -> Platform -> HaddockArgs -> [String]
renderPureArgs version comp platform args = concat
    [ ...
    , [ "--quickjump" | isVersion 2 19
                      , _ <- flagToList . argQuickJump $ args ]

    , [ "--hyperlinked-source" | isVersion 2 17
                               , True <- flagToList . argLinkedSource $ args ]
    ...
    , maybe [] (\(m,e,l) ->
                 ["--source-module=" ++ m
                 ,"--source-entity=" ++ e]
                 ++ if isVersion 2 14 then ["--source-entity-line=" ++ l]
                    else []
               ) . flagToMaybe . argLinkSource $ args
    ...
    , maybe [] ((:[]) . ("--use-contents="++)) . flagToMaybe . argContents $ args
    ...
    , map (\o -> case o of Hoogle -> "--hoogle"; Html -> "--html")
      . fromFlagOrDefault [] . argOutput $ args

    , renderInterfaces . argInterfaces $ args
    ...
    ]
    where
      renderInterfaces = map renderInterface

      renderInterface :: (FilePath, Maybe FilePath, Maybe FilePath, Visibility) -> String
      renderInterface (i, html, hypsrc, visibility) = "--read-interface=" ++
        (intercalate "," $ concat [ [ fromMaybe "" html ]
                                  , -- only render hypsrc path if html path
                                    -- is given and hyperlinked-source is
                                    -- enabled
                                    [ case (html, hypsrc) of
                                        (Nothing, _) -> ""
                                        (_, Nothing) -> ""
                                        (_, Just x)  | isVersion 2 17
                                                     , fromFlagOrDefault False . argLinkedSource $ args
                                                     -> x
                                                     | otherwise
                                                     -> ""
                                    ]
                                  , if haddockSupportsVisibility
                                      then [ case visibility of
                                               Visible -> "visible"
                                               Hidden  -> "hidden"
                                           ]
                                      else []
                                  , [ i ]
                                  ])

      isVersion major minor  = version >= mkVersion [major,minor]

[mpilgrem]

Further notes for myself:

Working through the above, I think I have the following Haddock flags and options:

--quickjump
--hyperlinked-source
--source-module=src/%{MODULE/./-}.html
--source-entity=src/%{MODULE/./-}.html#%{NAME}
--source-entity-line=src/%{MODULE/./-}.html#line-%{LINE}
--use-contents=/package/$pkg-$version [AFTER SUBSTITUTION of $pkg and $version]
--hoogle
--html
--read-interface=<html_path>,<hypsrc_path>[,visible|hidden],<i_path>
--read-interface=<html_path>,<hypsrc_path>[,visible|hidden],<i_path>
...

[mpilgrem]

For Stack, this seems to me to involve four five steps:

• 1. Enable, as an alterative, the passing of --for-hackage to Setup.hs haddock
• 2. Ensure Stack knows that toggling between --[no-]haddock-for-hackage means that things have to be rebuilt
• 3. Create the archive file for the Haddock documentation
• 4. Extend stack upload to include a -d, --documentation option
• 5. Improve the error messages for stack upload

[mpilgrem]

Cabal (the library) has a couple of (minor) issues with --for-hackage: see haskell/cabal#5210 ("cabal new-haddock --haddock-for-hackage claims to have created non-existent files") and haskell/cabal#9318 ("haddock --for-hackage enables Haddock's --hyperlinked-sources AND --source-* options").

[mpilgrem]

@pbrisbin, there is some tidying up to do around the edges, but the master branch version of Stack now does what I think you wanted it to do.

I've tested it on Stack itself, as follows:

Hackage could not build Haddock documentation for stack-2.13.1, logging:

Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: stack-2.13.1 (user goal)
[__1] next goal: template-haskell (dependency of stack)
[__1] rejecting: template-haskell-2.18.0.0/installed-2.18.0.0 (conflict: stack
=> template-haskell>=2.19.0.0)
[__1] rejecting: template-haskell-2.20.0.0, template-haskell-2.19.0.0,
template-haskell-2.18.0.0, template-haskell-2.17.0.0,
template-haskell-2.16.0.0, template-haskell-2.15.0.0,
template-haskell-2.14.0.0, template-haskell-2.13.0.0,
template-haskell-2.12.0.0, template-haskell-2.11.1.0,
template-haskell-2.11.0.0, template-haskell-2.10.0.0,
template-haskell-2.9.0.0, template-haskell-2.8.0.0, template-haskell-2.7.0.0,
template-haskell-2.6.0.0, template-haskell-2.5.0.0, template-haskell-2.4.0.1,
template-haskell-2.4.0.0, template-haskell-2.3.0.1, template-haskell-2.3.0.0,
template-haskell-2.2.0.0 (constraint from non-upgradeable package requires
installed instance)
[__1] fail (backjumping, conflict set: stack, template-haskell)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: stack, template-haskell

With the master branch version of Stack as the executable, I applied the following commands:

1. cd stack (to move into the project directory)
2. git checkout v2.13.1 (to checkout version 2.13.1)
3. stack haddock --force-dirty (to build the normal Haddock documentation for the project)
4. stack haddock --haddock-for-hackage (to build Haddock documentation suitable for Hackage)
5. stack upload . --documentation (documentation upload)

and, now, stack-2.13.1 has Haddock documentation on Hackage.

The 'latest' version of the online help is updated: https://docs.haskellstack.org/en/latest/build_command/#-no-haddock-for-haddock-flag and https://docs.haskellstack.org/en/latest/build_command/#-no-haddock-for-haddock-flag.

[pbrisbin]

Thanks @mpilgrem! The docs on Hackage for 2.13.1 look correct in the ways a naive stack haddock && tar && curl did not before. I checked that the Contents link goes back to the main modules listing, and that dependencies linked to their Hackage pages.

[pbrisbin]

Hi @mpilgrem I think there may be a bug with haddock --haddock-for-hackage when run with --dependencies-only. Let me describe a bit here, but let me know if you want a separate Issue reported.

On our CI, with an un-cached build, we run the following step:

stack --stack-yaml stack.yaml --no-terminal build --test --no-run-tests --dependencies-only --haddock --haddock-for-hackage

This fails at the very end, after finishing up for the last dependency:

2024-03-14T15:59:23.5892819Z yesod-test                                 > Registering library for yesod-test-1.6.16..
2024-03-14T15:59:23.7752305Z Completed 290 action(s).
2024-03-14T15:59:23.7784057Z /home/runner/work/freckle-app/freckle-app/.stack-work/dist/x86_64-linux-tinfo6/ghc-9.6.4/freckle-app-1.15.0.1-docs.tar.gz: withBinaryFile: does not exist (No such file or directory)

It seems like it's trying to do something related to the eventual docs, but since we're only doing the dependency docs, that archive is not there (yet).

Due to the quirks of stack-action, this failure will still store a cache, so when we do a re-run, that step is skipped and we proceed right to the non-dependencies build:

stack --stack-yaml stack.yaml --no-terminal build --test --no-run-tests --haddock --haddock-for-hackage

This works fine, since it builds the app docs to, which I guess it'll need at the end:

2024-03-12T18:25:33.1427198Z yesod-test                                 > Registering library for yesod-test-1.6.16..
2024-03-12T18:25:33.3337823Z freckle-app                                > configure (lib + test)
2024-03-12T18:25:33.3473748Z Configuring freckle-app-1.15.0.0...

...

2024-03-12T18:26:09.6409799Z Documentation created:
2024-03-12T18:26:09.6411053Z .stack-work/dist/x86_64-linux-tinfo6/ghc-9.6.4/doc/html/freckle-app-1.15.0.0-docs/index.html,
2024-03-12T18:26:09.6415986Z .stack-work/dist/x86_64-linux-tinfo6/ghc-9.6.4/doc/html/freckle-app-1.15.0.0-docs/freckle-app.txt

I'll admit I'm not exactly sure how doing Haddocks changes the notion of building and caching dependencies in their own step, it truly seems to build the world no matter what I do anyway, so please let me know if this is operator error.

[mpilgrem]

@pbrisbin, thanks for the report. I think the problem is this: Haddock documentation gets built for each package and then, at the end of the build, it all gets pulled together. In the case of Haddock --for-hackage, that building only applies to 'mutable' packages - project packages or local extra-deps - and it gets pulled together at the end as an archive file for each project package. However, by specifying --only-dependencies no project packages (and their Haddock documentation) have been built and - despite the project packages being targets - there is nothing to pull together. It had not occurred to me that --only-dependencies and --haddock-for-hackage would be combined. I'll open a separate issue for that.

[pbrisbin]

Thanks for the quick response!