Replacing latexmk -pvc by BufWritePost autocommand?

[michamos]

Hi everyone,

I have been using LaTeX-Box for some time now, and I find the approach of being lightweight and not implementing things that are already taken care of by other plugins (e.g. snippets) very appealing.

However, there is one aspect of LaTeX-Box that departs IMHO from this minimalistic philosophy, namely the handling of latexmk.
As far as I understand, this complexity is required in order to support the -pvc mode, as latexmk then runs continuously in the background to watch for changes to the document, and it is necessary to watch the process to load errors in the quickfix window and kill it when appropriate.

Although the auto-preview functionality of -pvc is extremely convenient, I think that it is possible to offer the same functionality in a much simpler way. Indeed, Vim knows when a buffer is saved, and it is possible to call latexmk automatically when that happens instead of having it run continuously. This also allows us to use the standard Vim compiler infrastructure and optionally Tim Pope’s vim-dispatch plugin for asynchronicity.

In practice, I suggest the following:

1. Keep the construction of the latexmk command with optional parameters, etc. (but not -pvc). Use it to set makeprg and put it as well as the errorformat in a compiler plugin.
2. Replace the body of LatexBox_latexmk() by a call to :make or :Make[!] (provided by vim-dispatch) according to the value of g:LatexBox_latexmk_async (and use :Copen in the async case for errors).
3. If g:LatexBox_latexmk_preview_continuously is set, register a BufWritePost autocommand to call LatexBox_latexmk(), and either use the -pv option for latexmk or let the previewer handle the reloading itself when it notices that the output file has changed (at least Skim and Evince can do this).

What do you think of this proposal?

[lervag]

Hi,

First, I totally agree that the latexmk part of LaTeX-Box departs from the basic idea of being minimalistic. However, I think the main reason is to support the asynchronuous mode with the vim-server, that is, not to support the -pvc mode.

If I am not mistaken, you were the one to submit the pull request that merged the vimserver approach with the no-vimserver approach. The reason we need both, is that the vimserver approach is superior because it gives asynchronisity, which most people want. However, it is not available on windows, and so we need the no-vimserver approach to be portable.

If we change the system such that it depends on an external plugin, this will in itself add some complexity. However, I do believe this might be the way to go in order to improve the code and make it both more flexible, minimalistic, and portable. I have previously looked at vimproc, however, I think vim-dispatch may look more promising.

In conclusion, I will support a proposal that simplifies the current system, even if it uses an external plugin. However, I do not agree that we should remove support of the -pvc flag for latexmk.

[LeszekSwirski]

Looks like syntastic went through much the same thought process. The result of the discussion seemed to be to stick with synchronous evaluation, because

• vim-dispatch only supports outputting to quickfix lists
• vim-proc expects the user to poll, rather than pass a callback
• They don't like having to rely on clientserver

My additional comments:

• vim-dispatch doesn't support killing running processes (although this may come one day, see vim-dispatch#28)
• We could call vim-proc with a manually specified clientserver-style callback, but this would reintroduce complexity into LaTeX-Box
• vim-proc requires a compiled dll/so file, vim-dispatch doesn't

[lcd047]

For what it's worth, I'd summarize the discussion at syntastic in slightly different terms:

• AsyncCommand relies on Python support being compiled into Vim, and uses the client-server mode. The main problem with it is that it seems to be prone to leaving zombie processes running in background, chewing 99% of the CPU cycles;
• Vim-dispatch relies on tmux, and uses an interesting trick to work: it puts the background job in a split pane, which goes away when the job is done; consequently, the other pane receives a SIGWINCH, which Vim translates into a VimResized event. The main problem with this (aside from using quickfix lists rather than loclists) is that having Vim's window resized every time you save a file would be "gross" (according to Tim Pope).
• Vimproc relies on a dll/so library to work (the C sources are provided). The main problem with it is that it expects the caller to poll to get the status of the background job, rather than providing some async notification when the job is done.
• Vim's client-server mode needs either an X clipboard (on UNIX and Mac OS-X), or OLE (on Windows), and thus it requires a GUI. I don't see anything wrong with relying on it aside from that, but the client-server mode alone is not enough to solve the initial problem.
• People have also mentioned vim-misc and vim-shell. I haven't looked at either.

Finally, I gave up on all this not for technical reasons, but because I don't care enough about it to spend my time finding ways to overcome Vim's clunkiness. The project's owner (who seems to be on an extended vacation for now) may or may not share my views. So please, do drop us a note if you can find a reasonable solution. :)

[lervag]

So, what are your conclusions, @LeszekSwirski? You suggest not to rely on the vimserver, and thus to remove the vimserver support?

Personally, I have become very fond of the -pvc flag for latexmk. The vim interface to latexmk is not really necessary, although it is rather convenient. Today, my LaTeX editing process concists of:

1. Open vim (or gvim)
2. Use ctrlp (or whatever) to open desired latex project (tex-file)
3. Issue <leader>ll to start latexmk (with -pvc)
4. Work on the document: everytime I write to disc, latexmk updates the output file
5. Stop latexmk if desired with <leader>lk

In the current version of LatexBox, we already depend heavily on latexmk. Is there a reason NOT to use the -pvc flag? If -pvc was mandatory, we could always put the command in the background, which means the vimserver is not necessary. I think the code would become much more simple in this case, and I think the -pvc flag would be generally considered convenient for most people.

One obvious reason to prefer not to use -pvc: If latexmk is running in the background, how do you notice if the compilation is not successful? Personally, I use xdotool, which can be used to update the window title of the pdf viewer after compilation. I do this by setting the following variables in my .latexmkrc file, but these settings can also be defined in the command line:

$compiling_cmd = "xdotool search --name \"%D\" set_window --name \"%D compiling...\"";
$success_cmd   = "xdotool search --name \"%D\" set_window --name \"%D OK\"";
$failure_cmd   = "xdotool search --name \"%D\" set_window --name \"%D FAILURE\"";

If I notice a FAILURE, I open the error window with <leader>le. One could perhaps also use $(compiling|success|failure)_cmd for the callback commands (which would require the vimserver, I guess)? Or perhaps they can be used in better and more creative ways?

I would also consider the suggestion by @michamos to not support the -pvc mode to be viable. If we remove the support of the background compilation with -pvc, the code would become more simple. However, I think we would still need the vimserver in order to compile asynchronuously. To not compile asynchronuously is not an option, I think, since many latex documents may take very long to compile (~30s and more), and it is not acceptable to be locked out of editing while waiting for compilation to finish.

A final question: In which cases is the vimserver not available? If I remember correctly, the contribution of @michamos that introduced the async option was introduced in order to increase portability. That is, I remember reading that the vimserver was not available on windows systems, and so we needed the async option. However, with the contribution of @LeszekSwirski, the vimserver approach should now work also on windows: Is it an option to remove the async option and to go back 100% to the vimserver approach?

[LeszekSwirski]

@lcd047 Just a couple of corrections/clarifications

• AsyncCommand doesn't require python, but it doesn't support killing processes, so it's mostly functionally a subset of vim-dispatch and vimproc
• Vim-dispatch doesn't rely on tmux, it's just one of the ways it can do async. If clientserver is available, then it uses it (see dispatch#callback). The resizing thus doesn't have to happen if a process is started as a background process.
• I've no problem with using vimserver where available, though I've had the experience that vim started in an Ubuntu terminal didn't have vimserver, while gvim did.
• Vim-shell/vim-misc is mostly a system() replacement which doesn't pop-up a cmd window on Windows (while Vim-dispatch "pops up" a minimised window, which I'm personally ok with)

@lervag I don't suggest removing vimserver at all, I think it's great. Personally, I would first suggest merging the codepaths for async and continuous preview, and only have the callback depend on vimserver (otherwise, the user has to manually <leader>le).

I also consider getting rid of -pvc sensible, because BufWritePost would do pretty much the same thing but doesn't rely on background processes (plus it'll allow callbacks, so quickfix can auto pop up on write).

After that, my personally preferred path would be to contribute process killing to vim-dispatch (and possibly live without process killing for now), and then switch to using that, because there's a lot of code duplication between the current LaTeX-Box code and Vim-Dispatch.

[LeszekSwirski]

Oh, and if people still want to use -pvc, which is perfectly acceptable, it's my opinion that they should just start it from outside of vim, following the Unix as an IDE philosophy.

[michamos]

Sorry for disappearing after opening this issue, I was pretty busy the last couple of days.

First I would like to rectify one of @lervag's comments, who gave me undue credit for @mtth's work on the async option. All I did was complain about the changed escaping of options :)

Back to this issue, I think there are really two different use cases:

1. editting a LaTeX document, and compilirng it when the user explicitly chooses to;
2. having the document auto-compile on save.

I think it is pretty reasonable for LaTeX-Box to support both modes. For 1. vim-dispatch looks like an excellent fit, with the foreground build :Make command. This provides asynchronicity and does not steal focus. This would introduce a dependency on an external plugin, but this seems reasonable to avoid code duplication (and installing a pure-vim plugin nowadays is pretty easy).

For 2., the current approach is with -pvc. As I already said, I think this is too complex, and could be replaced by a BufWritePost autocommand for the same effect, with the added benefit mentionned by @LeszekSwirski. Here also, I think vim-dispatch could be used, but for a background build :Make!, which would avoid the resizing issues.

[lervag]

Hi,

First: My apologies to @mtth for giving his due credit to the wrong person!

@michamos You write: "For 1. vim-dispatch looks like an excellent fit, with the foreground build :Make command. This provides asynchronicity and does not steal focus." However, as far as I understand :Make is not asynchronuous. At least not in X11 for gvim (I have personally tested this).

It seems that both @michamos and @LeszekSwirski think that it is a good idea to remove support of -pvc. I find the arguments to be good, and I also agree with @LeszekSwirski about "following the Unix as an IDE philosophy". I am not fully convinced that vim-dispatch is the way to go, mostly because of the asynchronicity. But I do agree that if we find a way to solve both use cases stated by @michamos that does not use -pvc, that does allow to automatically open the error window, and that is "simple", then I would of course be very happy!

[LeszekSwirski]

On second (third? fourth?) look, it looks like :Make is sometimes asynchronous; that is, it's asynchronous on tmux, windows and iterm (and possibly screen), but otherwise it just runs :make. :Make!, which is the background version, is asynchronous (it would run headless in @lervag's case), but it doesn't open the quickfix list. It appears that to use vim-dispatch we'd need to patch it to have custom callbacks, which is easy enough but I'm not 100% sure it's something Tim Pope would accept.

I entirely agree than we need asynchronous compilation, and can't go with vim-dispatch without it. The only reason I keep arguing in vim-dispatch's favour is because it's so close to being what we want it to be!

[LeszekSwirski]

I've submitted a PR to vim-dispatch, let's see what happens.

[lervag]

I'm just curious: How exactly would one use a custom callback function with your PR?

[LeszekSwirski]

Something along the lines of:

function! LatexBox_Callback(request)
  call LatexBox_LatexErrors(0)
endfunction

call dispatch#compile_command(!0, "latexmk %", "LatexBox_Callback")

[lervag]

Ah, I see! I was thinking in terms of :Make and :Dispatch, so I didn't consider actually calling the API itself (which is really quite obvious). Anyway, good work! :)

[LeszekSwirski]

Yeah, I didn't want to change the behaviour of :Make or :Dispatch, and it wasn't entirely obvious how one could add custom callbacks to them anyway without doing some magic command line parsing.