Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kiwix-serve: support passing socket as stdin (inetd-style) #610

Open
opk12 opened this issue Mar 22, 2023 · 11 comments
Open

kiwix-serve: support passing socket as stdin (inetd-style) #610

opk12 opened this issue Mar 22, 2023 · 11 comments

Comments

@opk12
Copy link

opk12 commented Mar 22, 2023

Traditionally on *nix, a service manager spawns the daemon and duplicates the socket to its standard input and standard output. The socket can even be AF_UNIX, bridged to the network interface by the service manager.

This approach is common nowadays (and supported by systemd), but it's called inetd-style because of historical reasons. More info at systemd "Converting inetd services".

Why it's better than binding to the network interface directly:

  • The system service manager centralizes the interface and port mappings for all daemons, so the config is cleaner.
  • TLS, logging, access control, and proxies can be added between the daemon and the network interface.
  • Containerizing the daemon into its own network and IPC namespaces, to block access to the network and to other daemons, is the standard approach on Linux nowadays, to protect from a compromised daemon. This requires ugly workarounds if the daemon wants to bind to the network interface directly.
@kelson42
Copy link
Contributor

kelson42 commented Apr 1, 2023

@rgaudin @mgautierfr @veloman-yunkan You validate the feature request?

@mgautierfr
Copy link
Member

It seems good to me.
And it may be also (as a bonus) a good way to test the server as we would have to run it on different port of things like that. (But not sure if our http client can do it)

@rgaudin
Copy link
Member

rgaudin commented Apr 4, 2023

Thanks for the link ; I've recently experimented with this and found it under-documented and somewhat difficult.

The main question would be which mode(s) to support: single server that gets instantiated on first request or per-connection instances.

From my limited understanding of how libkiwix works, per-connection instances would be terrible. I had difficulties with the Singleton mode (couldn't get a hold on the initial request) but I wasn't ready to invest much time in this and relied on third parties.

I believe apache has a module that allows this so it might be a useful implementation sample.

@opk12
Copy link
Author

opk12 commented Apr 4, 2023

Generally a (modern) daemon does the multiplexing, so that initialization only happens once. This corresponds to Accept=false in systemd and is advised in systemd.socket in the section Accept=. Multiple daemon instances can still be started from the outside if necessary, as in this example for QGis Server.

@veloman-yunkan
Copy link
Collaborator

If kiwix-serve gives up binding on the network interface directly, I don't see how it can serve concurrent requests efficiently (i.e. having a single shared cache of open ZIM files and search sessions). @opk12 Do you mean supporting operation via stdin/stdout as an option?

I think that my response somewhat repeats @rgaudin's comment above yet I thought that the concern better be stated straightforwardly.

@opk12 opk12 changed the title kiwix-serve: support passing socket as stdin and stdout (inetd-style) kiwix-serve: support passing socket as stdin (inetd-style) Apr 5, 2023
@opk12
Copy link
Author

opk12 commented Apr 5, 2023

That the format spec is free to be painted on the walls does not change the vendor lock-in effects if I can't access zim with favorite third-party tools; or I must use a web server that parses downloaded material, but also requires network access; or have to wait ages because compliance with the big redistributors is not a top priority (no LTS branch for Debian, app non-freeness caused by non-free build deps for F-droid) . So the very general context is adhering to proven, widespread, basic design patterns and integrating into the free software community / ecosystem, in favor of the technical or privacy-minded WMF projects user.

This specific feature request just changes where kiwix takes the socket from. Everyone says so far that it's better that kiwix makes connections from a socket ("single server that gets instantiated on first request" from above). A daemon can take a socket and forget where it's taken from, then make connections from it, regardless of how it was constructed. I've amended the title to remove the reference to stdout.

@mgautierfr
Copy link
Member

There is few different things in your last comment.

That the format spec is free to be painted on the walls does not change the vendor lock-in effects if I can't access zim with favorite third-party tools; or I must use a web server that parses downloaded material, but also requires network access; or have to wait ages because compliance with the big redistributors is not a top priority (no LTS branch for Debian, app non-freeness caused by non-free build deps for F-droid) . So the very general context is adhering to proven, widespread, basic design patterns and integrating into the free software community / ecosystem, in favor of the technical or privacy-minded WMF projects user.

This sound like a rant because we don't do enough. Guest what, this is free software. You can do more than open issue, pull request are welcomed.
You want a debian LTS package ? Please do. We had a contributor on debian packaging but he now lack of time.
You want a free build on F-droid ? Please do. Everything is free software, just package it.

We are a really small team, not working fulltime on kiwix. We do the best we can do depending on our priorities and our capacities (we are the "only" ones to know the code where a lot of people can do the packaging).

This specific feature request just changes where kiwix takes the socket from. Everyone says so far that it's better that kiwix makes connections from a socket ("single server that gets instantiated on first request" from above). A daemon can take a socket and forget where it's taken from, then make connections from it, regardless of how it was constructed. I've amended the title to remove the reference to stdout.

Here I agree with you. We are just discussing the feature to properly understand it and implement it the correct way.

@mgautierfr
Copy link
Member

I've recently experimented with this and found it under-documented and somewhat difficult.

@rgaudin,
The articles from Lennart are really intersting. See :
https://0pointer.net/blog/projects/socket-activation.html
https://0pointer.net/blog/projects/socket-activation2.html
https://0pointer.net/blog/projects/inetd.html (already given in this discussion)

The main question would be which mode(s) to support: single server that gets instantiated on first request or per-connection instances.

This is a valid question but it is a question for the administrator, not us.
Accepting connection given to us by fd allow us to do both without change in our code (at least to handle the two modes, we have to change code to accept fd)

Administrator just have to configure the service correctly to use the "instantiate of first request" mode if they want something efficient.

If kiwix-serve gives up binding on the network interface directly, I don't see how it can serve concurrent requests efficiently (i.e. having a single shared cache of open ZIM files and search sessions). @opk12 Do you mean supporting operation via stdin/stdout as an option?

@veloman-yunkan The idea is that systemd passes us the INET socket so we have to use it instead of binding to INADDR_ANY or the address given at command line argument.
It seems the option in libmicrohttpd is MHD_OPTION_LISTEN_SOCKET

The patch to cups (given in blog's article) http://0pointer.de/public/cups-patch-core.txt show us how we can get the socket from the fd.

@opk12
Copy link
Author

opk12 commented Apr 13, 2023

Sorry for the tone. You do too much already.; It's just the mentioned lack of integration.

@kelson42
Copy link
Contributor

@opk12 Thank you for opening this issue. We all agree that this would be a good ticket to implement. I will work to integrate it in a future milestone, but for now it does not belong to the top priorities. So, you will need a bit of patience ;)

@kelson42 kelson42 added this to the 3.9.0 milestone May 25, 2024
@srd424
Copy link

srd424 commented Oct 18, 2024

FWIW, looks like you can approximate direct socket activation like this:

# /etc/systemd/system/kiwix-proxy.socket
[Socket]
ListenStream=8085

[Install]
WantedBy=sockets.target
# /etc/systemd/system/kiwix-proxy.service
[Unit]
After=kiwix-serve.service kiwix-proxy.socket
Requires=kiwix-serve.service kiwix-proxy.socket

[Service]
ExecStart=/lib/systemd/systemd-socket-proxyd --exit-idle-time=10s 127.0.0.1:8084
Type=notify
# /etc/systemd/system/kiwix-serve.service
[Unit]
StopWhenUnneeded=yes

[Service]
ExecStart=kiwix-serve -p 8084 /export/kiwix/steved/*.zim

I don't think it's quite perfect because sometimes it appears it tries to forward the connection before kiwix-serve has finished initializing, but it comes fairly close. Native support would be great but this looks like a reasonable stop-gap!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants