PHP crashing with too many image requests

[goldmerc]

@jcupitt - Hi John! Hope you're well. Sorry, need to pic your brain again -

PHP 8.2
VIPS 8.14.3
PHP-VIPS 2.2.0

My website uses VIPS to thumbnail images and caches the image. If I open a page with too many uncached images (approx 30ish or more) it crashes PHP.

I'm a hobby coder and my server is weak - its the 2gb Linode. I've set it to only have 5 max workers. My php error log reads like this...

[28-Aug-2023 19:47:51] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[28-Aug-2023 19:48:03] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[28-Aug-2023 19:48:38] NOTICE: Terminating ...

Could this be a memory leak issue? or do I just need a more powerful server, so I can increase the number of workers?

Thanks in advance for your help. Sorry if this isn't actually a VIPS issue.

[goldmerc]

ps. the command I'm using is \Jcupitt\Vips\Image\thumbnail_buffer. Then I write the file to disk.

from looking online, should I be calling Vips\Config::shutDown() after the thumbnail_buffer command?

[jcupitt]

Hiya, I would have something to limit the number of simultaneous conversions. Or pay (ugh!) for a more powerful server.

[goldmerc]

Thanks John, sorry to be dim but when you say something to limit the number of simultaneous requests, do you have something in mind? or do you simply mean don't have too many uncached images on the same page?

Also, I just tried to use Config::shutdown and that method doesn't exist. I think it's now Vips\FFI::shutDown(). Should I be using that or is it not needed?

[jcupitt]

There are loads of ways of doing this. Most of them use the DB: when an image request comes in, you append the job details to a JOBS table, and you have a separate process, perhaps even on a different machine, which runs in a loop taking the top N jobs from the JOBS table and executing them.

You don't need to use shutdown in php, and it'll probably cause crashes used in a webserver. It's mostly there for debugging.

[goldmerc]

Sorry John, I misunderstood. Yes, I can queue the image thumbnailing but then I can't serve the images on demand as far as I understand. I suppose in the worst case, I'll just change my code to generate all the thumbnails in advance, queueing the operations as you suggest.

However, as a test, I just tried setting pm.max_requests to a low number...

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries.
setting pm.max_requests = 5; 

And now the pages can load without crashing PHP. The log shows the processes stopping and starting but PHP doesn't go down. As I mentioned, I'm just a hobby coder and I don't really understand all this stuff - I'm just tinkering.

[jcupitt]

Oh OK, well that sounds like a good solution.

[goldmerc]

Hi @jcupitt
As the issue seems to be something to do with PHP-FPM, I decided to run a script from the CLI to thumbnail my images in advance. It took over 12 hours but I generated over 500,000 thumbnails no problem. The server and PHP didn't crash.
I don't understand how come I can do so many image transformations without trouble but loading a web page with around 40 image transformations crashes PHP. It seems to be a PHP-FPM / VIPS issue.
I'm compiling libvips from source. I found this... #142 (comment)
At the moment the only build option I'm passing is --buildtype release. Could it help if I try building libvips with -Wl,-z,nodelete?
Or is that no longer relevant?
Thanks again as always!

[jcupitt]

I think it ought to be setting nodelete for you, but you could check.

[goldmerc]

So I tried building with...

LDFLAGS='nodelete'
meson setup build --buildtype release

But I'm still getting the same issue of php crashing. Do I need to do anything else to get the nodelete working?

[jcupitt]

No, that should be it, sorry.

To dig any more I think I'd need to set up a docker env with a php / apache server in debug mode :(

I don't have any free time for this now, we'll need to wait for a php expert to wander by.

[goldmerc]

Hi @jcupitt

No worries. I wouldn't expect you to do that for me! Sorry I don't know how to debug it myself. For the record I'm actually using nginx.

Anyway, I did try some experiments over the weekend. I changed my code to call vipsthumbnail directly using the symfony process class. Works absolutely fine without any need to change the php pool settings. Then I installed sharp.js and wrote a quick js script which I ran with node, also through the symfony process class. That worked just fine too. I've had no trouble using php-vips from the cli. So, it does look like the issues lies somewhere with php-vips and php-fpm. It may be an issue with how I've compiled vips? Or an issue with php-vips. I'm afraid it's beyond me.

Anyway, I have been super impressed with using sharp.js. It's so easy as I don't need to compile vips myself. I think I'm going to go down that route for now.

Thanks for all your help. And for vips which is brilliant! :-)