MAX_N_PID should be 4194304 on 64bit machines

[merkys]

While using t-coffee on Debian, it has been noticed that on 64bit machines process IDs (PIDs) exceed the default MAX_N_PID setting (currently 260000) and starts acting weird (please see the original Debian bug report). I would suggest either increasing the default value of MAX_N_PID to 4194304, or doing so only when compiled 64bit machines.

[EricDeveaud]

please do nt set an hard coded value. instead use the value that the OS and kernel may supply

MAX PID can be obtain via 2 calls
eg:
cat /proc/sys/kernel/pid_max
or
sysctl kernel.pid_max

it can also be determined programaticaly

the maxpid value is set by the os and can be edited by admins why 4194304 ? size_t

examples:

maestro-submit:~ > uname -s -p -o -m
Linux x86_64 x86_64 GNU/Linux
maestro-submit:~ > sysctl kernel.pid_max
kernel.pid_max = 65536

tars-submit0:~ > uname -s -p -o -m               
Linux x86_64 x86_64 GNU/Linux
tars-submit0:~ > sysctl kernel.pid_max
kernel.pid_max = 57344

regards

Eric

[cnotred]

Thanks a lot Eric, This is very useful. I will implement this ASAP Cedric Le 1/7/20 à 18:03, Eric Deveaud a écrit :

please do nt set an hard coded value. instead use the value that the OS and kernel may supply MAX PID can be obtain via 2 calls eg: |cat /proc/sys/kernel/pid_max| or |sysctl kernel.pid_max| it can also be determined programaticaly the maxpid value is set by the os and can be edited by admins why 4194304 ? size_t examples: |maestro-submit:~ > uname -s -p -o -m Linux x86_64 x86_64 GNU/Linux maestro-submit:~ > sysctl kernel.pid_max kernel.pid_max = 65536 | |tars-submit0:~ > uname -s -p -o -m Linux x86_64 x86_64 GNU/Linux tars-submit0:~ > sysctl kernel.pid_max kernel.pid_max = 57344 | regards Eric — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABML3KLV5AAV3SJZNL72OSDRZNM3RANCNFSM4OBHGCAA>.

-- ########################################## Dr Cedric Notredame, PhD Group Leader Notredame's lab - Comparative Bioinformatics Group Bioinformatics and Genomics Programme Room 440.03 Centre de Regulació Genòmica (CRG) Dr. Aiguader, 88 08003 Barcelona Spain Ph# + 34 93 316 02 71 Fax# + 34 93 316 00 99 Mobile# + 34 66 250 47 82 email [email protected] url www.tcoffee.org blog cedricnotredame.blogspot.com ORC-ID: 0000-0003-1461-0988 ###########################################

[merkys]

please do nt set an hard coded value. instead use the value that the OS and kernel may supply

Yes, this is a wise solution.

This is very useful. I will implement this ASAP

Do you mean you are going to implement taking the MAX_N_PID from the kernel in tcoffee build system?

[cnotred]

yes, I will implement a procedure that uses the kernel value. Unfortunately, I have just realized there does not seem to be any simple programmatic way to do this on mac OSX, does any body have a suggestion on this? Cedric Le 1/7/20 à 18:15, Andrius Merkys a écrit :

please do nt set an hard coded value. instead use the value that the OS and kernel may supply Yes, this is a wise solution. This is very useful. I will implement this ASAP Do you mean you are going to implement taking the |MAX_N_PID| from the kernel in tcoffee build system? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABML3KIADBYKRNE632C4NSTRZNOIPANCNFSM4OBHGCAA>.

-- ########################################## Dr Cedric Notredame, PhD Group Leader Notredame's lab - Comparative Bioinformatics Group Bioinformatics and Genomics Programme Room 440.03 Centre de Regulació Genòmica (CRG) Dr. Aiguader, 88 08003 Barcelona Spain Ph# + 34 93 316 02 71 Fax# + 34 93 316 00 99 Mobile# + 34 66 250 47 82 email [email protected] url www.tcoffee.org blog cedricnotredame.blogspot.com ORC-ID: 0000-0003-1461-0988 ###########################################

[EricDeveaud]

on my mac (Mojave) man 2 setaudit_addr says:

The ai_asid variable contains the audit session ID which is recorded with
     every event caused by the process.  It can be any value in the range 1 to
     PID_MAX (99999).  

http://www.opensource.apple.com/source/xnu/xnu-1699.24.23/bsd/sys/proc_internal.h
#define PID_MAX 99999

Eric

[Hirun]

https://apple.stackexchange.com/questions/51119/whats-the-maximum-pid-for-mac-os-x

Looking at sys/proc_internal.h in xnu-1699.24.23, I find that PID_MAX is 99999. The value is used in kern_fork.c in the function forkproc. Looking at that function, process IDs are not assigned equal to PID_MAX, so the highest possible pid is 99998

Seems like for Mac OS it is ok to hardcode the limit to 99998

[marco-mariotti]

Dear Cedric and others,
it seems the issue is still there in version 13.45.0.4846264. In my ubuntu server, the precompiled bin throws the error:

--ERROR: MAX_N_PID exceded -- Recompile changing the value of MAX_N_PID (current: 260000 Requested: 407611) OR setenv MAX_N_PID_4_TCOFFEE=407611

But it gets worse: t_coffee just won't die gracefully. Instead, the process hangs around and starts to quickly eat all the memory until the whole computer crashes... unless you notice it and kill it before that happens.

The conda recipe available for t_coffee has this problem, too.

BTW because t_coffee is installed as part of the phylogenetic reconstruction tools in etetoolkit (ete_toolchain, see http://etetoolkit.org/download/), all of this may happen under the hood, without the user being conscious of even running t_coffee. I had many server crashes before I could trace what was going on.

Looks like a high priority t_coffee bug to me, as it prevents its use in any 64bit Debian-like machine.

For those ending up in this page, here's a workaround to install a working version of t_coffee in any Linux system (disclaimer, I tested it in Ubuntu only):

• follow instructions to get the "bugged" t_coffee here
• locate the executable and set its location to a variable, either manually:
  TC2FIX=~/.t_coffee/bin/linux/t_coffee
• or, if you can't find it, automatically:
  TC2FIX=$(which t_coffee)
• finally, replace the bugged t_coffee with a command that sets MAX_N_PID_4_TCOFFEE before every run:
  mv $TC2FIX $TC2FIX.cmd && (echo '#!/usr/bin/env sh'; echo 'export MAX_N_PID_4_TCOFFEE=$(cat /proc/sys/kernel/pid_max)'; echo $TC2FIX.cmd '$@' ) > $TC2FIX && chmod a+x $TC2FIX

Afterwards you should be able to run t_coffee normally.

Marco

[nsoranzo]

The conda recipe available for t_coffee has this problem, too.

@marco-mariotti The t-coffee bioconda package (with a dash not an underscore) definitely doesn't have the issue because we workaround it in https://github.com/bioconda/bioconda-recipes/blob/master/recipes/t-coffee/t_coffee.sh

[marco-mariotti]

Thanks @nsoranzo ! I tested it and yes, it works.
Unfortunately there's two different bioconda recipes, for "t-coffee" and "t_coffee".
The second one refers to t_coffee version 11.0.8, and it is still bugged.

It'd be convenient to link the second recipe to the first one. I don't know who has the admin privileges to do that.

BTW the t-coffee bioconda recipe is a great way to install t_coffee that works across platforms (though arguably a bit heavy on dependencies). To @cnotred and others: perhaps you want to mention this in the installation instructions of t_coffee

[cnotred]

Thanks a lot for the report. I will fix this ASAP over the next few days.

On 16/12/22 19:08, Marco Mariotti wrote: Dear Cedric and others, it seems the issue is still there in version 13.45.0.4846264. In my ubuntu server, the precompiled bin throws the error: |--ERROR: MAX_N_PID exceded -- Recompile changing the value of MAX_N_PID (current: 260000 Requested: 407611) OR setenv MAX_N_PID_4_TCOFFEE=407611| But it gets worse: t_coffee just won't die gracefully. Instead, the process hangs around and starts to quickly eat all the memory until the whole computer crashes... unless you notice it and kill it before that happens. The conda recipe available for t_coffee has this problem, too. BTW because t_coffee is installed as part of the phylogenetic reconstruction tools in etetoolkit (ete_toolchain, see http://etetoolkit.org/download/), all of this may happen under the hood, without the user being conscious of even running t_coffee. I had many server crashes before I could trace what was going on. Looks like a high priority t_coffee bug to me, as it prevents its use in any 64bit Debian-like machine. For those ending up in this page, here's a workaround to install a working version of t_coffee in any Linux system (disclaimer, I tested it in Ubuntu only): * follow instructions to get the "bugged" t_coffee here <http://tcoffee-packages.s3-website.eu-central-1.amazonaws.com/#Stable/Latest/> * locate the executable and set its location to a variable, either manually: |TC2FIX=~/.t_coffee/bin/linux/t_coffee| * or, if you can't find it, automatically: |TC2FIX=$(which t_coffee)| * finally, replace the bugged t_coffee with a command that sets MAX_N_PID_4_TCOFFEE before every run: |mv $TC2FIX $TC2FIX.cmd && (echo '#!/usr/bin/env sh'; echo 'export MAX_N_PID_4_TCOFFEE=$(cat /proc/sys/kernel/pid_max)'; echo $TC2FIX.cmd '$@' ) > $TC2FIX && chmod a+x $TC2FIX | Afterwards you should be able to run |t_coffee| normally. Marco — Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABML3KNYHQGHPVWDW6LMTKDWNSVYXANCNFSM4OBHGCAA>. You are receiving this because you commented.Message ID: ***@***.***>

-- ########################################## Dr Cedric Notredame, PhD Group Leader Notredame's lab - Comparative Bioinformatics Group Bioinformatics and Genomics Programme Room 440.03 Centre de Regulació Genòmica (CRG) Dr. Aiguader, 88 08003 Barcelona Spain Ph# + 34 93 316 02 71 Fax# + 34 93 316 00 99 Mobile# + 34 66 250 47 82 ***@***.*** urlwww.tcoffee.org blog cedricnotredame.blogspot.com ORC-ID: 0000-0003-1461-0988 ###########################################