-
Notifications
You must be signed in to change notification settings - Fork 0
Software validation
You may think of a PKPD software as a calculator. You expect that operations in a calculator are correctly performed internally, so that the machine returns the correct answer. However, as a calculator, the answer is only accurate if it is correctly operated, and the problem at hand is correctly translated into sequential operations that give the correct answer. For instance, how many seconds are there in a day? The correct answer is 24x60x60=86400, and not 24+60+60=144. A validated calculator will give both answers, 86400 and 144 (and not any other number), depending on how it is operated. The result is only as good as the ability of the calculator operator to translate a real-world problem into a tractable mathematical problem that correctly approximates reality ("Essentially, all models are wrong, but some are useful" (George Box, a famous statistician)).
PK, PD, PKPD, PBPK models are nothing more than (more complicated) sequential operations aiming at the analysis and explanation of the observed experimental behaviour of some substance in the body. They normally use differential equations (DE) that is a mathematical tool that helps to explain time variations and physical interactions. Although more complicated, the operator principle is the same.
Every Xcipion installation must be validated to make sure that there is no problem with this version of the software and hardware. We must be sure that the software is returning the right answer if it is given certain data (in our example above if we input 24 times 60 times 60, the calculator must answer 86400). This is called Operational/Performance qualification (OQ/PQ or OPQ). This answer depends on the software as well as the hardware in which it is installed. Finding the parameters of a DE system that better fits some observed data involves several million arithmetic operations (our example of 86400 involved just three of them) and small differences in the hardware, combined with the decisions taken by the search algorithm based on the numbers it sees, result in slightly different answers for the DE parameters. In our 86400 example, it is as if the calculator is returning 86399.999985, the result is still correct (actually with your pocket calculator there is an interesting experiment: A) divide 4 by 3 (answer=1.333333); B) subtract 1 from the result (answer=0.33333); C) multiply the result by 4 (answer: 1.333333, we should be back at the starting point of step B); D) repeat steps B and C 30-40 times, the calculator behaviour is not what you would expect; this experiment has involved no more than 100 operations, at most, fitting DE parameters require several millions).
Additionally, the fitting problem is not uniquely determined and several different sets of DE parameters explain almost equally well the observed data (the statistical theory of sensitivity and identifiability is rich and very interesting for the advanced modeller). In our 86400 example, 24x24x60 is 86400, but 24x59.95x60.05 is also 86400.
Finally, the observed data is noisy (when we say that 30 minutes after the oral dose the drug concentration in plasma was 1 umol/L, our precision of the 30 minutes and 1 umol/L is limited; we do not mean 30 minutes=1,800,000,000 microseconds after the oral dose and 1.00000000000000000 umol/L with no analytical error). It is as if in our 86400 example, we have to look for three numbers whose product is some observed value. This observed value is sometimes 86401, other times 86396, 86407.4, ...
All this results in a variability of the DE parameters. The answer is considered to be correct as long as the parameters fall within an acceptable, predefined range. In our 86400 example, the answer is considered to be correct if the number of seconds per minute is estimated to be between 59.5 and 60.5.
Xcipion comes with a free suite of tests whose results are known (there is no other commercial test suite, the free suite covers all protocols in Xcipion). These tests construct the inputs for multiple protocols and verify that the answer is the expected one. To run these tests, go into the xcipion installation directory and run
./scipion tests --run >& tests.log
grep -i error tests.log
cat tests.log
The test process can go up to 2h depending on the machine. The grep command should only show the error message ERROR: Testing general error [83474] that is a verification that the error display mechanism is correctly working. The cat should show all the logging from the executions in green (no red message should be shown). If there are errors in the test, please, contact [email protected].
You may inspect the results of any of these tests by the command
./scipion project TestGabrielssonXXXXWorkflow &
where XXXX is the label of the tests (typical labels are PK01, PK02, ... PD01, PD02, ...). You may also open xcipion and open the test project you prefer.
./scipion &
Xcipion depends on some external libraries, a Python interpreter, a Linux operating system, etc. Although, it is not normally the case, the installation in any machine may be problematic if the dependencies are not satisfied. To install scipion you must download the source code from https://github.com/cossorzano/scipion/releases, uncompress it and install it. The commands are
wget https://github.com/cossorzano/scipion/archive/v17.05.tar.gz
tar zxvf v17.05.tar.gz
cd scipion-17.05
./scipion config >& config.log
./scipion install >& install.log
The config.log and install.log should not have any red message. In scipion-17.05/software/log, you also have the installation logging for each of the libraries/packages that xcipion has compiled.
After verifying that the installation is correct. We recommend to run the operational qualification to verify that the calculations in this software/hardware combination are correct.
All the code in Xcipion is open source and its changes are carefully tracked. You may see the code at https://github.com/cossorzano/scipion and its changes at https://github.com/cossorzano/scipion/commits/master. Each Xcipion installation has a unique hash code that can be retrieved with
git rev-parse HEAD
so that it can be known exactly with which code an analysis has been executed. Moreover, the software can be moved back to that state and the results exactly replicated.
Absolutely all operations inside Xcipion are recorded and their log files stored. Once a result is generated, we associate to it a hash code that depends on the file content and the time of production. If any of the two changes, the change is detected by the system and the result flagged as unreliable. The only way of getting rid of some result is by deleting it from the system (which is convenient while we are still deciding the best way to analyze some data). Once the project is finished, its current state shows all the necessary steps to go from the data import to the final result, and all its steps can be replicated.
Every new software installation must be qualified because results depend on both the software AND the machine on which it is run. Everything in Xcipion is free and open-source, and you may perform the qualification by yourself, or you may contact [email protected] for accessing already qualified machines running in the cloud (Amazon).