Execute the build on a ZIIP.

[DylanVermeersch]

Given that Groovy uses Java, can you execute the program on a ZIIP? I'm sorry if this question has already been answered, but I didn't find it.

[dennis-behm]

Hi @DylanVermeersch,
(I moved your question into discussions).

computations within the build scripts as well internal functions of the DBB Toolkit can be executed on a ZIIP, because they are basically a java process. The MVS execs like invoking compilers or linkage editor are standard MVS workload.

[DylanVermeersch]

Thanks for the fast reply, I'm using self made groovy scripts based on the build.groovy and the PLI.groovy scripts that can do some of the migration work. The scripts basically submit a couple JCL's to the z/OS system and then look into those outputs and screen it for meta data of applications that were asked by the program. I don't start an MVS Exec or MVS Job anywhere, does the program make one for me?

The reason why I'm asking this is that we're getting an error that says I use too much CPU on the CP so if I can switch the workload to the ZIIP my problem would be over. Only thing we use is the JCLExec.

[dennis-behm]

If you use the JCLExec API - you schedule a job to JES. Anything that you process through the jobs will most likely count to the standard workload.

Are you sure to have some ZIIP capacity left - it there is none, MVS will automatically dispatch to standard CP.

[DylanVermeersch]

We're sure we have enough capacity left. You're saying it is because we use the JCL output and that will always be CP workload?

Maybe a little more context to the program, the program submits maybe 5-6 JCL's wich finish in like less than a minute CPU time and the only thing we do then is go through the output of the JCL's, for example the DD's with OUT***. We save all those outputs to a separate file and then we just parse the file content. We use the JCLExec.saveOutput method to save it to a file when we also monitor the processes we can see that the program eventhough all the JCL's were executed, the program still uses (a lot) of CPU and all on the CP.

[FALLAI-Denis]

Hi @DylanVermeersch,

I understand that your problem is primarily linked to the amount of CPU allowed to your process.

There are various settings that control the amount of CPU allowed to a task, including the CPUTIMEMAX setting allocated in the OMVS segment (USS) account profile in use.

The use of ZIIP is not related to USS processes. ZIIP can be used by various processes. To date: Java, Db2, and as far as we are concerned our dump capture tool.

I don't think the ZIIP or GP processor is taken into consideration when calculating CPU consumption. I think it's global.

For a job submitted by JCL, the authorized CPU consumption may depend on the execution class used.

Ask your systems engineers to find out the limits they have given to CPU consumption.

Below is the declaration that we use for the account that executes the groovy scripts (launched from Jenkins):

OMVS INFORMATION          
----------------          
UID= 0123456789           
HOME= /u/JENKINS          
PROGRAM= /bin/sh          
CPUTIMEMAX= 0000086400    <-- disable CPU time limit for USS task, equivalent of JCL TIME=1440
ASSIZEMAX= NONE           
FILEPROCMAX= NONE         
PROCUSERMAX= NONE         
THREADSMAX= NONE          
MMAPAREAMAX= NONE         

[FALLAI-Denis]

Hi @DylanVermeersch,

I did runtime analyzes with a tool equivalent to IBM Application Performance Analyzer and found that groovy (zAppbuild) and Java (DBB) code executed on a ZIIP processor as expected.

Is the analysis processing of the output produced by the jobs coded in groovy or does it use external utilities such as shell commands (which in this case do not run on a ZIIP)?

On a test machine, we are also confronted with the phenomenon of capping, even if we are mainly executing groovy code: a process which usually takes 5 minutes ellapsed can last more than an hour. I think the capping globally affects the partition (tasks dispatching) and not the access to the processors.

I'm not a systems engineer, but in my opinion whether the code is running on a ZIIP processor or a GP processor, CPU consumption is measured globally, and if a maximum limit has been set then your processing will not be able to exceed the threshold fixed.

CPU billing and CPU consumption should not be confused. The CPU consumption of a ZIIP processor is not billed, but in my opinion it is taken into account in the consumption measurement mechanisms for task management.

[DylanVermeersch]

Hi @FALLAI-Denis,

That is strange, we execute on an lpar that has 2 ZIIP's but it shows almost no usage on the ZIIP's when we look at the usage with RMF Monitor III. Could it have something to do with the groovy code like I said earlier or is this something on the systems side that's preventing us from fully utilizing our ZIIP's?

[DylanVermeersch]

After some more testing it uses the ZIIP for a good amount of time but when it is reading the JES output of a submitted JCL then it almost fully uses the CP, any idea how this can be fixed? We use groovy code to get acces to our JCL output. We do it like this:
JCL1.execute()
JCL2.execute()
JCL....execute()
ListOfJCLS= [JCL1, JCL2, JCL...]

// get output from JCLS
ListOfJCLS.each { jcl ->
members.each { member ->
jcl.saveOutput("OUT${members.indexOf(member) + 1}", new File("new file...")
}
}

This is obviously not the actual groovy code but it is the same flow as the program and it is to show how we get the output.
We execute a bunch of JCL's and then save those JCL's in a list and later use that list to get the output of a DD called OUTxxx with a number and save that output to a new file. That's the only thing we do with the JCL's.

[dennis-behm]

I can only make guesses here - the underlying JclExec API is obviously interacting with SDSF and uses some USS APIs to perform it. Imho, saving the output should not cause a significant CP usage, but nobody knows what's happening in those USS APIs ;)

Are you already on DBB 2.0.1? Have you considered to use the new JobExec API?

[DylanVermeersch]

We are now using DBB 2.0.0 which doesn't have the JobExec API. How do we upgrade it to 2.0.1? Do we need te reinstall the whole DBB API or can we just import/update it?

The JobExec looks like it could be better since I think I know what is causing the excessive CP, we do the jcl.saveOutput() like 500+ times and I guess that this will always reconnect with sdsf and that that reconnection uses a lot of CP and not the actual saving of the job...
Correct me if I'm wrong though.

PS: I'm just an intern right now so I don't have much experience with installation and such.

[DylanVermeersch]

I have found a solution to the excessive CP usage when using JCLExec.saveOutput, instead of going through JES everytime I just altered my JCL slightly to store the output of the JCL in a dataset and then using CopyToHFS I have had a decrease of over 1000% in CP time, now also most of the time the ZIIP is used. The total build time is also like 12 minutes instead of 2 hours so very time efficient as well (for 920 files that needed to be copied). My guess is that CopyToHFS is a more CP efficient way of getting output from z/OS then the JCLExec.saveOutput because it doesn't use JES or SDSF, this may also be something that is fixed by the JobExec class but because we can't upgrade yet I had to get creative with my solution. Hope this can help at least some people with the same problem as me.

A big thank you to everyone who answered on this discussion.