+
+What is this analytic?
", +31 "The chart below shows the percentage of time that the CPU cores were idle compared to overall usage. Each point on the plot shows the data for the jobs for a particular user.
", +32 "Why is this analytic important?
", +33 "Making sure jobs use the right number of CPU cores helps ensure that the compute resources are used efficiently.
", +34 "How do I improve future jobs?
", +35 "For typical compute intensive jobs, the overall CPU usage should be > 90 % (i.e. CPU core idle < 10 %). Consider requesting fewer CPU cores for future jobs, or adjust the configuration settings of the software to make use of all the cores that have been requested.
", +36 "
+
+
+
+
+_Figure 1. Flow of information from job summary documents in MongoDB to the XDMoD datawarehouse._
+
+## Example
+
+The following example shows how to modify the dataset mapping file to support
+reading job data from the job summary document that is different from the
+default.
+
+In this example we describe how to extend the dataset mapping to map data
+from the lustre llite statistics to the Filesystem I/O statistics in XDMoD.
+The default mapping file reads nfs filesystem metrics based on the configuration
+settings in the [supremm_resources.json](supremm-configuration.html#supremm_resourcesjson) file. This example
+shows how to override this so the metrics are instead gathered from the lustre metrics.
+
+The steps to configure the software are as follows:
+1. Create a new dataset map.
+1. Configure XDMoD to use the new dataset map.
+1. Test the new dataset map.
+1. Re-ingest the job data into XDMoD.
+
+
+### 1. Create a new dataset mapping
+
+A new dataset mapping file should be created in `/usr/share/xdmod/etl/js/config/supremm/dataset_maps`. In this
+example, the new mapping file will be called `custom.js`. The file name should follow the Node.js
+module naming convention. The source of the new mapping file is shown below:
+```js
+// load the default mapping module
+var supremm = require('./supremm.js');
+
+// load the library that contains helper functions
+var map_helpers = require('../map_helpers.js');
+
+module.exports = function (config) {
+
+ // create a new copy of the default map
+ var supremm_map = new supremm(config);
+
+ // override the mapping attributes for netdir home and util:
+
+ // The second argument should be set to the name of the
+ // filesystem as it appears in the job level summary ----┐
+ // ▾
+ supremm_map.attributes.netdir_home_read = map_helpers.device('lustre', '/home', 'read_bytes-total');
+ supremm_map.attributes.netdir_home_write = map_helpers.device('lustre', '/home', 'write_bytes-total');
+
+ supremm_map.attributes.netdir_util_read = map_helpers.device('lustre', '/util', 'read_bytes-total');
+ supremm_map.attributes.netdir_util_write = map_helpers.device('lustre', '/util', 'write_bytes-total');
+
+ // can add more overrides here....
+
+ return supremm_map;
+}
+```
+This loads the default mapping and selectively overrides fields.
+
+### 2. Configure XDMoD to use the new map.
+
+The [supremm_resources.json](supremm-configuration.html#supremm_resourcesjson) file defines
+the dataset map to use for each resource. For the purpose of this example, we assume that
+the resource name is `wopr`. The configuration setting would then look similar to:
+```json
+{
+ "resources": [
+ {
+ "resource": "wopr",
+ "resource_id": 1,
+ "enabled": true,
+ "datasetmap": "custom",
+ "hardware": {
+ "gpfs": "gpfs0",
+ "network": [
+ "em1",
+ "eno1"
+ ]
+ }
+ ...
+```
+
+### 3. Testing the new mapping
+
+XDMoD does not require tests exist for a dataset mapping, however it is
+highly recommended to create at least one test case since it significantly
+helps with debugging problems in the mapping. If you do not wish to create test
+cases then this step can be skipped.
+
+The XDMoD software has a test mode that can be used to verify the dataset
+mapping. The test mode requires one or more input files each of which must
+contain a job summary document in json format. Each input file must have a corresponding
+output json file that contains the expected results. The test harness has
+a mechanism to generate the output file. The generated output file must
+be manually verified for the test to be meaningful.
+
+Steps to run the tests:
+
+1) Generate a json input file by exporting a job record from the MongoDB database in json format.
+The [mongoexport](https://docs.mongodb.com/manual/reference/program/mongoexport/) command can
+be used to export documents in json format. Note that newer versions of mongoexport output
+the data in an 'extended' json format. The test input files should be in the mongodb
+relaxed json format (see the [MongoDB json documentation](https://docs.mongodb.com/manual/reference/mongodb-extended-json/) for
+full details. Briefly, if the output file contains data similar to:
+```js
+ "acct": {
+ "local_job_id" : NumberLong(4014453)
+ }
+```
+or
+```json
+ "acct": {
+ "local_job_id" : {
+ "$numberLong": "4014453"
+ }
+ }
+```
+these should be edited to use a plain json number field
+```json
+ "acct": {
+ "local_job_id" : 4014453
+ }
+```
+
+2) Copy the input file to the `/usr/share/xdmod/etl/js/config/supremm/tests/[RESOURCE]/input`
+directory where `[RESOURCE]` is the name of the resource with the new
+mapping file as it appears in `supremm_resources.json`. The name of the file should
+match the document identifier from MongoDB (i.e. the `_id` field).
+
+ For example, if the input file was for job id 8291026 (mongo `_id` = 8291026-1518721536)
+on the `wopr` resource then the file would be called `/usr/share/xdmod/etl/js/config/supremm/tests/wopr/input/8291026-1518721536.json`
+
+3) Create the expected output file. The easiest way to create the output file is to create an empty json
+document in the output directory: `/usr/share/xdmod/etl/js/config/supremm/tests/[RESOURCE]/expected`.
+For example the output file corresponding to the example input file above would be
+`/usr/share/xdmod/etl/js/config/supremm/tests/wopr/expected/8291026-1518721536.json`
+with contents:
+
+```json
+{}
+```
+Then edit the file `/usr/share/xdmod/etl/js/lib/etl_profile.js` and change the
+line:
+```js
+ var regenerateOnFail = false;
+```
+to
+```js
+ var regenerateOnFail = true;
+```
+Then run the following:
+```bash
+$ cd /usr/share/xdmod/etl/js
+$ node etl.cli.js -t -d
+```
+This will generate a large number of error log messages since the expected
+results file is empty. But since the `regenerateOnFail` variable is set to true it
+will overwrite the expected result file with the new data. Manually
+check the file to confirm that the data are correct. In this example, there
+should be values in the file for the various `netdir` metrics added in the
+new custom mapping:
+```json
+{
+ "netdir_home_read": {
+ "value": 1235,
+ "error": 0
+ },
+ "netdir_home_write": {
+ "value": 1235,
+ "error": 0
+ },
+ "netdir_util_read": {
+ "value": 136192137,
+ "error": 0
+ },
+ "netdir_util_write": {
+ "value": 0,
+ "error": 0
+ }
+}
+```
+The `/usr/share/xdmod/etl/js/lib/etl_profile.js` should be edited again to revert the
+change to the `regenerateOnFail` variable.
+
+4) Finally re-run the test harness again and confirm that the tests pass:
+```bash
+$ cd /usr/share/xdmod/etl/js
+$ node etl.cli.js -t -d
+```
+```bash
+2020-03-10T18:12:42.401Z - info: Regression Testing...
+2020-03-10T18:12:42.513Z - info: Regression test of "8291026-1518721536.json" passed
+```
+
+### 4. Re-ingest data into XDMoD
+
+Reset the job ingest status for all jobs:
+
+```bash
+$ /usr/lib64/xdmod/xdmod-supremm-admin --action reset --resource [RESOURCE] -d
+```
+
+Run the ingest and aggregation script:
+```bash
+$ aggregate_supremm.sh
+```
diff --git a/11.0/jobefficiency.md b/11.0/jobefficiency.md
new file mode 100644
index 00000000..224a1be2
--- /dev/null
+++ b/11.0/jobefficiency.md
@@ -0,0 +1,98 @@
+
+The Job Efficiency Dashboard shows job information broken down by the categorized
+efficiency. The categorization algorithm is implemented in SQL and runs during the
+data aggregation step. The algorithm may be customized by editing the definition
+file
+```
+/etc/xdmod/etl/etl_macros.d/jobefficiency/job_categorization.sql
+```
+
+If the definition file is modified then any new jobs will be categorized using the updated
+algorithm. It does not automatically re-categorize existing jobs.
+
+The contents of the file must be valid SQL fragment. The SQL itself is used in a
+`SELECT` statement. An example of the default algorithm is shown below:
+```sql
+-- ----------------------------------------------------------------------------
+-- Classify a job based on the performance statistics
+-- ----------------------------------------------------------------------------
+CASE
+ WHEN cpu_user IS NULL THEN
+ -1
+ WHEN cpu_user < 0.1 AND COALESCE(max_memory, 1.0) < 0.5 THEN
+ 2
+ ELSE
+ 1
+END
+```
+The output must be one of three values:
+- `-1` if the algorithm is unable to categorize the job
+- `1` to mark the job as efficient
+- `2` to mark the job as inefficient
+
+The SQL query can use any value from the job performance fact table `modw_suprem`.`job`
+The documentation for each column in the table is provided in the `COMMENT` field
+in the table definition. The comment field can be viewed using the following
+statement in the `mysql` command line client:
+```mysql
+SHOW FULL COLUMNS FROM `modw_supremm`.`job`;
+```
+Not all performance metrics will be present for all jobs. If a column in the
+database is nullable then a null value is used to indicate that the corresponding
+metric was not present.
+
+## Examples
+
+The example below shows how to categorize jobs solely based on the CPU User metric.
+The example will mark jobs with CPU User less than 20% as inefficient.
+
+```sql
+-- ----------------------------------------------------------------------------
+-- Categorize jobs based only on the value of CPU User with a threshold
+-- of 20% (i.e. a ratio of 0.2)
+-- ----------------------------------------------------------------------------
+CASE
+ WHEN cpu_user IS NULL THEN
+ -1
+ WHEN cpu_user < 0.2 THEN
+ 2
+ ELSE
+ 1
+END
+```
+
+The example below shows how to use different criteria based on the other accounting
+information about the job. In this case, if the job ran on a partition with a name
+ that starts with 'gpu' then the job's efficiency
+is determined based on the GPU usage. Otherwise the CPU usage is used. In both cases
+a 10% (ratio of 0.1) threshold is used. The job's partition (also known as the queue)
+is stored as text in the `queue_id` column.
+```sql
+CASE
+ WHEN
+ queue_id LIKE 'gpu%'
+ THEN
+ CASE
+ WHEN gpu0_nv_utilization IS NULL THEN - 1
+ WHEN gpu0_nv_utilization < 0.1 THEN 1
+ ELSE 2
+ END
+ ELSE
+ CASE
+ WHEN cpu_user IS NULL THEN - 1
+ WHEN cpu_user < 0.1 THEN 1
+ ELSE 2
+ END
+END
+```
+
+# Reprocessing existing jobs.
+
+If the sql definition is updated then all new jobs will be categoried with the
+new definition. Existing data that has already been ingested into Open XDMoD
+will not automatically be reprocessed. All jobs can be reprocessed by running
+the following command:
+```bash
+/usr/share/xdmod/tools/etl/etl_overseer.php --last-modified-start-date 2000-01-01 -p jobefficiency.aggregation -p jobefficiency.joblist
+```
+This will reaggregate all job efficiency data.
diff --git a/11.0/supremm-architecture.md b/11.0/supremm-architecture.md
new file mode 100644
index 00000000..f0c054ed
--- /dev/null
+++ b/11.0/supremm-architecture.md
@@ -0,0 +1,47 @@
+
+Job performance data in XDMoD is obtained from collection software running on
+the compute nodes of the HPC resource. The software architecture comprises three major components.
+
+* Software that runs directly on HPC compute nodes and periodically collects performance information.
+* Job summarization software that uses the node-level performance data to generate job-level data.
+* An XDMoD module that enables the job-level information to be viewed and analysed in XDMoD.
+
+The job-level summary records are stored in a MongoDB document database.
+
+The Job Performance (SUPReMM) module in XDMoD may be extended to support multiple
+different data collection packages. The XDMoD team recommends
+using either Performance Co-Pilot (PCP) or Prometheus as the data source for new software installs. This document assumes that either
+PCP or Prometheus is used as the data source. For more information about other data
+sources please contact the development team via the mailing list or email
+(contact details on the [main overview page](http://open.xdmod.org)).
+
+A simplified high-level overview of the SUPReMM dataflow is shown in Figure 1.
+below. This diagram shows the principal features of the collection and
+processing architecture. The data aggregation and display components present in
+Open XDMoD are not shown.
+
+
+*Figure 1. SUPReMM data flow diagram*
+
+The data collection software running on the compute nodes provides the node-level
+performance metrics. This guide provides instructions for using either
+Performance Co-Pilot (PCP) or Prometheus as the data collection software. Metric
+data is configured to be logged every 30 seconds (and optionally also at the
+start and end of each compute job via hooks in the job prolog and epilog scripts).
+
+The node-level performance data is stored in a central location for subsequent
+processing by the summarization software. For PCP, the data are stored as
+PCP archive files on a shared filesystem. For Prometheus, the data are
+are stored in Prometheus's timeseries database.
+
+The accounting logs from the resource manager (such as Slurm SGE, PBS, LSF, etc) are ingested into the XDMoD datawarehouse.
+These accounting logs include information about the start and end times of each compute job
+as well as the compute nodes that were assigned to the job.
+
+The summarization software runs periodically via cron. The software uses the
+accounting information from the XDMoD datawarehouse and the information from the data
+collection software to generate a job-level summary for each HPC job. These job-level
+summaries are stored in a MongoDB document database.
+
+The summarized job data is then ingested into the XDMoD datawarehouse for display
+in the web-based user interface and analysis using XDMoD's Data analytic tools.
diff --git a/11.0/supremm-bootstrap.md b/11.0/supremm-bootstrap.md
new file mode 100644
index 00000000..a22c7ff1
--- /dev/null
+++ b/11.0/supremm-bootstrap.md
@@ -0,0 +1,258 @@
+
+The Job Performance data workflow is run as a series of batch processing
+scripts executed periodically via cron.
+
+Before setting up the batch processing workflow, we highly recommend manually
+running the various scripts in debug mode.
+
+All data processing scripts and commands should be run under an unprivileged
+user account. For a typical installation the `xdmod` user account is recommended
+since this will have the necessary access permissions to the XDMoD configuration
+files.
+
+## Prerequisites
+
+The following software components should have been installed and configured:
+
+1. Performance collection software running on compute nodes (PCP or Prometheus)
+1. XDMoD with the Job Performance (SUPReMM) module
+1. MongoDB
+1. Job Summarization software
+
+The following data should be present:
+
+1. XDMoD should have accounting data in the Jobs realm
+1. PCP archives that contain metrics collected from compute nodes that ran jobs present in XDMoD
+
+ **OR**
+
+ A Prometheus server that scrapes metrics from compute nodes that ran jobs present in XDMoD
+
+## Manual run
+
+The following steps should be performed in order:
+
+1. Ingest job accounting data into XDMoD
+1. Run the PCP archive indexer script (For resources configured with PCP)
+1. Run the job summarization script
+1. [Optional] Ingest the job batch scripts
+1. Run the XDMoD ingest and aggregation script
+1. Verify data in the XDMoD UI
+
+### 1. Ingest job accounting data into XDMoD
+
+Ensure that there is job accounting data in XDMoD that covers the time period of the data collection software.
+For example you can check the 'Number of Jobs Ended' metric in the Usage tab of the XDMoD UI
+and confirm that there are jobs for the same time period as the performance data.
+
+### 2. Run the PCP archive indexer script (PCP resources)
+
+**Note: If a resource is configured for Prometheus, then continue to the next step.** Running the
+archive indexer script on a Prometheus configured resource will not produce any output and will not
+affect the dataflow for a resource configured with Prometheus.
+
+The archive indexer script scans the PCP archive directory, parses each PCP
+archive and stores the archive metadata in the XDMoD datawarehouse. This index
+is then used by the job summarization script to efficiently obtain the list of
+archives for each job.
+
+The archive indexer script uses the archive file path to estimate the
+create time of each archive. The script will only process
+archives created in the previous 3 days by default. The first time the archive
+indexer is run, specify the "-a" flag to get it to processes all archives. It
+is also recommended to specify the debug output flag -d so that you can see
+that it is processing the files:
+
+ $ indexarchives.py -a -d
+
+A typical output of the script would be:
+
+ 2018-09-06T11:56:30.066 [DEBUG] Using config file /etc/supremm/config.json
+ 2018-09-06T11:56:30.068 [INFO] archive indexer starting
+ 2018-09-06T11:56:31.433 [INFO] Processed 1 of 1649 (hosttime 0.00125503540039, listdirtime 0.0011420249939, yieldtime 0.0) total 0.00125503540039 estimated completion 2018-09-06 11:56:33.501839
+ 2018-09-06T11:56:31.514 [DEBUG] processed archive /data/pcp-archives/hostname1.example.com/2018/09/04/20180904.00.10.index (fileio 0.067507982254, dbacins 9.67979431152e-05)
+ 2018-09-06T11:56:31.596 [DEBUG] processed archive /data/pcp-archives/hostname1.example.com/2018/09/05/20180905.00.10.index (fileio 0.0788278579712, dbacins 3.19480895996e-05)
+ 2018-09-06T11:56:31.652 [DEBUG] processed archive /data/pcp-archives/hostname1.example.com/2018/09/06/20180906.00.10.index (fileio 0.0517160892487, dbacins 3.09944152832e-05)
+ ...
+
+Verify that the output lists the PCP archives that you expect.
+
+If you see message similar to:
+
+ 2018-08-21T05:56:27.426 [DEBUG] Ignoring archive for host "login-a" because there are no jobs in the XDMoD datawarehouse that ran on this host.
+
+This means that the PCP archives for this host were skipped and will not be
+used for job summarization. If you see the hostname for a compute node that
+should have run jobs then the likely cause is that the job data was not ingested
+into XDMoD. Go back ingest the job data and then rerun the `indexarchives.py -a` command again.
+
+### 3. Run the summarization script:
+
+The summarization script is responsible for generating the job level summary records
+and inserting them in the MongoDB instance. The script reads the job information
+from the XDMoD datawarehouse and processes it according to the configured data source. The default
+behavior of the script is to process all jobs in the XDMoD datawarehouse that
+have not previously been summarized. If there are no data for
+a job then a summary record is still created, but, of course, there will
+be no performance metric information in the record.
+
+ $ summarize_jobs.py -d
+
+You should see log messages indicating that the jobs are being processed. You
+can hit CTRL-C to stop the process. The jobs that have been summarized by the
+script will be marked as processed in the database and the summaries should end
+up in MongoDB. Check to see that the summaries are in MongoDB by, for example, using
+the MongoDB command line client to query the database:
+
+ $ mongo [MONGO CONNECTION URI]
+ > var cols = db.getCollectionNames();
+ > for(var i=0; i < cols.length; i++) {
+ print(cols[i], db[cols[i]].count())
+ }
+
+### 4. [Optional] Run the job batch script ingest
+
+Job batch scripts are displayed in the "Job Script" tab in the "Job Viewer" in
+XDMoD. This tab is only displayed if the job script is available and the absence
+of the jobs scripts does not impact the rest of the interface.
+
+The `ingest_jobscripts.py` script parses the filesystem, reads each job script file
+and loads the contents into the XDMoD datawarehouse.
+If you have the batch scripts for each HPC job available, they are ingested as
+follows
+
+ $ ingest_jobscripts.py -a -d
+
+The `ingest_jobscripts.py` has similar commandline configuration as the `indexarchives.py`. The path
+is parsed to obtain the start time of the job and only the last 3 days worth of
+jobs are ingested by default.
+
+
+### 5. Run the XDMoD ingest and aggregation script
+
+Once there are job summary data in MongoDB it is ingested and aggregated into
+XDMoD by running the following command as the `xdmod` user on the XDMoD host:
+
+ $ aggregate_supremm.sh -d
+
+This script runs three separate commands. First the job summary data is ingested
+from MongoDB to the XDMoD datawarehouse. Then a script parses the job accounting
+data and records which jobs shared compute nodes. Then the data are aggregated
+to be displayed in the XDMoD UI.
+
+The ingestion and aggregation software automatically processes all job summaries
+that have not previously been processed. The shared jobs calculation by default
+processes jobs that ran over the previous three days. If there are jobs
+that shared compute nodes that are older than this three day window then
+run the shared jobs script with the `-a` flag to process all jobs in the database:
+
+ $ /usr/lib64/xdmod/supremm_sharedjobs.php -a -d
+ $ aggregate_supremm.sh -d
+
+This commands should be run as the `xdmod` user on the XDMoD host.
+It is not necessary to run the `supremm_sharedjobs.php` with the `-a` flag every
+time, only on initial ingest if there are older jobs.
+
+### 6. Check Open XDMoD Portal
+
+After the XDMoD ingest and aggregation script has completed successfully
+you should log in to the portal and confirm that the data are present.
+
+The Job Performance data is displayed in the 'SUPREMM' realm in XDMoD. There are
+access controls enabled on the data as follows:
+
+- Accounts with 'User' role may only see job performance data for jobs that they ran;
+- Accounts with 'Principal Investigator' role may only see job performance data for
+ their jobs and the job that they were PI on;
+- Accounts with 'Center Staff' and 'Center Director' role may see all jobs.
+
+You should login using an account that has either 'Center Staff' or 'Center Director'
+role and confirm that there is an additional tab visible named "Job Viewer". In addition to the new
+tab, the "SUPREMM" realm should be visible in the "Usage" and "Metric
+Explorer" tabs.
+
+If there were jobs in the database then there should be data for the 'Number of Jobs Ended' metric.
+If there were valid metrics then you should see data for the 'Avg CPU %: User metric'.
+
+**Note that the admin user that was created in the Open XDMoD has a 'User' role
+mapped to the 'Unknown' user by default.**{:.note} This means that all plots of
+SUPREMM realm data will show "No data available for the criteria specified" and
+no jobs will be able to be viewed in the "Job Viewer" tab. The 'Center Director'
+role may be added to the admin user account via the Internal Dashboard.
+
+
+# Troubleshooting
+
+The administrator internal dashboard has a 'SUPReMM Dataflow' tab that shows the data flow information
+for each configured resource. The internal dashboard is available using the "Dashboard"
+button on the top toolbar in the XDMoD UI for user accounts that have the 'Manager' role.
+An example of the dataflow diagram is shown in Figure 1. below. **Note: The admin dashboard currently only
+displays information for resources configured with PCP.**{:.note}
+
+
+
+*Figure 1. Example screenshot of the dataflow diagram in the XDMoD internal dashboard. The dataflow
+diagram displays detailed statistics on the data stored at various
+points in the processing workflow.*
+
+The data flow page shows information for each configured HPC resource separately.
+Use the dropdown on the top left to select each resource and check that the
+statistics reported in each of the blocks are as expected.
+
+### PCP archive files
+
+The 'node count' metric should match the number of distinct compute nodes with PCP data
+stored on the filesystem.
+The 'node level archives' metric should match the number of node level archives. If these
+do not match up then rerun the `indexarchives.py` script following the instructions
+in [Step 2.](supremm-bootstrap.md#2-run-the-pcp-archive-indexer-script)
+
+### Job summary documents
+
+The 'total' metric indicates the total number of job summaries stored in the MongoDB instance. These
+are generated by the `summarize_jobs.py` script described in
+[Step 3.](supremm-bootstrap.md#3-run-the-summarization-script)
+The 'processed' count indicates the number of jobs that have been successfully ingested
+into the XDMoD datawarehouse. If the 'pending' count is non-zero (and the processed
+job count is non-zero) then the job data needs to be ingested into XDMoD following the instructions
+in [Step 5.](supremm-bootstrap.md#5-run-the-xdmod-ingest-and-aggregation-script)
+
+### SUPREMM job records in Open XDMoD datawarehouse
+
+The 'total' metric indicates the total number of job records ingested into XDMoD from MongoDB. If
+this value is less than the number of jobs in MongoDB then the job data needs to be ingested
+into XDMoD following the instructions
+in [Step 5.](supremm-bootstrap.md#5-run-the-xdmod-ingest-and-aggregation-script)
+
+# Advanced troubleshooting
+
+The recommended strategy for advanced troubleshooting is to identify an HPC job that
+- Has accounting data present in the XDMoD datawarehouse
+- Has raw performance data for the compute nodes on which the job ran
+- Had a wall time long enough for multiple data points to be recorded. For example if you have setup a 30 second collection period, then choose a job that ran for at least ten minutes.
+
+The information about this job should be manually traced though each step in the dataflow. The
+`xdmod-supremm-jobinfo` script provides low-level debug information about a given job
+by directly querying the XDMoD datawarehouse and MongoDB.
+
+ /usr/lib64/xdmod/xdmod-supremm-jobinfo -r RESOURCE -j JOBID
+
+An example output of this script for a job that has all data present and correct is shown below:
+
+ $ ./xdmod-supremm-jobinfo -r resourcename -j 8038957
+ Job 8038957 (end time 2018-01-01 16:40:50)
+ Host list (1 hosts):
+ "compute-15-33"
+ Archive list:
+ Host "compute-d15-33"
+ "/data/pcp-logs/compute-d15-33/2018/01/01/20180101.00.10"
+ "/data/pcp-logs/compute-d15-33/2018/01/01/job-8038957-begin-20180101.03.46.33"
+ "/data/pcp-logs/compute-d15-33/2018/01/01/job-8038957-end-20180101.16.40.50"
+
+
+ Data from modw_supremm:
+ 13 8038957 1514842850
+ Summary in mongo: _id, summarization, procDump, process_memory, infiniband, cpuperf, gpfs, cpu, network, cores, nfs, uncperf, memory, load1, acct, catastrophe, block, nodememory, processed
+ Summary in mongo: _id, lnet, memused, cpuuser, ib_lnet, version, nfs, simdins, process_mem_usage, hosts, membw, memused_minus_diskcache, block
+
+**Note: Resources configured with Prometheus do not have archives associated with them.**{:.note}
diff --git a/11.0/supremm-compute-pcp.md b/11.0/supremm-compute-pcp.md
new file mode 100644
index 00000000..0bbac249
--- /dev/null
+++ b/11.0/supremm-compute-pcp.md
@@ -0,0 +1,207 @@
+**These instructions only apply to resources that will use PCP software. Configuration instructions for Prometheus are [here](supremm-compute-prometheus.html).**
+
+This section gives example configuration settings for Performance Co-Pilot (PCP) running on the compute nodes
+of an HPC cluster. These configuration guidelines are based on the PCP data collection setup
+at CCR Buffalo, which uses PCP version 4.3.2 that is supplied with Centos 7.
+
+Note that the PCP version 5.3.7 supplied with Rocky 8 does not support using date
+variables in the pmlogger output path. The template pmlogger control file
+is not suitable for use with PCP 5.
+
+## Prerequisites
+
+PCP should be installed on every compute node as described in the [install section](supremm-install-pcp.md).
+
+## Configuration
+
+After the PCP software has been installed on each compute node it should be configured so that
+data are collected:
+
+* At the start of every job
+* 10, 20 and 30 seconds after job start
+* Periodically during the job (recommended 30 seconds)
+* At the end of every job
+
+The archive data for each node needs to be readable by the job summarization software
+process. We recommend that the archive data is stored on a shared filesystem such as a parallel filesystem
+or network attached storage.
+
+In order to constrain the number of files in a given directory,
+we recommend storing the archives in a directory structure
+that includes the hostname and the date in different subdirectories. The recommended
+directory path is:
+
+ [YYYY]/[MM]/[HOSTNAME]/[YYYY-MM-DD]
+
+Where YYYY is the 4 digit year, MM 2 digit month, DD is the 2 digit day of the month
+and HOSTNAME is the hostname of the compute node. The two main reasons for storing
+the data in this way are (1) to minimize the number of files per directory and
+(2) to reduce the I/O of the summarization software's indexing script. The
+indexing script is designed to run daily and scans the filesystem for new PCP
+archives. The script is able to exclude old directories from processing based
+on the timestamp. This reduces the amount of I/O since only candidate
+directories are scanned.
+
+In previous versions of the summarization software the recommended path
+was:
+
+ [HOSTNAME]/[YYYY]/[MM]/[DD]
+
+this directory structure is still supported by the indexing script and may
+still be used. The reason for changing the recommendation is that the new directory
+structure limits the total number of files under a given directory. This
+helps reduce the runtime of backup software. If the filesystem I/O performance with
+ the existing directory structure is not an issue then there is no need to change to the new one.
+
+The archive indexing also supports the path structure where all the
+PCP archives for a host are under the hostname directory:
+
+ [HOSTNAME]
+
+The disadvantage of this directory structure is that the number of files in the
+hostname directory can become very large. If you do use this structure it is
+recommended to create a script that is run periodically to remove or otherwise
+archive older PCP files to limit the number of files in each hostname
+directory. Large numbers of files in the log output directory will likely cause
+poor performance of the PCP logging software and the summarization software.
+
+
+Configuration Templates
+-----------------------
+
+The [Job Summarization software][] includes template files that can be used to
+configure PCP collection on the compute nodes (if using PCP version 4). The package itself should not
+be installed on the compute nodes, however you may wish to install it
+on a test node in order to obtain the PCP configuration file templates.
+Alternatively, the scripts may be extracted directly from the source tar file or
+from the [github repository][].
+
+The PCP configuration templates are applicable to PCP version 4.3.2 that is
+supplied with Centos 7 and may need to be modified to work with more recent versions
+of the PCP software. The template logger control files should not
+be used with PCP version 5 because this version does not support using
+date substitutions in the logger pathname.
+
+
+The template files must be edited before use.
+
+These templates are available:
+------------------------------
+#### /usr/share/supremm/templates/pcp-4.3.x/pmlogger/control
+
+This template file **must** be edited to specify the path to the directory
+where the PCP archives are to be saved.
+
+The path `PCP_LOG_DIR/pmlogger` should be changed to the path where the PCP archives are to be saved.
+
+The edited template should be saved in the `/etc/pcp/pmlogger` directory and any existing files under
+`/etc/pcp/pmlogger/control.d` must be removed to ensure that there is only one primary logger
+configured.
+
+Note that the string `LOCALHOSTNAME` in the file is expanded by the pcp logger software to the hostname
+of the compute node running the logger.
+
+The template includes the following directive that disables the compression behavior so that the
+archive files do not get compressed while the summarization software is
+processing them:
+
+```
+$PCP_COMPRESSAFTER=never
+```
+
+The template also specifies the `-T24h10m`` option to the pmlogger process to ensure new log files
+are created every day.
+
+#### /usr/share/supremm/templates/pcp-4.3.x/pmlogger/pmlogger-supremm.config
+* Moved to /etc/pcp/pmlogger
+* Can be updated to change metrics logged or frequency
+ * You may wish to reduce logging frequency from the default 30 seconds until confirming impact on your system and storage utilization
+
+#### /usr/share/supremm/templates/slurm/slurm-prolog
+
+This is a template perl script that **must** be edited. The string `/