Make JSON examples minimal

[duncandewhurst]

Following on from the discussion started in #1661 (comment):

To ease comprehension, we should have minimal (but coherent) examples. That is, the JSON files in docs/examples should contain only the fields needed to illustrate the specific documentation pages that they feature in. Minimal examples are also easier to maintain, for example, to reflect a schema change.

The purpose of this issue differs from the purpose of #1449, which is to avoid examples that are incoherent, or that are too abstract (e.g. Anytown buys widgets).

I'll review the examples to identify those whose clarity would be improved by removing fields. I'll do that after #1661 is done to avoid any duplication of effort.

[duncandewhurst]

Moving relevant comments from #1449:

@duncandewhurst in #1449 (comment):

@jpmckinney to check that we are on the same page when it comes to making examples minimal, please could you review the changes I've made to the tender updates and amendments example (original, minimal version).

In addition to the changes proposed in my previous update, since the notes which previously appeared at the end of the example each relate to a single JSON snippet, I integrated them into the paragraphs introducing each snippet.

@jpmckinney in #1449 (comment):

Looks good! It would be nice to be able to emphasize the lines that changed. I opened OpenDataServices/sphinxcontrib-opendataservices#47

I think to go along with #1639 there are a few places where we would replace "tender" with "opportunity" (mainly where tender isn't followed by a word like "release" or "update release"). Or rewording like:

The buyer adds a main procurement category to the tender

The buyer now indicates the opportunity's main procurement category.

[duncandewhurst]

@jpmckinney I've made your suggested changes.

Regarding the second example on that page, it feels unnecessary to me. Given that the purpose of the page is to explain the differences in modelling new information, updates, and amendments, I think it would be sufficient to add a sentence at the end of the first example along the lines of:

Contract updates and amendments are modelled in the same way: the 'contract', 'contractUpdate' and 'contractAmendment' release tags distinguish the type of change and contract amendments are listed in the contract.amendments field.

Regarding the third example on that page, we can decide what to do with it as a part of #1531 (which removes the easy releases terminology).

[jpmckinney]

Yes, happy to remove the 2nd example. We can put the new sentence in an admonition titled "Contract updates and amendments"

[duncandewhurst]

Noting that as part of going through all the examples for this issue, I think it'll be useful to use a consistent approach to setting ocid and id. I suggest the following:

• ocid: Set to 'ocds-213czf-0000' for the first contracting process in each subfolder of examples. If there's more than one contracting process in the subfolder, set the ocid of the subsequent processes to 'ocds-213czf-0001', 'ocds-213czf-0002' etc.
• id: Set incrementally within the scope of the contracting process the release relates to, e.g. '1', '2' etc.

[jpmckinney]

Sounds good to me re: ocid and id.

[duncandewhurst]

I reviewed the examples not already touched by #1680, #1661 and #1531. The following examples would be improved by removing fields. I'll check these off as I work through them:

• Unsuccessful processes (see suggestions below)
• Framework agreements
• Processes with pre-qualification and pre-selection
• Contract suspension (Removed in release-schema: Add finalStatus to tender, award and contract. Clarify tender.datePublished. #1648)
• Organization identifiers
• Personal identifiers
• Organizational units (Done apart from second example, see question below)
• Organization classifications
• Beneficial ownership information

I also noticed a couple of other opportunities for improvements and fixes:

Unsuccessful processes

I think this example is trying to do too much: it illustrates linking planning and contracting processes, linking unsuccessful and restarted processes, and how to assign different ocids to unsuccessful and restarted process when they share an identifier in a publisher's system.

Suggested actions:

• Move the content and example related to linking planning and contracting processes to contracting and planning processes
• Draft a minimal example to illustrate how to link unsuccessful and restarted processes in the common case where they are treated as separate contracting processes in the publisher's system
• Address the Paraguay case (shared identifiers) in an admonition, e.g.

Unsuccessful and restarted processes with a shared identifier
In some jurisdictions, such as Paraguay, if a first attempt is unsuccessful in procuring items, the second attempt is considered to be part of the same contracting process and shares the same identifier. However, in OCDS, the two attempts need to be modelled as separate contracting processes with different ocids. To create an ocid for the second process in the absence of a unique identifier in your data sources, you can append an incrementing number to the ocid of the first process, e.g. {first process ocid}-1.

Framework agreements

Whilst updating the examples, I noticed a couple of conflicts with guidance:

tender.procurementMethod

The guidance states:

[set] tender.procurementMethod according to the competitive conditions of the first stage of the framework agreement. Since framework agreements typically involve qualification, this is most often 'selective'.

However, both examples have the following:

Since any supplier is able to submit a response to the invitation to participate, tender.procurementMethod is set to 'selective'.

In both cases, I've updated it to match the guidance:

Only qualified suppliers can respond to the invitation so tender.procurementMethod is set to 'selective'

award.suppliers

The guidance states:

A single award notice can be used to disclose many award decisions; however [...] such notices ought to be split into multiple awards in OCDS.

However, the second example has a single award with two suppliers (not in a consortia).

I've updated the example to split the award into two, one for each supplier.

[duncandewhurst]

Organizational units

I don't understand why an extension is needed in the second example. Couldn't the division code be disclosed in .additionalIdentifiers?

[jpmckinney]

In both cases, I've updated it to match the guidance:

Only qualified suppliers can respond to the invitation so tender.procurementMethod is set to 'selective'

Hmm, I think any supplier can indeed request to participate, but only qualified suppliers will then be able to submit a bid .

I don't understand why an extension is needed in the second example. Couldn't the division code be disclosed in .additionalIdentifiers?

You're right. It's not needed. This was clarified in #1510 Maybe we do something like https://github.com/sdd1982/branch which indicates the branch of government (executive, legislative, judicial).

[jpmckinney]

The other suggestions look good!

[duncandewhurst]

Hmm, I think any supplier can indeed request to participate, but only qualified suppliers will then be able to submit a bid .

Oh yes, of course. I'll update the text.

You're right. It's not needed. This was clarified in #1510 Maybe we do something like https://github.com/sdd1982/branch which indicates the branch of government (executive, legislative, judicial).

Hmm, I'm not sure that extension fits the idea of an organizational unit (i.e. an administrative division of an organization). It seems more like a classification of the branch of government to which an organization belongs. I think we can probably just remove the second example?

[duncandewhurst]

Contracting processes and planning processes

The guidance states that:

A planning process should not contain the releaseTag 'tender' even if it contains a tender object.

However, the description of the 'tender' tag suggests that it can be used in a planning process (my emphasis):

Information about, for example, the needed items and their estimated value, procurement method, award criteria, and various deadlines. This information concerns either the contracting process or the planning process. For a contracting process, this information typically concerns the period starting with contracting documents (for example, procurement documents) being available to potential suppliers and ending with the bid submission deadline. For a planning process, this information typically concerns the period before the contracting documents are available to potential suppliers.

Should the references to planning processes be removed?

Edit: Also we'll need to replace "should" with "ought to" on the guidance page.

[jpmckinney]

You're right. It's not needed. This was clarified in #1510 Maybe we do something like https://github.com/sdd1982/branch which indicates the branch of government (executive, legislative, judicial).

Hmm, I'm not sure that extension fits the idea of an organizational unit (i.e. an administrative division of an organization). It seems more like a classification of the branch of government to which an organization belongs. I think we can probably just remove the second example?

Oh, right, I didn't check the page title! Yes, we can just remove that example – since there's nothing about it that's different from creating an extension for any other reason.

Should the references to planning processes be removed?

Ah, indeed, it was made to be a copy of the tender field description. Yes, we can remove the mentions of planning process (and the now unnecessary "For a contracting process, ").

[duncandewhurst]

Organization classifications

The worked example for the organization classification extension begins with a lot of guidance:

An organization classification needs to consist of at least two parts: an identifier for the list (scheme) from which the classification is taken, and an identifier for the category from that list being applied. It is useful to also publish a text label and/or URI that users can draw on to interpret the classification.

Each classification block contains fields to provide information about the description (a textual description or title for the classification code), id (the classification code), uri (to uniquely identify the classification code) and scheme. The scheme value can be drawn from the open classificationScheme codelist (where the Category value is "organization"), or it can be a local scheme. Schemes are given to classify the activities of procuring authorities (i.e. procuring entities and/or buyers).

Where an appropriate scheme is not listed in the classificationScheme codelist, publishers can specify their own scheme. Publishers can either reuse an alternative scheme, or provide their own. Where publishers provide their own local schemes, they ought to prefix their scheme code with an ISO-3166-1 alpha-3 country code to preserve its global uniqueness. Details of this local scheme, and a list of possible codes, ought to be described in the publication policy.

There is a lot of crossover and some discrepancies ('needs to' vs 'ought to') with the guidance that appears at the beginning of the page:

Each classification.id field ought to contain a code from the particular scheme given in the classification.scheme field. Details about the particular organization characteristic that is being disclosed ought to be provided in the classification.description field. The given classification.scheme can be an existing scheme (drawn from the open classificationScheme codelist), or a local scheme for a particular publisher. In both cases, we encourage publishers to provide details of all schemes and classification codes used in their publication policy, to help users understand the data.

To reduce repetition and separate the guidance and examples (as on other guidance/example pages) I suggest that we integrate the guidance from the beginning of the worked example with the guidance at the start of the page. Sound good?

[jpmckinney]

Yes, sounds good. If relevant, feel free to add structure (headings) to the guidance part, as it might get long.

[duncandewhurst]

One more suggestion about organization classifications: In terms of modelling, I don't see why example 3 (disclosing whether an organization is chaired by a woman) should differ from example 2.2 (disclosing whether an organization is owned by a woman). Given that the third option is discouraged anyway, I think that we can discard it and add an admonition to the second example along the lines of:

Mapping a boolean data element
If you are mapping a boolean data element that describes a characteristic of an organization, e.g. a flag that is set to True when an organization is chaired by a woman, the preferred approach is to model it as a local classification scheme with a single code representing the True value.

[jpmckinney]

Yes, I'm happy to remove examples of extensions, as once publishers learn about extensions, they will come up with use cases like these on their own.

The admonition is a good idea. In general, a list of codes can be modelled as a set of boolean fields (e.g. SME, VSCE, etc.). Perhaps the admonition can cover boolean elements more broadly.

If you are mapping boolean data elements that describe characteristics of an organization, try to create logical groups of characteristics and create a local classification scheme for each group. It is allowed for a scheme to contain a single code.

For example: The City of Boston's Department of Supplier Diversity defines minority-owned businesses (MBE), women-owned businesses (WBE) and small local business enterprises (SLBE), among others. Depending on your use cases, two logical groups might be ownership-based characteristics and location-based characteristics. Instead of three booleans, this information can be modelled as a USA-MA-BOS-OWNER scheme with 'MBE' and 'WBE' codes and a USA-MA-BOS-LOCATION scheme with an 'SLBE' code.

[jpmckinney]

@duncandewhurst Is there anything in this issue not resolved by #1680?

[duncandewhurst]

Only my suggestion in #1666 (comment), which I've done for all the examples I've edited, but there may be some others that still need updating:

* `ocid`: Set to 'ocds-213czf-0000' for the first contracting process in each subfolder of `examples`. If there's more than one contracting process in the subfolder, set the `ocid` of the subsequent processes to 'ocds-213czf-0001', 'ocds-213czf-0002' etc.

* `id`: Set incrementally within the scope of the contracting process the release relates to, e.g. '1', '2' etc.

When updating the ocid and id, we'll need to also check if any accompanying text in Markdown files needs updating.

[odscjen]

@jpmckinney do you want the check for ocid and id from #1666 (comment) completed for all other examples as part of #1680? If so I can do that seeing as Duncan is on leave, or we can create a new issue and do that as a new PR once #1680 is merged?

[jpmckinney]

Let's create a new issue as I already started reviewing #1680.