DFC Connector Data Capture Feature & Store

[jgaehring]

Objective

Enable remote data capture functionality in the DFC connector, as requested by
the FDC Governance Circle, so that data may be captured within the DFC
Network and relayed to an independent triple store that will act as a Data
Commons.

Proposal

While we discussed the possible necessity of incorporating the data capture
mechanism into the code generator's templates, I've realized that may not ever
be necessary or even desirable. In all three implementations of the connector,
the core request/response logic can be found within the main Connector class
or its modules (such as the JsonldStream importer and exporter in the case of
the TypeScript implementation), which are all contained within the static code
directories and not produced through code generation. Because these
import/export methods are indirectly invoked by all semantic object subclasses'
getters, setters, adders and removers, it would be the ideal place to inject
optional hooks that could extend the import/export behavior.

A good model for this kind of extension might be the axios library's
interceptor pattern:

// Add a request interceptor
axios.interceptors.request.use(function (config) {
    // Do something before request is sent
    return config;
  }, function (error) {
    // Do something with request error
    return Promise.reject(error);
  });

// Add a response interceptor
axios.interceptors.response.use(function (response) {
    // Any status code that lie within the range of 2xx cause this function to trigger
    // Do something with response data
    return response;
  }, function (error) {
    // Any status codes that falls outside the range of 2xx cause this function to trigger
    // Do something with response error
    return Promise.reject(error);
  });

Internally the axios interceptors are private members of the
InterceptorManager, with a separate instantiation for both request and
response cycle. The interceptors can also be "ejected":

const myInterceptor = axios.interceptors.request.use(function () {/*...*/});
axios.interceptors.request.eject(myInterceptor);

Some consideration should be given to the API for the connector and the
corresponding getters and setters that will actually invoke the capturing logic.
The getters and setter can differ in behavior, with some being synchronous and
others asynchronous, while the capturing behavior will always be asynchronous.
But we could generally take an approach such as the following:

const loggerRef = connector.interceptors.import.use(logger);

Where logger could be a function (or two functions, to handle both success and
error results), or an instance of a Logger class with a wider variety of
configurable options, or both.

As for the triple store, where logs will be sent, there are a lot of options. To
begin, the DFC prototype could be used for running integration tests in the
local development environment. If that achieves much of desired outcomes, a fork
of that could be prepared for deployment. A more customized solution could be
built with SemApps, but might require more development. Another extenuating
factor is the degree to which OFN's stakeholders would like this store to be
integrated with OFN's core software and regional server instances.

As for the triple store to send logs to, there are many options, depending upon
how tightly integrated with the core OFN software and server instances OFN's
stakeholders wish this to be, as opposed to a totally independent server that
core OFN knows nothing about. It may be more difficult to judge with much
accuracy the cost and time required to stand up a maintainable instance of the
triple store based on these decisions and a more detailed conversation. In any
case, however, the proposed logging interceptor should work just the same, since
the only parameter it will strictly require should be a location to send the
logs to. Different logging interceptors can be adapted to different behaviors as
desired, and even combined, since this would enable multiple interceptors. The
flexibility of the interceptor pattern may in fact allow for more incremental
development of the triple store and how it is deployed to production.

Requirements

• Implement the .import.use() and export.use() method, a general interface
  for the function or Interceptor class they would each accept as arguments,
  and the implementations of those functions or classes as the actually
  ImportLogger and ExportLogger. Obviously, the names for all these classes
  and methods can be decided upon later. These will first be implemented in
  TypeScript.
• Write appropriate unit tests for these interceptors and the data capture
  implementation(s), extending the existing TypeScript connector tests as
  appropriate. These will only mock the intended triple store behavior.
• Pending further discussion, develop integration tests that can run against a
  local instance of a triple store, possibly based on the DFC prototype or
  SemApps, that can receive and store JSON-LD logs. Preferably this local
  instance will be containerized so it's easy to replicate on a staging server,
  or perhaps as the basis for store that can eventually go into production for
  the data commons.

Milestones

1. TypeScript connector's import.use() and export.use() methods, interfaces,
   classes, and corresponding unit tests.
2. Local triple store and integration tests of the connector's interceptor API
   and the data capture interceptors specifically.
3. Staging server and/or production deployment of the triple store.

Estimated Time and Cost

Milestones 1 and 2 will each require roughly 15 hours of development time, and
their order is more or less interchangeable. Depending decisions on how best to
develop, test, and deploy the triple store, milestone 3 could vary widely,
potentially as little as 6-12 development hours, or over 30 dev hours, if more
customization is required beyond simply running an off-the-shelf solution.
Similarly, milestone 4 is difficult to assess at this time, but would require at
least the same amount of dev hours, possibly more.

#	Description	Dev Hrs	Est. Cost	Duration
1	Connector features	24 - 30	$2520 - $3150	1 - 2 wks
2	Integration testing	6 - 30	$630-3150	1 - 3 wks
3	Staging/production deployments	12 - 60	$1260-6300	2 - 6 wks

The contingencies in milestones 2 and 3 makes this a very imprecise estimation,
costing anywhere from $4,410 to $12,600 and taking 1 to 3 months to
complete. We can speak in further detail on the expectations for the triple
store as we go ahead with the connector features, or wait until a clearer set of
requirements can be determined for all 3 milestones.

[lecoqlibre]

Thanks Jamie.

@RaggedStaff Can you provide more details about the objective of the project?

[RaggedStaff]

The FDC project are wanting to ensure data that is processed through the DFC interfaces within the UK network is available (in an anonymised/aggregated form) to allow analysis/research on local food ecosystems in the UK.

So the requirement is to capture data from DFC requests, anonymise & store it in a datastore that can be used for aggregate analysis.

Consent for data capture will be provided when a Producer/Hub joins the FDC and is a requirement to participate in the commons.

The data will be owned collectively & the governance circle, which is made up of representatives from participating platforms, industry bodies (e.g. LWA: the local branch of La Via Campesina & Sustain) and producers.

Our initial scope would be to create a mechanism to capture data, anonymise it, and then store it somewhere (unaggregated initially).

Also noting that @jgaehring suggested the potential to have audit logging within the Connector, so that activities undertaken with functionality from extensible modules (like this analytics capture) would always be documented somewhere. I'm in favour of this, but not sure if we (FDC) can fully fund that work.

[jgaehring]

Regarding audit logging, @RaggedStaff and I briefly discussed the possibility of storing those audit logs as linked data themselves, belonging to and co-located with the source of the original data that was captured and to which the logs pertain. Seems like like an elegant (ie, doesn't require a separate storage mechanism) but not especially simple or easy way of handling this (unless there's it's an already implemented solution that meets the criteria and is easy to adopt). All to say, it would indeed be outside the scope of the capturing feature itself, and it would need an alternative source of funding. On that not, however, it could prove to be an attractive feature to prospective funders, so worth keeping somewhere in the middle part of one's mind.

[RaggedStaff]

@lecoqlibre has proposed to introduce mixins to the Semantizer.

We can utilise these from the connector (prob via a new static module) controlled by a boolean env variable, that defaults to FALSE.

We will also require further configuration (i.e. where to send the data)