Delegating image extraction to CohortExtractor from RDMP

[tznind]

This discussion currently relates to the 'HIC mirror' and integrating the extraction practices of SmiServices with the current (small scale) extraction capabilities of RDMP.Dicom that HIC use today.

Use Case

Data analyst wants to extract multiple datasets including imaging modalities (e.g. CT) for a project cohort (list of CHI/ECHIs) directly from RDMP. Then run an extraction of all the anonymous clinical and accompanying images to disk with a single click.

User adds datasets to extraction and clicks Execute

Current RDMP support for image extraction

Current image extraction from RDMP via the Rdmp.Dicom plugin are covered here:
https://github.com/HicServices/RdmpDicom/blob/develop/Documentation/DataExtraction.md

Proposed changes

Add a new component in a plugin to RDMP that instead of handling the image extraction itself in a block single threaded manner instead sends messages to RabbitMQ (see below for messages).

Issues:

• Currently ExtractionRequestMessage and the RabbitMQ sender logic is in SmiServices repo while the imaging plugin is upstream in Rdmp.Dicom repo. So either we need a second plugin in the scope of SmiServices or do a lot of code duplication
• The UID mapping to project specific mapping capability needs to be added to the extraction pipeline (support for breaking the ability to easily link images between projects). This is a feature of FoDicomAnonymiser that would need to be replicated.
• The extraction pipeline currently substitutes UIDs in the RDMP extraction pipeline. We can't really do that if it is an async later on process
• The extraction pipeline currently outputs file paths too, that would need to change as it depends on how microservices are configured as to where they output the anonymous images to.
• The RDMP extraction could be at Study / Series or Image level or anything inbetween. If we are extracting based on StudyInstanceUID for example then we might in a single extraction run get duplication and order is not garuanteed. This would introduce complexity for using ExtractFileCollectionInfoMessage

Messages

The plugin must take the role of CohortExtractor and all upstream services (extractorCL). As such it needs to produce the following messages:

Looks like we need to send multiple messages:

• ExtractionRequestInfoMessage (what I want extracted is X Tag with Y Value)
  • What is 'KeyValueCount' is that just an autonum?
• ExtractFileMessage (Hey you CTP!, copy out the file at DicomFilePath path)
• ExtractFileCollectionInfoMessage (Hey Packager, this is what I told CTP to do)

Release Plugin

Since extraction will be async we need a plugin that works at release stage in RDMP that will say 'not ready for release because images are not anonymised yet'.

[tznind]

So we could require that the image table HAS to have PatientID, StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID AND RelativeFileArchiveUri in the table to be supported by the extraction tool (i.e. CT_ImageTable, MR_ImageTable etc).

That would mean we could do the UID swapping prior to sending to RabbitMQ.

[tznind]

Release Plugin

Should look something like this:

public class JustSayNo : IPluginDataFlowComponent<ReleaseAudit>, IPipelineRequirement<Project>, IPipelineRequirement<ReleaseData>
{
	private Project _project;
	private ReleaseData _release;

	public void Abort(IDataLoadEventListener listener)
	{
		throw new NotImplementedException();
	}

	public void Check(ICheckNotifier notifier)
	{
		if(_release == null)
		{
			return;
		}

		foreach(var kvp in _release.ConfigurationsForRelease)
		{
			var config = kvp.Key;
			foreach(var releaseable in kvp.Value)
			{
				//TODO:  if releaseable.DataSet is imaging
				// TODO: Check MongoDb Packaging  Db to see if it is finished extracting images

				notifier.OnCheckPerformed(new CheckEventArgs($"Images not ready yet for {releaseable.DataSet} in {config}", CheckResult.Fail));
			}
		}
	}

	public void Dispose(IDataLoadEventListener listener, Exception pipelineFailureExceptionIfAny)
	{
		
	}

	public void PreInitialize(Project value, IDataLoadEventListener listener)
	{
		_project = value;
	}

	public void PreInitialize(ReleaseData value, IDataLoadEventListener listener)
	{
		_release = value;
	}

	public ReleaseAudit ProcessPipelineData(ReleaseAudit toProcess, IDataLoadEventListener listener, GracefulCancellationToken cancellationToken)
	{
		return toProcess;
	}
}

When checking a configuration with the above code you would get :

[rkm]

Notes from call:

• Update diagram of extraction
• Send SOPInstance messages
• Store Guid
• Change to ExternalDatabaseServer for rabbit settings
• Cohort Packager respond to status messages (is guid done yet?)
• Rabbit MQ uri format?
• Mapping table in CTP
  • https://mircwiki.rsna.org/index.php?title=The_CTP_DICOM_Anonymizer#.40lookup.28ElementName.2CKeyType.2Caction.29