Skip to content

2.3.0
Compare
Choose a tag to compare
@rjrudin rjrudin released this 26 Jul 17:55
· 98 commits to master since this release
2.3.0
0eaf6b1

This minor release provides significant new functionality in support of the 1.0.0 release of the new MarkLogic Flux data movement tool. Much of this functionality is documented in the Flux documentation. We will soon have complete documentation of all the new options in this repository's documentation as well.

In the meantime, the new options in this release are listed below.

Read Options

  1. spark.marklogic.read.javascriptFile and spark.marklogic.read.xqueryFile allow for custom code to be read from a file path.
  2. spark.marklogic.read.partitions.javascriptFile and spark.marklogic.read.partitions.xqueryFile allow for custom code to be read from a file path.
  3. Can now read document rows by specifying a list of newline-delimited URIs via the spark.marklogic.read.documents.uris option.
  4. Can now read rows containing semantic triples in MarkLogic via spark.marklogic.read.triples.graphs, spark.marklogic.read.triples.collections, spark.marklogic.read.triples.query, spark.marklogic.read.triples.stringQuery, spark.marklogic.read.triples.uris, spark.marklogic.read.triples.directory, spark.marklogic.read.triples.options, spark.marklogic.read.triples.filtered, and spark.marklogic.read.triples.baseIri.
  5. Can now read Flux and MLCP archives by setting spark.marklogic.read.files.type to archive or mlcp_archive.
  6. Can control which categories of metadata are read from Flux archives via spark.marklogic.read.archives.categories.
  7. Can now specify the encoding of a file to read via spark.marklogic.read.files.encoding.
  8. Can now see progress logged of reading data from MarkLogic via spark.marklogic.read.logProgress.
  9. Can specify whether to fail on a file read error via spark.marklogic.read.files.abortOnFailure.

Write Options

  1. spark.marklogic.write.threadCount has been altered to reflect the common user understanding of "number of threads used to connect to MarkLogic". If you need to specify a thread count per partition, use spark.marklogic.write.threadCountPerPartition.
  2. Can now see progress logged of data written to MarkLogic via spark.marklogic.write.logProgress.
  3. spark.marklogic.write.javascriptFile and spark.marklogic.write.xqueryFile allow for custom code to be read from a file path.
  4. Settingspark.marklogic.write.archivePathForFailedDocuments to a file path will result in any failed documents being added to an archive zip file at that file path.
  5. spark.marklogic.write.jsonRootName allows for a root field to be added to a JSON document constructed from an arbitrary row.
  6. spark.marklogic.write.xmlRootName and spark.marklogic.write.xmlNamespace allow for an XML document to be constructed from an arbitrary row.
  7. Options starting with spark.marklogic.write.json. will be used to configure how the connector serializes a Spark row into a JSON object.
  8. Can use spark.marklogic.write.graph and spark.marklogic.write.graphOverride to specify the graph when writing RDF triples to MarkLogic.
  9. Deprecated spark.marklogic.write.fileRows.documentType in favor of using spark.marklogic.write.documentType to force a document type on documents written to MarkLogic with an extension unrecognized by MarkLogic.
  10. Can use spark.marklogic.write.files.prettyPrint to pretty-print JSON and XML files written by the connector.
  11. Can use spark.marklogic.write.files.encoding to write files in a different encoding.
  12. Can use spark.marklogic.write.files.rdf.format to specify an RDF type when writing triples to RDF files.
  13. Can use spark.marklogic.write.files.rdf.graph to specify a graph when writing RDF files.
Assets 4
Loading