-
Notifications
You must be signed in to change notification settings - Fork 78
Migrating a Roxy project to ml gradle
Version 2.9.0 of ml-gradle includes support for migrating a Roxy project into an ml-gradle project. More support will be provided in future release, but what's supported in 2.9.0 is described below.
Migrating a Roxy project into an ml-gradle project involves two high-level tasks - migrating the source files, properties, and build steps from the Roxy project files, and then exporting your Roxy project resources from MarkLogic, which requires that the Roxy project already be deployed to MarkLogic. These two tasks are described in more detail below.
The FAQ for Roxy Users gives a conceptual overview of how the structures change when moving from Roxy to ml-gradle.
To migrate a Roxy project into an ml-gradle project, you need two things - you need Gradle installed, and you need a Gradle build file that applies the ml-gradle plugin. You can create this file in your Roxy project directory, but it's recommended to try this out first in a new directory so that you can examine the results separate from your Roxy project, as there's some overlap with the "src" directory.
So let's assume you make a new directory for your ml-gradle project - then just create a file named "build.gradle" with the following in it (use the latest version of ml-gradle, which is currently 3.1.0):
plugins {
id "com.marklogic.ml-gradle" version "3.1.0"
}
In addition, you'll most likely want to use the Gradle properties plugin to support property values for different environments:
plugins {
id "net.saliman.properties" version "1.4.6"
id "com.marklogic.ml-gradle" version "3.1.0"
}
You can find more information about properties in a Gradle project in the Configuring ml-gradle page.
Running "gradle tasks" shows the following new group of tasks in 2.9.0:
Ml-gradle Roxy tasks
--------------------
mlRoxyMigrateBuildSteps - Migrate build steps from deploy/app_specific.rb into custom Gradle tasks. Use -ProxyProjectPath to define the location of your Roxy project, and -PappSpecificPath to define a path other than deploy/app_specific.rb
mlRoxyMigrateFiles - Migrate Roxy source files into this Gradle project. Use -ProxyProjectPath to define the location of your Roxy project.
mlRoxyMigrateProject - Run all tasks for migrating a Roxy project into this Gradle project. Use -ProxyProjectPath to define the location of your Roxy project.
mlRoxyMigrateProperties - Migrate Roxy properties into the gradle.properties file in this project. Use -ProxyProjectPath to define the location of your Roxy project.
Note that the command-line property "roxyProjectPath" is required for all of the tasks.
The simplest thing to run then is "mlRoxyMigrateProject", which runs every other task:
gradle mlRoxyMigrateProject -ProxyProjectPath=/path/to/roxy/project
You can also run each task individually too, if you want to proceed more carefully and examine the output of each task before invoking another one:
gradle mlRoxyMigrateFiles -ProxyProjectPath=/path/to/roxy/project
gradle mlRoxyMigrateBuildSteps -ProxyProjectPath=/path/to/roxy/project
gradle mlRoxyMigrateProperties -ProxyProjectPath=/path/to/roxy/project
There are no dependencies between these tasks, so you can run them in any order.
The sections below have more information on each task.
The "mlRoxyMigrateFiles" task copies files from the Roxy "src" and "rest-api" directories into the correct locations in the ml-gradle project. All MarkLogic modules in an ml-gradle project are under "src/main/ml-modules" - the mapping of Roxy files to directories under that directory is:
- src -> /root
- rest-api/config -> /options
- rest-api/ext -> /services
- rest-api/transforms -> transforms
Version 2.9.0 of ml-gradle also includes support for deleting test modules. See the property reference for more information - specifically, the mlDeleteTestModules and mlDeleteTestModulesPattern properties. Assuming you're using the Gradle properties plugin, you can simply e.g. set mlDeleteTestModules=true in gradle-(environmentName).properties.
The "mlRoxyMigrateProperties" task copies Roxy properties from deploy/default.properties and deploy/build.properties into the standard Gradle properties file, which is gradle.properties.
This task will map known Roxy properties into their associated Gradle properties. Any Roxy property that doesn't map to a Gradle property is copied over as-is. As of version 2.9.0, the following Roxy properties are mapped:
- app-name -> mlAppName
- app-port -> mlRestPort
- user -> mlUsername
- password -> mlPassword
- group -> mlGroupName
- content-forests-per-host -> mlContentForestsPerHost
Please see Configuring ml-gradle for more information on Gradle properties.
NOTE - ml-gradle will create a REST API server by default unless the property "mlNoRestServer" is set to "true". Version 3.0.0 will automatically add this property to gradle.properties. But with 2.9.0, you'll most likely want to manually add this property, as your exported application (see below) will likely have all the app servers it wants already.
The "mlRoxyMigrateBuildSteps" task attempts to migrate custom Roxy build steps in the ./deploy/app_specific.rb file into custom Gradle tasks. This support is limited - it relies on parsing text, and it only works for build steps that executes queries. For each such build step, this task will add to the build.gradle file a new task that makes a call to /v1/eval with the same XQuery as the associated Roxy build step.
Instead of trying to convert the contents of the Roxy ml-config.xml into Manage API files, the recommended approach is to first deploy your Roxy application, and then use the ml-gradle export task for exporting the resources in your Roxy application as Manage API files.
The Wiki page for exporting resources provides instructions on how to select resources to export.
Resources are by default exported to the location that ml-gradle expects to find them - src/main/ml-config. A few caveats about this:
- You can use a prefix or regex to specify the resources to export, but you may need to use the properties file approach to get everything. It's common for applications to have resources like privileges that don't start with a common prefix, and thus you may need to specify them explicitly in a properties file.
- The export feature in ml-gradle doesn't yet support exporting every resource type. You can always manually export tasks via the Manage API on port 8002 of your MarkLogic server.
- The export feature prints out helpful messages about the resources it has exported. Pay attention to these, as you may need to make some manual edits. For example, MarkLogic does not allow for a user password to be exported for obvious reasons, so a default password is used that you'll most likely want to change.
Once you've migrated all the Roxy files and exported your application resources, try deploying the application (the "-i" flag tells Gradle to log at the info level, which is normally helpful when using ml-gradle, as you'll see every call that's made to the Manage API):
gradle -i mlDeploy
ml-gradle doesn't have a pre-defined task for loading data from a specific directory like Roxy does - e.g. the "data" directory and "ml local deploy content". Instead, you can configure as many data-loading tasks as you want using MLCP (Content Pump). Please see this guide for more information.
The reason why ml-gradle doesn't have a pre-defined task for loading data is so that ml-gradle doesn't depend on a specific version of MLCP. Instead, you can use whatever version of MLCP you need via Gradle's normal dependency management.
Here are some of the problems you may run into that require manual changes or exporting additional resources:
- Something wasn't exported from your Roxy application that's required for deployment - i.e. a role requires a privilege, and the role was exported but not the privilege.
- Two or more roles have circular dependencies. This is difficult to resolve with the Manage API, as each role is created in a separate transaction. You'll need to define two files for one of the roles - one that doesn't depend on any roles, and one that includes the role dependencies.
- Some build steps weren't migrated that are needed. You'll need to manually recreate these as Gradle tasks - file an issue if you need help with this.