Add support for filtering

[sherriff]

Use case:

We have a bunch of Java applications installed using Solaris PKGs. To avoid duplication it would be nice to be able to use templates and just specify the differences using properties.

Example usage:

1. Have a start-up script template which can be reused by multiple projects. Java options, installation basedir and name of start-up jar file are examples of typical variables.
2. Reuse manifest files for Solaris SMF. Typical variables that can be substituted are: Path to start-up script and service name.
3. Reuse action scripts (post-install, pre-remove). Typical variables needed: installation basedir, application name

It seems likely that it is necessary to add configuration option to specify where the different files should be fetched from. E.g. target/unix/files/ and target/unix/scripts instead of src/main/unix/files and src/main/unix/scripts.

See also http://jira.codehaus.org/browse/MUNIX-8 and http://jira.codehaus.org/browse/MUNIX-9.

[trygvis]

Would something like this work?

<copyFile>
  <path>goals.txt</path>
  <toFile>/usr/share/hudson/README.txt</toFile>
  <!-- For a single replacement -->
  <replace>
    <pattern>SCRIPT_NAME</pattern>
    <value>app1</value>
  </replace>

  <!-- For multiple replacements -->
  <replacements>
    <replace>
      <pattern>SCRIPT_NAME</pattern>
      <value>app1</value>
    </replace>
    <replace>
      <pattern>SCRIPT_NAME</pattern>
      <value>app1</value>
    </replace>
  </replacements>
</copyFile>

Re "where to copy files from", I think this is covered already by just specifying whatever you want as <path> inside a <copyFile> or <from> inside a <copyDirectory>.

[sherriff]

Comments and questions:

1. I like the convention over configuration principle that have been used for example to add specific functionality when files are placed in src/main/unix/files and src/main/unix/scripts. A simple boolean to change whether variables in files and scripts folders are replaced or not would then be sufficient. E.g.:

   <configuration>
     <defaults> 
       <filtering>true</filtering>
     </defaults>
   </configuration>
   

2. Adding support for replacements in copyFile/copyDir seems like a good idea. I guess this can be the underlying implementation for the default filtering option in 1 as well?

3. I don't quite understand the syntax of the value tag.

   a. I would prefer to be able to turn replacements/filtering on/off with a flag and then expect the replacements that Maven normally would apply. I think this will result in less configuration than specifying all replacements explicitly. This solution would, however, probably solve the problem.

   b. So if I want to use standard maven properties as placeholders to replace it would look something like this?

   in unix-m-p config:

     <replacements>
         <replace>
           <pattern>${application.install.baseDirectory}</pattern>
           <value>/opt/application1</value>
         </replace>
   

   And I'd have to duplicate this information in the pom if the same value should be used other places? E.g.

     <properties>
       <application.install.baseDirectory>/opt/application1</application.install.baseDirectory>
      </properties>
   

4. I suggest to support replacements only, not a single replacement tag.

[sherriff]

I guess it is a separate issue, but with regards to "where to copy files from". Yes, this is supported by copyFile and copyDirectory. However, I want to use the default "copy rules" that are in effect for src/main/unix/files and scripts as much as possible. The design suggestion in bullet 1 in my previous comment this would be hidden for the user, but the implementation would use a folder containing filtered files instead of the unfiltered templates.

[trygvis]

(1) is a good idea that will control filtering on/off on the implicit copyDirectory. Don't know if I need one for file and one for scripts, but I guess the flexibility is good to have.

(2) yes, this will be how everything is implemented.

(3) The value is the exact value that will be used as a value for every regexp match. As outlined in the original example you would do something like this:

<replacements>
  <replace>
    <!--
        ${application.install.baseDirectory} would be replaced by the Maven XML
        reader. We don't want that.
    -->
    <pattern>@application.install.baseDirectory@</pattern>
    <value>${application.install.baseDirectory}</value>
  </replace>
</replacements>

...

<properties>
  <application.install.baseDirectory>/opt/application1</application.install.baseDirectory>
</properties>

But as you say, using the build-in filtering mechanism would be useful. This means that having this:

<properties>
  <application.install.baseDirectory>/opt/application1</application.install.baseDirectory>
</properties>

...

<copyDirectory>
  <path>target/foo</path>
  <to>/usr/share/foo</to>
  <filtering>true</filtering>
</copyDirectory>

would imply this:

<copyDirectory>
  <path>target/foo</path>
  <to>/usr/share/foo</to>
  <replacements>
    <replace>
      <pattern>\${application\.install\.baseDirectory\}</pattern>
      <value>/opt/application1</value>
    </replace>
  </replacements>
</copyDirectory>

[trygvis]

An alternative approach which is less work and possibly cleaner as it separates copying from filtering:

<copyDirectory>
  ...
</copyDirectory>
<copyFile>
  ...
</copyFile>

<filter>
  <!-- path refers to installed files -->
  <path>/opt/foo/etc</path>

  <!-- similar to resources plugin, as suggested above -->
  <filtering>true</filtering>

  <!-- as above -->
  <replacements>
    ...
  </replacements>
<filter>

Pros:

• Less code, I don't have to add support for filtering in 5 tasks (copy directory/file/artifact, extract file/artifact)
• Cleaner as each task type (copy vs filter) is more specific and hopefully easier to understand

Cons:

• The existing resources and assembly plugins does copy and filtering in one go which is what people might expect.
• The Zip package support will

[cholden14]

Any ETA on Filtering? I am about to have to rewrite our packaging but need filtering to make this particular plugin work.

[trygvis]

The code in the repository should work for the zip packaging. Which packaging are you using?

[cholden14]

Thanks for the reply I am using solaris and rpm's. I have a set of files for the pre and post functions that I want to use as a dependency for each project and have them filtered with application specific information (user, groups, perms).

Any plans to expand the filtering to work with all packages types?

[trygvis]

Planned, yes, timeframe, no. Given that people want it makes it more likely, but I don't have any workhours to spend on it now so don't hold your breath. It is doable to implement yourself if you have the time, use the zip code as a start.

[trygvis]

@cholden14 Is this still something you need? I'm working on this now.

[cholden14]

Yes sir it is!

Sent from my iPhone

On Feb 13, 2013, at 6:06 AM, Trygve Laugstøl [email protected] wrote:

@cholden14 Is this still something you need? I'm working on this now.

—
Reply to this email directly or view it on GitHub.

[trygvis]

Let me know when you've been able to try it out. I'm working on updating the handbook too.

[cholden14]

Trygve

I will look as soon as I can, might be Wednesday

Thank you!

Sent from my iPhone

On Feb 18, 2013, at 8:45 AM, Trygve Laugstøl [email protected] wrote:

Let me know when you've been able to try it out. I'm working on updating the handbook too.

—
Reply to this email directly or view it on GitHub.

[cholden14]

Trygvis,

Forgive me but which filtering option did you implement? I mean in terms of basic syntax?

Thanks
Chris

[trygvis]

Example syntax:

    <filterFiles>
      <includes>
        <include>/opt/hudson/etc/*</include>
      </includes>
      <excludes>
        <exclude>/opt/hudson/etc/unfiltered.properties</exclude>
      </excludes>
      <!-- Regexes does not work just yet. Need a day more -->
      <regexes>
        <regex>
          <!-- A simple regex -->
          <pattern>MY_PLACEHOLDER</pattern>
          <replacement>MY_VALUE</replacement>
        </regex>
        <regex>
          <!-- A regex using replacements -->
          <pattern>^([-a-z]*)=(/.*)</pattern>
          <replacement>$1=/foo$2</replacement>
        </regex>
      </regexes>
    </filterFiles>

However, while updating the handbook I ran into an issue. Only the keys project.version, project.artifactId and project.groupId will be filtered.

I've more or less fixed the code so that all properties (system, user and project) are used and getting the regexp to work.

[cholden14]

Trygvis,

Having trouble getting this to build.. keeps wanting to upload to the sonatype. Any suggestions?

[cholden14]

Nevermind.. idiot typo.. testing the filtering today Trygvis

[cholden14]

Trygvis,

Ok done some testing and keep getting null pointer exceptions for any of my substitution attempts:

                   <filterFiles>
                        <includes>
                            <include>install/*</include>
                        </includes>
                        <regexes>
                        <regexe>
                            <pattern>${pkg.user}</pattern>
                            <replacement>${user}</replacement>
                        </regexe>
                        </regexes>
                    </filterFiles>

any thoughts?

[trygvis]

That filter won't work, ${pkg.user} is not a very good regexp. It should work if you just drop the entire regexes section and make sure ${pkg.user} is defined inside <properties>. If you do not have a regex list will create a set of expressions from the current list of properties.

I tried to document it so if you did read it please let me know what I can improve. It is quite crude yet. Also check out the integration tests for some examples: https://github.com/trygvis/unix-maven-plugin/blob/master/unix-maven-plugin/src/it/test-zip-1/pom4test.xml

Please paste the NPE, that's something that obviously needs fixing :)

[cholden14]

Trygvis,

OK So I have the following properties specified in what I refer to as my packaging module:

com.theice.tribe:tribe-assembly:zip some_user some_group 2044 1501 some_groups some_groups start_script deploy_location crontab_file postinstallhook.sh

My files to filter look like so for the properties to filter:

USER=${pkg.user}
GROUP=${pkg.group}
UID=${pkg.uid}
GID=${pkg.gid}

I have also tried:

USER=pkg.user
GROUP=pkg.group
UID=pkg.uid
GID=pkg.gid

In my files to filter what should they places I want substituted to look like? Sorry to be so dense on this but I bet it's something simple I am missing.

I have looked over the examples and I think this should work or maybe I simply can't filter the solaris specific postinstall/preinstall files?

Thanks for your help!

[trygvis]

Your file should look like what you have tried:

USER=${pkg.user}

I don't have any tests for pkg, only rpm so if you could try that fast that would be nice. You can also try to run the plugin in debugging mode with -Dmaven.unix.debug and/or -X. Feel free to email me the logs privately.

You should see output like this:

[INFO] Filter Files:
[INFO]  Includes:
[INFO]   /opt/hudson/etc/*
[INFO]  Excludes:
[INFO]   /opt/hudson/etc/filter-2.conf
[INFO]   /opt/hudson/etc/unfiltered.properties
[INFO]  Replacers:
[INFO]   pattern=\Q${version.maven-resources-plugin}\E, replacement=2.5
[INFO]   pattern=\Q${version.maven-project-info-reports-plugin}\E, replacement=2.6
[INFO]   pattern=\Q${version.maven-pmd-plugin}\E, replacement=2.5
[INFO]   pattern=\Q${version.maven-jxr-plugin}\E, replacement=2.3
[INFO]   pattern=\Q${version.maven-javadoc-plugin}\E, replacement=2.9
[INFO]   pattern=\Q${version.maven-deploy-plugin}\E, replacement=2.5
[INFO]   pattern=\Q${version.maven-dependency-plugin}\E, replacement=2.5
....

Note that you should NOT have any <regexp> entries at all.