#37649 use filesystem events to trigger file upload when available, p…

CODE_OF_CONDUCT.md

@@ -0,0 +1,7 @@
+# Code of Conduct
+
+Do not offend.
+
+Do not be easily offended.
+
+Go with the flow.

CONTRIBUTING.md

@@ -0,0 +1,16 @@
+# Before You Start
+
+Get in touch with us to discuss your contribution: <mailto:[email protected]> 
+
+# Contributing
+
+All changes or additions that potentially benefit a significant portion of the user base are welcome. E.g. adding 
+support for a new type of file source/sink, say S3 storage. 
+
+# When to Fork Instead of Contributing
+
+If your change is specific to your particular environment or usage scenario, or targets a niche product or protocol, 
+then it is better to fork the project as we are unlikely to merge such a change into the main repository. E.g. 
+adding an HTTP endpoint that exposes a proprietary telemetry data format.  
+
+

README.md

@@ -7,7 +7,9 @@ There is no endpoint (beside actuator/health) that are offered here.
 The CDR Client is triggered by a scheduler and synchronizes by the given delay time the files from the CDR API.
 
 ### Functionality
-For each defined connector the CDR Client calls the defined endpoints of the CDR API.
+For each defined connector the CDR Client calls the defined endpoint of the CDR API.
+
+#### Document Download
 
 For each connector one file after the other is pulled. Each file is written into a temporary folder defined as 'local-folder'.
 The file is named after the received 'cdr-document-uuid' header that is a unique identifier created by the CDR API.
@@ -17,13 +19,27 @@ After successfully deleting the file in the CDR API, the file is moved to the co
 The temporary folders need to be monitored by another tool to make sure that no files are forgotten (should only happen if the move
 to the destination folder is failing).
 
-For each connector one file after the other is pushed from the defined 'source-folder'. After the file is successfully uploaded it will be deleted.
+#### Document Upload
+
+Document upload uses a combination of directory polling and event driven uploads. The polling process inspects the 
+contents of every source folder at the configured interval and uploads all `.xml` files it finds to the CDR API. The 
+event driven process listens for filesystem events from the same directories and uploads all `.xml` files as they 
+are created. The two approaches are combined so
+
+* at start of the client all files that might have arrived while the client was not running are uploaded
+* folders on (remote) filesystems that do not support filesystem events can be used as source folders
+
+If the filesystem that hosts a source folder supports filesystem events, then the polling process normally won't find 
+any files to process and immediately goes back to sleep. If the polling process wakes up right at the moment a new file 
+arrives, it might happen that both processes pick up the same file for processing. However, only one of the two will 
+continue to process the file, depending on which one is first to register the file for processing.
+
+After the file is successfully uploaded it will be deleted.
 If the upload failed with a response code of 4xx the file will be appended with '.error' and an additional file with the same name as the sent file, but with
 the extension '.log' will be created and the received response body will be saved to this file.
-If the upload failed with a response code of 5xx the file will be retried a defined amount of times, 
-see retry-delay in the [application-client.yaml](./src/main/resources/config/application-client.yaml) file. After reaching the max retry count the file will 
-be appended with '.error' and an additional file with the same name as the sent file, but with the extension '.log' will be created and the received response 
-body will be saved to this file.
+If the upload failed with a response code of 5xx the file will be retried indefinitely, assuming the root cause is 
+an infrastructure issue that will ultimately be resolved (and uploading another file would fail too, for the same 
+reason). See retry-delay in the [application-client.yaml](./src/main/resources/config/application-client.yaml) file.
 
 ## Local development
 To test some usecases there is a [docker-compose.yaml](./docker-compose/docker-compose.yaml) with wiremock that simulates the CDR API. Run with ```docker-compose down && docker-compose up --build```.

build.gradle.kts

@@ -7,13 +7,17 @@ version = "3.0.1-SNAPSHOT"
 java.sourceCompatibility = JavaVersion.VERSION_17
 
 val jvmVersion: String by project
+val kotlinCoroutinesVersion: String by project
 val springCloudVersion: String by project
 val jacocoVersion: String by project
 val kotlinLoggingVersion: String by project
 val mockkVersion: String by project
 val logstashEncoderVersion: String by project
 val micrometerTracingVersion: String by project
-val detektKotlinVersion: String by project
+val kfsWatchVersion: String by project
+val kacheVersion: String by project
+val springMockkVersion: String by project
+val awaitilityVersion: String by project
 
 val outputDir: Provider<Directory> = layout.buildDirectory.dir(".")
 
@@ -47,27 +51,35 @@ dependencyManagement {
     imports {
         mavenBom("org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}")
     }
+    // https://github.com/detekt/detekt/issues/6198#issuecomment-2265183695
+    configurations.matching { it.name == "detekt" }.all {
+        resolutionStrategy.eachDependency {
+            if (requested.group == "org.jetbrains.kotlin") {
+                useVersion(io.gitlab.arturbosch.detekt.getSupportedKotlinVersion())
+            }
+        }
+    }
 }
 
 dependencies {
     implementation("org.springframework.boot:spring-boot-starter-web")
-    implementation("org.springframework.boot:spring-boot-starter-web-services")
     implementation("org.springframework.boot:spring-boot-starter-actuator")
-    implementation("org.springframework.cloud:spring-cloud-commons")
     implementation("com.squareup.okhttp3:okhttp")
     implementation("org.jetbrains.kotlin:kotlin-reflect")
-    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
-    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core")
+    implementation("org.jetbrains.kotlin:kotlin-stdlib")
+    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:${kotlinCoroutinesVersion}")
+    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor:${kotlinCoroutinesVersion}") // to enable @Scheduled on Kotlin suspending functions
     implementation("io.github.oshai:kotlin-logging:${kotlinLoggingVersion}")
     implementation("net.logstash.logback:logstash-logback-encoder:${logstashEncoderVersion}")
     implementation("io.micrometer:micrometer-tracing:${micrometerTracingVersion}")
     implementation("io.micrometer:micrometer-tracing-bridge-otel:${micrometerTracingVersion}")
+    implementation("io.github.irgaly.kfswatch:kfswatch:$kfsWatchVersion")
+    implementation("com.mayakapps.kache:kache:$kacheVersion")
 
     kapt("org.springframework.boot:spring-boot-configuration-processor")
 
     testImplementation("org.jacoco:org.jacoco.core:${jacocoVersion}")
     testImplementation("org.springframework.boot:spring-boot-starter-test")
-    testImplementation("org.springframework.boot:spring-boot-starter-webflux")
     testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test")
     testImplementation("com.squareup.okhttp3:mockwebserver") {
         // Unfortunately we cannot exclude JUnit 4 as MockWebServer implements interfaces from that version
@@ -78,7 +90,9 @@ dependencies {
     }
     testImplementation("org.junit.jupiter:junit-jupiter")
     testImplementation("io.micrometer:micrometer-tracing-test")
-    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
+    testImplementation("com.ninja-squad:springmockk:${springMockkVersion}")
+    testImplementation("org.awaitility:awaitility:${awaitilityVersion}")
+
 }
 
 springBoot {
@@ -177,15 +191,6 @@ detekt {
     baseline = file("$projectDir/detekt_baseline.xml") // Module specific suppression list.
 }
 
-// https://github.com/detekt/detekt/issues/6198
-project.afterEvaluate {
-    configurations["detekt"].resolutionStrategy.eachDependency {
-        if (requested.group == "org.jetbrains.kotlin") {
-            useVersion(detektKotlinVersion)
-        }
-    }
-}
-
 tasks.register("publishVersion") {
     group = "publishing"
     description = "Publishes boot jar"

config/detekt.yml

@@ -2,10 +2,10 @@ build:
   maxIssues: 0
   excludeCorrectable: false
   weights:
-    # complexity: 2
-    # LongParameterList: 1
-    # style: 1
-    # comments: 1
+  # complexity: 2
+  # LongParameterList: 1
+  # style: 1
+  # comments: 1
 
 config:
   validation: true
@@ -34,11 +34,11 @@ processors:
 console-reports:
   active: true
   exclude:
-     - 'ProjectStatisticsReport'
-     - 'ComplexityReport'
-     - 'NotificationReport'
-     - 'FindingsReport'
-     - 'FileBasedFindingsReport'
+    - 'ProjectStatisticsReport'
+    - 'ComplexityReport'
+    - 'NotificationReport'
+    - 'FindingsReport'
+    - 'FileBasedFindingsReport'
   #  - 'LiteFindingsReport'
 
 output-reports:
@@ -175,7 +175,6 @@ complexity:
     ignoreDeprecated: false
     ignorePrivate: false
     ignoreOverridden: false
-    ignoreAnnotatedFunctions: []
 
 coroutines:
   active: true

gradle.properties

@@ -1,26 +1,35 @@
 kotlin.code.style=official
+
+# Export those variables in your environment, e.g. in `.profile`, to point to the respective JDK installations
+# Unless your JDKs are installed in the system default location, then Gradle will find them automatically.
+org.gradle.java.installations.fromEnv=JDK17,JDK21
+
 ############
 # Version Management
 ############
 ############
 # Kotlin
 ############
 jvmVersion=17
-kotlinVersion=1.9.24
-kotlinxSerializationVersion=1.6.3
+kotlinVersion=2.0.21
+kotlinCoroutinesVersion=1.9.0
 ############
 # Plugins
 ############
-detektVersion=1.23.6
-springBootVersion=3.3.0
-springDependencyManagementVersion=1.1.5
+detektVersion=1.23.7
+springBootVersion=3.3.4
+springDependencyManagementVersion=1.1.6
 ############
 # Dependencies
 ############
-detektKotlinVersion=1.9.23
 jacocoVersion=0.8.12
-kotlinLoggingVersion=6.0.9
-logstashEncoderVersion=7.4
-micrometerTracingVersion=1.3.1
-mockkVersion=1.13.11
-springCloudVersion=2023.0.2
+kotlinLoggingVersion=7.0.0
+logstashEncoderVersion=8.0
+micrometerTracingVersion=1.3.5
+mockkVersion=1.13.12
+springCloudVersion=2023.0.3
+kfsWatchVersion=1.3.0
+kacheVersion=2.1.0
+springMockkVersion=4.0.2
+awaitilityVersion=4.2.2
+

gradle/wrapper/gradle-wrapper.properties

@@ -1,6 +1,7 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-8.8-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip
 networkTimeout=10000
+validateDistributionUrl=true
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists

gradlew

@@ -15,6 +15,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# SPDX-License-Identifier: Apache-2.0
+#
 
 ##############################################################################
 #
@@ -55,7 +57,7 @@
 #       Darwin, MinGW, and NonStop.
 #
 #   (3) This script is generated from the Groovy template
-#       https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
+#       https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
 #       within the Gradle project.
 #
 #       You can find Gradle at https://github.com/gradle/gradle/.
@@ -83,10 +85,9 @@ done
 # This is normally unused
 # shellcheck disable=SC2034
 APP_BASE_NAME=${0##*/}
-APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
-
-# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
+# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
+APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
+' "$PWD" ) || exit
 
 # Use the maximum available, or set MAX_FD != -1 to use that value.
 MAX_FD=maximum
@@ -133,26 +134,29 @@ location of your Java installation."
     fi
 else
     JAVACMD=java
-    which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+    if ! command -v java >/dev/null 2>&1
+    then
+        die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
 
 Please set the JAVA_HOME variable in your environment to match the
 location of your Java installation."
+    fi
 fi
 
 # Increase the maximum file descriptors if we can.
 if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
     case $MAX_FD in #(
       max*)
         # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
-        # shellcheck disable=SC3045 
+        # shellcheck disable=SC2039,SC3045
         MAX_FD=$( ulimit -H -n ) ||
             warn "Could not query maximum file descriptor limit"
     esac
     case $MAX_FD in  #(
       '' | soft) :;; #(
       *)
         # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
-        # shellcheck disable=SC3045 
+        # shellcheck disable=SC2039,SC3045
         ulimit -n "$MAX_FD" ||
             warn "Could not set maximum file descriptor limit to $MAX_FD"
     esac
@@ -197,11 +201,15 @@ if "$cygwin" || "$msys" ; then
     done
 fi
 
-# Collect all arguments for the java command;
-#   * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
-#     shell script including quotes and variable substitutions, so put them in
-#     double quotes to make sure that they get re-expanded; and
-#   * put everything else in single quotes, so that it's not re-expanded.
+
+# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
+
+# Collect all arguments for the java command:
+#   * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
+#     and any embedded shellness will be escaped.
+#   * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
+#     treated as '${Hostname}' itself on the command line.
 
 set -- \
         "-Dorg.gradle.appname=$APP_BASE_NAME" \

gradlew.bat

@@ -13,6 +13,8 @@
 @rem See the License for the specific language governing permissions and
 @rem limitations under the License.
 @rem
+@rem SPDX-License-Identifier: Apache-2.0
+@rem
 
 @if "%DEBUG%"=="" @echo off
 @rem ##########################################################################
@@ -43,11 +45,11 @@ set JAVA_EXE=java.exe
 %JAVA_EXE% -version >NUL 2>&1
 if %ERRORLEVEL% equ 0 goto execute
 
-echo.
-echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
-echo.
-echo Please set the JAVA_HOME variable in your environment to match the
-echo location of your Java installation.
+echo. 1>&2
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
+echo. 1>&2
+echo Please set the JAVA_HOME variable in your environment to match the 1>&2
+echo location of your Java installation. 1>&2
 
 goto fail
 
@@ -57,11 +59,11 @@ set JAVA_EXE=%JAVA_HOME%/bin/java.exe
 
 if exist "%JAVA_EXE%" goto execute
 
-echo.
-echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
-echo.
-echo Please set the JAVA_HOME variable in your environment to match the
-echo location of your Java installation.
+echo. 1>&2
+echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
+echo. 1>&2
+echo Please set the JAVA_HOME variable in your environment to match the 1>&2
+echo location of your Java installation. 1>&2
 
 goto fail