Skip to content

Commit

Permalink
Merge pull request #202 from ARGOeu/devel
Browse files Browse the repository at this point in the history
Prepare release 1.3
  • Loading branch information
themiszamani authored Jul 22, 2021
2 parents 5565f2c + 1887763 commit 10fb722
Show file tree
Hide file tree
Showing 482 changed files with 58,874 additions and 1,492 deletions.
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,8 @@ __pycache__
.DS_STORE
conf.cfg
.pytest_cache

# Java related
.classpath
.project
.settings/
15 changes: 8 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
# Set-up a python centric enviroment in order to easily choose py version:2.7
# bonus: Java 7 and mvn also included
language: python
# Target py version 2.7
# Target py version 3.6
python:
- "2.7"
- "3.6"

script:
- pip install -r ./bin/requirements.txt
- pytest
- cd flink_jobs/ams_ingest_metric/ && travis_wait mvn test
- cd ../batch_ar && travis_wait mvn test
- cd ../batch_status && travis_wait mvn test
- cd ../stream_status && travis_wait mvn test
- cd ../ams_ingest_sync && travis_wait mvn test
- cd flink_jobs/ams_ingest_metric/ && travis_wait mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn test
- cd ../batch_ar && travis_wait mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn test
- cd ../batch_status && travis_wait mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn test
- cd ../stream_status && travis_wait mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn test
- cd ../ams_ingest_sync && travis_wait mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn test


79 changes: 79 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@

pipeline {
agent none
options {
checkoutToSubdirectory('argo-streaming')
newContainerPerStage()
}
environment {
PROJECT_DIR='argo-streaming'
REQUIREMENTS="${PROJECT_DIR}/bin/requirements.txt"
}
stages {
stage('Configuration scripts Tests') {
agent {
docker {
image 'argo.registry:5000/epel-7-py36'
args '-u jenkins:jenkins'
}
}
steps {
echo 'Testing compute engine auto configuration scripts'
sh """
pip3 install -r ${REQUIREMENTS} --user
pytest --junit-xml=${PROJECT_DIR}/junit.xml --cov=${PROJECT_DIR} --cov-report=xml
"""
junit '**/junit.xml'
cobertura coberturaReportFile: '**/coverage.xml'
}
post {
always {
cleanWs()
}
}
}
stage('Flink Jobs Testing & Packaging') {
agent {
docker {
image 'argo.registry:5000/epel-7-java18'
args '-u jenkins:jenkins'
}
}
steps {
echo 'Packaging & Testing Flink Jobs'
sh """
mvn clean package cobertura:cobertura -Dcobertura.report.format=xml -f ${PROJECT_DIR}/flink_jobs/stream_status/pom.xml
mvn clean package cobertura:cobertura -Dcobertura.report.format=xml -f ${PROJECT_DIR}/flink_jobs/batch_ar/pom.xml
mvn clean package cobertura:cobertura -Dcobertura.report.format=xml -f ${PROJECT_DIR}/flink_jobs/batch_status/pom.xml
mvn clean package cobertura:cobertura -Dcobertura.report.format=xml -f ${PROJECT_DIR}/flink_jobs/ams_ingest_metric/pom.xml
mvn clean package cobertura:cobertura -Dcobertura.report.format=xml -f ${PROJECT_DIR}/flink_jobs/ams_ingest_sync/pom.xml
mvn clean package cobertura:cobertura -Dcobertura.report.format=xml -f ${PROJECT_DIR}/flink_jobs/status_trends/pom.xml
"""
junit '**/target/surefire-reports/*.xml'
cobertura coberturaReportFile: '**/target/site/cobertura/coverage.xml'
archiveArtifacts artifacts: '**/target/*.jar'
}
post {
always {
cleanWs()
}
}
}
}
post {
success {
script{
if ( env.BRANCH_NAME == 'master' || env.BRANCH_NAME == 'devel' ) {
slackSend( message: ":rocket: New version for <$BUILD_URL|$PROJECT_DIR>:$BRANCH_NAME Job: $JOB_NAME !")
}
}
}
failure {
script{
if ( env.BRANCH_NAME == 'master' || env.BRANCH_NAME == 'devel' ) {
slackSend( message: ":rain_cloud: Build Failed for <$BUILD_URL|$PROJECT_DIR>:$BRANCH_NAME Job: $JOB_NAME")
}
}
}
}
}
79 changes: 79 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,3 +368,82 @@ Ingest Sync | Ingesting sync data from `{{ams-endpoint}}`/v1/projects/`{{project
Batch AR | Ar Batch job for tenant:`{{tenant}}` on day:`{{day}}` using report:`{{report}}`
Batch Status | Status Batch job for tenant:`{{tenant}}` on day:`{{day}}` using report:`{{report}}`
Streaming Status | Streaming status using data from `{{ams-endpoint}}`/v1/projects/`{{project}}`/subscriptions/`[`{{metric_subscription}}`,`{{sync_subscription}}`]

## Status Trends
Flink batch Job that calculate status trends for critical,warning,unknown status
Job requires parameters:

`--yesterdayData` : file location of previous day's data
`--todayData` : file location of today day's data
`--N` : (optional) number of displayed top results
`--mongoUri` : uri to the mongo db , to store results
`--apiUri` : uri to the web-api
`--key` : users's token, used for authentication
`--proxy` : (optional) proxy url
`--clearMongo` : (optional) defines if the collections in mongo will be cleared from previous documents or not. if false or is missing collection will remain as it is


Flink batch Job that calculate flip flop trends for service endpoints metrics
Job requires parameters:

`--yesterdayData` : file location of previous day's data
`--todayData` : file location of today day's data
`--N` : (optional) number of displayed top results
`--mongoUri` : uri to the mongo db , to store results
`--apiUri` : uri to the web-api
`--key` : users's token, used for authentication
`--proxy` : (optional) proxy url
`--clearMongo` : (optional) defines if the collections in mongo will be cleared from previous documents or not. if false or is missing collection will remain as it is


Flink batch Job that calculate flip flop trends for service endpoints
Job requires parameters:

`--yesterdayData` : file location of previous day's data
`--todayData` : file location of today day's data
`--N` : (optional) number of displayed top results
`--mongoUri` : uri to the mongo db , to store results
`--apiUri` : uri to the web-api
`--key` : users's token, used for authentication
`--proxy` : (optional) proxy url
`--clearMongo` : (optional) defines if the collections in mongo will be cleared from previous documents or not. if false or is missing collection will remain as it is


Flink batch Job that calculate flip flop trends for service
Job requires parameters:

`--yesterdayData` : file location of previous day's data
`--todayData` : file location of today day's data
`--N` : (optional) number of displayed top results
`--mongoUri` : uri to the mongo db , to store results
`--apiUri` : uri to the web-api
`--key` : users's token, used for authentication
`--proxy` : (optional) proxy url
`--clearMongo` : (optional) defines if the collections in mongo will be cleared from previous documents or not. if false or is missing collection will remain as it is


Flink batch Job that calculate flip flop trends for groups
Job requires parameters:

`--yesterdayData` : file location of previous day's data
`--todayData` : file location of today day's data
`--N` : (optional) number of displayed top results
`--mongoUri` : uri to the mongo db , to store results
`--apiUri` : uri to the web-api
`--key` : users's token, used for authentication
`--proxy` : (optional) proxy url
`--clearMongo` : (optional) defines if the collections in mongo will be cleared from previous documents or not. if false or is missing collection will remain as it is

Flink batch Job that calculate flip flop trends for all levels of groups
Job requires parameters:

`--yesterdayData` : file location of previous day's data
`--todayData` : file location of today day's data
`--N` : (optional) number of displayed top results
`--mongoUri` : uri to the mongo db , to store results
`--apiUri` : uri to the web-api
`--key` : users's token, used for authentication
`--proxy` : (optional) proxy url
`--clearMongo` : (optional) defines if the collections in mongo will be cleared from previous documents or not. if false or is missing collection will remain as it is


Loading

0 comments on commit 10fb722

Please sign in to comment.