-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(helm): add live logs deployment (#827) #827
Open
jlemesh
wants to merge
4
commits into
reanahub:master
Choose a base branch
from
jlemesh:feature_add_opensearch_deployment
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat(helm): add live logs deployment (#827) #827
jlemesh
wants to merge
4
commits into
reanahub:master
from
jlemesh:feature_add_opensearch_deployment
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 27, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 27, 2024 06:36
17afbaa
to
c56a418
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #827 +/- ##
==========================================
+ Coverage 29.80% 31.08% +1.27%
==========================================
Files 26 26
Lines 2486 2487 +1
==========================================
+ Hits 741 773 +32
+ Misses 1745 1714 -31
|
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 27, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 27, 2024 06:50
c56a418
to
ba8d585
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 28, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 28, 2024 09:34
ba8d585
to
ea1fb45
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 28, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 29, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 29, 2024 05:56
a526d7d
to
c8239fb
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 29, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 30, 2024 07:23
c8239fb
to
5746bf0
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 30, 2024 07:27
5746bf0
to
38da2a1
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 30, 2024 08:53
38da2a1
to
657e739
Compare
jlemesh
changed the title
feat(helm): add opensearch deployment (#827)
feat(helm): add live logs deployment (#827)
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Aug 30, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
August 30, 2024 14:22
657e739
to
2be0faf
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 9, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 9, 2024 09:12
4faf1f0
to
a7b6a49
Compare
tiborsimko
reviewed
Oct 10, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 14, 2024 08:15
bb901b3
to
3f72a94
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 14, 2024 08:28
3f72a94
to
62dd874
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 14, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 14, 2024 09:14
62dd874
to
372f8cf
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 15, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 15, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 15, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 15, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 15, 2024 08:12
372f8cf
to
52766e0
Compare
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 15, 2024
jlemesh
added a commit
to jlemesh/reana
that referenced
this pull request
Oct 15, 2024
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 15, 2024 12:48
52766e0
to
47e35de
Compare
jlemesh
force-pushed
the
feature_add_opensearch_deployment
branch
from
October 17, 2024 11:02
47e35de
to
7377233
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #824
OpenSearch
Optionally deploy OpenSearch alongside REANA (
enabled: false
by default).In dev environment, deploy one instance with security features off (i. e. no TLS, no authn/authz)
In prod, also only one OpenSearch node is configured. To add more nodes, each of these nodes needs to have its own
PersistentVolumeClaim
andPersistentVolume
as each node writes to a directory with the same name, but different contents - this will not work with currentreana-shared-persistent-volume
orreana-infrastructure-persistent-volume
. It is possible to usevolumeClaimTemplates
with theStorageClass
of choice (configured in OpenSearch Helm chartpersistence.storageClass
).Volumes configuration tested with nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner, for one and multiple nodes.
OpenSearch allows deploying nodes of different types, e. g.
master
anddata
. This configuration would not allow to do it, we need to add the secondopensearch
dependency, e. g.opensearch-master
, and use the same Helm chart with different configuration. I think it is an overkill at this stage, as this is needed for bigger clusters.TLS certificates for OpenSearch nodes are generated automatically by a Helm function, similar to how it is done for Ingress, but without automatic rotation on each
helm upgrade
. It is possible for a user to supply his own certificates, by putting them in a secret and mounting to an OpenSearch pod (inopensearch.secretMounts
). Admin TLS certificates provide superadmin permissions to whoever uses them, and are stored in the same secret - there is no specific admin user.Two users are configured -
reana
andfluentbit
. Their passwords need to be prepared by first spinning up environment with Opensearch and runninghash.sh
script (described in docs):After getting the hashes for the passwords, they should be supplied to Helm with
--set opensearch.customSecurityConfig.internalUsers.reana.hash='$So$Me$pASsWOrD.HasH' --set opensearch.customSecurityConfig.internalUsers.fluentbit.hash='$So$Me$pASsWOrD.HasH'
.Also enabled SSL cert reload, which allows updating SSL certs without restarting OpenSearch instance. Link to docs provided in the comments for those who will want to use it.
FluentBit
Optionally deploy FluentBit alongside REANA (
enabled: false
by default). Collects job and workflow logs withtail
input plugin and pushes logs to OpenSearch. Allows configuring custom TLS certificates for OpenSearch connection.reana-dev
The command breaks Helm configuration if it contains long (multiline) strings. PyYAML library first reads the YAML config file, converts multiline strings to one long string delimited with
"
and then wraps it whiledump
ing, i. e.:If the string is used with Helm template values (
tpl
function), it adds whitespaces in places where\
was used, which results in malformed configuration for OpenSearch and FluentBit.dump(width=100000)
will add\
only if the string is longer than 100000 characters.It also requires user to escape certain characters in YAML values like
$
and\
. PyYAML converts|
strings to""
and Helm command is supplied with bad values.To avoid errors when loading/dumping
values.yaml
file, skip its loading altogether as there is no need to load it explicitly as it is always loaded by Helm itself. In casereleasehelm
mode is used forreana-dev cluster-deploy
, just use an empty dict for values and add overrides into it,values.yaml
will be loaded by default by Helm.How to test
Checkout reanahub/reana-workflow-controller#602 and reanahub/reana-job-controller#468 to retrieve live logs in REANA components. Checkout and install reanahub/reana-client#731 to watch live logs via CLI.
Dev setup
Deploy REANA as usual and run:
Prod setup
Deploy REANA as usual then run: