Skip to content

Commit

Permalink
fix: helm deploy for initial mongodb script
Browse files Browse the repository at this point in the history
  • Loading branch information
wl4g committed Apr 23, 2023
1 parent 422498e commit 4cfde68
Show file tree
Hide file tree
Showing 32 changed files with 368 additions and 137 deletions.
26 changes: 25 additions & 1 deletion docs/en/deploy-production.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,31 @@

> This article will introduce how to deploy Rengine to the production environment, where Rengine ApiServer, Controller, Executor, Jobs are deployed to the Kubernetes platform, and HBase is built using CDH6(including HBase and Kafka (or Flink) Cluster) refer to: [blogs.wl4g.com/archives/3368](https://blogs.wl4g.com/archives/3368)

## Deploy on Kubernetes

- [Deploy Rengine Stack (ApiServer, Executor, Controller, Job) to Kubernetes with helm.](../../tools/deploy/helm/rengine-stack/README.md)

- Deploy success

```bash
kubectl -n rengine get pods

NAME READY STATUS RESTARTS AGE
rengine-redis-cluster-0 2/2 Running 0 5m19s
rengine-redis-cluster-1 2/2 Running 0 5m19s
rengine-redis-cluster-2 2/2 Running 0 5m19s
rengine-redis-cluster-3 2/2 Running 0 5m19s
rengine-redis-cluster-4 2/2 Running 0 5m19s
rengine-redis-cluster-5 2/2 Running 0 5m19s
rengine-zookeeper-0 1/1 Running 0 5m19s
rengine-zookeeper-1 1/1 Running 0 5m19s
rengine-zookeeper-2 1/1 Running 0 5m19s
rengine-mongodb-74bcdbd7b9-926v9 1/1 Running 0 5m19s
rengine-minio-566b978c86-x76qg 1/1 Running 0 5m19s
rengine-apiserver-baseline-5b6c9789d9-sxsf5 1/1 Running 0 5m19s
rengine-controller-baseline-78c568946-bhst6 1/1 Running 0 5m19s
rengine-executor-baseline-74f46464ff-qbtd9 1/1 Running 0 5m19s
rengine-ui-baseline-65d966dccc-j2sb8 1/1 Running 0 5m19s
rengine-init-minio 1/1 Completed 0 5m19s
rengine-init-update-rootpassword 1/1 Completed 0 5m19s
```
24 changes: 23 additions & 1 deletion docs/en/deploy-standalone.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,29 @@
```bash
git clone https://github.com/wl4g/rengine.git
cd rengine
./run.sh run-standalone -U --prune-all-volumes

$ ./run.sh run-standalone -U --prune-all-volumes

$ ./run.sh run-standalone -S
f0779f017ff1 rengine_controller wl4g/rengine-controller:1.0.0 Up 2 minutes
0e7e01ede490 rengine_ui wl4g/rengine-ui:1.0.0 Up 2 minutes
c62284e09d59 rengine_apiserver wl4g/rengine-apiserver:1.0.0 Up 2 minutes
7dc3d38ca03c rengine_mongodb_express mongo-express:0.54.0 Up 2 minutes
ecccb6bf5bab rengine_executor wl4g/rengine-executor-native:1.0.0 Up 2 minutes
5a85d28db4e0 rengine_mongodb bitnami/mongodb:4.4.6 Up 2 minutes
6bba5cfb1b0c rengine_job_tm_default wl4g/rengine-job:1.0.0 Up 2 minutes
66b5df0e5cd3 rengine_job_jm_default wl4g/rengine-job:1.0.0 Up 2 minutes
935c57e75ea7 rengine_kafka_manager registry.cn-shenzhen.aliyuncs.com/wl4g/kafka-manager:v3.0.0.6-2 Up 2 minutes
74f2e3edaf76 rengine_kafka bitnami/kafka:2.2.0 Up 2 minutes
d5344bed2674 rengine_redis_node_5 bitnami/redis-cluster:7.0 Up 2 minutes
cd48ed90c734 rengine_hbase wl4g/hbase:hbase-2.1.0-phoenix-5.1.1 Up 2 minutes
828cd427c341 rengine_redis_node_3 bitnami/redis-cluster:7.0 Up 2 minutes
9f3fae3d4b15 rengine_zookeeper bitnami/zookeeper:3.6.2 Up 2 minutes
03a0b3f1b15f rengine_redis_node_4 bitnami/redis-cluster:7.0 Up 2 minutes
6341acf3ef3c rengine_minio minio/minio:RELEASE.2022-08-26T19-53-15Z Up 2 minutes
eb0e9192aeb5 rengine_redis_node_0 bitnami/redis-cluster:7.0 Up 2 minutes
f71b4e187d7d rengine_redis_node_2 bitnami/redis-cluster:7.0 Up 2 minutes
16c56797b29c rengine_redis_node_1 bitnami/redis-cluster:7.0 Up 2 minutes
```

## Initial
Expand Down
3 changes: 3 additions & 0 deletions docs/en/devel.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ Usage: ./run.sh [OPTIONS] [arg1] [arg2] ...
--skip-build Skip recompile build before building image.
-u,--ui Build image for UI.
--skip-build Skip recompile build before building image.
-d,--initdb Build image for initdb.
--skip-build Skip recompile build before building image.
-A,--all Build image for all components (but excludes the executor-native).
--skip-build Skip recompile build before building image.
push-image Push component images.
Expand All @@ -58,6 +60,7 @@ Usage: ./run.sh [OPTIONS] [arg1] [arg2] ...
-e,--executor Push image for executor.
-E,--executor-native Push image for executor (native).
-u,--ui Push image for UI.
-d,--initdb Push image for initdb.
-A,--all Push image for all components.
build-push Build with Maven and push images for all components.
prune-image Prune unused all images. (tag=none)
Expand Down
3 changes: 2 additions & 1 deletion executor/src/main/resources/application.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,8 @@ quarkus:
#datasource: {}
## see:https://github.com/quarkusio/quarkus-quickstarts/blob/main/mongodb-quickstart/src/main/resources/application.properties
mongodb:
connection-string: mongodb://localhost:27017
## see:https://www.mongodb.com/community/forums/t/mongodb-authentication-failed-with-spring-data-uri/109256/13
connection-string: mongodb://localhost:27017/rengine?authSource=admin
health:
enabled: true
metrics:
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@
<!-- <java.version>1.8</java.version> -->
<java.version>11</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<infra.version>3.1.22</infra.version>
<infra.version>3.1.26</infra.version>
<spring-native.version>0.12.1</spring-native.version>
<spring-boot.version>2.7.3</spring-boot.version>
<spring-cloud.version>2021.0.3</spring-cloud.version>
Expand Down
2 changes: 1 addition & 1 deletion rengine-ui
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,23 @@ public class RengineServiceProperties {
@NotBlank
URI executorEndpoint = URI.create("http://localhost:28002");

@NotNull
UploadServiceProperties upload = new UploadServiceProperties();

@NotNull
DictServiceProperties dict = new DictServiceProperties();

@NotNull
ControllerLogServiceProperties controllerLog = new ControllerLogServiceProperties();

@Getter
@Setter
@ToString
@NoArgsConstructor
public static class UploadServiceProperties {
String minioEndpoint = "http://localhost:9000";
}

/**
* @see {@link com.wl4g.rengine.executor.execution.ExecutionConfig.ServiceConfig}
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,17 +93,20 @@ public UploadApplyResult apply(@NotNull UploadApply model) {
// New create temporary STS credentials.
try {
final Credentials credentials = minioManager.createSTSCredentials(upload.getObjectPrefix());
final MinioClientProperties config = minioManager.getConfig();
final MinioClientProperties minioConfig = minioManager.getConfig();
return UploadApplyResult.builder()
.endpoint(config.getEndpoint())
.region(minioManager.getConfig().getRegion())
// .bucket(details.getBucket())
// Note: An independent external endpoint should be used to
// avoid the same address as the internal endpoint used by
// apiserver and executor services.
// .endpoint(config.getEndpoint())
.endpoint(config.getUpload().getMinioEndpoint())
.region(minioConfig.getRegion())
.bucket(RengineConstants.DEFAULT_MINIO_BUCKET)
.accessKey(credentials.accessKey())
.secretKey(credentials.secretKey())
.sessionToken(credentials.sessionToken())
.partSize(minioManager.getConfig().getUserUpload().getLibraryPartSize().toBytes())
.fileLimitSize(minioManager.getConfig().getUserUpload().getLibraryFileLimitSize().toBytes())
.partSize(minioConfig.getUserUpload().getLibraryPartSize().toBytes())
.fileLimitSize(minioConfig.getUserUpload().getLibraryFileLimitSize().toBytes())
.objectPrefix(upload.getObjectPrefix())
.extensions(safeList(UploadType.of(upload.getUploadType()).getExtensions()).stream()
.map(t -> t.getSuffix())
Expand Down Expand Up @@ -156,18 +159,21 @@ public UploadSaveResult save(@NotNull UploadSave model) {
// with precise authorized write permissions.
try {
final Credentials credentials = minioManager.createSTSCredentials(upload.getObjectPrefix());
final MinioClientProperties config = minioManager.getConfig();
final MinioClientProperties minioConfig = minioManager.getConfig();
return UploadSaveResult.builder()
.id(upload.getId())
.endpoint(config.getEndpoint())
// Note: An independent external endpoint should be used to
// avoid the same address as the internal endpoint used by
// apiserver and executor services.
// .endpoint(config.getEndpoint())
.endpoint(config.getUpload().getMinioEndpoint())
.region(minioManager.getConfig().getRegion())
// .bucket(details.getBucket())
.bucket(RengineConstants.DEFAULT_MINIO_BUCKET)
.accessKey(credentials.accessKey())
.secretKey(credentials.secretKey())
.sessionToken(credentials.sessionToken())
.partSize(minioManager.getConfig().getUserUpload().getLibraryPartSize().toBytes())
.fileLimitSize(minioManager.getConfig().getUserUpload().getLibraryFileLimitSize().toBytes())
.partSize(minioConfig.getUserUpload().getLibraryPartSize().toBytes())
.fileLimitSize(minioConfig.getUserUpload().getLibraryFileLimitSize().toBytes())
.objectPrefix(upload.getObjectPrefix())
.extensions(safeList(uploadType.getExtensions()).stream().map(t -> t.getSuffix()).collect(toList()))
.build();
Expand Down
1 change: 1 addition & 0 deletions service/src/main/resources/application-mongo.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ spring:
## see:org.springframework.boot.autoconfigure.data.mongo.MongoDatabaseFactoryDependentConfiguration
## see:org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration
## see:org.springframework.data.mongodb.config.AbstractMongoClientConfiguration#mongoTemplate
## see:https://www.mongodb.com/community/forums/t/mongodb-authentication-failed-with-spring-data-uri/109256/13
mongodb:
#uri: ${RENGINE_DB_URI:mongodb+srv://rengine:[email protected]/rengine}
uri: ${RENGINE_DB_URI:mongodb://localhost:27017/rengine}
Expand Down
2 changes: 2 additions & 0 deletions service/src/main/resources/application-services.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
rengine:
services:
executor-endpoint: http://localhost:28002
upload:
minio-endpoint: http://localhost:9000
dict:
dict-cached-prefix: rengine:services:dict:cache
dict-cached-expire: 86400_000
Expand Down
34 changes: 34 additions & 0 deletions tools/build/docker/Dockerfile.initdb
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
## Copyright 2017 ~ 2025 the original authors James Wong<[email protected]>.
##
## Licensed under the Apache License, Version 2.0 (the "License");
## you may not use this file except in compliance with the License.
## You may obtain a copy of the License at
##
## http://www.apache.org/licenses/LICENSE-2.0
##
## Unless required by applicable law or agreed to in writing, software
## distributed under the License is distributed on an "AS IS" BASIS,
## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
## See the License for the specific language governing permissions and
## limitations under the License.
##

FROM busybox:1.28

LABEL maintainer="James Wong<[email protected]>" \
description="Rengine is a cloud native, distributed, generic rule engine platform." \
repoName=${REPO_NAME} \
branch=${BRANCH} \
pr=${PR} \
tag=${TAG} \
commit=${COMMIT_ID} \
buildTime=${BUILD_TIME} \
buildURL=${BUILD_URL}

ARG APP_VERSION
ENV APP_VERSION=${APP_VERSION}

## Note: It should be placed in the initdb dir, because the standard k8s pod volumeMount is the mount directory by default.
COPY tools/db/rengine_init.js /initdb/rengine_init.js

ENTRYPOINT [ "/bin/sh", "-c" ]
Loading

0 comments on commit 4cfde68

Please sign in to comment.