diff --git a/README.md b/README.md index 01a1cbe..0651ce5 100644 --- a/README.md +++ b/README.md @@ -149,11 +149,12 @@ redis-connect\bin> redisconnect.bat start - - - - - + + + + + +
diff --git a/examples/db2/demo/config/jobmanager.properties b/examples/db2/demo/config/jobmanager.properties index fe50554..429fd31 100644 --- a/examples/db2/demo/config/jobmanager.properties +++ b/examples/db2/demo/config/jobmanager.properties @@ -16,7 +16,7 @@ #job.claim.attempt.interval=5000 #job.claim.batch.size.per.attempt=4 #job.claim.max.capacity=4 -#job.claim.heartbeat.lease.renewal.interval=5000 +#job.claim.heartbeat.lease.renewal.interval=10000 ##### REST properties #rest.api.enabled=true diff --git a/examples/mongodb/README.md b/examples/mongodb/README.md new file mode 100644 index 0000000..ff1e85a --- /dev/null +++ b/examples/mongodb/README.md @@ -0,0 +1,12 @@ +## Setting up MongoDB (Source) + +Please see MongoDB Setup for reference. + +## Setting up Redis Enterprise Databases (Target) + +Before using Redis Connect to capture the changes committed on SQL Server into Redis Enterprise Database, first create a database for the metadata management and metrics provided by Redis Connect by creating a database with [RedisTimeSeries](https://redislabs.com/modules/redis-timeseries/) module enabled, see [Create Redis Enterprise Database](https://docs.redislabs.com/latest/rs/administering/creating-databases/#creating-a-new-redis-database) for reference. Then, create (or use an existing) another Redis Enterprise database (Target) to store the changes coming from SQL Server. Additionally, you can enable [RediSearch 2.0](https://redislabs.com/blog/introducing-redisearch-2-0/) module on the target database to enable secondary index with full-text search capabilities on the existing hashes where SQL Server changed events are being written at then [create an index, and start querying](https://oss.redislabs.com/redisearch/Commands/) the document in hashes. + +| ℹ️ | +|:--------------------------------------------| +| Docker demo: Follow the [Docker demo](demo) | +| K8s Setup: Follow the [k8s-docs](k8s-docs) | \ No newline at end of file diff --git a/examples/mongodb/demo/README.md b/examples/mongodb/demo/README.md new file mode 100644 index 0000000..0847a98 --- /dev/null +++ b/examples/mongodb/demo/README.md @@ -0,0 +1,179 @@ +# Demo Outline +:white_check_mark: Setup MongoDB database (Source)
+:white_check_mark: Setup and start Redis Enterprise database (Target)
+:white_check_mark: Setup and start Redis Connect
+:white_check_mark: Perform Initial load and CDC with Redis Connect
+ +# Prerequisites +Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed. +
Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect Connector instances deployed on separate hosts.
+
Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts. +i.e.
+ +```bash +wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \ +mkdir -p redis-connect/demo && \ +mkdir -p redis-connect/k8s-docs && \ +unzip main.zip "redis-connect-dist-main/examples/mongodb/*" -d redis-connect && \ +cp -R redis-connect/redis-connect-dist-main/examples/mongodb/demo/* redis-connect/demo && \ +cp -R redis-connect/redis-connect-dist-main/examples/mongodb/k8s-docs/* redis-connect/k8s-docs && \ +rm -rf main.zip redis-connect/redis-connect-dist-main && \ +cd redis-connect && \ +chmod a+x demo/*.sh && \ +cd demo +``` + +## Setup MongoDB database (Source) + +https://www.mongodb.com/atlas/database + +## Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target) +
Execute [setup_re.sh](setup_re.sh)
+```bash +demo$ ./setup_re.sh +``` +**NOTE** + +The above script will create a 1-node Redis Enterprise cluster in a docker container, [Create a target database with RediSearch module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a job management and metrics database with RedisTimeSeries module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a RediSearch index for emp Hash](https://redislabs.com/blog/getting-started-with-redisearch-2-0/), [Start a docker instance of grafana with Redis Data Source](https://redisgrafana.github.io/) and [Start an instance of RedisInsight](https://docs.redislabs.com/latest/ri/installing/install-docker/). + +## Start Redis Connect + +
Review options by running Redis Connect docker container +

+ +```bash +demo$ docker run \ +-it --rm --privileged=true \ +--name redis-connect-$(hostname) \ +-v $(pwd)/config:/opt/redislabs/redis-connect/config \ +-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \ +--net host \ +redislabs/redis-connect +``` + +

+
+ +
Expected output: +

+ +```bash +------------------------------- +Redis Connect startup script. +******************************* +Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf +Example environment variables and volume mapping for docker based deployments +-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL] +-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL] +-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL] +-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL] +-v :/opt/redislabs/redis-connect/config +-v :/opt/redislabs/redis-connect/config/samples/credentials +-v :/opt/redislabs/redis-connect/extlib [OPTIONAL] +-p 8282:8282 + +Usage: [-h|cli|start] +options: +-h: Print this help message and exit. +-v: Print version. +cli: init Redis Connect CLI +start: init Redis Connect Instance (Cluster Member) +------------------------------- +``` + +

+
+ +
Start Redis Connect Instance +

+ +```bash +docker run \ +-it --rm --privileged=true \ +--name redis-connect-$(hostname) \ +-v $(pwd)/config:/opt/redislabs/redis-connect/config \ +-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \ +--net host \ +redislabs/redis-connect start +``` + +

+
+ +
Expected output: +

+ +```bash + +``` + +

+
+ +**Open browser to access Swagger UI -** [http://localhost:8282/swagger-ui/index.html]() +
_For quick start, use '**cdc_job**' as **jobName**_ +

+ +**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}` +
_For quick start, use the sample `cdc-job.json` configuration:_ SQL Server +

+
+ +**Or Use `curl` to create the `cdc-job` configuration**
+`demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"` + +------------------------------- + +### Initial Loader Steps + +**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}` +
Use '**load**' as _**jobType**_ +

+ +**Or Use `curl` to start the initial load for `cdc-job`**
+`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/load" -H "accept: */*"` + +
Query for the above inserted record in Redis (target) +

+ +```bash +demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 ft.search idx_emp "@empno:[1 11]"' + +``` + +

+
+ +------------------------------- + +### CDC Steps + +**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}` +
Use '**stream**' as _**jobType**_ +

+ +**Or Use `curl` to start the stream for `cdc-job`**
+`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/stream" -H "accept: */*"` + +**Confirm Job Claim -** `/connect/api/vi/jobs/claim/{jobStatus}` +
_For quick start, use '**all**' as **jobStatus**_ +

+ +**Or Use `curl` to query the `cdc-job` status**
+`demo$ curl -X GET "http://localhost:8282/connect/api/v1/cluster/jobs/claim/all" -H "accept: */*"` + +Expected output: `[{"jobId":"{connect}:job:cdc-job","jobName":"cdc-job","jobStatus":"CLAIMED","jobOwner":"30@virag-cdc","jobType":"STREAM"}]` + +INSERT/UPDATE a document in MongoDB (source) +

+ +

Query for the above inserted record in Redis (target) +

+ +```bash +demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 ft.search idx_emp "@empno:[1002 1002]"' + +``` + +

+
diff --git a/examples/mongodb/demo/config/jobmanager.properties b/examples/mongodb/demo/config/jobmanager.properties new file mode 100644 index 0000000..429fd31 --- /dev/null +++ b/examples/mongodb/demo/config/jobmanager.properties @@ -0,0 +1,38 @@ +################################################################################################## +##### Properties to start a Redis Connect cluster. Default values are commented out. ##### +################################################################################################## + +##### Cluster properties +#cluster.name=default +#cluster.leader.heartbeat.lease.renewal.interval=5000 +#cluster.election.attempt.interval=5000 +#cluster.timeseries.metrics.enabled=false + +##### Job Manager Services properties +#job.manager.services.enabled=true +#job.manager.services.threadpool.size=2 +#job.reap.attempt.interval=7000 +#job.claim.service.enabled=true +#job.claim.attempt.interval=5000 +#job.claim.batch.size.per.attempt=4 +#job.claim.max.capacity=4 +#job.claim.heartbeat.lease.renewal.interval=10000 + +##### REST properties +#rest.api.enabled=true +#rest.api.port=8282 + +##### Job Manager Database properties +redis.connection.url=redis://127.0.0.1:14001 +#redis.connection.sslEnabled=false +#redis.truststore= +#redis.keystore= +#redis.connection.insecure=false +#redis.connection.timeout.duration=5 +#redis.connection.auto.reconnect=true +#redis.connection.suspend.reconnect.on.protocol.failure=true + +##### Credentials properties +credentials.file.path=/opt/redislabs/redis-connect/config/samples/credentials +#credentials.rotation.event.listener.enabled=false +#credentials.rotation.event.listener.interval=60000 \ No newline at end of file diff --git a/examples/mongodb/demo/config/logback-cli.xml b/examples/mongodb/demo/config/logback-cli.xml new file mode 100644 index 0000000..7692971 --- /dev/null +++ b/examples/mongodb/demo/config/logback-cli.xml @@ -0,0 +1,113 @@ + + + + + + + + ${LOG_REDIS_CONNECT_HEARTBEAT_PATH} + + logs/archived/redis-connect-heartbeat.%d{yyyy-MM-dd}.%i.log.gz + + 10MB + + 20GB + + 60 + + + %d %p %c{1.} [%t] %m%n + + + + ${LOG_REDIS_CONNECT_MANAGER_PATH} + + logs/archived/redis-connect-manager.%d{yyyy-MM-dd}.%i.log.gz + + 10MB + + 20GB + + 60 + + + %d %p %c{1.} [%t] %m%n + + + + ${LOG_REDIS_CONNECT_PATH}-${PID}.log + + logs/archived/redis-connect-${PID}.%d{yyyy-MM-dd}.%i.log.gz + + 10MB + + 20GB + + 60 + + + %d %p %c{1.} [%t] %m%n + + + + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/examples/mongodb/demo/config/logback.xml b/examples/mongodb/demo/config/logback.xml new file mode 100644 index 0000000..fa3a5b9 --- /dev/null +++ b/examples/mongodb/demo/config/logback.xml @@ -0,0 +1,113 @@ + + + + + + + + ${LOG_REDIS_CONNECT_HEARTBEAT_PATH} + + logs/archived/redis-connect-heartbeat.%d{yyyy-MM-dd}.%i.log.gz + + 10MB + + 20GB + + 60 + + + %d %p %c{1.} [%t] %m%n + + + + ${LOG_REDIS_CONNECT_MANAGER_PATH} + + logs/archived/redis-connect-manager.%d{yyyy-MM-dd}.%i.log.gz + + 10MB + + 20GB + + 60 + + + %d %p %c{1.} [%t] %m%n + + + + ${LOG_REDIS_CONNECT_PATH}-${PID}.log + + logs/archived/redis-connect-${PID}.%d{yyyy-MM-dd}.%i.log.gz + + 10MB + + 20GB + + 60 + + + %d %p %c{1.} [%t] %m%n + + + + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_jobmanager.properties b/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_jobmanager.properties new file mode 100644 index 0000000..03d1626 --- /dev/null +++ b/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_jobmanager.properties @@ -0,0 +1,4 @@ +jobmanager.username= +jobmanager.password= +redis.truststore.password= +redis.keystore.password= diff --git a/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_mongodb_cdc-job.properties b/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_mongodb_cdc-job.properties new file mode 100644 index 0000000..7333281 --- /dev/null +++ b/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_mongodb_cdc-job.properties @@ -0,0 +1,2 @@ +source.username=redisconnect +source.password=Redis123 \ No newline at end of file diff --git a/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_redis_cdc-job.properties b/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_redis_cdc-job.properties new file mode 100644 index 0000000..2ad529f --- /dev/null +++ b/examples/mongodb/demo/config/samples/credentials/redisconnect_credentials_redis_cdc-job.properties @@ -0,0 +1,4 @@ +target.username= +target.password= +redis.truststore.password= +redis.keystore.password= diff --git a/examples/mongodb/demo/config/samples/payloads/cdc-job.json b/examples/mongodb/demo/config/samples/payloads/cdc-job.json new file mode 100644 index 0000000..8a43f00 --- /dev/null +++ b/examples/mongodb/demo/config/samples/payloads/cdc-job.json @@ -0,0 +1,47 @@ +{ + "partitions" : 1, + "pipeline": { + "pipelineBufferSize" : 1024, + "stages" : [ + { + "database": { + "credentialsFilePath" : "/opt/redislabs/redis-connect/config/samples/credentials", + "databaseURL": "redis://127.0.0.1:14000", + "databaseType": "REDIS" + }, + "stageName": "REDIS_JSON_SINK", + "index" : 1, + "checkpointStageIndicator": true, + "checkpointTransactionalityEnabled": true, + "metricsEnabled" : true, + "metricsRetentionInHours" : 1 + } + ] + }, + "source": { + "database": { + "credentialsFilePath" : "/opt/redislabs/redis-connect/config/samples/credentials", + "databaseURL": "mongodb+srv://", + "databaseType": "MONGODB", + "customConfiguration": { + "mongodb.hosts" : "", + "database.include.list" : "sample_airbnb", + "collection.include.list" : "sample_airbnb.testSnapshot", + "mongodb.ssl.enabled" : true, + "mongodb.members.auto.discover" : true + } + }, + "tables": { + "sample_airbnb.testSnapshot": { + "columns": [ + { "targetColumn": "primary_key", "sourceColumn": "_id", "primaryKey": true } + ], + "initialLoad": { + "partitions": 4 + }, + "schemaAndTableName": "sample_airbnb.testSnapshot", + "passThroughEnabled" : true + } + } + } +} \ No newline at end of file diff --git a/examples/mongodb/demo/restore_network.sh b/examples/mongodb/demo/restore_network.sh new file mode 100755 index 0000000..3be5948 --- /dev/null +++ b/examples/mongodb/demo/restore_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +sudo docker network connect network2 re-node1-cluster1 +sudo docker network connect network3 re-node1-cluster1 +sudo docker network connect network1 re-node1-cluster2 +sudo docker network connect network3 re-node1-cluster2 +sudo docker network connect network1 re-node1-cluster3 +sudo docker network connect network2 re-node1-cluster3 diff --git a/examples/mongodb/demo/setup_re.sh b/examples/mongodb/demo/setup_re.sh new file mode 100755 index 0000000..d93bb15 --- /dev/null +++ b/examples/mongodb/demo/setup_re.sh @@ -0,0 +1,91 @@ +#!/bin/bash +sudo docker kill re-node1;sudo docker rm re-node1; +sudo docker kill redisinsight;sudo docker rm redisinsight; +sudo docker kill grafana; sudo docker rm grafana; +# shellcheck disable=SC2046 +sudo docker rmi -f $(sudo docker images | grep redislabs | awk '{print $3}') +# Start 1 docker container since we can't do HA with vanilla docker instance. Use docker swarm, RE on VM's or RE's K8s operator to achieve HA, clustering etc. +echo "Starting Redis Enterprise as Docker containers..." +sudo docker run -d --cap-add sys_resource -h re-node1 --name re-node1 -p 18443:8443 -p 19443:9443 -p 14000-14005:12000-12005 -p 18070:8070 redislabs/redis:latest +# Create Redis Enterprise cluster +echo "Waiting for the servers to start..." +sleep 60 +echo "Creating Redis Enterprise cluster..." +sudo docker exec -it --privileged re-node1 "/opt/redislabs/bin/rladmin" cluster create name re-cluster.local username demo@redis.com password redislabs +echo "" +# Test the cluster +sudo docker exec -it re-node1 bash -c "/opt/redislabs/bin/rladmin info cluster" + +# Get the module info to be used for database creation +tee -a list_modules.sh </dev/null +sudo docker network rm network2 2>/dev/null +sudo docker network rm network3 2>/dev/null + +# shellcheck disable=SC2046 +sudo docker rmi -f $(sudo docker images | grep redislabs | awk '{print $3}') + +# Create new bridge networks +echo "Creating new subnets..." +sudo docker network create network1 --subnet=172.18.0.0/16 --gateway=172.18.0.1 +sudo docker network create network2 --subnet=172.19.0.0/16 --gateway=172.19.0.1 +sudo docker network create network3 --subnet=172.20.0.0/16 --gateway=172.20.0.1 + +# Add entries to /etc/hosts so A-A can work without DNS setup +cluster1=$(grep cluster1.local /etc/hosts | cut -d ' ' -f 2) +if [ ! -z "$cluster1" ] +then + echo "cluster1.local entry exists in /etc/hosts. Skipping.." +else + echo "Adding cluster1.local entry to /etc/hosts.." + echo "172.18.0.2 cluster1.local" | sudo tee -a /etc/hosts +fi +cluster2=$(grep cluster2.local /etc/hosts | cut -d ' ' -f 2) +if [ ! -z "$cluster2" ] +then + echo "cluster2.local entry exists in /etc/hosts. Skipping.." +else + echo "Adding cluster2.local entry to /etc/hosts.." + echo "172.19.0.2 cluster2.local" | sudo tee -a /etc/hosts +fi +cluster3=$(grep cluster3.local /etc/hosts | cut -d ' ' -f 2) +if [ ! -z "$cluster3" ] +then + echo "cluster3.local entry exists in /etc/hosts. Skipping.." +else + echo "Adding cluster3.local entry to /etc/hosts.." + echo "172.20.0.2 cluster3.local" | sudo tee -a /etc/hosts +fi + +# Start 3 sudo docker containers. Each container is a node in a separate network +echo "Starting Redis Enterprise as Docker containers..." +sudo docker run -d --cap-add sys_resource -h re-node1-cluster1 --name re-node1-cluster1 -p 8443:8443 -p 9443:9443 -p 14000:12000 -p 14001:12001 -p 8071:8070 --network=network1 --ip=172.18.0.2 redislabs/redis:latest +sudo docker run -d --cap-add sys_resource -h re-node1-cluster2 --name re-node1-cluster2 -p 8445:8443 -p 9445:9443 -p 14002:12000 -p 8072:8070 --network=network2 --ip=172.19.0.2 redislabs/redis:latest +sudo docker run -d --cap-add sys_resource -h re-node1-cluster3 --name re-node1-cluster3 -p 8447:8443 -p 9447:9443 -p 14004:12000 -p 8073:8070 --network=network3 --ip=172.20.0.2 redislabs/redis:latest + +# Connect the networks +sudo docker network connect network2 re-node1-cluster1 +sudo docker network connect network3 re-node1-cluster1 +sudo docker network connect network1 re-node1-cluster2 +sudo docker network connect network3 re-node1-cluster2 +sudo docker network connect network1 re-node1-cluster3 +sudo docker network connect network2 re-node1-cluster3 + +# Create 3 Redis Enterprise clusters - one for each network +echo "Waiting for the servers to start..." + +sleep 60 + +echo "Creating clusters" +sudo docker exec -it re-node1-cluster1 /opt/redislabs/bin/rladmin cluster create name cluster1.local username demo@redis.com password redislabs +sudo docker exec -it re-node1-cluster2 /opt/redislabs/bin/rladmin cluster create name cluster2.local username demo@redis.com password redislabs +sudo docker exec -it re-node1-cluster3 /opt/redislabs/bin/rladmin cluster create name cluster3.local username demo@redis.com password redislabs + +# Test the cluster +sudo docker exec -it re-node1-cluster1 bash -c "/opt/redislabs/bin/rladmin info cluster" + +# Get the module info to be used for database creation +tee -a list_modules.sh <.*)\\\\}.*"}, {"regex": "(?.*)"}], "module_list": [ {"module_args": "PARTITIONS AUTO", "module_name": "$search_module_name", "semantic_version": "$search_semantic_version"} ] }, "instances": [{"cluster": {"url": "https://cluster1.local:9443","credentials": {"username": "demo@redis.com", "password": "redislabs"}, "name": "cluster1.local"}, "compression": 6}, {"cluster": {"url": "https://cluster2.local:9443", "credentials": {"username": "demo@redis.com", "password": "redislabs"}, "name": "cluster2.local"}, "compression": 6}, {"cluster": {"url": "https://cluster3.local:9443", "credentials": {"username": "demo@redis.com", "password": "redislabs"}, "name": "cluster3.local"}, "compression": 6}], "name": "Target" }' https://localhost:9443/v1/crdbs +curl -v -k -L -u demo@redis.com:redislabs --location-trusted -H "Content-type:application/json" -d '{"name": "JobManager", "type":"redis", "replication": false, "memory_size": 1024000000, "port": 12001, "module_list": [{"module_args": "", "module_name": "$timeseries_module_name", "semantic_version": "$timeseries_semantic_version"} ] }' https://localhost:9443/v1/bdbs +EOF + +sleep 20 + +sed -i "s/ //g" create_demodb.sh +sudo docker cp create_demodb.sh re-node1-cluster1:/opt/create_demodb.sh +sudo docker exec --user root -it re-node1-cluster1 bash -c "chmod 777 /opt/create_demodb.sh" +sudo docker exec -it re-node1-cluster1 bash -c "/opt/create_demodb.sh" +echo "" + +echo "Database port mappings per node. We are using mDNS so use the IP and exposed port to connect to the databases." +echo "node1:" +sudo docker port re-node1-cluster1 | grep -E "12000|12001" +sudo docker port re-node1-cluster2 | grep -E "12000|12001" +sudo docker port re-node1-cluster3 | grep -E "12000|12001" +echo "------- RLADMIN status -------" +sleep 60 +sudo docker exec -it re-node1-cluster1 bash -c "rladmin status" +echo "" +sudo docker exec -it re-node1-cluster1 bash -c "crdb-cli coordinate crdb-list" +echo "" +echo "You can open a browser and access Redis Enterprise Admin UI at https://127.0.0.1:8443 (replace localhost with your ip/host) with username=demo@redis.com and password=redislabs." +echo "To connect using RedisInsight or redis-cli, please use the exposed port from the node where master shard for the database resides." +echo "Creating RedisInsight in docker container.." +sudo docker run -d --name redisinsight -p 18001:8001 -v redisinsight:/db redislabs/redisinsight:latest +echo "Creating Grafana with redis-datasource in docker container.." +sudo docker run -d -p 3000:3000 --name=grafana -e "GF_INSTALL_PLUGINS=redis-datasource" grafana/grafana +echo "" +echo "Creating idx_emp index for search.." +sleep 60 +sudo docker exec -it re-node1-cluster1 bash -c "/opt/redislabs/bin/redis-cli -p 12000 ft.create idx_emp on hash prefix 1 'emp:' schema empno numeric sortable fname text sortable lname text job tag sortable mgr numeric hiredate text sal numeric comm numeric dept numeric" +sudo docker exec -it re-node1-cluster2 bash -c "/opt/redislabs/bin/redis-cli -p 12000 ft.create idx_emp on hash prefix 1 'emp:' schema empno numeric sortable fname text sortable lname text job tag sortable mgr numeric hiredate text sal numeric comm numeric dept numeric" +sudo docker exec -it re-node1-cluster3 bash -c "/opt/redislabs/bin/redis-cli -p 12000 ft.create idx_emp on hash prefix 1 'emp:' schema empno numeric sortable fname text sortable lname text job tag sortable mgr numeric hiredate text sal numeric comm numeric dept numeric" +echo "You can open a browser and access RedisInsight client UI at http://127.0.0.1:18001 (replace localhost with your ip/host) and add databases to monitor." +echo "Please visit, https://docs.redis.com/latest/ri/using-redisinsight/add-instance/ for steps to add these databases to RedisInsight." +echo "DISCLAIMER: This is best for local development or functional testing. Please see, https://docs.redis.com/latest/rs/getting-started/getting-started-docker" + +# Cleanup +rm list_modules.sh +sudo docker exec --user root -it re-node1-cluster1 bash -c "rm /opt/list_modules.sh" +sudo docker exec --user root -it re-node1-cluster1 bash -c "rm /opt/module_list.txt" +rm create_demodb.sh +sudo docker exec --user root -it re-node1-cluster1 bash -c "rm /opt/create_demodb.sh" + diff --git a/examples/mongodb/demo/split_network.sh b/examples/mongodb/demo/split_network.sh new file mode 100755 index 0000000..7e20340 --- /dev/null +++ b/examples/mongodb/demo/split_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +sudo docker network disconnect network2 re-node1-cluster1 +sudo docker network disconnect network3 re-node1-cluster1 +sudo docker network disconnect network1 re-node1-cluster2 +sudo docker network disconnect network3 re-node1-cluster2 +sudo docker network disconnect network1 re-node1-cluster3 +sudo docker network disconnect network2 re-node1-cluster3 diff --git a/examples/mongodb/demo/stop_re_crdb.sh b/examples/mongodb/demo/stop_re_crdb.sh new file mode 100755 index 0000000..74132db --- /dev/null +++ b/examples/mongodb/demo/stop_re_crdb.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +sudo docker stop re-node1-cluster1 re-node1-cluster2 re-node1-cluster3 redisinsight grafana +sudo docker rm re-node1-cluster1 re-node1-cluster2 re-node1-cluster3 redisinsight grafana +sudo docker network rm network1 2>/dev/null +sudo docker network rm network2 2>/dev/null +sudo docker network rm network3 2>/dev/null diff --git a/examples/mongodb/k8s-docs/README.md b/examples/mongodb/k8s-docs/README.md new file mode 100644 index 0000000..219f8a5 --- /dev/null +++ b/examples/mongodb/k8s-docs/README.md @@ -0,0 +1 @@ +# Redis Connect in K8s diff --git a/examples/mssql/demo/config/jobmanager.properties b/examples/mssql/demo/config/jobmanager.properties index fe50554..429fd31 100644 --- a/examples/mssql/demo/config/jobmanager.properties +++ b/examples/mssql/demo/config/jobmanager.properties @@ -16,7 +16,7 @@ #job.claim.attempt.interval=5000 #job.claim.batch.size.per.attempt=4 #job.claim.max.capacity=4 -#job.claim.heartbeat.lease.renewal.interval=5000 +#job.claim.heartbeat.lease.renewal.interval=10000 ##### REST properties #rest.api.enabled=true diff --git a/examples/mysql/demo/config/jobmanager.properties b/examples/mysql/demo/config/jobmanager.properties index fe50554..429fd31 100644 --- a/examples/mysql/demo/config/jobmanager.properties +++ b/examples/mysql/demo/config/jobmanager.properties @@ -16,7 +16,7 @@ #job.claim.attempt.interval=5000 #job.claim.batch.size.per.attempt=4 #job.claim.max.capacity=4 -#job.claim.heartbeat.lease.renewal.interval=5000 +#job.claim.heartbeat.lease.renewal.interval=10000 ##### REST properties #rest.api.enabled=true diff --git a/examples/oracle/demo/config/jobmanager.properties b/examples/oracle/demo/config/jobmanager.properties index fe50554..429fd31 100644 --- a/examples/oracle/demo/config/jobmanager.properties +++ b/examples/oracle/demo/config/jobmanager.properties @@ -16,7 +16,7 @@ #job.claim.attempt.interval=5000 #job.claim.batch.size.per.attempt=4 #job.claim.max.capacity=4 -#job.claim.heartbeat.lease.renewal.interval=5000 +#job.claim.heartbeat.lease.renewal.interval=10000 ##### REST properties #rest.api.enabled=true diff --git a/examples/postgres/demo/config/jobmanager.properties b/examples/postgres/demo/config/jobmanager.properties index fe50554..429fd31 100644 --- a/examples/postgres/demo/config/jobmanager.properties +++ b/examples/postgres/demo/config/jobmanager.properties @@ -16,7 +16,7 @@ #job.claim.attempt.interval=5000 #job.claim.batch.size.per.attempt=4 #job.claim.max.capacity=4 -#job.claim.heartbeat.lease.renewal.interval=5000 +#job.claim.heartbeat.lease.renewal.interval=10000 ##### REST properties #rest.api.enabled=true diff --git a/examples/vertica/demo/config/jobmanager.properties b/examples/vertica/demo/config/jobmanager.properties index fe50554..429fd31 100644 --- a/examples/vertica/demo/config/jobmanager.properties +++ b/examples/vertica/demo/config/jobmanager.properties @@ -16,7 +16,7 @@ #job.claim.attempt.interval=5000 #job.claim.batch.size.per.attempt=4 #job.claim.max.capacity=4 -#job.claim.heartbeat.lease.renewal.interval=5000 +#job.claim.heartbeat.lease.renewal.interval=10000 ##### REST properties #rest.api.enabled=true diff --git a/images/icons/MongoDB Icon.png b/images/icons/MongoDB Icon.png new file mode 100644 index 0000000..0e2cdc7 Binary files /dev/null and b/images/icons/MongoDB Icon.png differ