Upgrade to 3.7.2 and 7.7.2
Upgrade to 3.7.2 and 7.7.2
1.Prerequisites
Before proceeding with this upgrade, please make sure and verify the below prerequisites are met.
-
RDAF Deployment CLI version: 1.3.2
-
Infra Services tag: 1.0.3 / 1.0.3.3 (HA Proxy)
-
Platform Services and RDA Worker tag: 3.7.x
-
OIA Application Services tag: 7.7.x
-
CloudFabrix recommends taking VMware VM snapshots where RDA Fabric infra/platform/applications are deployed
Note
- Check the Disk space of all the Platform and Service Vm's using the below mentioned command, the highlighted disk size should be less than 80%
rdauser@oia-125-216:~/collab-3.5-upgrade$ df -kh Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 357M 6.0G 6% /run /dev/mapper/ubuntu--vg-ubuntu--lv 48G 12G 34G 26% / tmpfs 32G 0 32G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/loop0 64M 64M 0 100% /snap/core20/2318 /dev/loop2 92M 92M 0 100% /snap/lxd/24061 /dev/sda2 1.5G 309M 1.1G 23% /boot /dev/sdf 50G 3.8G 47G 8% /var/mysql /dev/loop3 39M 39M 0 100% /snap/snapd/21759 /dev/sdg 50G 541M 50G 2% /minio-data /dev/loop4 92M 92M 0 100% /snap/lxd/29619 /dev/loop5 39M 39M 0 100% /snap/snapd/21465 /dev/sde 15G 140M 15G 1% /zookeeper /dev/sdd 30G 884M 30G 3% /kafka-logs /dev/sdc 50G 3.3G 47G 7% /opt /dev/sdb 50G 29G 22G 57% /var/lib/docker /dev/sdi 25G 294M 25G 2% /graphdb /dev/sdh 50G 34G 17G 68% /opensearch /dev/loop6 64M 64M 0 100% /snap/core20/2379- Check all MariaDB nodes are sync on HA setup using below commands before start upgrade
Tip
Please run the below commands on the VM host where RDAF deployment CLI was installed and
rdafk8s setupcommand was run. Themariadbconfiguration is read from/opt/rdaf/rdaf.cfgfile.MARIADB_HOST=`cat /opt/rdaf/rdaf.cfg | grep -A3 mariadb | grep datadir | awk '{print $3}' | cut -f1 -d'/'` MARIADB_USER=`cat /opt/rdaf/rdaf.cfg | grep -A3 mariadb | grep user | awk '{print $3}' | base64 -d` MARIADB_PASSWORD=`cat /opt/rdaf/rdaf.cfg | grep -A3 mariadb | grep password | awk '{print $3}' | base64 -d` mysql -u$MARIADB_USER -p$MARIADB_PASSWORD -h $MARIADB_HOST -P3307 -e "show status like 'wsrep_local_state_comment';"Please verify that the
mariadbcluster state is in Synced state.+---------------------------+--------+ | Variable_name | Value | +---------------------------+--------+ | wsrep_local_state_comment | Synced | +---------------------------+--------+Please run the below command and verify that the
mariadbcluster size is 3.mysql -u$MARIADB_USER -p$MARIADB_PASSWORD -h $MARIADB_HOST -P3307 -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'";
Warning
Make sure all of the above pre-requisites are met before proceeding with the upgrade process.
Warning
Though RDA Fabric deployment supports zero downtime upgrade, it is recommended to schedule a maintenance window for upgrading RDAF Platform and AIOps services to newer version.
Important
Please make sure full backup of the RDAF platform system is completed before performing the upgrade.
Kubernetes: Please run the below backup command to take the backup of application data.
Run the below command on RDAF Management system and make sure the Kubernetes PODs are NOT in restarting mode (it is applicable to only Kubernetes environment)
- Verify that RDAF deployment
rdafcli version is 1.3.2 on the VM where CLI was installed for docker on-prem registry managing Kubernetes or Non-kubernetes deployments.
- On-premise docker registry service version is 1.0.3
ff6b1de8515f cfxregistry.CloudFabrix.io:443/docker-registry:1.0.3 "/entrypoint.sh /bin…" 7 days ago Up 7 days deployment-scripts-docker-registry-1
- RDAF Infra services status
Run the below command to get RDAF Infra service status & version details. Please make sure all services are in running state.
- RDAF Platform services status
Run the below command to get RDAF Platform services status & version details. Please make sure all services are in running state.
- RDAF Application services status
Run the below command to get RDAF App services status & version details. Please make sure all services are in running state.
Warning
Make sure all of the above pre-requisites are met before proceeding with the upgrade process.
Warning
Though RDA Fabric deployment supports zero downtime upgrade, it is recommended to schedule a maintenance window for upgrading RDAF Platform and AIOps services to newer version.
Important
Please make sure full backup of the RDAF platform system is completed before performing the upgrade.
Note: Please make sure this backup-dir is mounted across all infra,cli vms.- Verify that RDAF deployment
rdafcli version is 1.3.2 on the VM where CLI was installed for docker on-prem registry managing Kubernetes or Non-kubernetes deployments.
- On-premise docker registry service version is 1.0.3
ff6b1de8515f cfxregistry.CloudFabrix.io:443/docker-registry:1.0.3 "/entrypoint.sh /bin…" 7 days ago Up 7 days deployment-scripts-docker-registry-1
- RDAF Infra services status
Run the below command to get RDAF Infra service status & version details. Please make sure all services are in running state.
- RDAF Platform services status
Run the below command to get RDAF Platform services status & version details. Please make sure all services are in running state.
- RDAF Application services status
Run the below command to get RDAF App services status & version details. Please make sure all services are in running state.
2. Download the New Docker Images
Download the new docker image tags for RDAF Platform and OIA (AIOps) Application services and wait until all of the images are downloaded
To fetch registry please use the below command
Run the below command to verify above mentioned tags are downloaded for all of the RDAF Platform and OIA (AIOps) Application services.
Please make sure the 3.7.2 image tag is downloaded for the below RDAF Platform services.
-
rda-client-api-server
-
rda-registry
-
rda-scheduler
-
rda-collector
-
rda-identity
-
rda-fsm
-
rda-asm
-
rda-stack-mgr
-
rda-access-manager
-
rda-resource-manager
-
rda-user-preferences
-
onprem-portal
-
onprem-portal-nginx
-
rda-worker-all
-
onprem-portal-dbinit
-
cfxdx-nb-nginx-all
-
rda-event-gateway
-
rda-chat-helper
-
rdac
-
rdac-full
-
cfxcollector
-
bulk_stats
Please make sure the 7.7.2 image tag is downloaded for the below RDAF OIA (AIOps) Application services.
-
rda-app-controller
-
rda-alert-processor
-
rda-file-browser
-
rda-smtp-server
-
rda-ingestion-tracker
-
rda-reports-registry
-
rda-ml-config
-
rda-event-consumer
-
rda-webhook-server
-
rda-irm-service
-
rda-alert-ingester
-
rda-collaboration
-
rda-notification-service
-
rda-configuration-service
-
rda-alert-processor-companion
3. Update OpenSearch User Permissions
- Run the below script to update the OpenSearch user permissions
wget https://macaw-amer.s3.us-east-1.amazonaws.com/releases/rdaf-platform/1.3.2/opensearch_policy_permissions.py
Note
Please make sure socat command is availabe on all RDAF Infrastructure service nodes. Without this command utility, the below python script may fail.
- Please execute the command shown below in the CLI VM
Note
The expected output of the above command will be similar to the below output
- Run the below script to update the OpenSearch user permissions
wget https://macaw-amer.s3.us-east-1.amazonaws.com/releases/rdaf-platform/1.3.2/opensearch_policy_permissions.py
- Please execute the command shown below in the CLI VM
Note
The expected output of the above command will be similar to the below output
4. Upgrade Steps
4.1 Upgrade Platform Services
Step-1: Run the below command to initiate upgrading all the RDAF Platform services
Note
After upgrading the above mentioned service, use the following commands to verify that the service is up and running
As the upgrade procedure is a non-disruptive upgrade, it puts the currently running PODs into Terminating state and newer version PODs into Pending state.
Step-2: Run the below command to check the status of the existing and newer PODs and make sure atleast one instance of each Platform service is in Terminating state.
Step-3: Run the below command to put all Terminating RDAF platform service PODs into maintenance mode. It will list all of the POD Ids of platform services along with rdac maintenance command that required to be put in maintenance mode.
Note
If maint_command.py script doesn't exist on RDAF deployment CLI VM, it can be downloaded using the below command.
Step-4: Copy & Paste the rdac maintenance command as below.
Step-5: Run the below command to verify the maintenance mode status of the RDAF platform services.
Step-6: Run the below command to delete the Terminating RDAF platform service PODs
for i in `kubectl get pods -n rda-fabric -l app_category=rdaf-platform | grep 'Terminating' | awk '{print $1}'`; do kubectl delete pod $i -n rda-fabric --force; done
Please wait till the new platform service is in Up state and run the below command to verify their status and make sure upgraded services are running with 3.7.2 version.
+---------------+----------------+---------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+---------------+----------------+---------------+--------------+-------+
| rda-api- | 192.168.131.47 | Up 1 Days ago | 4955cfdf09ed | 3.7.2 |
| server | | | | |
| rda-api- | 192.168.131.46 | Up 2 Days ago | cfe50b289750 | 3.7.2 |
| server | | | | |
| rda-registry | 192.168.131.45 | Up 5 Days ago | 607c946648a8 | 3.7.2 |
| rda-registry | 192.168.131.44 | Up 5 Days ago | 883a60eb17c6 | 3.7.2 |
| rda-identity | 192.168.131.44 | Up 5 Days ago | 86aac73187c3 | 3.7.2 |
| rda-identity | 192.168.131.46 | Up 5 Days ago | 25d7b9481739 | 3.7.2 |
| rda-fsm | 192.168.131.45 | Up 5 Days ago | c7aa638ec34f | 3.7.2 |
| rda-fsm | 192.168.131.44 | Up 5 Days ago | 1bfcc06fc44b | 3.7.2 |
| rda-asm | 192.168.131.44 | Up 5 Days ago | 77e8581fcded | 3.7.2 |
| rda-asm | 192.168.131.45 | Up 5 Days ago | f5326a49a44f | 3.7.2 |
| rda-chat- | 192.168.131.45 | Up 5 Days ago | 56643ca71349 | 3.7.2 |
| helper | | | | |
| rda-chat- | 192.168.131.44 | Up 5 Days ago | bf18487c1ff0 | 3.7.2 |
| helper | | | | |
| rda-access- | 192.168.131.47 | Up 5 Days ago | 9b123a96f654 | 3.7.2 |
| manager | | | | |
| rda-access- | 192.168.131.45 | Up 5 Days ago | 2e832da2ba20 | 3.7.2 |
| manager | | | | |
| rda-resource- | 192.168.131.45 | Up 5 Days ago | eb72b20e509c | 3.7.2 |
| manager | | | | |
| rda-resource- | 192.168.131.44 | Up 5 Days ago | 931065aabbcd | 3.7.2 |
| manager | | | | |
| rda-scheduler | 192.168.131.47 | Up 2 Days ago | 2d5336985ff7 | 3.7.2 |
| rda-scheduler | 192.168.131.46 | Up 2 Days ago | c6665498ec5e | 3.7.2 |
| rda-asset- | 192.168.131.46 | Up 5 Days ago | 9bee54aabaab | 3.7.2 |
| dependency | | | | |
| rda-asset- | 192.168.131.47 | Up 5 Days ago | 75cb208232ff | 3.7.2 |
| dependency | | | | |
| rda-collector | 192.168.131.44 | Up 5 Days ago | 917ff0e96aa4 | 3.7.2 |
| rda-collector | 192.168.131.45 | Up 5 Days ago | 5476b2d12a2b | 3.7.2 |
| rda-user- | 192.168.131.44 | Up 5 Days ago | f42d84d3cebd | 3.7.2 |
| preferences | | | | |
| rda-user- | 192.168.131.45 | Up 5 Days ago | 6ea4c5fc3c23 | 3.7.2 |
| preferences | | | | |
| rda-portal- | 192.168.131.46 | Up 2 Days ago | 8644577686f8 | 3.7.2 |
| backend | | | | |
| rda-portal- | 192.168.131.46 | Up 1 Days ago | f2312e184ba1 | 3.7.2 |
| frontend | | | | |
| rda-portal- | 192.168.131.47 | Up 2 Days ago | 7053eaeb167c | 3.7.2 |
| backend | | | | |
| rda-portal- | 192.168.131.47 | Up 2 Days ago | 34044ca874da | 3.7.2 |
| frontend | | | | |
+---------------+----------------+---------------+--------------+-------+
Run the below command to check if all services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=1, Brokers=[0, 1, 2] |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | kafka-consumer | ok | Health: [{'0bde1fc06a7542c3a6a4c47d85f64d9d.inbound-events': 0, '0bde1fc06a7542c3a6a4c47d85f64d9d.mapped-events': 0}, {}] |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | kafka-consumer | ok | Health: [{'0bde1fc06a7542c3a6a4c47d85f64d9d.inbound-events': 0, '0bde1fc06a7542c3a6a4c47d85f64d9d.mapped-events': 0}, {}] |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=0, Brokers=[0, 1, 2]
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Run the below command to initiate upgrading all the RDAF Platform services with zero downtime
Note
timeout <10> mentioned in the above command represents as Seconds
Note
The rolling-upgrade option upgrades the Platform services running in high-availability mode on one VM at a time in sequence. It completes the upgrade of Platform services running on VM-1 before upgrading them on VM-2, followed by VM-3, and so on.
During this upgrade sequence, RDAF platform continues to function without any impact to the application traffic.
After completing the Platform services upgrade on all VMs, it will ask for user confirmation to delete the older version Platform service PODs. The user has to provide YES to delete the old pods
2025-01-20 03:58:18,406 [rdaf.component.platform] INFO - Gathering platform container details.
2025-01-20 03:58:18,541 [rdaf.component.platform] INFO - Gathering rdac pod details.
+----------+------------+---------+------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+------------+---------+------------------+--------------+-------------+------------+
| 7e262f73 | scheduler | 3.7.1 | 2 days, 6:18:01 | ff0068a64009 | None | True |
| 21facf4d | api-server | 3.7.1 | 2 days, 18:56:32 | 1f3799933711 | None | True |
+----------+------------+---------+------------------+--------------+-------------+------------+
Continue moving above pods to maintenance mode? [yes/no]: yes
2025-01-20 04:00:55,768 [rdaf.component.platform] INFO - Initiating Maintenance Mode...
2025-01-20 04:01:01,494 [rdaf.component.platform] INFO - Following container are in maintenance mode
+----------+------------+---------+------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+------------+---------+------------------+--------------+-------------+------------+
| 21facf4d | api-server | 3.7.1 | 2 days, 18:59:13 | 1f3799933711 | maintenance | False |
| 7e262f73 | scheduler | 3.7.1 | 2 days, 6:20:42 | ff0068a64009 | maintenance | False |
+----------+------------+---------+------------------+--------------+-------------+------------+
2025-01-20 04:01:01,495 [rdaf.component.platform] INFO - Waiting for timeout of 120 seconds...
2025-01-20 04:03:01,594 [rdaf.component.platform] INFO - Upgrading service: rda_scheduler on host 192.168.109.50
Run the below command to initiate upgrading all the RDAF Platform Services without** zero downtime
Note
After upgrading the above mentioned services, use the following commands to verify that the services are up and running
Please wait till all of the platform service PODs are in Running state and run the below command to verify their status and make sure all of them are running with 3.7.2 version.
+---------------+-----------------+-----------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+---------------+-----------------+-----------+--------------+-------+
| rda_api_serve | 192.168.107.200 | Up 3 days | e1cb7dfdfabd | 3.7.2 |
| r | | | | |
| rda_registry | 192.168.107.200 | Up 2 days | 33209709a605 | 3.7.2 |
| rda_scheduler | 192.168.107.200 | Up 3 days | 8c7c158d960b | 3.7.2 |
| rda_collector | 192.168.107.200 | Up 2 days | ef42b08c1eed | 3.7.2 |
| rda_asset_dep | 192.168.107.200 | Up 2 days | 4ccec304dda6 | 3.7.2 |
| endency | | | | |
| rda_identity | 192.168.107.200 | Up 2 days | 2482f720f509 | 3.7.2 |
| rda_asm | 192.168.107.200 | Up 2 days | bb63508c51cb | 3.7.2 |
| rda_fsm | 192.168.107.200 | Up 2 days | 8d0191582a05 | 3.7.2 |
| rda_chat_help | 192.168.107.200 | Up 2 days | c69f059cda83 | 3.7.2 |
| er | | | | |
| cfx-rda- | 192.168.107.200 | Up 2 days | 1b20630ea20e | 3.7.2 |
| access- | | | | |
| manager | | | | |
| cfx-rda- | 192.168.107.200 | Up 2 days | 66c87957aa9f | 3.7.2 |
| resource- | | | | |
| manager | | | | |
| cfx-rda-user- | 192.168.107.200 | Up 2 days | 33b14be6e284 | 3.7.2 |
| preferences | | | | |
| portal- | 192.168.107.200 | Up 3 days | d2bcc80dfa7e | 3.7.2 |
| backend | | | | |
| portal- | 192.168.107.200 | Up 3 days | 5cf0dec64ddf | 3.7.2 |
| frontend | | | | |
+---------------+-----------------+-----------+--------------+-------+
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------|
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-status | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | minio-connectivity | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-initialization-status | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | kafka-connectivity | ok | Cluster=NTc1NWU1MTQxYmY3MTFlZg, Broker=1, Brokers=[1, 2, 3] |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-status | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | minio-connectivity | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-initialization-status | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | kafka-connectivity | ok | Cluster=NTc1NWU1MTQxYmY3MTFlZg, Broker=3, Brokers=[1, 2, 3] |
| rda_app | alert-processor | c6cc7b04ab33 | b4ebfb06 | | service-status | ok | |
| rda_app | alert-processor | c6cc7b04ab33 | b4ebfb06 | | minio-connectivity | ok | |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
4.2 Upgrade RDA Worker Services
Step-1: Please run the below command to initiate upgrading the RDA Worker service PODs.
Step-2: Run the below command to check the status of the existing and newer PODs and make sure atleast one instance of each RDA Worker service POD is in Terminating state.
NAME READY STATUS RESTARTS AGE
rda-worker-77f459d5b9-9kdmg 1/1 Running 0 73m
rda-worker-77f459d5b9-htsmr 1/1 Running 0 74m
Step-3: Run the below command to put all Terminating RDAF worker service PODs into maintenance mode. It will list all of the POD Ids of RDA worker services along with rdac maintenance command that is required to be put in maintenance mode.
Step-4: Copy & Paste the rdac maintenance command as below.
Step-5: Run the below command to verify the maintenance mode status of the RDAF worker services.
Step-6: Run the below command to delete the Terminating RDAF worker service PODs
for i in `kubectl get pods -n rda-fabric -l app_component=rda-worker | grep 'Terminating' | awk '{print $1}'`; do kubectl delete pod $i -n rda-fabric --force; done
Note
Wait for 120 seconds between each RDAF worker service upgrade by repeating above steps from Step-2 to Step-6 for rest of the RDAF worker service PODs.
Step-7: Please wait for 120 seconds to let the newer version of RDA Worker service PODs join the RDA Fabric appropriately. Run the below commands to verify the status of the newer RDA Worker service PODs.
+------------+----------------+---------------+--------------+---------+
| Name | Host | Status | Container Id | Tag |
+------------+----------------+---------------+--------------+---------+
| rda-worker | 192.168.108.17 | Up 1 Hour ago | d3e5eab39e98 | 3.7.2 |
| rda-worker | 192.168.108.18 | Up 1 Hour ago | 5a60f0342956 | 3.7.2 |
+------------+----------------+---------------+--------------+---------+
Step-8: Run the below command to check if all RDA Worker services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=2, Brokers=[0, 1, 2] |
| rda_app | alert-ingester | rda-alert-in | b67d62b4 | | kafka-consumer | ok | Health: [{'0bde1fc06a7542c3a6a4c47d85f64d9d.inbound-events': 0, '0bde1fc06a7542c3a6a4c47d85f64d9d.mapped-events': 0}, {}] |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=2, Brokers=[0, 1, 2] |
| rda_app | alert-ingester | rda-alert-in | a4b6ed5a | | kafka-consumer | ok | Health: [{'0bde1fc06a7542c3a6a4c47d85f64d9d.inbound-events': 0, '0bde1fc06a7542c3a6a4c47d85f64d9d.mapped-events': 2}, {}] |
| rda_app | alert-processor | rda-alert-pr | 8340f204 | | service-status | ok | |
| rda_app | alert-processor | rda-alert-pr | 8340f204 | | minio-connectivity | ok | |
| rda_app | alert-processor | rda-alert-pr | 8340f204 | | service-dependency:cfx-app-controller | ok | 2 pod(s) found for cfx-app-controller |
| rda_app | alert-processor | rda-alert-pr | 8340f204 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
- Upgrade RDA Worker Services
Please run the below command to initiate upgrading the RDA Worker Service with zero downtime
Note
timeout <10> mentioned in the above command represents as seconds
Note
The rolling-upgrade option upgrades the Worker services running in high-availability mode on one VM at a time in sequence. It completes the upgrade of Worker services running on VM-1 before upgrading them on VM-2, followed by VM-3, and so on.
After completing the Worker services upgrade on all VMs, it will ask for user confirmation, the user has to provide YES to delete the older version Worker service PODs.
2024-08-12 02:56:11,573 [rdaf.component.worker] INFO - Collecting worker details for rolling upgrade
2024-08-12 02:56:14,301 [rdaf.component.worker] INFO - Rolling upgrade worker on 192.168.133.96
+----------+----------+---------+---------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+----------+---------+---------+--------------+-------------+------------+
| c8a37db9 | worker | 3.7 | 3:32:31 | fffe44b43708 | None | True |
+----------+----------+---------+---------+--------------+-------------+------------+
Continue moving above pod to maintenance mode? [yes/no]: yes
2024-08-12 02:57:17,346 [rdaf.component.worker] INFO - Initiating maintenance mode for pod c8a37db9
2024-08-12 02:57:22,401 [rdaf.component.worker] INFO - Waiting for worker to be moved to maintenance.
2024-08-12 02:57:35,001 [rdaf.component.worker] INFO - Following worker container is in maintenance mode
+----------+----------+---------+---------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+----------+---------+---------+--------------+-------------+------------+
| c8a37db9 | worker | 3.7 | 3:33:52 | fffe44b43708 | maintenance | False |
+----------+----------+---------+---------+--------------+-------------+------------+
2024-08-12 02:57:35,002 [rdaf.component.worker] INFO - Waiting for timeout of 3 seconds.
Please run the below command to initiate upgrading the RDA Worker Service without zero downtime
Please wait for 120 seconds to let the newer version of RDA Worker service containers join the RDA Fabric appropriately. Run the below commands to verify the status of the newer RDA Worker service containers.
| Infra | worker | True | 6eff605e72c4 | a318f394 | rda-site-01 | 13:45:13 | 4 | 31.21 | 0 | 0 |
| Infra | worker | True | ae7244d0d10a | 554c2cd8 | rda-site-01 | 13:40:40 | 4 | 31.21 | 0 | 0 |
+------------+----------------+---------------+--------------+---------+
| Name | Host | Status | Container Id | Tag |
+------------+----------------+---------------+--------------+---------+
| rda_worker | 192.168.133.92 | Up 10 hours | 22a6ac023781 | 3.7.2 |
| rda_worker | 192.168.133.96 | Up 10 hours | cd758ca38c35 | 3.7.2 |
+------------+----------------+---------------+--------------+---------+
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | service-status | ok | |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | minio-connectivity | ok | |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | service-dependency:configuration-service | ok | 1 pod(s) found for configuration-service |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | service-initialization-status | ok | |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | kafka-connectivity | ok | Cluster=ZGQ3MGExMWVjOTY1MTFlZQ, Broker=1, Brokers=[1] |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | kafka-consumer | ok | Health: [{'13c03dbb7281410db821eca533c4ed23.inbound-events': 0, '13c03dbb7281410db821eca533c4ed23.mapped-events': 0}, {'13c03dbb7281410db821eca533c4ed23.event-request': 0}] |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-status | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | minio-connectivity | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-dependency:cfx-app-controller | ok | 1 pod(s) found for cfx-app-controller |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-dependency:configuration-service | ok | 1 pod(s) found for configuration-service |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-initialization-status | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | DB-connectivity | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | kafka-consumer | ok | Health: [{'13c03dbb7281410db821eca533c4ed23.ingested-alerts': 0}] |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | kafka-connectivity | ok | Cluster=ZGQ3MGExMWVjOTY1MTFlZQ, Broker=1, Brokers=[1] |
| rda_app | alert-processor-companion | ea50bea2659b | 5e42c587 | | service-status | ok | |
| rda_app | alert-processor-companion | ea50bea2659b | 5e42c587 | | minio-connectivity | ok | |
| rda_app | alert-processor-companion | ea50bea2659b | 5e42c587 | | service-dependency:alert-processor | ok | 1 pod(s) found for alert-processor |
| rda_app | alert-processor-companion | ea50bea2659b | 5e42c587 | | service-initialization-status | ok | |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
4.3 Upgrade OIA Application Services
Step-1: Please run the below command to initiate upgrading all the OIA Application Services
Step-2: Run the below command to check the status of the newly upgraded PODs.
Step-3: Run the below command to put all Terminating OIA application service PODs into maintenance mode. It will list all of the POD Ids of OIA application services along with rdac maintenance command that are required to be put in maintenance mode.
Step-4: Copy & Paste the rdac maintenance command as below.
Step-5: Run the below command to verify the maintenance mode status of the OIA application services.
Step-6: Run the below command to delete the Terminating OIA application service PODs
for i in `kubectl get pods -n rda-fabric -l app_name=oia | grep 'Terminating' | awk '{print $1}'`; do kubectl delete pod $i -n rda-fabric --force; done
Note
Wait for 120 seconds and Repeat above steps from Step-2 to Step-6 for rest of the OIA application service PODs.
Please wait till all of the new OIA application service PODs are in Running state and run the below command to verify their status and make sure they are running with 7.7.2 version.
-
rda-app-controller
-
rda-alert-processor
-
rda-file-browser
-
rda-smtp-server
-
rda-ingestion-tracker
-
rda-reports-registry
-
rda-ml-config
-
rda-event-consumer
-
rda-webhook-server
-
rda-irm-service
-
rda-alert-ingester
-
rda-collaboration
-
rda-notification-service
-
rda-configuration-service
-
rda-alert-processor-companion
Please wait till all of the new OIA application service containers are in Up state and run the below command to verify their status and make sure they are running with 7.7.2 version.
+-------------------------------+----------------+-------------------+--------------+-----------+
| Name | Host | Status | Container Id | Tag |
+-------------------------------+----------------+-------------------+--------------+-----------+
| rda-alert-ingester | 192.168.131.49 | Up 25 Minutes ago | 630148ddf93e | 7.7.2 |
| rda-alert-ingester | 192.168.131.47 | Up 26 Minutes ago | 6b5f3f192bc8 | 7.7.2 |
| rda-alert-processor | 192.168.131.49 | Up 24 Minutes ago | 5d4f4a0769af | 7.7.2 |
| rda-alert-processor | 192.168.131.50 | Up 24 Minutes ago | 874455e3acae | 7.7.2 |
| rda-alert-processor-companion | 192.168.131.50 | Up 25 Minutes ago | 12736dacb2b8 | 7.7.2 |
| rda-alert-processor-companion | 192.168.131.49 | Up 24 Minutes ago | bea9cc38a351 | 7.7.2 |
| rda-app-controller | 192.168.131.46 | Up 25 Minutes ago | 0cafac284b91 | 7.7.2 |
| rda-app-controller | 192.168.131.47 | Up 26 Minutes ago | ad0933767e27 | 7.7.2 |
| rda-collaboration | 192.168.131.50 | Up 25 Minutes ago | d73a2944d6d0 | 7.7.2 |
| rda-collaboration | 192.168.131.46 | Up 25 Minutes ago | d423aba37e64 | 7.7.2 |
| rda-configuration-service | 192.168.131.47 | Up 26 Minutes ago | 45966a32efad | 7.7.2 |
| rda-configuration-service | 192.168.131.49 | Up 24 Minutes ago | 7afbc227014c | 7.7.2 |
| rda-event-consumer | 192.168.131.50 | Up 24 Minutes ago | 254556e3e318 | 7.7.2 |
| rda-event-consumer | 192.168.131.49 | Up 25 Minutes ago | 84cd6f062d2d | 7.7.2 |
| rda-file-browser | 192.168.131.46 | Up 26 Minutes ago | 5e5dad769632 | 7.7.2 |
| rda-file-browser | 192.168.131.49 | Up 25 Minutes ago | 4189d6b4daca | 7.7.2 |
| rda-ingestion-tracker | 192.168.131.47 | Up 26 Minutes ago | 97af6df3ca99 | 7.7.2 |
| rda-ingestion-tracker | 192.168.131.46 | Up 25 Minutes ago | 18b4bd425b9f | 7.7.2 |
| rda-irm-service | 192.168.131.50 | Up 25 Minutes ago | bc0e571426b1 | 7.7.2 |
| rda-irm-service | 192.168.131.49 | Up 23 Minutes ago | 237894baf3fa | 7.7.2 |
| rda-ml-config | 192.168.131.46 | Up 25 Minutes ago | 85a842762dcf | 7.7.2 |
| rda-ml-config | 192.168.131.47 | Up 25 Minutes ago | 08b106c49533 | 7.7.2 |
| rda-notification-service | 192.168.131.49 | Up 25 Minutes ago | 586d3cd4df49 | 7.7.2 |
| rda-notification-service | 192.168.131.50 | Up 24 Minutes ago | 17427df09a9d | 7.7.2 |
| rda-reports-registry | 192.168.131.50 | Up 25 Minutes ago | 31814258f9cd | 7.7.2 |
| rda-reports-registry | 192.168.131.46 | Up 26 Minutes ago | 7142d0c895bf | 7.7.2 |
| rda-smtp-server | 192.168.131.49 | Up 24 Minutes ago | bf268609c808 | 7.7.2 |
| rda-smtp-server | 192.168.131.47 | Up 25 Minutes ago | a91a7bed1eab | 7.7.2 |
| rda-webhook-server | 192.168.131.46 | Up 26 Minutes ago | 832b7a1c4416 | 7.7.2 |
| rda-webhook-server | 192.168.131.49 | Up 23 Minutes ago | ed38d468f2d7 | 7.7.2 |
+-------------------------------+----------------+-------------------+--------------+-----------+
+-------+----------------------------------------+-------------+----------------+----------+-------------+-----------------+--------+--------------+---------------+--------------+
| Cat | Pod-Type | Pod-Ready | Host | ID | Site | Age | CPUs | Memory(GB) | Active Jobs | Total Jobs |
|-------+----------------------------------------+-------------+----------------+----------+-------------+-----------------+--------+--------------+---------------+--------------|
| App | alert-ingester | True | rda-alert-inge | b67d62b4 | | 0:46:14 | 8 | 31.33 | | |
| App | alert-ingester | True | rda-alert-inge | a4b6ed5a | | 0:45:29 | 8 | 31.33 | | |
| App | alert-processor | True | rda-alert-proc | 8340f204 | | 0:44:44 | 8 | 31.33 | | |
| App | alert-processor | True | rda-alert-proc | 3907e4f4 | | 0:44:12 | 8 | 31.33 | | |
| App | alert-processor-companion | True | rda-alert-proc | d139659c | | 0:45:48 | 8 | 31.33 | | |
| App | alert-processor-companion | True | rda-alert-proc | 24bbbec4 | | 0:44:12 | 8 | 31.33 | | |
| App | asset-dependency | True | rda-asset-depe | 5b7681ff | | 3 days, 2:32:59 | 8 | 31.33 | | |
| App | asset-dependency | True | rda-asset-depe | fb42e84f | | 3 days, 2:32:59 | 8 | 31.33 | | |
| App | authenticator | True | rda-identity-7 | 298181dd | | 3 days, 2:32:58 | 8 | 31.33 | | |
| App | authenticator | True | rda-identity-7 | 3e1563c0 | | 3 days, 2:32:54 | 8 | 31.33 | | |
| App | cfx-app-controller | True | rda-app-contro | 66ac8a50 | | 0:46:08 | 8 | 31.33 | | |
| App | cfx-app-controller | True | rda-app-contro | 9c7ebab3 | | 0:46:14 | 8 | 31.33 | | |
| App | cfxdimensions-app-access-manager | True | rda-access-man | cf516334 | | 3 days, 2:32:56 | 8 | 31.33 | | |
| App | cfxdimensions-app-access-manager | True | rda-access-man | a1a3aad4 | | 3 days, 2:32:58 | 8 | 31.33 | | |
| App | cfxdimensions-app-collaboration | True | rda-collaborat | bbc422e4 | | 0:46:09 | 8 | 31.33 | | |
| App | cfxdimensions-app-collaboration | True | rda-collaborat | b898fb32 | | 0:45:15 | 8 | 31.33 | | |
| App | cfxdimensions-app-file-browser | True | rda-file-brows | cf54d8d5 | | 0:46:17 | 8 | 31.33 | | |
| App | cfxdimensions-app-file-browser | True | rda-file-brows | 5a884fc9 | | 0:45:19 | 8 | 31.33 | | |
| App | cfxdimensions-app-irm_service | True | rda-irm-servic | 2dddb44d | | 0:45:28 | 8 | 31.33 | | |
| App | cfxdimensions-app-irm_service | True | rda-irm-servic | 95fc4d30 | | 0:43:53 | 8 | 31.33 | | |
Run the below command to check if all services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------|
| rda_app | alert-ingester | rda-alert-in | f9314916 | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=0, Brokers=[0, 1, 2] |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=1, Brokers=[0, 1, 2] |
| rda_app | alert-processor | rda-alert-pr | e7e1e389 | | service-status | ok | |
| rda_app | alert-processor | rda-alert-pr | e7e1e389 | | minio-connectivity | ok | |
| rda_app | alert-processor | rda-alert-pr | e7e1e389 | | service-dependency:cfx-app-controller | ok | 2 pod(s) found for cfx-app-controller |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
- Please run the below command to initiate upgrading all the OIA Application services with zero downtime
Note
timeout <10> mentioned in the above command represents as Seconds
Note
The rolling-upgrade option upgrades the OIA application services running in high-availability mode on one VM at a time in sequence. It completes the upgrade of OIA application services running on VM-1 before upgrading them on VM-2, followed by VM-3, and so on.
After completing the OIA application services upgrade on all VMs, it will ask for user confirmation to delete the older version OIA application service PODs.
2025-01-20 06:11:51,921 [rdaf.component.oia] INFO - Gathering OIA app container details.
2025-01-20 06:11:52,187 [rdaf.component.oia] INFO - Gathering rdac pod details.
+----------+--------------------------+-------------+-------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+--------------------------+-------------+-------------------+--------------+-------------+------------+
| 84a846f6 | configuration-service | 7.7.1 | 10 days, 23:29:55 | 4a39340400df | None | True |
| 214d4f3c | alert-ingester | 7.7.1 | 10 days, 23:29:31 | af83a5ce33fc | None | True |
| e2d4a478 | event-consumer | 7.7.1 | 10 days, 23:28:53 | 478340dd41a3 | None | True |
| f705daca | alert-processor | 7.7.1 | 10 days, 23:28:13 | 1445b3ed4772 | None | True |
| 28d120c2 | cfxdimensions-app- | 7.7.1 | 10 days, 23:27:34 | 64efbb9bde20 | None | True |
| | collaboration | | | | | |
| 2f361992 | cfxdimensions-app- | 7.7.1 | 10 days, 23:26:57 | 3a9485387e24 | None | True |
| | irm_service | | | | | |
+----------+--------------------------+-------------+-------------------+--------------+-------------+------------+
Continue moving above pods to maintenance mode? [yes/no]: yes
2025-01-20 06:14:10,062 [rdaf.component.oia] INFO - Initiating Maintenance Mode...
2025-01-20 06:14:14,571 [rdaf.component.oia] INFO - Waiting for services to be moved to maintenance.
2025-01-20 06:14:36,595 [rdaf.component.oia] INFO - Following container are in maintenance mode
+----------+--------------------------+-------------+-------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+--------------------------+-------------+-------------------+--------------+-------------+------------+
| 214d4f3c | alert-ingester | 7.7.1 | 10 days, 23:32:11 | af83a5ce33fc | maintenance | False |
| f705daca | alert-processor | 7.7.1 | 10 days, 23:30:53 | 1445b3ed4772 | maintenance | False |
| 28d120c2 | cfxdimensions-app- | 7.7.1 | 10 days, 23:30:14 | 64efbb9bde20 | maintenance | False |
| | collaboration | | | | | |
| 2f361992 | cfxdimensions-app- | 7.7.1 | 10 days, 23:29:37 | 3a9485387e24 | maintenance | False |
| | irm_service | | | | | |
| 84a846f6 | configuration-service | 7.7.1 | 10 days, 23:32:35 | 4a39340400df | maintenance | False |
| e2d4a478 | event-consumer | 7.7.1 | 10 days, 23:31:33 | 478340dd41a3 | maintenance | False |
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
2025-01-20 06:14:36,597 [rdaf.component.oia] INFO - Waiting for timeout of 10 seconds...
+-----------------------------------+----------------+--------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+-----------------------------------+----------------+--------------+--------------+-------+
| cfx-rda-configuration-service | 192.168.109.53 | Up 7 minutes | 34ab5ecc6648 | 7.7.2 |
| cfx-rda-configuration-service | 192.168.109.54 | Up 4 minutes | 2a0dc3d0db7a | 7.7.2 |
| cfx-rda-alert-ingester | 192.168.109.53 | Up 7 minutes | e92775d11dfc | 7.7.2 |
| cfx-rda-alert-ingester | 192.168.109.54 | Up 3 minutes | b9dc153f57e0 | 7.7.2 |
| cfx-rda-webhook-server | 192.168.109.53 | Up 3 weeks | 3374b37c8e1c | 7.7 |
| cfx-rda-webhook-server | 192.168.109.54 | Up 3 weeks | 69e759386d8f | 7.7 |
| cfx-rda-event-consumer | 192.168.109.53 | Up 7 minutes | f5f038c1f211 | 7.7.2 |
| cfx-rda-event-consumer | 192.168.109.54 | Up 3 minutes | 2b24ef8b3c33 | 7.7.2 |
| cfx-rda-alert-processor | 192.168.109.53 | Up 7 minutes | 7b43f647d6e0 | 7.7.2 |
| cfx-rda-alert-processor | 192.168.109.54 | Up 3 minutes | acbdb3ea8b2c | 7.7.2 |
| cfx-rda-irm-service | 192.168.109.53 | Up 6 minutes | a1a225dd98c6 | 7.7.2 |
| cfx-rda-irm-service | 192.168.109.54 | Up 3 minutes | 4e32c0b81b0f | 7.7.2 |
| cfx-rda-ml-config | 192.168.109.53 | Up 3 weeks | 872bbca736c8 | 7.7 |
| cfx-rda-ml-config | 192.168.109.54 | Up 3 weeks | 09cd07430bae | 7.7 |
| cfx-rda-collaboration | 192.168.109.53 | Up 7 minutes | 02e692fe2b09 | 7.7.2 |
| cfx-rda-collaboration | 192.168.109.54 | Up 3 minutes | bdd254d8942e | 7.7.2 |
+-----------------------------------+----------------+--------------+--------------+-------+
Please run the below command to initiate upgrading all the OIA Application services without zero downtime
Please wait till all of the new OIA application service containers are in Up state and run the below command to verify their status and make sure they are running with 7.7.2 version.
-
rda-app-controller
-
rda-alert-processor
-
rda-file-browser
-
rda-smtp-server
-
rda-ingestion-tracker
-
rda-reports-registry
-
rda-ml-config
-
rda-event-consumer
-
rda-webhook-server
-
rda-irm-service
-
rda-alert-ingester
-
rda-collaboration
-
rda-notification-service
-
rda-configuration-service
-
rda-alert-processor-companion
Please wait till all of the new OIA application service containers are in Up state and run the below command to verify their status and make sure they are running with 7.7.2 version.
+--------------------+-----------------+-------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+--------------------+-----------------+-------------+--------------+-------+
| cfx-rda-app- | 192.168.107.200 | Up 25 hours | fec20e84e25f | 7.7.2 |
| controller | | | | |
| cfx-rda-reports- | 192.168.107.200 | Up 25 hours | 0056bae37b57 | 7.7.2 |
| registry | | | | |
| cfx-rda- | 192.168.107.200 | Up 25 hours | f3088e0e8b78 | 7.7.2 |
| notification- | | | | |
| service | | | | |
| cfx-rda-file- | 192.168.107.200 | Up 25 hours | 945f5e59b652 | 7.7.2 |
| browser | | | | |
| cfx-rda- | 192.168.107.200 | Up 25 hours | b63e38c19a82 | 7.7.2 |
| configuration- | | | | |
| service | | | | |
| cfx-rda-alert- | 192.168.107.200 | Up 25 hours | c42b361d65b4 | 7.7.2 |
| ingester | | | | |
| cfx-rda-webhook- | 192.168.107.200 | Up 25 hours | 6aedb39b6a75 | 7.7.2 |
| server | | | | |
| cfx-rda-smtp- | 192.168.107.200 | Up 25 hours | c89577b993cd | 7.7.2 |
| server | | | | |
| cfx-rda-event- | 192.168.107.200 | Up 25 hours | 56af42e86b22 | 7.7.2 |
| consumer | | | | |
| cfx-rda-alert- | 192.168.107.200 | Up 25 hours | de4254b26f9c | 7.7.2 |
| processor | | | | |
| cfx-rda-irm- | 192.168.107.200 | Up 25 hours | 25b03b77197c | 7.7.2 |
| service | | | | |
| cfx-rda-ml-config | 192.168.107.200 | Up 25 hours | 8761372d5023 | 7.7.2 |
| cfx-rda- | 192.168.107.200 | Up 25 hours | 395c964bc4f8 | 7.7.2 |
| collaboration | | | | |
| cfx-rda-ingestion- | 192.168.107.200 | Up 25 hours | 665627c00c29 | 7.7.2 |
| tracker | | | | |
| cfx-rda-alert- | 192.168.107.200 | Up 25 hours | ea50bea2659b | 7.7.2 |
| processor- | | | | |
| companion | | | | |
+--------------------+-----------------+-------------+--------------+-------+
Run the below command to verify all OIA application services are up and running.
+-------+----------------------------------------+-------------+--------------+----------+-------------+-------------------+--------+--------------+---------------+--------------+
| Cat | Pod-Type | Pod-Ready | Host | ID | Site | Age | CPUs | Memory(GB) | Active Jobs | Total Jobs |
|-------+----------------------------------------+-------------+--------------+----------+-------------+-------------------+--------+--------------+---------------+--------------|
| App | alert-ingester | True | c42b361d65b4 | dfbf4922 | | 1 day, 1:22:54 | 8 | 58.85 | | |
| App | alert-processor | True | de4254b26f9c | 2e5abbb0 | | 1 day, 1:22:08 | 8 | 58.85 | | |
| App | alert-processor-companion | True | ea50bea2659b | 5e42c587 | | 1 day, 1:21:09 | 8 | 58.85 | | |
| App | asset-dependency | True | a74820c74cf0 | 683435b0 | | 23 days, 18:51:12 | 8 | 58.85 | | |
| App | authenticator | True | 1e7b99800267 | cc69f638 | | 23 days, 18:51:12 | 8 | 58.85 | | |
| App | cfx-app-controller | True | fec20e84e25f | a2fb6762 | | 1 day, 1:23:45 | 8 | 58.85 | | |
| App | cfxdimensions-app-access-manager | True | 44bda7333ea7 | 0521b562 | | 23 days, 18:51:12 | 8 | 58.85 | | |
| App | cfxdimensions-app-collaboration | True | 395c964bc4f8 | 9a98a9d0 | | 1 day, 1:21:33 | 8 | 58.85 | | |
| App | cfxdimensions-app-file-browser | True | 945f5e59b652 | 707fa5af | | 1 day, 1:23:21 | 8 | 58.85 | | |
| App | cfxdimensions-app-irm_service | True | 25b03b77197c | 4d3819c8 | | 1 day, 1:21:56 | 8 | 58.85 | | |
| App | cfxdimensions-app-notification-service | True | f3088e0e8b78 | 3ccaa598 | | 1 day, 1:23:33 | 8 | 58.85 | | |
| App | cfxdimensions-app-resource-manager | True | 0252d754c1f9 | bbc7eedf | | 23 days, 18:51:12 | 8 | 58.85 | | |
| App | chat-helper | True | 5977effb3046 | 8113a4f6 | | 23 days, 18:51:14 | 8 | 58.85 | | |
| App | configuration-service | True | b63e38c19a82 | 30285aa4 | | 1 day, 1:23:07 | 8 | 58.85 | | |
| App | event-consumer | True | 56af42e86b22 | 41c0be8c | | 1 day, 1:22:20 | 8 | 58.85 | | |
| App | fsm | True | 13300b2f43b8 | 7e7b1485 | | 23 days, 18:51:11 | 8 | 58.85 | | |
| App | ingestion-tracker | True | 665627c00c29 | 3961e945 | | 1 day, 1:21:21 | 8 | 58.85 | | |
| App | ml-config | True | 8761372d5023 | e9bbe8f7 | | 1 day, 1:21:44 | 8 | 58.85 | | |
| App | reports-registry | True | 0056bae37b57 | 83c893b9 | | 1 day, 1:23:45 | 8 | 58.85 | | |
| App | smtp-server | True | c89577b993cd | ab4c849d | | 1 day, 1:22:32 | 8 | 58.85 | | |
| App | user-preferences | True | 78f53b6df44f | acd755bc | | 23 days, 18:51:11 | 8 | 58.85 | | |
| App | webhook-server | True | 6aedb39b6a75 | dcbedcd7 | | 1 day, 1:22:44 | 8 | 58.85 | | |
| Infra | api-server | True | e1cb7dfdfabd | 38583aad | | 1 day, 1:26:33 | 8 | 58.85 | | |
| Infra | asm | True | f4a413578f5a | d2e30756 | | 23 days, 18:51:14 | 8 | 58.85 | | |
| Infra | collector | True | 89d7c26a5f45 | e12a7292 | | 23 days, 18:51:14 | 8 | 58.85 | | |
| Infra | registry | True | 388d1478fd13 | 29738e99 | | 23 days, 18:51:12 | 8 | 58.85 | | |
| Infra | scheduler | True | 8c7c158d960b | 0b6e183d | *leader* | 1 day, 1:27:06 | 8 | 58.85 | | |
| Infra | worker | True | a8d109f44b8a | 2053f941 | rda-site-01 | 23 days, 18:49:27 | 8 | 58.85 | 4 | 3760 |
+-------+----------------------------------------+-------------+--------------+----------+-------------+-------------------+--------+--------------+---------------+--------------+
Run the below command to check services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | service-status | ok | |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | minio-connectivity | ok | |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | service-dependency:configuration-service | ok | 1 pod(s) found for configuration-service |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | service-initialization-status | ok | |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | kafka-connectivity | ok | Cluster=ZGQ3MGExMWVjOTY1MTFlZQ, Broker=1, Brokers=[1] |
| rda_app | alert-ingester | c42b361d65b4 | dfbf4922 | | kafka-consumer | ok | Health: [{'13c03dbb7281410db821eca533c4ed23.inbound-events': 0, '13c03dbb7281410db821eca533c4ed23.mapped-events': 0}, {'13c03dbb7281410db821eca533c4ed23.event-request': 0}] |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-status | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | minio-connectivity | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-dependency:cfx-app-controller | ok | 1 pod(s) found for cfx-app-controller |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-dependency:configuration-service | ok | 1 pod(s) found for configuration-service |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | service-initialization-status | ok | |
| rda_app | alert-processor | de4254b26f9c | 2e5abbb0 | | DB-connectivity | ok |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
4.4 Upgrade Event Gateway Services
Important
This Upgrade is for Non-K8s only
Step 1. Prerequisites
- Event Gateway with 3.7 tag should be already installed
Note
If a user deployed the event gateway using the RDAF CLI, follow Step 2 and skip Step 3 or if the user did not deploy event gateway in RDAF CLI go to Step 3
Step 2. Upgrade Event Gateway Using RDAF CLI
-
To upgrade the event gateway, log in to the rdaf cli VM and execute the following command.
Note
The below procedure is applicable if the event gateway is installed through Docker Compose File
Step 3. Upgrade Event Gateway Using Docker Compose File
-
Login to the Event Gateway installed VM
-
Navigate to the location where Event Gateway was previously installed, using the following command
-
Edit the docker-compose file for the Event Gateway using a local editor (e.g. vi) update the tag and save it
version: '3.1' services: rda_event_gateway: image: cfxregistry.cloudfabrix.io/ubuntu-rda-event-gateway:3.7.2 restart: always network_mode: host mem_limit: 6G memswap_limit: 6G volumes: - /opt/rdaf/network_config:/network_config - /opt/rdaf/event_gateway/config:/event_gw_config - /opt/rdaf/event_gateway/certs:/certs - /opt/rdaf/event_gateway/logs:/logs - /opt/rdaf/event_gateway/log_archive:/tmp/log_archive logging: driver: "json-file" options: max-size: "25m" max-file: "5" environment: RDA_NETWORK_CONFIG: /network_config/rda_network_config.json EVENT_GW_MAIN_CONFIG: /event_gw_config/main/main.yml EVENT_GW_SNMP_TRAP_CONFIG: /event_gw_config/snmptrap/trap_template.json EVENT_GW_SNMP_TRAP_ALERT_CONFIG: /event_gw_config/snmptrap/trap_to_alert_go.yaml AGENT_GROUP: event_gateway_site01 EVENT_GATEWAY_CONFIG_DIR: /event_gw_config LOGGER_CONFIG_FILE: /event_gw_config/main/logging.yml -
Please run the following commands
-
Use the command as shown below to ensure that the RDA docker instances are up and running.
-
Use the below mentioned command to check docker logs for any errors
-
Login to event gateway installed VM and edit the endpoint.yaml file as shown below
| endpoint.yaml (existing config) | endpoint.yaml (updated config) |
|---|---|
|
| endpoint.yaml (existing config) | endpoint.yaml (updated config) |
|---|---|
|
Tip
In version 3.6.1 or above, the RDA Event Gateway agent introduces enhanced Syslog TCP/UDP endpoints, developed in Go (lang), to boost event processing rates significantly and optimize system resource utilization.
-
New Syslog TCP Endpoint Type: syslog_tcp_go
-
New Syslog UDP Endpoint Type: syslog_udp_go
In version 3.7.2, the RDA Event Gateway agent extended the previously mentioned enhancement to include the following endpoint as well.
- New TCP JSON Endpoint Type: tcp_json_go