Upgrade to 3.7.1 and 7.7.1
Upgrade to 3.7.1 and 7.7.1
1.Prerequisites
Before proceeding with this upgrade, please make sure and verify the below prerequisites are met.
-
RDAF Deployment CLI version: 1.3.2
-
Infra Services tag: 1.0.3 / 1.0.3.3(HA Proxy)
-
Platform Services and RDA Worker tag: 3.7
-
OIA Application Services tag: 7.7
2. Download the New Docker Images
Download the new docker image tags for RDAF Platform and OIA (AIOps) Application services and wait until all of the images are downloaded
To fetch registry please use the below command
Run the below command to verify above mentioned tags are downloaded for all of the RDAF Platform and OIA (AIOps) Application services.
Please make sure the 3.7.1 image tag is downloaded for the below RDAF Platform (AIOps) services.
-
rda-scheduler
-
rda-collector
-
rda-fsm
-
rda-asm
-
rda-client-api-server
-
onprem-portal
-
onprem-portal-nginx
Please make sure the 7.7.1 image tag is downloaded for the below RDAF OIA (AIOps) Application services.
-
rda-configuration-service
-
rda-event-consumer
-
rda-irm-service
-
rda-collaboration
-
rda-alert-processor
-
rda-alert-ingester
3. Upgrade Steps
3.1 Upgrade Platform Services
Step-1: Run the below command to initiate upgrading following RDAF Platform services ASM Service, FSM Service, Collector Service, Scheduler Service, API Server Service, Onprem Portal Service, Onprem Portal Nginx Service
rdafk8s platform upgrade --tag 3.7.1 --service rda-asm --service rda-fsm --service rda-collector --service rda-scheduler --service rda-api-server --service rda-portal
Note
After upgrading the above mentioned service, use the following commands to verify that the service is up and running
As the upgrade procedure is a non-disruptive upgrade, it puts the currently running PODs into Terminating state and newer version PODs into Pending state.
Step-2: Run the below command to check the status of the existing and newer PODs and make sure atleast one instance of each Platform service is in Terminating state.
Step-3: Run the below command to put all Terminating RDAF platform service PODs into maintenance mode. It will list all of the POD Ids of platform services along with rdac maintenance command that required to be put in maintenance mode.
Step-4: Copy & Paste the rdac maintenance command as below.
Step-5: Run the below command to verify the maintenance mode status of the RDAF platform services.
Step-6: Run the below command to delete the Terminating RDAF platform service PODs
for i in `kubectl get pods -n rda-fabric -l app_category=rdaf-platform | grep 'Terminating' | awk '{print $1}'`; do kubectl delete pod $i -n rda-fabric --force; done
Please wait till the new platform service is in Up state and run the below command to verify their status and make sure below service is running with 3.7.1 version.
-
rda-asm
-
rda-fsm
-
rda-collector
-
rda-scheduler
-
rda-api-server
-
rda-portal
+----------------------+--------------+----------------+--------------+---------+
| Name | Host | Status | Container Id | Tag |
+----------------------+--------------+----------------+--------------+---------+
| rda-api-server | 192.168.131.45 | Up 1 Weeks ago | 58b99f123c02 | 3.7.1 |
| rda-api-server | 192.168.131.47 | Up 6 Days ago | 7382d1c378f1 | 3.7.1 |
| rda-registry | 192.168.131.45 | Up 1 Weeks ago | 8a2fcf983c92 | 3.7 |
| rda-registry | 192.168.131.44 | Up 1 Weeks ago | 40d4cedb6ab5 | 3.7 |
| rda-identity | 192.168.131.46 | Up 1 Weeks ago | 1f122f117fff | 3.7 |
| rda-identity | 192.168.131.47 | Up 1 Weeks ago | 44e3a6b9d1ca | 3.7 |
| rda-fsm | 192.168.131.46 | Up 1 Weeks ago | 3640194f462b | 3.7.1 |
| rda-fsm | 192.168.131.47 | Up 1 Weeks ago | b47a01b4e770 | 3.7.1 |
| rda-asm | 192.168.131.46 | Up 1 Weeks ago | a89382738d69 | 3.7.1 |
| rda-asm | 192.168.131.47 | Up 1 Weeks ago | 25cae0594c09 | 3.7.1 |
| rda-chat-helper | 192.168.131.44 | Up 1 Weeks ago | d6853d7b7fac | 3.7 |
| rda-chat-helper | 192.168.131.45 | Up 1 Weeks ago | e6d74db71730 | 3.7 |
| rda-access-manager | 192.168.131.47 | Up 1 Weeks ago | 8ecac89875ec | 3.7 |
| rda-access-manager | 192.168.131.45 | Up 1 Weeks ago | f25107c5eb16 | 3.7 |
| rda-resource-manager | 192.168.131.45 | Up 1 Weeks ago | 45e2819f2e2a | 3.7 |
| rda-resource-manager | 192.168.131.44 | Up 1 Weeks ago | 6ef3fb1acb34 | 3.7 |
| rda-scheduler | 192.168.131.47 | Up 1 Weeks ago | 4edfbaadc7aa | 3.7.1 |
| rda-scheduler | 192.168.131.46 | Up 1 Weeks ago | 28243c594897 | 3.7.1 |
+--------------------+----------------+-----------------+--------------+-------+
Run the below command to check if all services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------|
| rda_infra | api-server | rda-api-serv | 3c36575d | | service-status | ok | |
| rda_infra | api-server | rda-api-serv | 3c36575d | | minio-connectivity | ok | |
| rda_infra | api-server | rda-api-serv | 1fd1778b | | service-status | ok | |
| rda_infra | api-server | rda-api-serv | 1fd1778b | | minio-connectivity | ok | |
| rda_infra | asm | rda-asm-f6b8 | 39a53ac4 | | service-status | ok | |
| rda_infra | asm | rda-asm-f6b8 | 39a53ac4 | | minio-connectivity | ok | |
| rda_infra | asm | rda-asm-f6b8 | 199a31d2 | | service-status | ok | |
| rda_infra | asm | rda-asm-f6b8 | 199a31d2 | | minio-connectivity | ok | |
| rda_infra | scheduler | rda-schedule | ee7565aa | | service-status | ok | |
| rda_infra | scheduler | rda-schedule | ee7565aa | | minio-connectivity | ok | |
| rda_infra | scheduler | rda-schedule | ee7565aa | | DB-connectivity | ok | |
| rda_infra | scheduler | rda-schedule | ee7565aa | | scheduler-webserver-connectivity | ok | |
| rda_infra | scheduler | rda-schedule | 779a624d | | service-status | ok | |
| rda_infra | scheduler | rda-schedule | 779a624d | | minio-connectivity | ok | |
| rda_infra | scheduler | rda-schedule | 779a624d | | DB-connectivity | ok | |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
Warning
For Non-Kubernetes deployment, upgrading RDAF Platform and AIOps application services is a disruptive operation when rolling-upgrade option is not used. Please schedule a maintenance window before upgrading RDAF Platform and AIOps services to newer version.
Run the below command to initiate upgrading following RDAF Platform services ASM Service, FSM Service, Collector Service, Scheduler Service, API Server Service, Portal Backend Service and Portal Frontend Service with zero downtime
rdaf platform upgrade --tag 3.7.1 --rolling-upgrade --service rda_asm --service rda_fsm --service rda_collector --service rda_scheduler --service rda_api_server --service portal-backend --service portal-frontend --timeout 10
Note
timeout <10> mentioned in the above command represents as Seconds
Note
The rolling-upgrade option upgrades the Platform services running in high-availability mode on one VM at a time in sequence. It completes the upgrade of Platform services running on VM-1 before upgrading them on VM-2, followed by VM-3, and so on.
During this upgrade sequence, RDAF platform continues to function without any impact to the application traffic.
After completing the Platform services upgrade on all VMs, it will ask for user confirmation to delete the older version Platform service PODs. The user has to provide YES to delete the old pods
2024-12-03 05:53:54,144 [rdaf.component.platform] INFO - Gathering platform container details.
2024-12-03 05:53:54,360 [rdaf.component.platform] INFO - Gathering rdac pod details.
+----------+------------+-----------+-------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+------------+-----------+-------------------+--------------+-------------+------------+
| d87d14d0 | asm | 3.7 | 10 days, 23:22:46 | 0d2c02a9aa86 | None | True |
| abba515a | fsm | 3.7 | 10 days, 23:22:32 | 8b7dcf5c9cda | None | True |
| c126ae4b | collector | 3.7 | 3 days, 20:29:31 | 21793d96c546 | None | True |
| 86dfb725 | scheduler | 3.7 | 10 days, 23:22:02 | 464def561de4 | None | True |
| 43ea7403 | api-server | 3.7 | 10 days, 23:21:38 | 54c1036f2add | None | True |
+----------+------------+-----------+-------------------+--------------+-------------+------------+
Continue moving above pods to maintenance mode? [yes/no]: yes
2024-12-03 05:56:00,217 [rdaf.component.platform] INFO - Initiating Maintenance Mode...
2024-12-03 05:56:08,316 [rdaf.component.platform] INFO - Waiting for services to be moved to maintenance.
2024-12-03 05:56:30,552 [rdaf.component.platform] INFO - Following container are in maintenance mode
+----------+------------+-----------+-------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+------------+-----------+-------------------+--------------+-------------+------------+
| 43ea7403 | api-server | 3.7 | 10 days, 23:24:18 | 54c1036f2add | maintenance | False |
| d87d14d0 | asm | 3.7 | 10 days, 23:25:26 | 0d2c02a9aa86 | maintenance | False |
| c126ae4b | collector | 3.7 | 3 days, 20:32:11 | 21793d96c546 | maintenance | False |
| abba515a | fsm | 3.7 | 10 days, 23:25:12 | 8b7dcf5c9cda | maintenance | False |
| 86dfb725 | scheduler | 3.7 | 10 days, 23:24:41 | 464def561de4 | maintenance | False |
+----------+------------+-----------+-------------------+--------------+-------------+------------+
2024-12-03 05:56:30,553 [rdaf.component.platform] INFO - Waiting for timeout of 10 seconds...
+--------------------------+----------------+--------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+--------------------------+----------------+--------------+--------------+-------+
| rda_api_server | 192.168.109.50 | Up 7 minutes | 0b5ec5136dac | 3.7.1 |
| rda_api_server | 192.168.109.51 | Up 2 minutes | 80f61700de62 | 3.7.1 |
| rda_registry | 192.168.109.50 | Up 3 weeks | 14202e1a8b33 | 3.7 |
| rda_registry | 192.168.109.51 | Up 3 weeks | 5cace600c998 | 3.7 |
| rda_scheduler | 192.168.109.50 | Up 7 minutes | 3a935a2ec7a6 | 3.7.1 |
| rda_scheduler | 192.168.109.51 | Up 2 minutes | f64a753842d2 | 3.7.1 |
| rda_collector | 192.168.109.50 | Up 7 minutes | 18a6352d0ccb | 3.7.1 |
| rda_collector | 192.168.109.51 | Up 2 minutes | 88f5d486391d | 3.7.1 |
| rda_asset_dependency | 192.168.109.50 | Up 3 weeks | 5978017174d3 | 3.7 |
| rda_asset_dependency | 192.168.109.51 | Up 3 weeks | 2d135b6b477a | 3.7 |
| rda_identity | 192.168.109.50 | Up 3 weeks | 1bf40b89e762 | 3.7 |
| rda_identity | 192.168.109.51 | Up 3 weeks | 08e6ad7e2a07 | 3.7 |
| rda_asm | 192.168.109.50 | Up 8 minutes | b3f072b73d2d | 3.7.1 |
| rda_asm | 192.168.109.51 | Up 2 minutes | a573fae8bea5 | 3.7.1 |
| rda_fsm | 192.168.109.50 | Up 8 minutes | ee1868aebec7 | 3.7.1 |
| rda_fsm | 192.168.109.51 | Up 2 minutes | a0de4da45838 | 3.7.1 |
| portal-backend | 192.168.109.50 | Up 7 minutes | 36c031cd36ab | 3.7.1 |
| portal-backend | 192.168.109.51 | Up 2 minutes | 644a6a1b5eee | 3.7.1 |
| portal-frontend | 192.168.109.50 | Up 6 minutes | 9368bd4477bf | 3.7.1 |
| portal-frontend | 192.168.109.51 | Up 2 minutes | 3131d458b590 | 3.7.1 |
+--------------------------+----------------+--------------+--------------+-------+
Run the below command to initiate upgrading following RDAF Platform ASM Service, FSM Service, Collector Service, Scheduler Service, API Server Service, Portal Backend Service and Portal Frontend Service without zero downtime
rdaf platform upgrade --tag 3.7.1 --service rda_asm --service rda_fsm --service rda_collector --service rda_scheduler --service rda_api_server --service portal-backend --service portal-frontend
Note
After upgrading the above mentioned services, use the following commands to verify that the services are up and running
Please wait till the new platform service is in Up state and run the below command to verify their status and make sure below service is running with 3.7.1 version.
-
rda_asm
-
rda_fsm
-
rda_collector
-
rda_scheduler
-
rda_api_server
-
portal-backend
-
portal-frontend
+--------------------+----------------+------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+--------------------+----------------+------------+--------------+-------+
| rda_api_server | 192.168.109.50 | Up 7 days | 54c1036f2add | 3.7.1 |
| rda_api_server | 192.168.109.51 | Up 7 days | 29b6a62ad3c0 | 3.7.1 |
| rda_registry | 192.168.109.50 | Up 2 weeks | 14202e1a8b33 | 3.7 |
| rda_registry | 192.168.109.51 | Up 2 weeks | 5cace600c998 | 3.7 |
| rda_scheduler | 192.168.109.50 | Up 7 days | 464def561de4 | 3.7.1 |
| rda_scheduler | 192.168.109.51 | Up 7 days | 7f85be9732c3 | 3.7.1 |
| rda_collector | 192.168.109.50 | Up 7 days | 21793d96c546 | 3.7.1 |
| rda_collector | 192.168.109.51 | Up 7 days | 27d22cde6cf4 | 3.7.1 |
| rda_asset_dependen | 192.168.109.50 | Up 2 weeks | 5978017174d3 | 3.7 |
| cy | | | | |
| rda_asset_dependen | 192.168.109.51 | Up 2 weeks | 2d135b6b477a | 3.7 |
| cy | | | | |
| rda_identity | 192.168.109.50 | Up 2 weeks | 1bf40b89e762 | 3.7 |
| rda_identity | 192.168.109.51 | Up 2 weeks | 08e6ad7e2a07 | 3.7 |
| rda_asm | 192.168.109.50 | Up 7 days | 0d2c02a9aa86 | 3.7.1 |
| rda_asm | 192.168.109.51 | Up 7 days | ab9ce2a5bee4 | 3.7.1 |
| rda_fsm | 192.168.109.50 | Up 7 days | 8b7dcf5c9cda | 3.7.1 |
| rda_fsm | 192.168.109.51 | Up 7 days | 9cd56ca014e5 | 3.7.1 |
+--------------------+----------------+------------+--------------+-------+
Run the below command to check if all services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------|
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-status | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | minio-connectivity | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-initialization-status | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | kafka-connectivity | ok | Cluster=NTc1NWU1MTQxYmY3MTFlZg, Broker=1, Brokers=[1, 2, 3] |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-status | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | minio-connectivity | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-initialization-status | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | kafka-connectivity | ok | Cluster=NTc1NWU1MTQxYmY3MTFlZg, Broker=3, Brokers=[1, 2, 3] |
| rda_app | alert-processor | c6cc7b04ab33 | b4ebfb06 | | service-status | ok | |
| rda_app | alert-processor | c6cc7b04ab33 | b4ebfb06 | | minio-connectivity | ok | |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
3.2 Upgrade OIA Application Services
Step-1: Please run the below command to initiate upgrading the Following OIA Application Services RDA Configuration, RDA Event Consumer, RDA IRM, RDA Collaboration, RDA Alert Processor and RDA Alert Ingester
rdafk8s app upgrade OIA --tag 7.7.1 --service rda-configuration-service --service rda-alert-ingester --service rda-event-consumer --service rda-alert-processor --service rda-collaboration --service rda-irm-service
Step-2: Run the below command to check the status of the newly upgraded PODs.
Step-3: Run the below command to put all Terminating OIA application service PODs into maintenance mode. It will list all of the POD Ids of OIA application services along with rdac maintenance command that are required to be put in maintenance mode.
Step-4: Copy & Paste the rdac maintenance command as below.
Step-5: Run the below command to verify the maintenance mode status of the OIA application services.
Step-6: Run the below command to delete the Terminating OIA application service PODs
for i in `kubectl get pods -n rda-fabric -l app_name=oia | grep 'Terminating' | awk '{print $1}'`; do kubectl delete pod $i -n rda-fabric --force; done
Note
Wait for 120 seconds and Repeat above steps from Step-2 to Step-6 for rest of the OIA application service PODs.
Please wait till all of the new OIA application service PODs are in Running state and run the below command to verify their status and make sure they are running with 7.7.1 version.
-
rda-configuration-service
-
rda-alert-ingester
-
rda-event-consumer
-
rda-alert-processor
-
rda-collaboration
-
rda-irm-service
+-------------------------------+----------------+----------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+-------------------------------+----------------+----------------+--------------+-------+
| rda-alert-ingester | 192.168.131.49 | Up 1 Weeks ago | 4bbd91fd317f | 7.7.1 |
| rda-alert-ingester | 192.168.131.47 | Up 1 Weeks ago | 51f104511a21 | 7.7.1 |
| rda-alert-processor | 192.168.131.50 | Up 1 Weeks ago | 6071ba5a41dd | 7.7.1 |
| rda-alert-processor | 192.168.131.49 | Up 1 Weeks ago | eade4566af8d | 7.7.1 |
| rda-alert-processor-companion | 192.168.131.47 | Up 1 Weeks ago | caa9f90b813d | 7.7 |
| rda-alert-processor-companion | 192.168.131.46 | Up 1 Weeks ago | 260b1513c8fa | 7.7 |
| rda-app-controller | 192.168.131.46 | Up 1 Weeks ago | 9b0977f4c39a | 7.7 |
| rda-app-controller | 192.168.131.47 | Up 1 Weeks ago | ca6b9b1ec3fb | 7.7 |
| rda-collaboration | 192.168.131.49 | Up 1 Weeks ago | 751178b01278 | 7.7.1 |
| rda-collaboration | 192.168.131.50 | Up 1 Weeks ago | 71c5b71b9d61 | 7.7.1 |
| rda-configuration-service | 192.168.131.46 | Up 1 Weeks ago | 3a977e492c1a | 7.7.1 |
| rda-configuration-service | 192.168.131.49 | Up 1 Weeks ago | 7fd316c374f6 | 7.7.1 |
| rda-event-consumer | 192.168.131.49 | Up 1 Weeks ago | 35a9672e5498 | 7.7.1 |
| rda-event-consumer | 192.168.131.46 | Up 1 Weeks ago | 07be9e191ae7 | 7.7.1 |
+-------------------------------+----------------+----------------+--------------+-------+
+-------+----------------------------------------+-------------+----------------+----------+-------------+-------------------+--------+--------------+---------------+--------------+
| Cat | Pod-Type | Pod-Ready | Host | ID | Site | Age | CPUs | Memory(GB) | Active Jobs | Total Jobs |
|-------+----------------------------------------+-------------+----------------+----------+-------------+-------------------+--------+--------------+---------------+--------------|
| Infra | collector | True | rda-collector- | 874761cd | | 5 days, 9:04:23 | 8 | 31.33 | | |
| Infra | registry | True | rda-registry-6 | 069f3456 | | 23 days, 20:03:11 | 8 | 31.33 | | |
| Infra | registry | True | rda-registry-6 | 0ae8af6f | | 23 days, 20:03:06 | 8 | 31.33 | | |
| Infra | collector | True | rda-collector- | 874761cd | | 5 days, 9:04:23 | 8 | 31.33 | | |
| Infra | registry | True | rda-registry-6 | 069f3456 | | 23 days, 20:03:11 | 8 | 31.33 | | |
| Infra | registry | True | rda-registry-6 | 0ae8af6f | | 23 days, 20:03:06 | 8 | 31.33 | | |
| Infra | scheduler | True | rda-scheduler- | ee7565aa | *leader* | 3 days, 0:43:20 | 8 | 31.33 | | |
| Infra | scheduler | True | rda-scheduler- | 779a624d | | 3 days, 0:43:02 | 8 | 31.33 | | |
| Infra | worker | True | rda-worker-7cf | 7563615a | rda-site-01 | 3 days, 0:36:29 | 8 | 31.33 | 0 | 3281 |
| Infra | worker | True | rda-worker-7cf | 0cbdeb0d | rda-site-01 | 3 days, 0:35:31 | 8 | 31.33 | 2 | 3252 |
+-------+----------------------------------------+-------------+----------------+----------+-------------+-------------------+--------+--------------+---------------+--------------+
Run the below command to check if all services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------|
| rda_app | alert-ingester | rda-alert-in | f9314916 | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | f9314916 | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=0, Brokers=[0, 1, 2] |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | service-status | ok | |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | minio-connectivity | ok | |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | service-initialization-status | ok | |
| rda_app | alert-ingester | rda-alert-in | 8fc5bbcb | | kafka-connectivity | ok | Cluster=IrA5ccri7mBeUvhzvrimEg, Broker=1, Brokers=[0, 1, 2] |
| rda_app | alert-processor | rda-alert-pr | e7e1e389 | | service-status | ok | |
| rda_app | alert-processor | rda-alert-pr | e7e1e389 | | minio-connectivity | ok | |
| rda_app | alert-processor | rda-alert-pr | e7e1e389 | | service-dependency:cfx-app-controller | ok | 2 pod(s) found for cfx-app-controller |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
- Please run the below command to initiate upgrading the OIA Application services RDA Configuration, RDA Alert Ingester, RDA Event Consumer, RDA Alert Processor, RDA Collaboration and RDA IRM Service with zero downtime
rdaf app upgrade OIA --tag 7.7.1 --rolling-upgrade --service cfx-rda-configuration-service --service cfx-rda-alert-ingester --service cfx-rda-event-consumer --service cfx-rda-alert-processor --service cfx-rda-collaboration --service cfx-rda-irm-service --timeout 10
Note
timeout <10> mentioned in the above command represents as Seconds
Note
The rolling-upgrade option upgrades the OIA application services running in high-availability mode on one VM at a time in sequence. It completes the upgrade of OIA application services running on VM-1 before upgrading them on VM-2, followed by VM-3, and so on.
After completing the OIA application services upgrade on all VMs, it will ask for user confirmation to delete the older version OIA application service PODs.
2024-12-03 06:11:51,921 [rdaf.component.oia] INFO - Gathering OIA app container details.
2024-12-03 06:11:52,187 [rdaf.component.oia] INFO - Gathering rdac pod details.
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
| 84a846f6 | configuration-service | 7.7 | 10 days, 23:29:55 | 4a39340400df | None | True |
| 214d4f3c | alert-ingester | 7.7 | 10 days, 23:29:31 | af83a5ce33fc | None | True |
| e2d4a478 | event-consumer | 7.7 | 10 days, 23:28:53 | 478340dd41a3 | None | True |
| f705daca | alert-processor | 7.7 | 10 days, 23:28:13 | 1445b3ed4772 | None | True |
| 28d120c2 | cfxdimensions-app- | 7.7 | 10 days, 23:27:34 | 64efbb9bde20 | None | True |
| | collaboration | | | | | |
| 2f361992 | cfxdimensions-app- | 7.7 | 10 days, 23:26:57 | 3a9485387e24 | None | True |
| | irm_service | | | | | |
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
Continue moving above pods to maintenance mode? [yes/no]: yes
2024-12-03 06:14:10,062 [rdaf.component.oia] INFO - Initiating Maintenance Mode...
2024-12-03 06:14:14,571 [rdaf.component.oia] INFO - Waiting for services to be moved to maintenance.
2024-12-03 06:14:36,595 [rdaf.component.oia] INFO - Following container are in maintenance mode
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
| Pod ID | Pod Type | Version | Age | Hostname | Maintenance | Pod Status |
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
| 214d4f3c | alert-ingester | 7.7 | 10 days, 23:32:11 | af83a5ce33fc | maintenance | False |
| f705daca | alert-processor | 7.7 | 10 days, 23:30:53 | 1445b3ed4772 | maintenance | False |
| 28d120c2 | cfxdimensions-app- | 7.7 | 10 days, 23:30:14 | 64efbb9bde20 | maintenance | False |
| | collaboration | | | | | |
| 2f361992 | cfxdimensions-app- | 7.7 | 10 days, 23:29:37 | 3a9485387e24 | maintenance | False |
| | irm_service | | | | | |
| 84a846f6 | configuration-service | 7.7 | 10 days, 23:32:35 | 4a39340400df | maintenance | False |
| e2d4a478 | event-consumer | 7.7 | 10 days, 23:31:33 | 478340dd41a3 | maintenance | False |
+----------+--------------------------+-----------+-------------------+--------------+-------------+------------+
2024-12-03 06:14:36,597 [rdaf.component.oia] INFO - Waiting for timeout of 10 seconds...
+-----------------------------------+----------------+--------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+-----------------------------------+----------------+--------------+--------------+-------+
| cfx-rda-configuration-service | 192.168.109.53 | Up 7 minutes | 34ab5ecc6648 | 7.7.1 |
| cfx-rda-configuration-service | 192.168.109.54 | Up 4 minutes | 2a0dc3d0db7a | 7.7.1 |
| cfx-rda-alert-ingester | 192.168.109.53 | Up 7 minutes | e92775d11dfc | 7.7.1 |
| cfx-rda-alert-ingester | 192.168.109.54 | Up 3 minutes | b9dc153f57e0 | 7.7.1 |
| cfx-rda-webhook-server | 192.168.109.53 | Up 3 weeks | 3374b37c8e1c | 7.7 |
| cfx-rda-webhook-server | 192.168.109.54 | Up 3 weeks | 69e759386d8f | 7.7 |
| cfx-rda-event-consumer | 192.168.109.53 | Up 7 minutes | f5f038c1f211 | 7.7.1 |
| cfx-rda-event-consumer | 192.168.109.54 | Up 3 minutes | 2b24ef8b3c33 | 7.7.1 |
| cfx-rda-alert-processor | 192.168.109.53 | Up 7 minutes | 7b43f647d6e0 | 7.7.1 |
| cfx-rda-alert-processor | 192.168.109.54 | Up 3 minutes | acbdb3ea8b2c | 7.7.1 |
| cfx-rda-irm-service | 192.168.109.53 | Up 6 minutes | a1a225dd98c6 | 7.7.1 |
| cfx-rda-irm-service | 192.168.109.54 | Up 3 minutes | 4e32c0b81b0f | 7.7.1 |
| cfx-rda-ml-config | 192.168.109.53 | Up 3 weeks | 872bbca736c8 | 7.7 |
| cfx-rda-ml-config | 192.168.109.54 | Up 3 weeks | 09cd07430bae | 7.7 |
| cfx-rda-collaboration | 192.168.109.53 | Up 7 minutes | 02e692fe2b09 | 7.7.1 |
| cfx-rda-collaboration | 192.168.109.54 | Up 3 minutes | bdd254d8942e | 7.7.1 |
+-----------------------------------+----------------+--------------+--------------+-------+
Please run the below command to initiate upgrading the OIA Application services RDA Configuration, RDA Alert Ingester, RDA Event Consumer, RDA Alert Processor, RDA Collaboration and RDA IRM Service without zero downtime
rdaf app upgrade OIA --tag 7.7.1 --service cfx-rda-configuration-service --service cfx-rda-alert-ingester --service cfx-rda-event-consumer --service cfx-rda-alert-processor --service cfx-rda-collaboration --service cfx-rda-irm-service
Please wait till all of the new OIA application service containers are in Up state and run the below command to verify their status and make sure they are running with 7.7.1 version.
-
rda-configuration-service
-
rda-alert-ingester
-
rda-event-consumer
-
rda-alert-processor
-
rda-collaboration
-
rda-irm-service
+-----------------------------------+--------------+------------+--------------+---------+
| Name | Host | Status | Container Id | Tag |
+-----------------------------------+--------------+------------+--------------+---------+
| cfx-rda-app-controller | 192.168.109.53 | Up 2 weeks | e0a3b011092b | 7.7 |
| cfx-rda-app-controller | 192.168.109.54 | Up 2 weeks | dd729df4567f | 7.7 |
| cfx-rda-reports-registry | 192.168.109.53 | Up 2 weeks | d62ddb342bc2 | 7.7 |
| cfx-rda-reports-registry | 192.168.109.54 | Up 2 weeks | 4b30336152fe | 7.7 |
| cfx-rda-notification-service | 192.168.109.53 | Up 2 weeks | 6f2a8c2ff9fa | 7.7 |
| cfx-rda-notification-service | 192.168.109.54 | Up 2 weeks | 5f70fcea5902 | 7.7 |
| cfx-rda-file-browser | 192.168.109.53 | Up 2 weeks | 9736dd5ca119 | 7.7 |
| cfx-rda-file-browser | 192.168.109.54 | Up 2 weeks | aa510a84edea | 7.7 |
| cfx-rda-configuration-service | 192.168.109.53 | Up 7 days | 4a39340400df | 7.7.1 |
| cfx-rda-configuration-service | 192.168.109.54 | Up 7 days | 5d43a8fa039d | 7.7.1 |
| cfx-rda-alert-ingester | 192.168.109.53 | Up 7 days | af83a5ce33fc | 7.7.1 |
| cfx-rda-alert-ingester | 192.168.109.54 | Up 7 days | acd2d64374fc | 7.7.1 |
| cfx-rda-webhook-server | 192.168.109.53 | Up 2 weeks | 3374b37c8e1c | 7.7 |
| cfx-rda-webhook-server | 192.168.109.54 | Up 2 weeks | 69e759386d8f | 7.7 |
| cfx-rda-smtp-server | 192.168.109.53 | Up 2 weeks | 12a338584d26 | 7.7 |
| cfx-rda-smtp-server | 192.168.109.54 | Up 2 weeks | 22fd2869fba4 | 7.7 |
| cfx-rda-event-consumer | 192.168.109.53 | Up 7 days | 478340dd41a3 | 7.7.1 |
| cfx-rda-event-consumer | 192.168.109.54 | Up 7 days | e2831d3b0a57 | 7.7.1 |
| cfx-rda-alert-processor | 192.168.109.53 | Up 7 days | 1445b3ed4772 | 7.7.1 |
| cfx-rda-alert-processor | 192.168.109.54 | Up 7 days | d82f035b146e | 7.7.1 |
| cfx-rda-irm-service | 192.168.109.53 | Up 7 days | 3a9485387e24 | 7.7.1 |
| cfx-rda-irm-service | 192.168.109.54 | Up 7 days | e089f3ce972f | 7.7.1 |
| cfx-rda-ml-config | 192.168.109.53 | Up 2 weeks | 872bbca736c8 | 7.7 |
| cfx-rda-ml-config | 192.168.109.54 | Up 2 weeks | 09cd07430bae | 7.7 |
| cfx-rda-collaboration | 192.168.109.53 | Up 7 days | 64efbb9bde20 | 7.7.1 |
| cfx-rda-collaboration | 192.168.109.54 | Up 7 days | ebceebe572b0 | 7.7.1 |
+-----------------------------------+----------------+------------+--------------+-------+
Run the below command to verify all OIA application services are up and running.
+-------+----------------------------------------+-------------+----------------+----------+-------------+----------+--------+--------------+---------------+--------------+
| Cat | Pod-Type | Pod-Ready | Host | ID | Site | Age | CPUs | Memory(GB) | Active Jobs | Total Jobs |
|-------+----------------------------------------+-------------+----------------+----------+-------------+----------+--------+--------------+---------------+--------------|
| App | alert-ingester | True | rda-alert-inge | 6a6e464d | | 19:22:36 | 8 | 31.33 | | |
| App | alert-ingester | True | rda-alert-inge | 7f6b42a0 | | 19:22:53 | 8 | 31.33 | | |
| App | alert-processor | True | rda-alert-proc | a880e491 | | 19:23:21 | 8 | 31.33 | | |
| App | alert-processor | True | rda-alert-proc | b684609e | | 19:23:18 | 8 | 31.33 | | |
| App | alert-processor-companion | True | rda-alert-proc | 874f3b33 | | 19:22:24 | 8 | 31.33 | | |
| App | alert-processor-companion | True | rda-alert-proc | 70cadaa7 | | 19:22:05 | 8 | 31.33 | | |
| App | asset-dependency | True | rda-asset-depe | bde06c15 | | 19:47:50 | 8 | 31.33 | | |
| App | asset-dependency | True | rda-asset-depe | 47b9eb02 | | 19:47:38 | 8 | 31.33 | | |
| App | authenticator | True | rda-identity-d | faa33e1b | | 19:47:52 | 8 | 31.33 | | |
| App | authenticator | True | rda-identity-d | 36083c36 | | 19:47:46 | 8 | 31.33 | | |
| App | cfx-app-controller | True | rda-app-contro | 5fd3c3f4 | | 19:23:09 | 8 | 31.33 | | |
| App | cfx-app-controller | True | rda-app-contro | d66e5ce8 | | 19:22:56 | 8 | 31.33 | | |
| App | cfxdimensions-app-access-manager | True | rda-access-man | ecbb535c | | 19:47:46 | 8 | 31.33 | | |
| App | cfxdimensions-app-access-manager | True | rda-access-man | 9a05db5a | | 19:47:36 | 8 | 31.33 | | |
| App | cfxdimensions-app-collaboration | True | rda-collaborat | 61b3c53b | | 19:22:18 | 8 | 31.33 | | |
| App | cfxdimensions-app-collaboration | True | rda-collaborat | 09b9474e | | 19:21:57 | 8 | 31.33 | | |
| App | cfxdimensions-app-file-browser | True | rda-file-brows | 00495640 | | 19:22:45 | 8 | 31.33 | | |
| App | cfxdimensions-app-file-browser | True | rda-file-brows | 640f0653 | | 19:22:29 | 8 | 31.33 | | |
| App | cfxdimensions-app-irm_service | True | rda-irm-servic | 27e345c5 | | 19:21:43 | 8 | 31.33 | | |
| App | cfxdimensions-app-irm_service | True | rda-irm-servic | 23c7e082 | | 19:21:56 | 8 | 31.33 | | |
| App | cfxdimensions-app-notification-service | True | rda-notificati | bbb5b08b | | 19:23:20 | 8 | 31.33 | | |
| App | cfxdimensions-app-notification-service | True | rda-notificati | 9841bcb5 | | 19:23:02 | 8 | 31.33 | | |
+-------+----------------------------------------+-------------+----------------+----------+-------------+----------+--------+--------------+---------------+--------------+
Run the below command to check services has ok status and does not throw any failure messages.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------|
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-status | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | minio-connectivity | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | service-initialization-status | ok | |
| rda_app | alert-ingester | 7f75047e9e44 | daa8c414 | | kafka-connectivity | ok | Cluster=NTc1NWU1MTQxYmY3MTFlZg, Broker=1, Brokers=[1, 2, 3] |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-status | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | minio-connectivity | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-dependency:configuration-service | ok | 2 pod(s) found for configuration-service |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | service-initialization-status | ok | |
| rda_app | alert-ingester | f9ec55862be0 | f9b9231c | | kafka-connectivity | ok | Cluster=NTc1NWU1MTQxYmY3MTFlZg, Broker=2, Brokers=[1, 2, 3] |
| rda_app | alert-processor | c6cc7b04ab33 | b4ebfb06 | | service-status | ok | |
| rda_app | alert-processor | c6cc7b04ab33 | b4ebfb06 | | minio-connectivity | ok | |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------------+