Upgrade to 3.1.0 and 7.0.6
This section provides instructions on how to upgrade existing deployment of RDAF platform and it's application OIA (Operations Intelligence & Analytics), which is also referred as AIOps.
1. Upgrade from 7.0.x to 7.0.6
Upgrade Prerequisites
Below are the pre-requisites which need to be in place before upgrafing the OIA (AIOps) application services.
RDAF Deployment CLI Version Upgrade: From 1.0.6 or higher to 1.1.2
RDAF Infrastructure Services Tag Version: From 1.0.1 or higher to 1.0.2 (Note: Not applicable if the services are already running at 1.0.2 version)
RDAF Core Platform & Worker Services Tag Version: From 3.0.9 to 3.1.0
RDAF Client (RDAC) Tag Version: From 3.0.9 to 3.1.0
- Upgrade RDAF Deployment CLI (for Kubernetes or Non-Kubernetes)
- Fetch upgrade images on Docker On-premise Registry and download all RDAF Platform's service images (Infrastructure, Core Platform, Application and Worker services)
Warning
Please complete all of the above pre-requisites before installing the OIA (AIOps) application services.
On-premise docker-registry
Login into RDAF on-premise docker-registry VM or RDAF platform VM as rdauser using SSH client on which rdaf CLI was installed and run the below command to verify status of the docker-registry service.
+-----------------+---------------+------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+-----------------+---------------+------------+--------------+-------+
| docker-registry | 111.92.12.140 | Up 4 weeks | 71b8036fc64f | 1.0.1 |
+-----------------+---------------+------------+--------------+-------+
RDAF Infrastructure, Platform and Application services:
Login into RDAF on-premise docker-registry VM or RDAF platform VM as rdauser using SSH client on which rdaf CLI was installed and run the below command to verify status of the RDAF platform's infrastructure, core platform, application and worker services.
+----------------+--------------+-----------------+--------------+------------------------------+
| Name | Host | Status | Container Id | Tag |
+----------------+--------------+-----------------+--------------+------------------------------+
| haproxy | 111.92.12.41 | Up 6 days | 245a37201207 | 1.0.2 |
| keepalived | 111.92.12.41 | Not Provisioned | N/A | N/A |
| nats | 111.92.12.41 | Up 6 days | 15469a93d96f | 1.0.2 |
| minio | 111.92.12.41 | Up 6 days | 3fd3f97bf25b | RELEASE.2022-11-07T23-47-39Z |
| mariadb | 111.92.12.41 | Up 6 days | 0fa1a0027993 | 1.0.2 |
| opensearch | 111.92.12.41 | Up 6 days | dae308716400 | 1.0.2 |
| zookeeper | 111.92.12.41 | Up 6 days | 4d8f61b4ab17 | 1.0.2 |
| kafka | 111.92.12.41 | Up 6 days | 0dee08cd9c59 | 1.0.2 |
| redis | 111.92.12.41 | Up 6 days | d1eccf90846e | 1.0.2 |
| redis-sentinel | 111.92.12.41 | Up 6 days | 683beb7b913e | 1.0.2 |
+----------------+--------------+-----------------+--------------+------------------------------+
+--------------------------+--------------+-----------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+--------------------------+--------------+-----------+--------------+-------+
| cfx-rda-access-manager | 111.92.12.41 | Up 6 days | e487cdf24b46 | 3.0.9 |
| cfx-rda-resource-manager | 111.92.12.41 | Up 6 days | a7a21a31a26e | 3.0.9 |
| cfx-rda-user-preferences | 111.92.12.41 | Up 6 days | 9306d8da4b5a | 3.0.9 |
| portal-backend | 111.92.12.41 | Up 6 days | 55df761dad1d | 3.0.9 |
| portal-frontend | 111.92.12.41 | Up 6 days | 2183f00efa64 | 3.0.9 |
| rda_api_server | 111.92.12.41 | Up 6 days | 3ba6256d1694 | 3.0.9 |
| rda_asset_dependency | 111.92.12.41 | Up 6 days | d1a8b76bb114 | 3.0.9 |
| rda_collector | 111.92.12.41 | Up 6 days | 441427d2bb1e | 3.0.9 |
| rda_identity | 111.92.12.41 | Up 6 days | 2c1215d9155a | 3.0.9 |
| rda_registry | 111.92.12.41 | Up 6 days | 7358e6ee6298 | 3.0.9 |
| rda_scheduler | 111.92.12.41 | Up 6 days | ee72c66f8c80 | 3.0.9 |
+--------------------------+--------------+-----------+--------------+-------+
+------------+--------------+-----------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+------------+--------------+-----------+--------------+-------+
| rda_worker | 111.92.12.43 | Up 6 days | 88f4916ce18e | 3.0.9 |
| rda_worker | 111.92.12.43 | Up 6 days | 88f491612345 | 3.0.9 |
+------------+--------------+-----------+--------------+-------+
+------------------------------+--------------+-----------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+------------------------------+--------------+-----------+--------------+-------+
| all-alerts-cfx-rda-dataset- | 111.92.12.42 | Up 6 days | 58a75c01c51f | 7.0.5 |
| caas | | | | |
| cfx-rda-alert-ingester | 111.92.12.42 | Up 6 days | bc9a78953b73 | 7.0.5 |
| cfx-rda-alert-processor | 111.92.12.42 | Up 6 days | 28401e5c2570 | 7.0.5 |
| cfx-rda-app-builder | 111.92.12.42 | Up 6 days | be8f100056fd | 7.0.5 |
| cfx-rda-app-controller | 111.92.12.42 | Up 6 days | a7a4ef35097d | 7.0.5 |
| cfx-rda-collaboration | 111.92.12.42 | Up 6 days | d9d980b28a2b | 7.0.5 |
| cfx-rda-configuration- | 111.92.12.42 | Up 6 days | db1a45835e1a | 7.0.5 |
| service | | | | |
| cfx-rda-event-consumer | 111.92.12.42 | Up 6 days | baf09bad3ce1 | 7.0.5 |
| cfx-rda-file-browser | 111.92.12.42 | Up 6 days | 32ccdfca8d8f | 7.0.5 |
| cfx-rda-ingestion-tracker | 111.92.12.42 | Up 6 days | 1030345f2179 | 7.0.5 |
| cfx-rda-irm-service | 111.92.12.42 | Up 6 days | 89d931f7d7b8 | 7.0.5 |
| cfx-rda-ml-config | 111.92.12.42 | Up 6 days | 57fc39489a08 | 7.0.5 |
| cfx-rda-notification-service | 111.92.12.42 | Up 6 days | 408dbebb33c5 | 7.0.5 |
| cfx-rda-reports-registry | 111.92.12.42 | Up 6 days | 3296cba8b3e4 | 7.0.5 |
| cfx-rda-smtp-server | 111.92.12.42 | Up 6 days | 0f9884b6e7c8 | 7.0.5 |
| cfx-rda-webhook-server | 111.92.12.42 | Up 6 days | a4403dee414e | 7.0.5 |
| current-alerts-cfx-rda- | 111.92.12.42 | Up 6 days | d6cc63214103 | 7.0.5 |
| dataset-caas | | | | |
+------------------------------+--------------+-----------+--------------+-------+
Important
Please take RDAF platform's full data backup before performing an upgrade. For more information on RDAF platform's backup and restore commands using `rdaf` CLI, please refer at <a href="https://docs.fabrix.ai/installation_guides/rdaf_cli/#1310-rdaf-backup" title="RDAF Platform Backup">RDAF Platform Backup</a>
Download RDAF Platform & OIA Images
- Login into On-premise docker registry VM as
rdauserusing SSH client and run the below command to download RDAF platform's and OIA (AIOps) application service's updated images.
- Please wait until all of the RDAF platform's and OIA (AIOps) application service's images are downloaded. Run the below command to verify if the images are downloaded appropriately.
Upgrade RDAF deployment CLI on RDAF Platform VM
Please follow and repeat the steps to download and upgrade the rdaf deployment CLI on RDAF platform VM using the steps outlined under RDAF CLI Upgrade on On-premise docker registry VM section.
Upgrade RDAF Platform & OIA Services
RDAF Platform Services Upgrade:
Run the below command to upgrade RDAF platform's services to version
Once above command is completed, run the below command to verify all of RDAF platform's services are upgraded to the specified version and all of their corresponding containers are in running state.
RDAF Client CLI Upgrade:
Run the below command to upgrade the RDAF client CLI rdac to latest version.
After rdac CLI is upgraded, run the below commands to see all of the running RDAF platform's services pods.
+-------+----------------------------------------+--------------+----------+-------------+-----------------+--------+--------------+---------------+--------------+
| Cat | Pod-Type | Host | ID | Site | Age | CPUs | Memory(GB) | Active Jobs | Total Jobs |
|-------+----------------------------------------+--------------+----------+-------------+-----------------+--------+--------------+---------------+--------------|
| App | cfxdimensions-app-access-manager | d412efb99f2e | ccb83d20 | | 4:13:45 | 8 | 31.21 | | |
| App | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de | | 1 day, 18:33:27 | 8 | 31.21 | | |
| App | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | 4:13:31 | 8 | 31.21 | | |
| App | user-preferences | 520bca813ddf | f4ca7d44 | | 4:13:14 | 8 | 31.21 | | |
| Infra | api-server | 0656b4230f44 | 6d4d40ab | | 0:33:06 | 8 | 31.21 | | |
| Infra | collector | 6336341682ad | 042af0af | | 4:11:19 | 8 | 31.21 | | |
| Infra | registry | cae649622fba | 4e4c4a4d | | 4:11:03 | 8 | 31.21 | | |
| Infra | scheduler | 3ab379305be1 | b2bb9915 | *leader* | 4:10:59 | 8 | 31.21 | | |
+-------+----------------------------------------+--------------+----------+-------------+-----------------+--------+--------------+---------------+--------------+
Run the below command to verify functional health of each platform's service and verify all of their status is in OK state.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------|
| rda_infra | api-server | 0656b4230f44 | 6d4d40ab | | service-status | ok | |
| rda_infra | api-server | 0656b4230f44 | 6d4d40ab | | minio-connectivity | ok | |
| rda_app | asset-dependency | e006dfd39d9b | 9f02a8f1 | | service-status | ok | |
| rda_app | asset-dependency | e006dfd39d9b | 9f02a8f1 | | minio-connectivity | ok | |
| rda_app | authenticator | 1782a79e36c5 | adda9bc0 | | service-status | ok | |
| rda_app | authenticator | 1782a79e36c5 | adda9bc0 | | minio-connectivity | ok | |
| rda_app | authenticator | 1782a79e36c5 | adda9bc0 | | DB-connectivity | ok | |
| rda_app | cfxdimensions-app-access-manager | d412efb99f2e | ccb83d20 | | service-status | ok | |
| rda_app | cfxdimensions-app-access-manager | d412efb99f2e | ccb83d20 | | minio-connectivity | ok | |
| rda_app | cfxdimensions-app-access-manager | d412efb99f2e | ccb83d20 | | service-dependency:registry | ok | 1 pod(s) found for registry |
| rda_app | cfxdimensions-app-access-manager | d412efb99f2e | ccb83d20 | | service-initialization-status | ok | |
| rda_app | cfxdimensions-app-access-manager | d412efb99f2e | ccb83d20 | | DB-connectivity | ok | | |
| rda_app | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de | | service-status | ok | |
| rda_app | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de | | minio-connectivity | ok | |
| rda_app | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de | | service-initialization-status | ok | |
| rda_app | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de | | DB-connectivity | ok | |
| rda_app | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | service-status | ok | |
| rda_app | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | minio-connectivity | ok | |
| rda_app | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | service-dependency:registry | ok | 1 pod(s) found for registry |
| rda_app | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | service-dependency:cfxdimensions-app-access-manager | ok | 1 pod(s) found for cfxdimensions-app-access-manager |
| rda_app | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | service-initialization-status | ok | |
| rda_app | cfxdimensions-app-resource-manager | ec87d2ee6387 | 33ee28ca | | DB-connectivity | ok | |
| rda_infra | collector | 6336341682ad | 042af0af | | service-status | ok | |
| rda_infra | collector | 6336341682ad | 042af0af | | minio-connectivity | ok | |
| rda_infra | collector | 6336341682ad | 042af0af | | opensearch-connectivity:default | ok | |
| rda_infra | scheduler | 3ab379305be1 | b2bb9915 | | service-status | ok | |
| rda_infra | scheduler | 3ab379305be1 | b2bb9915 | | minio-connectivity | ok | |
| rda_infra | scheduler | 3ab379305be1 | b2bb9915 | | DB-connectivity | ok | |
| rda_app | user-preferences | 520bca813ddf | f4ca7d44 | | service-status | ok | |
| rda_app | user-preferences | 520bca813ddf | f4ca7d44 | | minio-connectivity | ok | |
| rda_app | user-preferences | 520bca813ddf | f4ca7d44 | | service-dependency:registry | ok | 1 pod(s) found for registry |
| rda_app | user-preferences | 520bca813ddf | f4ca7d44 | | service-initialization-status | ok | |
| rda_app | user-preferences | 520bca813ddf | f4ca7d44 | | DB-connectivity | ok | | |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
RDAF Worker Service Upgrade:
Run the below command to upgrade RDAF worker services to latest version.
After upgrading the RDAF worker service using the above command, run the below command to verify it's running status and the version.
+------------+--------------+-------------+--------------+-------+
| Name | Host | Status | Container Id | Tag |
+------------+--------------+-------------+--------------+-------+
| rda_worker | 111.92.12.60 | Up 1 minute | 4ce2a8f13d16 | 3.1.0 |
+------------+--------------+-------------+--------------+-------+
Run the below command to verify the functional health of the each RDAF worker service and verify that all of their status is in OK state.
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
| Cat | Pod-Type | Host | ID | Site | Health Parameter | Status | Message |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------|
| rda_infra | api-server | 0656b4230f44 | 6d4d40ab | | service-status | ok | |
...
...
| rda_infra | worker | 4ce2a8f13d16 | d627124d | rda-site-01 | service-status | ok | |
| rda_infra | worker | 4ce2a8f13d16 | d627124d | rda-site-01 | minio-connectivity | ok | |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
Create Kafka Topics for OIA Application Services:
Download the below script and execute it on where rdafk8s setup was run during the initial RDAF platform setup. Please make sure the file /opt/rdaf/rdaf.cfg exist which is required for the below script to execute successfully.
RDAF OIA Application Services Upgrade:
Run the below command to upgrade the RDAF OIA (AIOps) application services to latest version.
Once above command is completed, run the below command to verify all of the RDAF OIA application services are upgraded to the specified version and all of their corresponding containers are in running state.
Wait for 3 to 5 minutes and run the below command to verify the functional health of each RDAF OIA application service and verify all of their status is in OK state.