Upgrade from 8.1.0.1 to 8.1.1 Using Offline Bundles.
1. Upgrade From 8.1.0.1 to 8.1.1 Using Offline Bundles
-
RDAF Platform: From 8.1.0.1 to 8.1.1
-
OIA (AIOps) Application: From 8.1.0.1 to 8.1.1
-
RDAF Deployment
rdafCLI: From 1.4.1 to 1.4.2 -
RDAF Client
rdacCLI: From 8.1.0.1 to 8.1.1
2. Prerequisites
Before proceeding with this upgrade, please make sure and verify the below prerequisites are met.
-
RDAF Deployment CLI version: 1.0.4
-
Infra Services tag: 1.0.4
-
Platform Services and RDA Worker tag: 8.1.0.1
-
OIA Application Services tag: 8.1.0.1
-
CloudFabrix recommends taking VMware VM snapshots where RDA Fabric infra/platform/applications are deployed
Important
-
If the Webhook URL is currently configured with port 7443, it should be updated to port 443. Below are the steps to update Webhook URL:
-
Login to UI → Click on Administration → Organization → click on Configure → click on Alert Endpoints → click on required Endpoint and edit to update the port (Note: Please ignore if the Alert Endpoint is already configured with port 443)
3. Upgrade Steps
Important
Ensure that the highlighted disk size (Use% column) should be less than 50%. If any disk exceeds this percentage, consider taking appropriate actions like cleaning up unused files or expanding storage.
rdauser@oia-125-216:~/collab-3.7-upgrade$ df -kh
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 357M 6.0G 6% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 48G 12G 34G 26% /
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/loop0 64M 64M 0 100% /snap/core20/2318
/dev/loop2 92M 92M 0 100% /snap/lxd/24061
/dev/sda2 1.5G 309M 1.1G 23% /boot
/dev/sdf 50G 3.8G 47G 8% /var/mysql
/dev/loop3 39M 39M 0 100% /snap/snapd/21759
/dev/sdg 50G 541M 50G 2% /minio-data
/dev/loop4 92M 92M 0 100% /snap/lxd/29619
/dev/loop5 39M 39M 0 100% /snap/snapd/21465
/dev/sde 15G 140M 15G 1% /zookeeper
/dev/sdd 30G 884M 30G 3% /kafka-logs
/dev/sdc 50G 3.3G 47G 7% /opt
/dev/sdb 50G 29G 22G 57% /var/lib/docker
/dev/sdi 25G 294M 25G 2% /graphdb
/dev/sdh 50G 34G 17G 68% /opensearch
/dev/loop6 64M 64M 0 100% /snap/core20/2379
3.1 RDAF Deployment CLI Upgrade
Note
Upgrade the RDAF Deployment CLI on both the on-premise Docker registry VM and the RDAF Platform's management VM if they are provisioned separately.
Perform these steps on the VM where the RDAF Deployment CLI was installed, whether it is managing Docker registry or non-Kubernetes deployments.
Note
This process ensures the RDAF CLI is upgraded without requiring internet access on the management VM.
- Download the RDAF Deployment CLI's newer version 1.4.2 bundle and copy it to RDAF management VM on which
rdafdeployment CLI was installed.
wget https://macaw-amer.s3.us-east-1.amazonaws.com/releases/rdaf-platform/1.4.2/offline-ubuntu-1.4.2.tar.gz
- Extract the
rdafCLI software bundle contents
- Change the directory to the extracted directory
- Upgrade the
rdafCLI to version 1.4.2
- Verify the installed
rdafCLI version
3.2 Upgrade On-Prem Registry
- Please download the below python script (
rdaf_upgrade_141_142.py)
wget https://macaw-amer.s3.us-east-1.amazonaws.com/releases/rdaf-platform/1.4.2/rdaf_upgrade_141_142.py
The below step will generate values.yaml.latest files for all RDAF Infrastructure, Platform and Application services in the /opt/rdaf/deployment-scripts directory.
- Please run the downloaded python upgrade script rdaf_upgrade_141_142.py as shown below
Note
The above command will show the available options for the upgrade script
usage: rdaf_upgrade_141_142.py [-h] {upgrade,haproxy_upgrade,cleanup_haproxy} ...
options:
-h, --help show this help message and exit
options:
{upgrade,haproxy_upgrade,cleanup_haproxy}
Available options
upgrade upgrade the setup
haproxy_upgrade Upgrade HAProxy with VIP
cleanup_haproxy cleaning up haproxy and keepalived
- Please run the downloaded python upgrade script rdaf_upgrade_141_142.py as shown below
rdauser@hari-infra13360:~$ python rdaf_upgrade_141_142.py upgrade
cleaning up expiring certificates...
Cleanup complete!
cleaning up expiring certificates...
Cleanup complete!
Updating policy json configuration.
Creating backup policy.json
Encrypting policy user credentials.
Updating the policy.json in platform and service hosts.
Copying policy.json to hosts: 192.168.133.63
Copying policy.json to hosts: 192.168.133.66
Copying policy.json to hosts: 192.168.133.65
Copying policy.json to hosts: 192.168.133.64
Updating the opensearch tenant user permissions...
{"status":"OK","message":"'role-74f772b55ef14890929b7857d20766be-dataplane-policy' updated."}
{"status":"OK","message":"'role-74f772b55ef14890929b7857d20766be' updated."}
Creating backup of existing haproxy.cfg on host 192.168.133.60
Updating haproxy configs on host 192.168.133.60..
Creating backup of existing haproxy.cfg on host 192.168.133.61
Updating haproxy configs on host 192.168.133.61..
Copied /opt/rdaf/deployment-scripts/worker.yaml to /opt/rdaf/deployment-scripts/192.168.133.65
Copied /opt/rdaf/deployment-scripts/worker.yaml to /opt/rdaf/deployment-scripts/192.168.133.66
Copying /opt/rdaf/rdaf.cfg to host 192.168.133.61
Creating directory /opt/rdaf/config/runtime and setting ownership to user 1000 and group to group 1000 on host 192.168.133.61
Copying /opt/rdaf/rdaf.cfg to host 192.168.133.63
Creating directory /opt/rdaf/config/runtime and setting ownership to user 1000 and group to group 1000 on host 192.168.133.63
Copying /opt/rdaf/rdaf.cfg to host 192.168.133.65
Creating directory /opt/rdaf/config/runtime and setting ownership to user 1000 and group to group 1000 on host 192.168.133.65
Copying /opt/rdaf/rdaf.cfg to host 192.168.133.62
Creating directory /opt/rdaf/config/runtime and setting ownership to user 1000 and group to group 1000 on host 192.168.133.62
Copying /opt/rdaf/rdaf.cfg to host 192.168.133.66
Creating directory /opt/rdaf/config/runtime and setting ownership to user 1000 and group to group 1000 on host 192.168.133.66
Copying /opt/rdaf/rdaf.cfg to host 192.168.133.64
Creating directory /opt/rdaf/config/runtime and setting ownership to user 1000 and group to group 1000 on host 192.168.133.64
backing up existing values.yaml..
Removing rda_asset_dependency and AIA entries from the values.yaml file
[+] Stopping 1/1
✔ Container platform-rda_asset_dependency-1 Stopped 10.5s
Going to remove platform-rda_asset_dependency-1
[+] Removing 1/0
✔ Container platform-rda_asset_dependency-1 Removed 0.0s
Removing rda_asset_dependency entries from the platform_yaml
[+] Stopping 1/1
✔ Container platform-rda_asset_dependency-1 Stopped 10.6s
Going to remove platform-rda_asset_dependency-1
[+] Removing 1/0
✔ Container platform-rda_asset_dependency-1 Removed 0.0s
Removing rda_asset_dependency entries from the platform_yaml
backing up existing nats.conf on host 192.168.133.60
JetStream section removed successfully.
backing up existing nats.conf on host 192.168.133.61
JetStream section removed successfully.
The upgrade script makes the following changes:
- OpenSearch Certificate Cleanup
Cleans up expired OpenSearch certificates.
Connects to all VMs via SSH to perform the cleanup.
- Policy File Update
Copies policy.json to /opt/rdaf/config/policy.json on platform and service hosts.
Takes backup of the existing policy.json.
Updates policy user credentials within the file.
- IP Address Directory Creation
Creates a directory for each platform and worker host at /opt/rdaf/deployment-scripts/192.168.xx.xx.
Moves corresponding YAML files into their respective IP address directories.
- Runtime Folder Creation
Creates an empty runtime folder at /opt/rdaf/config.
- AIA Dependency Removal
Removes AIA dependency configuration from values.yaml.
- Asset Dependency Service Removal
Removes the asset-dependency service entry from platform.yaml.
- NATS JetStream Removal
Removes the JetStream configuration section from /opt/rdaf/config/nats.conf.
- HAProxy Configuration Update
Creates a backup of the existing haproxy.cfg file.
Updates /opt/rdaf/config/haproxy/haproxy.cfg with the following configuration under backend webhook:
backend webhook
mode http
balance roundrobin
stick-table type ip size 10k expire 10m
stick on src
option httpchk GET /healthcheck
http-check expect rstatus (2|3)[0-9][0-9]
http-check disable-on-404
http-response set-header Cache-Control no-store
http-response set-header Pragma no-cache
default-server inter 10s downinter 5s fall 3 rise 2
cookie SERVERID insert indirect nocache maxidle 30m maxlife 24h httponly secure
server rdaf-webhook-1 192.168.108.51:8888 check cookie rdaf-webhook-1
server rdaf-webhook-2 192.168.108.52:8888 check cookie rdaf-webhook-2
- Portal Backend Update in
values.yaml
File path: /opt/rdaf/deployment-scripts/values.yaml
Updates the portal-backend environment variables section to be dynamically injected via CLI instead of hardcoded:
portal-backend:
mem_limit: 4G
memswap_limit: 4G
environment:
CFX_URL_PREFIX: ''
DATABASE_SQLALCHEMY_POOL_SIZE: 10
DATABASE_SQLALCHEMY_MAX_OVERFLOW: 10
deployment: true
cap_add:
- SYS_PTRACE
privileged: true
Important
Add the geodr_api_server service section after the api_server section in the values.yaml file.
- Add
geodr_api_serverService invalues.yaml
rda_geodr_api_server:
mem_limit: 2G
memswap_limit: 2G
privileged: true
cap_add:
- SYS_PTRACE
environment:
RDA_ENABLE_TRACES: 'no'
DISABLE_REMOTE_LOGGING_CONTROL: 'no'
RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
deployment: false
hosts:
- 192.168.108.51
- 192.168.108.52
Note
For the FSM Service below, the parameters highlighted in yellow must be updated manually.
- FSM Environment Updates in
values.yaml
File path: /opt/rdaf/deployment-scripts/values.yaml
Under rda_fsm service, the value PURGE_STALE_INSTANCES_DAYS is updated from 120 to 90.
Adds a new environment variable FSM_INSTANCE_CACHE_SIZE with value 2000
rda_fsm:
mem_limit: 4G
memswap_limit: 4G
privileged: true
cap_add:
- SYS_PTRACE
environment:
RDA_ENABLE_TRACES: 'yes'
DISABLE_REMOTE_LOGGING_CONTROL: 'no'
RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
PURGE_COMPLETED_INSTANCES_DAYS: 1
PURGE_STALE_INSTANCES_DAYS: 90
FSM_INSTANCE_CACHE_SIZE: 2000
KAFKA_CONSUMER_BATCH_MAX_SIZE: 100
KAFKA_CONSUMER_BATCH_MAX_TIME_SECONDS: 1
KAFKA_CONSUMER_BATCH_MAX_TIME_SECONDS: 1
deployment: true
4. External OpenSearch Upgrade
Note
If an external OpenSearch is configured, ensure it is upgraded to version 1.0.4.1 by running the following commands.
- Please download the external OpenSearch package using the following command.
wget https://macaw-amer.s3.us-east-1.amazonaws.com/releases/RDA/8.1.1/rda-platform-opensearch-1.0.4.1.tar.gz
Note
Make sure all the above tar files are copied to /opt/rdaf-registry/import/ on the VM hosting RDAF registry server role.
- Use the following command to import external OpenSearch package or tag file into the registry.
- To upgrade External Opensearch please use these following command.
- To check the status of External OpenSearch, use the following command.
+---------------------+-----------------+------------+--------------+---------+
| Name | Host | Status | Container Id | Tag |
+---------------------+-----------------+------------+--------------+---------+
| opensearch_external | 192.168.107.187 | Up 34 hours | 6fb1babd1e05 | 1.0.4.1|
| opensearch_external | 192.168.107.188 | Up 34 hours | 95a8a7b61135 | 1.0.4.1|
| opensearch_external | 192.168.107.189 | Up 34 hours | dc776fc0adb6 | 1.0.4.1|
+---------------------+-----------------+------------+--------------+---------+
5. Fetching Latest Tags
To update to the latest tags, ensure that all files are copied into the directory /opt/rdaf-registry/import.
Note
Infrastructure files are imported separately, while platform and application files are managed using packaged tar files.
- Use the following command to download the comprehensive package containing all necessary platform components.
- Execute this command to obtain the full package required for OIA integration.
- Retrieve the full package needed for on-premises setup using the command below.
6. Running the Registry Import to Fetch Tags
- Use the following command to import each package or tag file into the registry.
rdaf registry import --file All-onprem.tar.gz
rdaf registry import --file All-Platform.tar.gz
rdaf registry import --file All-OIA.tar.gz
rdaf registry import --file mc-RELEASE.2024-11-21T17-21-54Z.tar
rdaf registry import --file minio.tar.gz
rdaf registry import --file rda-platform-arangodb-starter.tar.gz
rdaf registry import --file rda-platform-arangodb.tar.gz
rdaf registry import --file rda-platform-busybox.tar.gz
rdaf registry import --file rda-platform-filebeat.tar.gz
rdaf registry import --file rda-platform-haproxy.tar.gz
rdaf registry import --file rda-platform-kafka.tar.gz
rdaf registry import --file rda-platform-kube-arangodb.tar.gz
rdaf registry import --file rda-platform-kubectl.tar.gz
rdaf registry import --file rda-platform-logstash.tar.gz
rdaf registry import --file rda-platform-mariadb.tar.gz
rdaf registry import --file rda-platform-nats-boot-config.tar.gz
rdaf registry import --file rda-platform-nats-box.tar.gz
rdaf registry import --file rda-platform-nats.tar.gz
rdaf registry import --file rda-platform-opensearch.tar.gz
rdaf registry import --file rda-platform-prometheus-nats-exporter.tar.gz
rdaf registry import --file rda-platform-telegraf.tar.gz
- To view the list of tags available in the registry, use the below given command
+------------------------------------------+--------------------------------------------------------------------------------+
| Service-Name | Tags |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-access-manager | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-asm | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-irm-service | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-registry | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-scheduler | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rdac-full | |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-smtp-server | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-onprem-portal | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| minio | RELEASE.2024-12-18T13-15-44Z |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-notification-service | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-nats-box | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-kafka | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-worker-all | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-haproxy | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-app-controller | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-identity | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-kube-arangodb | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-busybox | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-configuration-service | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-nats | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-client-api-server | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-opensearch | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfxcollector | |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-logstash | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| cfx-rda-collaboration | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-nats-server-config-reloader | |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-telegraf | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-prometheus-nats-exporter | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
| ubuntu-rda-event-gateway | 8.1.1 |
+------------------------------------------+--------------------------------------------------------------------------------+
| rda-platform-nats-boot-config | 1.0.4 |
+------------------------------------------+--------------------------------------------------------------------------------+
- If necessary, kindly remove outdated image tags from the on-premise registry that are no longer in use by executing the following command:
- Please proceed with the full upgrade following the instructions in this Document
Note
There's no need to execute the registry commands.