Skip to content

Copy Correlation/Suppression Policies From DB To PStream

For 8.1 Release

1. Overview

To simplify the creation of RDA packs, dashboards, snapshots, and other artifacts, suppression and correlation policies should be copied from the database to PStream.

2. Prerequisites for Executing the Script

  • Check for Existing PStream

    - Navigate to Main Menu → Configuration → RDA Administration → Persistent Streams to verify if the oia_alertpolicy PStream already exists.

  • Create the PStream (if not found)

    - If the oia_alertpolicy PStream is missing, create it by selecting Add under Main Menu → Configuration → RDA Administration → Persistent Streams, and provide the appropriate definition.

Click to view the definition of oia_alertpolicy pstream 

    {
        "unique_keys": [
            "id"
        ],
        "_mappings": {
            "properties": {
                "createdat": {
                    "type": "date"
                },
                "updatedat": {
                    "type": "date"
                },
                "repeatendsat": {
                    "type": "date"
                },
                "enableat": {
                    "type": "date"
                },
                "disableat": {
                    "type": "date"
                }
            }
        },
        "default_values": {},
        "case_insensitive": true
    }

  • Script needs to be executed inside a service container. Preferably configuration-service which has less amount of load

  • Download CopyPoliciesFromDBToPStream.py script from below given location

wget https://macaw-amer.s3.us-east-1.amazonaws.com/releases/rdaf-platform/1.4.1/copy_policies_from_db_to_pstream_job.tar
  • Use the following command to untar the downloaded file copy_policies_from_db_to_pstream_job.tar.
tar -xvf copy_policies_from_db_to_pstream_job.tar
  • Change the directory to copy_policies_from_db_to_pstream_job
cd copy_policies_from_db_to_pstream_job
  • The copy_policies_from_db_to_pstream_job directory contains the following scripts, This script copies policies from the alertpolicy table in the database to the oia_alertpolicy PStream., use the command mentioned below.
CopyPoliciesFromDBToPStream.py
  • copy the downloaded file inside the configuration-service container. Command
docker cp CopyPoliciesFromDBToPStream.py <configuration-service-container-id>:/tmp

3. Command To Execute Script

  • Command to copy data to from alertpolicy table to oia_alertpolicy pstream
python CopyPoliciesFromDBToPStream.py

Note

If the script fails for any reason, simply re-run it to retry the data copy process.

4. Monitoring and Validation

4.1 Sample Log Output

  • Use the logs below to verify successful execution
2025-06-20 06:57:37,466 [PID=3235:TID=MainThread:__main__:copyToDBData:922] INFO - Started getting policies data from DB.
2025-06-20 06:57:37,487 [PID=3235:TID=MainThread:__main__:copyToDBData:942] INFO - Number of policies fetched from DB: 44
2025-06-20 06:57:37,487 [PID=3235:TID=MainThread:__main__:copyToDBData:944] INFO - Started writing policies to oia_alertpolicy Pstream.
2025-06-20 06:57:38,092 [PID=3235:TID=MainThread:__main__:copyToDBData:946] INFO - Successfully Copied policies: 44 from DB to pstream took:626 Millis