Redundancy

Server redundancy facilitates the following behavior:

  • Automatic failover of UNP, providing high-availability and resiliency for your notification platform. The secondary servers stay in sync with the UNP primary server. In the case of a server failover, sending notifications moves to the secondary server until the primary server is back online.
  • Distributed notifications, providing you with the ability to specify, through the use of priority groups of triggers, which notifications are sent through which servers.

Redundancy is a licensed feature.

Note: The redundancy environment for UNP is set up as a high-availability cluster, also referred to as active/active, which uses multiple servers that provide continued service in cases where system components fail. Traffic intended for a failed server is passed to the next priority server, as defined by your Priority Groups.

In redundancy mode, a primary and secondary relationship is maintained to provide data replication between servers, where the primary server maintains the publisher database and the secondary servers are subscriber databases.

Redundancy is activated by enabling the feature during installation of UNP on any secondary server. Load balancing allows SIP trunks to redirect to a secondary server. It also automatically redirects users to the UNP web interface on a secondary server.

This section describes use models for how you can set up redundancy in your UNP environment.

Use this use model when you want to ensure that your notifications continue if your primary server goes down.

Failover model notification flow

How to set this up?

  1. Install at least two servers in a redundancy setup. See Install and configure UNP for instructions
  2. Set up your default Priority Groups to define the failover order.
  3. You do not need to adjust any triggers, as they automatically use the default Priority Group. Therefore, upon failover, your notifications will continue on the next available server.

What happens?

  1. Set up your UNP environment on the Primary server, which creates a Publisher database.
  2. UNP regularly synchronizes your Publisher database with the subscriber databases on your secondary servers
  3. A trigger occurs, where the default Priority Group defines which server should send the notifications.
  4. UNP sends the notification to the defined endpoints.
  5. In a failover situation, UNP routes the notification through the next server listed in the Priority Group.

Use this model when you want to distribute notifications such that they are sent through a server local to a location.

For example, you run a school district and you want notifications specific to a school to be sent through a server located at the same location as the endpoints receiving them.

Distributed multi-location notification flow

How would you set this up?

  1. Install at least two servers in a redundancy setup. See Install and configure UNP for instructions.

    For example: DistrictOffice (Primary), Elementary (Secondary), HighSchool (Secondary)

  2. For this example, set up three Priority Groups:
    1. DEFAULT: DistrictOffice, Elementary, HighSchool
    2. ElementaryGroup: Elementary, DistrictOffice, HighSchool
    3. HighSchoolGroup: HighSchool, DistrictOffice, Elementary

  3. You will want to create triggers that are specific to the various sites (sometimes creating multiple variations of the same trigger) such that the priority groups are defined to have the proper server listed first.

    For example, You might set your CAP triggers, for natural disaster notifications, and bell schedules to always use the server located on site.

    Therefore, you would create two versions of all your CAP triggers, such that there is one for each school location. For your bell schedule, you would set the Priority Groups field for the location-specific calendars to use either the Elementary or High School priority groups.

  4. You will then create notifications, that match the location-specific trigger to the endpoints at the specific location.

What happens?

  1. Set up your UNP environment on the Primary server, creating a Publisher database.
  2. UNP regularly syncs your Publisher database with the subscriber databases on your secondary servers
  3. A trigger occurs, where the specified Priority Group defines which server should carry the notifications; Elementary school notifications go to the Elementary server. High School notifications go to the HighSchool server, and all other notifications go to DistrictOffice server.
  4. UNP sends the notification to the defined endpoints.
  5. In a failover situation, UNP sends the notification through whichever remaining server has the highest priority. For example, in the above scenario, the Elementary and High School notifications would fail over to the District Office server.
  • Redundancy is a licensed feature. If your customer license isn't authorized for redundancy, you will not be able to install UNP on any secondary server. Contact Sales for licensing.

  • SIP trunks need to be redirected to the secondary servers so that SIP activated notifications can continue to be triggered. (For example, users can continue to dial a SIP line number (SIP trigger) to activate notifications when UNP is running on the secondary server.)

  • UNP users need to be redirected to the UI on a secondary server. This is configured in your load balancer. If you're not using a load balancer, then users need to manually open the UNP UI on the secondary server. Their login credentials remain the same.
  • Network Time Protocol (NTP) must be setup to synchronize time on the primary and secondary servers.

  • Configuration changes, creating/editing/deleting triggers, notifications, user accounts, contact records, etc. are done only on the primary server. Changes on the primary server are synced to the secondary servers. The secondary servers are used for continuing to send notifications in the case of failover or when set as the first server in a priority group, see Redundancy Use Models. Therefore, even if you log into the secondary server with an admin user role, you can only view configuration settings.

    For Global Settings on a secondary server, an admin can make changes only to the following fields: Multicast (Multicast IP Address) and Network (Network Interface Select).

  • Restarting UNP is always done on the primary server first.
  • If a UNP user forgets his or her login password while UNP is running on a secondary server, the password cannot be reset until the primary server is back online.
  • The Sent Notifications page has a Server column that shows which server the notification was sent through. When you log into a secondary server, this page will only show notifications sent from this server.
  • The System Status page for the primary server has multiple tabs, one for each server, from which you can view module statuses.
  • People cannot create or modify their contact accounts through the Self-Management Portal or register their mobile devices when UNP is in failover mode. However, users and contacts can continue to send notifications from their already registered mobile devices.

The basic steps to set up redundancy are:

Load balancers can be added to a UNP environment to route incoming web requests used for registration, authentication, and GPI triggers to the correct server. When UNP servers are included in a load balanced environment, and the primary UNP server fails, the load balancer redirects incoming web requests to the UNP subscriber server until the primary server takes over again.

Some examples of software capable of HTTP load balancing are HAProxy, Nginx, and Varnish. However, there are many load balancing applications available. Your company network specialist should determine which one is appropriate for your environment.

Load balancer flow

Here's what you'll need to configure:

Have you reviewed port requirements in the System Requirements topic?

  • SQL Server – Make sure your SQL server has TCP/IP protocol enabled, which allows remote connections.
  • IP Devices – You may need to register your IP devices with your load balancer and the individual servers in order for them to receive notifications. It all depends on your device firmware. For example, AND IP speakers need to have the load balancer IP address added to its list of registered servers. Atlas, Algo, CyberData, and Valcom IP speakers send to all servers registered through DHCP option 72. To be certain as to whether your particular IP device needs to be configured with the load balancer IP address, check your IP device vendor documentation.

  • Configure SIP trunk failover – SIP trunks need to be redirected to a secondary server during failover. Without SIP trunk failover, notifications that use a SIP trigger cannot be sent; notifications that don't use a SIP trigger continue to be triggered from a secondary server. This can be done in the load balancer. Some PBX systems support configuring SIP trunk failover natively (Cisco, for example). Consult your PBX vendor documentation.

    High-level steps are:

    1. Create a new line and set ITPS Domain Name to the UNP primary server. Use the defaults for everything else. Make sure you leave Check OOS enabled. Create another new line and set ITPS Domain Name to the secondary server.
    2. Create a new ARS – Delete the short codes added by default. Add a new short code and set the Line Group ID to the line number assigned to the secondary server when you created the new line above. Create another ARS, same as you did the first one, but this time set the Line Group ID to the line number assigned to the primary server and set the Alternate Route Priority level to the ARS you just created for the secondary servers.
    3. Create a new Short Code for the SIP line pattern you assigned to the ARS.
    4. After you save, apply the changes immediately.

    Consult your Avaya documentation for step-by-step instructions.

    CUCM supports native failover.

    1. Point your CUCM authentication URL to the load balanced URL.
    2. When setting up your SIP trunk, include Route Group and Route List.

    Consult your Cisco documentation for complete instructions.

    • MiVoice Business – Go to Configuration > Phone Systems > SIP; go to Settings; enable Disable Reinvites.
    • MiVoice Connect – Follow the steps for creating a SIP trunk, except for redundancy you'll create a server, trunk group, etc. for the primary server (pub) and then again for any secondary servers (sub). Steps are outlined in the SIP trunk topic.
    • ShoreTel Director – Consult your ShoreTel documentation.
  • Configure your load balancer to check the status of UNP to determine when the primary is down. Use a GET command to this URL:

    • http://<UNP ip address>/SA-Announce/RevolutionStatus

    The UNP UI is separate from the UNP module. The UI can show that it's running when the UNP module is down. The RevolutionStatus URL provides the status of the UNP module, which is what triggers failover.

  • If you want your UNP users automatically redirected to a secondary server in failover mode, configure your load balancer with the URL for each UNP server.
  • If you have set up CAP alert push feeds, point them to the load balanced URL. You'll also need to configure the load balancer with the push URL for each UNP server.

    • Tip: Push CAP feed type is used when monitoring private systems. Most CAP emergency alerting systems are public and use a Poll CAP feed type, which don't need to be configured in the load balancer.

  • Global Settings > Network must remain at the default value of Auto-Select. This setting is only changed when multiple network cards are present. Multiple network cards are not supported when utilizing the UNP redundancy feature.

Our lab uses HAProxy. As an example, here's the code we added to the HAProxy configuration file to configure redirecting HTTP and HTTPS web requests:

frontend http

        bind 192.168.11.59:80

        option forwardfor

        default_backend bk_http

backend bk_http

 option httpchk GET /SA-Announce/RevolutionStatus

 http-check disable-on-404

 http-check expect status 200

 server s1_http 192.168.11.51:80 check

 server s2_http 192.168.11.251:80 check backup

frontend https

        bind 192.168.11.59:443

        option forwardfor

        option tcplog

        mode tcp

        default_backend bk_https

backend bk_https

 mode tcp

 option ssl-hello-chk

 server s1_https 192.168.11.51:443 check

 server s2_https 192.168.11.251:443 check backup
  • All servers must meet the standard UNP System Requirements.
  • Primary and secondary (subscriber) servers must run the same Windows OS.
  • Time must be synchronized between the primary and secondary servers. This can be achieved by pulling the time from the same source, such as Active Directory or Network Time Protocol (NTP).
  • SQL database must be on the same server as UNP. Primary and secondary servers must run the same SQL version. Remote database setup is not supported when UNP is set up for redundancy.
  • Primary and secondary servers must run the same version of UNP.

Note: Existing customers should follow the Product Upgrade steps.

Are you an existing UNP customer? Follow the Upgrade instructions below.

  • Installation and configuration is done on the primary server first. At minimum, complete the basic UNP configuration (phones, speakers, etc.), and create and test notifications on the primary server before installing UNP on any secondary server.
  • After UNP is installed on a secondary server, it syncs with the primary server and copies down your configurations. Once sync is complete, both servers display on the System Status page of the primary server under Redundancy Servers; secondary servers only show their information.
  • You cannot simultaneously install on the primary and a secondary server.
  • Primary and secondary servers use the same license and system keys.
  • Further configuration is done on the primary server, which is then copied to all secondary servers on the next sync.

Have you verified the additional system requirements needed for a redundancy environment?

Follow the standard Installation Steps for the primary server.

Once installation is complete, review the Configuration Overview topic for a list of the basic configuration steps, features you can configure, advanced configurations, and links to creating notifications.

Tip: You don't necessarily need to configure everything on the primary server before installing on the secondary server, but the basics should done so that syncing can occur after you install on the secondary server.

Before You Begin

Have you already installed and configured UNP on the primary server?

  • Copy the UNP installer to the secondary server.
  • You'll need the following information from the UNP installation on the primary server:
    • Account and Authorization IDs
    • Admin login and password you created when installing UNP on the primary server.
  • If this is an offline installation, you'll also need to copy the UNP license you used for the primary server installation on to the secondary server.

Subscriber Server Installation

  1. Do one of the following:
    • If you're already logged in as an administrator, double-click on the executable.
    • If you're not logged in as an administrator, right-click on the executable and select Run as administrator.

    The install wizard unpacks appropriate files and prompts for license agreements.

  2. Do one of the following:
    • Offline Installation – Click Browse and select the license file you copied to your server.
    • Online Installation – There's nothing to enter because the license file is automatically downloaded as part of the installation.
  3. License Check
    • Enter the Account ID you entered when installing UNP on the primary server.
    • Copy in the Authorization ID you entered when installing UNP on the primary server.
  4. Redundant Subscriber

    5. Check Is this a Redundant Subscriber Server Setup.

    (You left this unchecked when installing on the primary server. Now you need to check it to tell UNP this is the subscriber (secondary) server.)

  5. Redundancy Settings

    • Enter the IP address of the UNP primary server.

    • Enter the UNP admin login credentials you created when you installed on the primary server. (Admin login on the primary server is Admin Username on the subscriber server.)

    Note: UNP version and SQL instance, along with SQL read/write passwords, are automatically read from the primary server install.

  6. Click Install. The installer may display that it's skipping parts of the installation. This occurs when the installer detects that the proper .NET or SQL is already installed.
  7. Click Finish.
  8. Reboot the server when install completes. Windows Services default to using delayed start and will take a few minutes to come up.

To test your two-server, failover-only setup,

  1. Open Windows Services on the primary server and stop the UNP service.
  2. Log in to the secondary (subscriber) server, open the UNP UI, and verify the system status is secondary.
  3. Send a couple of test notifications from the secondary server UNP UI and check Sent Notifications to make sure they were successful.
  4. Log back in to the primary server and start the UNP service using Windows Services.
  5. Open the primary server UNP UI and check the system status is primary.
  6. Go to Sent Notifications and verify the notifications sent from the secondary server are displayed on the Secondary tab.
  7. Send a couple of test notifications from the primary server UNP UI and go to Sent Notifications to verify they were successful.

 

Note: For customers who have moved from SA-Announce to UNP, please note this difference in upgrade order. In SA-Announce, secondary (subscriber) servers were upgraded before primary (publisher) server.

You already have UNP installed in a standalone environment (non-redundant mode) and now want to implement redundancy.

Important: It's important to use your existing UNP server as the primary server in your redundancy environment so that your existing configuration is preserved. Once UNP is installed on the secondary servers, your existing primary server configuration and data are replicated to the secondary servers. Any data on a secondary server is removed before populating it with the primary server data.

In this scenario,

Upgrade Standalone UNP Server

  1. Open the UNP web interface on your UNP server.
  2. Click the submenu in the upper-right corner, and click Download UNP Installer. This ensures you have the latest installer.
  3. Double-click the executable file.
  4. Click Upgrade and go through the prompts. You must select version 2018.4.0 or newer.

Important: Do not use deprovision on the System Status page in the UNP UI. Deprovision is not a temporary halt to redundancy. Deprovision is used to revert UNP from a redundancy environment back to a standalone environment.

Tip: You may want to make a backup of the primary server or take a snapshot of the primary server VM before upgrading.

  1. Open the UNP web interface on your UNP primary server.
  2. Click the submenu in the upper-right corner, and click Download UNP Installer. This ensures you have the latest installer.
  3. Close the UNP UI.
  4. On the primary server, double-click the executable file. Once the installer starts, failover occurs, switching UNP to the secondary server.
  5. Click Upgrade and go through the prompts.
  6. Once the primary server restarts after the upgrade is complete, UNP is back to running on the primary server. Syncing to the secondary servers won't start because there is now an expected UNP version mismatch. Proceed with upgrading a secondary server.

Important: If you mistakenly updated a secondary server before the primary, or if the primary server took longer than expected and did not finish upgrading and rebooting until after a secondary, you will need to restart the UNP service on the affected secondary server.

  1. Do one of the following:
    • Copy the UNP installer to the secondary server.
    • Log in to the UNP UI on the secondary server, download the latest installer from the submenu, and then close the UI.
  2. Double-click the executable file.
  3. Click Upgrade and go through the prompts.

  4. Log in to UNP and go to the System Status page to verify UNP is syncing between the primary and secondary servers.

We recommend you run the same redundancy setup testing as you did when you initially set it up. See Test Redundancy Setup above.

Managing your redundancy environment

In a case where something happens with your primary server, you can recover the publisher database from a backup of one of your secondary servers.

  1. Back up a subscriber database on one of your secondary servers. See Back up your environment.
  2. Repair your primary server, as needed.
  3. Restore the database on you primary server using the secondary backup. See Restore your environment.
  4. Update your Registry:
    1. Open the Registry Editor.
    2. Go to HKEY_LOCAL_MACHINE/Software/Wow6432Node/Syn-Apps
    3. Update IPAddress to contain the IP Address of the new primary server.
    4. Remove the IsRedundant entry (right-click, then select Delete.)
    5. Save your changes.
  5. Restart UNP on your primary server.

We recommend powering down the VMs used for redundancy when taking snapshot or backup of the primary UNP server so that synchronization remains intact.

There are two scenarios for deprovisioning redundancy:

  • You need to replace your secondary server.
  • You're switching from a redundancy environment to a standalone environment.
  1. Go to System Status on the primary server.
  2. Scroll down to the Redundancy Server section.
  3. Select one of the secondary servers.
  4. Click Advanced Settings.
  5. Click Deprovision.

This breaks the redundancy setup and syncing between the servers no longer occurs. UNP on the primary server continues to be fully functional, however it is no longer functional on the secondary server.

Once deprovisioning is complete, the Redundancy Server section no longer displays the secondary server on the System Status page. You no longer have access to view notifications sent from the secondary server. (The same occurs if choose to stop renewing redundancy in your UNP license.)

Deprovisioning is not a temporary pause. Even though UNP is still installed on the secondary server, no activations can occur. It is recommended to turn of all Unified Notification Platform services through the Services app or uninstall UNP from the secondary server to prevent future network communication between UNP modules of the primary and secondary server.

If after deprovisioning you want to put redundancy in place again, you must uninstall UNP from the secondary (subscriber) server, then rerun the UNP installer for a clean install.

  • Redundancy is logged in SyncLog. If you are having issues with the primary and a secondary server syncing, call Support to walk you through using the reconfigure function available in System Status > Redundancy Servers > Advanced Settings.
  • Verify that Global Settings > Network is set to Auto-Select.

  • You cannot install UNP on more secondary servers than allowed in your license. If you attempt to install onto an additional server, the installation will end with an error.

    After you update your license to support additional servers, you should do the following:

    1. Refresh your license on the primary server
    2. Exit the installer on the secondary server that issued the error and restart the installation.