Alerts

An alert is a response of the system to a custom user-defined event or condition.

Alerts are raised by triggers (alert conditions). A trigger can be an event, a state or a change of state of a system component/hardware device. See the triggers section for details.

When an alert is raised, the system responds to it:

Administering Alerts

Triggers

Alerts are activated by triggers. Every alert may have several triggers associated with it. Each trigger defines a condition in which the alert should be raised. There are two types of triggers, described below:

Each alert may define zero or more triggers of every type. If no triggers are defined, the alert is never raised. If several triggers are defined, they act separately -- each trigger may raise the alert (they don't all have to exist at the same time -- just one is enough to raise the alert).

Event Triggers

An Event trigger raises the alert when a certain event fires in a context, and conforms to a condition specified in trigger settings. This event may be generated by LinkServer itself or may come from a Data Terminal.

The event trigger's properties are described here. Each trigger causes LinkServer to "listen" for the event specified by the Event setting in every context that matches the Context Mask setting. When such an event is detected, the server evaluates the expression specified by the Expression setting. This expression usually refers to some data associated with the event, but it may also refer to any other data, such as the values of some context variables.

Event Correlation

In terms of AggreGate alerts, event correlation is a way to activate the alert by an event trigger when one ("primary") event occurs, and deactivate the alert upon another ("correlated") event.

To enable event correlation for a certain event trigger, specify the name of the Correlated event in the trigger settings. In this case:

The following picture illustrates activation/deactivation of event trigger by Primary and Correlated events:

Variable Triggers

A variable trigger may raises the alert:

Selection between these two methods is performed by the Monitor State Changes property of the trigger. Other properties of the variable trigger are described here.

Every variable trigger periodically inspects the value of the variable in every context matching the defined by the Context Mask setting. Polling period is defined in seconds, by the Check Period parameter. A Data Table containing the value of the variable is retrieved from the context, and then something happens, according to the Monitor State Changes setting:

If the Context Mask parameter points to more than one context, each context is separately inspected, and the alert is raised every time the variable value satisfies the condition (or changes, depending on the Monitor State Changes setting) in any of these contexts.

Delays and Hysteresis

Variable triggers have a Delay parameter that help to set up alerts like "Warning, temperature is less than 10 degrees for over one hour".

The Delay parameter is relevant only when Monitor State Changes is off. It defines the interval between time when the trigger expression becomes true and the time the alert is raised. If set to zero (the default), the alert is raised immediately when Expression evaluates to TRUE. When Delay is non-zero, the server waits for the specified number of seconds before activating trigger and raising and alert. If Expression evaluates back to FALSE before the Delay time expires, the alert is not raised and the time counter is reset.

Variable trigger is activated if Expression is TRUE for longer than Delay time. It is deactivated if:

A combination of Delay and Deactivation Delay is called hysteresis.

The following picture shows how the variable trigger is activated/deactivated when Deactivator is not specified:

When Deactivator is specified, activation/deactivation of trigger is performed according to the following picture:

Alert Notifications

When an alert is raised by one of its triggers, LinkServer starts dealing with notifications. Notification settings are described here.

NOTIFYING ALERT OWNER

The LinkServer user owning the alert is notified if the Notify Owner If Connected To The Server setting is enabled and this user is logged in to a LinkServer User Interface. For example, in AggreGate Client, a popup message is shown to the user:

The alert notification popup contains:

If the Acknowledgement Is Required parameter is enabled, the user is prompted to enter an acknowledgement text for the alert in the same popup box. Here is what it looks like in AggreGate Client:

If owner notification is enabled, you can specify a Notification sound (.WAV file) that will be played in LinkServer User Interface before the notification popup is shown.

SENDING E-MAIL NOTIFICATIONS

LinkServer can prepare and send an alert notification e-mail message. This message may be sent to three recipient categories:

Sending mail to alert owner is enabled by the Send E-mail To Owner setting. E-mail Recipients table contains a list of LinkServer users that will receive e-mail notification.

It is also possible to send e-mail notifications to custom e-mail addresses. These addresses should be listed in the Additional E-mail Recipients field and separated by commas.

Contents of the email:

This is what such an email looks like:

Plain text version of alert email:

SENDING SMS NOTIFICATIONS

LinkServer can send an alert notification via SMS to a number of cellular phones. Specify recipients using the SMS recipients setting located in the Alert Notifications section.

The sender's number is defined by Sender Number Global Configuration option.

EXECUTING ACTIONS ON ALERTS

LinkServer is capable of running actions when an alert is raised. These are called "corrective actions" because they're intended to correct the problem automatically or by interacting with humans.

Actions may be run in interactive or non-interactive mode. In non-interactive mode, all action input is pre-defined and action output is sent to the server log or written to event history. This mode is also used by the scheduler. In interactive mode, the action interacts with the alert owner by using UI Procedures. He must be logged in to an LinkServer User Interface (e.g. AggreGate Client) to allow interactive actions execution. See the action execution modes section for details.

Non-interactive execution is controlled by the Automatic Corrective Actions property. This property lets you configure multiple actions to run on alert, and has several parameters: Every action will be launched from every context matching the Context Mask parameter. Action name is defined by the Action parameter. Parameters is the list of action-specific parameters (this might be confusing, but keep in mind that Parameters here is, itself, a parameter. Get it?).  These parameters are used to substitute user input when the alert processing engine executes an action in non-interactive mode. For example, if a given action requires confirmation (i.e. asks user something like "Delete query?" and lets the user click OK or Cancel), this field will contain a Delete query? parameter with two possible choices: OK or Cancel.

Interactively executed actions are listed in a table called Interactive Corrective Actions. Every action has two parameters: Context Mask and Action, which work just like in non-interactive execution (see above). There is no Parameters field in this list, since all input for interactive actions is provided by a human user.

Alert States

Alerts can be in one of the following states:

•	Enabled ()

•

Active ()

•	Escalated ()

•	Disabled ()

The initial state for a newly created alert is Enabled. This means that all triggers are enabled and may raise the alert. You can disable the alert (put it in Disabled state, technically speaking) by switching off the Alert is enabled basic option. A disabled alert is never raised and doesn't do anything, because all triggers are inactive. For descriptions of Active and Escalated states see Active Alerts and Alert Escalation below.

Active Alerts

Alert is active if:

The activation reason is shown in a tooltip that appears when the mouse hovers over the alert in LinkServer User Interfaces.

Pending Alerts

When an alert is raised, a special Alert event is generated and persistently stored in LinkServer database. Alert events that are not acknowledged are called "pending alert instances". See the events topic for more information about events and event acknowledgement. The pending instances of every alert may be managed using the View Pending Alerts action, defined in the Alert context. This action shows when and why the alert was raised, and lets you delete or acknowledge pending alerts. Alert instances that have already been acknowledged are not shown in the list.

When configuring an alert, one of the properties is called Allow Pending Alerts. If you disable this (i.e, disallow pending alerts for this particular alert you're configuring), the alert won't be shown in the Pending Alerts list even if it wasn't acknowledged. It also won't be escalated.

Here is what the list of pending alerts looks like (screenshot made in AggreGate Client):

When one or more unacknowledged alert events exist, the alert remains in Active state. This state is indicated in LinkServer User Interfaces by a special icon. The active state indicates that some attention is required to solve the problem indicated by an alert. When the problem is solved, system operator may launch the View Pending Alerts action and acknowledge all pending alert instances. Alert will be put back in the "Enabled" (normal) state, indicated by the icon.

Non-acknowledged instances may force alert to switch into the Active state only if the Allow pending alerts setting is enabled.

Alert Acknowledgement

Every alert raised is persistently stored as a LinkServer event (see Alert Event for details). These events may be acknowledged exactly like all other LinkServer events. Acknowledging an alert is the same as acknowledging its "Alert" event. The acknowledgement is stored in the event history, just like any other event acknowledgement. Its parameters (author, time and text) may be viewed when browsing event history (for example, using the Event Log in AggreGate Client).

E-Mail Acknowledgements

LinkServer provides an E-mail alert acknowledgement service letting operators acknowledge events by replying to alert notification e-mail message. To start using this service, enable mail receiving in Mail Settings section of LinkServer Global Configuration and configure incoming mail server properties (POP3 server address, login and password). Make sure that Check Incoming Mail scheduled job is enabled (it's disabled by default).

To acknowledge an alert by e-mail, reply to the alert notification e-mail and enter the acknowledgement text in the first line of reply. Alert acknowledgement will be added once the reply is delivered and received by LinkServer. LinkServer checks for new mail periodically, according to Incoming mail check period global configuration option.

Alert Escalation

Active alerts may be escalated. Escalation is usually used to draw attention to some critical conditions. In most cases, escalated alerts, indicated with the icon, require the system operators to take quick action.

For an alert to be escalated, there has to be at least one pending (unacknowledged) instance of it, and it must satisfy at least one of the escalation rules defined by the Alert Escalation property. There are two escalation types:

With number-based escalation, an alert is escalated if the number of pending alert instances exceeds the limit specified by the Number of pending alerts requited to initiate escalation setting.

With time-based escalation, an alert is escalated if any pending instance (alert event) is not acknowledged within the time frame specified by its Time before escalation setting.

When an alert is escalated, an Escalation Notification is emailed to all recipients defined by the Notifications setting. This notification contains the time of escalation, and the reason for it:

The reason is also shown in a tooltip that appears when the mouse hovers over the alert in LinkServer User Interfaces. Here is what it looks like in AggreGate Client:

To put an alert back in pending or normal state, acknowledge some of its pending instances so that escalation rules will not be satisfied.

Alert Event

When alert is raised, a special LinkServer event is generated in the Alert context. It is called "Alert event" or "Alert instance", and has several uses:

•	When this event is not acknowledged, it is called "pending alert instance". When an alert has pending instances, the alert itself is put into the Active state. Its icon changes to . This is similar to how an email client works -- you can see your mailbox as "Unread" if it has unread messages in it.

Format of Alert Event:

Example of Alert event:

Built-In Alerts

Device Offline Alert

Device Offline alert is built into LinkServer basic distribution package. It is raised when any Data Terminal is disconnected from the server for longer than 10 minutes.

According to its Variable Triggers, this alert is triggered only for Devices whose Enable Offline Alert setting is active.

Device Offline alert is owned by default administrator.

Alert Configuration

Alert Properties

This defines the basic options for an alert.

These may be accessed via the childInfo variable.

Event Triggers

This defines a list of Event Triggers.

These may be accessed via the eventTriggers variable.

Variable Triggers

Every variable trigger periodically inspects value the variable in every context matching the mask defined by the Context Mask setting. Polling period is defined in seconds, by the Check Period parameter. A Data Table containing the value of the variable is retrieved from the context, and then something happens, according to the value of the Monitor State Changes setting:

This property defines a list of Variable Triggers.

These may be accessed via the variableTriggers variable.

Alert Notifications

This property defines alert notification rules.

These may be accessed via the notifications variable.

Alert Escalation

This property defines processing options for pending alerts and alert escalation rules.

These may be accessed via the escalation variable.

Automatic Corrective Actions

This property defines the actions to be executed in non-interactive mode when the alert is raised. A good example illustrating application of this feature is execution of an external application on alert.

These may be accessed via the alertActions variable.

interactive Corrective actions

This property defines the actions to be executed in interactive mode when the alert is raised while its owner is connected to the server.

These may be accessed via the interactiveActions variable.

Examples

Now that we've covered all the theory, let's see some examples of real-world alert configuration.

IMPACT WARNING ALERT

Let's assume that our hardware device is a forklift controller that may generate Impact event when forklift collides with some obstacle. This event contains Level field that represents impact level, higher levels indicate more severe impacts.

The Impact Warning alert has a single Event Trigger which fires due to the Impact event in the Data Terminal context if the Integer level field in event data is greater than 50 (i.e. alert is raised only for severe impacts) .

Alert Info

Triggers Activated by Events

Alert Notifications

SEVERAL HARDWARE DEVICES IN TROUBLE

This complex example show how integration of two powerful features, alerts and queries, may be helpful to detect error conditions that involve several Device.

Assuming that ten temperature sensors are connected to LinkServer and owned by user john. The Data Terminal context of every sensor has a Temperature variable (variable name: temperature) whose value has an Integer field, celsius, containing the temperature in Celsius. Suppose the temperature of every individual sensor is not critical, but if three or more sensors report temperature greater than 100 degrees this may lead to problems in the future, and an alert should be raised to let the operators know there's a problem.

We'll use a query that helps to find all thermometers reporting temperature over than 100 degrees:

SELECT * FROM users.john.devices.*:temperature WHERE temperature$celsius > 100

This query returns a table (Data Table) containing one record per every sensor showing high temperature.

We can execute the above query by calling the executeQuery function from the Root context and passing the query text to it as an input parameter. So the idea is to set up an alert with a trigger that periodically executes this query and checks the number of records in its output. If the number is greater than three, the trigger fires.

Variable triggers may not be used to check function output directly, but here we must check a function's output -- executeQuery is a function. As covered above, alerts can be triggered in two ways -- when an event (so-called "event triggers") happens, or according to the value of some variable ("variable trigger"). For our example, we'll use a "variable" trigger, but we'll hack it a little bit. A variable trigger always has a context, a variable and an expression. Now, expressions can actually invoke functions.

So we're going to create an alert which has a meaningless context, a meaningless variable (which is still valid for that context), and a very meaningful expression. This expression won't even refer to the context and variable of the alert. We care only about the expression in this case, because it causes the query to be executed for us. The other parts (context and variable) are just placeholders. As placeholders, we'll use the "" (root) context and the version read-only variable, containing the server version.

Out alert will be raised when the expression evaluates to true.

To access the number of records contained in the output of the executeQuery function we will use the records property (see References for details about the reference resolving process). So, the final reference that will return the number of thermometers registering a high temperature looks like this:

{:executeQuery("SELECT * FROM users.john.devices.*:temperature WHERE temperature$celsius > 100")#records}

How it works:

Alert Info

Variable Triggers

Alert Notifications