Datadog

A Datadog webhook delivers monitor notifications from the Datadog Webhooks integration into Plural. Each request is one alert. Datadog's default webhook payload does not carry a status Plural can read, so you give the webhook a custom payload that does.

An alert does not start a job by itself. A job starts only when a webhook trigger on a workbench matches the alert, and only when the alert starts firing: the first firing notification for an alert, or a firing notification for an alert that had resolved. Repeat notifications for an alert that is still firing do not start another job, and an alert starts at most one job per workbench per hour.

What Plural reads from the payload

Use a payload like this one. Every value is a Datadog template variable:

{
  "id": "$ALERT_CYCLE_KEY",
  "title": "$EVENT_TITLE",
  "body": "$TEXT_ONLY_MSG",
  "status": "$ALERT_TYPE",
  "alert_type": "$ALERT_TYPE",
  "priority": "$ALERT_PRIORITY",
  "link": "$LINK",
  "project": "$TAGS[plrl_project]",
  "cluster": "$TAGS[plrl_cluster]"
}

• id identifies the alert, so a later notification with the same id updates it. $ALERT_CYCLE_KEY stays the same from the time a monitor triggers until it recovers, so the recovery resolves the same alert. Keep it quoted: an id that is not a string is dropped.
• status sets the state, falling back to alert_transition. Plural matches these lowercase values exactly: firing, triggered, trigger, alert, alerting, warning, error, and critical are firing. Everything else is resolved, including a missing status.
• priority sets the severity: P1 is critical, P2 high, P3 medium, P4 and P5 low. Without a priority, alert_type sets it: error and critical are critical, warning is medium, and info is low.
• title becomes the alert title, defaulting to "Datadog Alert". title and body together become the alert summary. Without a body, Plural uses text_title or message.
• link (or url) is the link back to Datadog.
• meta is stored as annotations. tags, as a JSON list of key:value strings, is stored as tags.

$ALERT_TYPE is error, warning, success, or info, so it gives both a status and a severity Plural understands. Do not use $ALERT_TRANSITION for status. Its values are capitalized (Triggered, Recovered), and Plural reads them as resolved.

Associate alerts with Plural resources

Plural keeps an alert only when it belongs to a Plural project, cluster, or service, or when a workbench trigger matches it. Otherwise the request is acknowledged and ignored. Plural looks in these places, in order:

• Project: meta.project, then project, then a plrl_project, project, or plural_project tag.
• Cluster (by handle): meta.cluster, then cluster, then a plrl_cluster, cluster, or plural_cluster tag.
• Service: meta.service or service as <cluster-handle>/<service-name>, or a plrl_service, service, or plural_service tag holding the service name when the cluster is also known.