Request an invite

Alerts to incidents

When a monitor goes down, or Prometheus Alertmanager or Grafana fires an alert, Upstreamr writes a draft incident for you. Nobody outside your team sees it until you publish it.

Every plan. Drafts from your monitors are on by default; inbound alerts need an endpoint, under Alerts to incidents.

Guides and contents

How a draft opens

A monitor that goes down, once its probes agree, opens a draft titled “component is down” for the component it watches. Your owners and admins get the usual alert, with a link to the draft. The status page shows the component’s status as before; no incident is published and no subscriber is told.

  • One draft per component outage: other monitors and alerts on the same component join it.
  • No draft while the component is under maintenance.
  • A monitor that goes down and up again within 30 minutes reopens its draft instead of opening another.
  • Monitors without a component open no draft: they aren’t on any page.

Review and publish

Drafts wait under Incidents → Drafts, counted in the menu. Review & publish opens the usual incident form with the draft filled in: change the title, the first update, the impact or the components, start from a template, then Publish incident. It goes on every page that shows the component, and subscribers are notified as for any incident. Discard drops a draft you don’t need.

The monitor’s error and the alert’s description stay in the admin; they are never published.

When it recovers

If every monitor and alert behind a draft recovers before anyone publishes it, the draft closes by itself and stays in the history as Recovered before publishing.

Once an incident is published, its page in the admin says when everything behind it has recovered and offers the proposed update: Post “Resolved” update. If you switch on Resolve published incidents by themselves, Upstreamr posts that update after five minutes without a new failure.

Settings

Under Alerts to incidents:

  • Open a draft when a monitor goes down: on by default.
  • Draft text: the built-in text, or one of your incident templates, with {component} and the other placeholders filled in.
  • Publishing: Wait for review is the default. You can choose Publish immediately, or Publish if still down after a delay of 1 to 120 minutes; a draft that recovers before then is never published. Drafts without a component always wait for you.

Alertmanager, Grafana and other tools

Choose Add endpoint to get a URL for your alerting tool. It is shown once; only a hash is stored. Each alert names its component with a component label (or another label you choose); a component’s name works as it is, and other values can be mapped to a component. A firing alert opens or joins a draft; the same alert resolved closes it.

receivers:
  - name: upstreamr
    webhook_configs:
      - url: 'https://app.upstreamr.com/api/v1/inbound/inb_…'
        send_resolved: true
        max_alerts: 50

In Grafana, add a contact point of type Webhook with the same URL. Any other tool can post JSON:

curl -X POST 'https://app.upstreamr.com/api/v1/inbound/inb_…' \
  -H 'content-type: application/json' \
  -d '{"title":"API error rate above 5%","status":"firing",
       "component":"API","severity":"critical","fingerprint":"api-5xx"}'

critical or major means a major outage, warning or minor a partial outage, info degraded performance. Send "status":"resolved" with the same fingerprint when it’s fixed. At most 50 alerts per request and 60 requests a minute per endpoint.