Skip to main content

Configure Incident Workflows

Learn how to create and manage automated incident workflows in Harness AI SRE.

Overview​

Incident workflows help you:

  • Automate response actions
  • Standardize processes
  • Coordinate teams
  • Track progress
  • Ensure compliance

Workflow Types​

Detection Workflow​

name: "Service Alert to Incident"
trigger:
  type: alert
  conditions:
    service: [service]
    severity: [P1, P2]
actions:
  - create_incident:
      template: "Service Incident"
      fields:
        service: [alert.service]
        severity: [alert.severity]
  - notify:
      channel: "#[service]-incidents"
      message: "🚨 New [severity] incident created for [service]"

Response Workflow​

name: "P1 Incident Response"
trigger:
  type: incident
  conditions:
    severity: P1
    status: detected
actions:
  - create_zoom:
      name: "[service] P1 Incident Bridge"
      participants: ["[team]"]
  - notify_oncall:
      team: [team]
      message: "🔴 Join P1 incident call: [zoom_url]"
  - trigger_runbook:
      name: "Incident Response"
      variables:
        service: [incident.service]
        severity: [incident.severity]

Update Workflow​

name: "Status Update Notification"
trigger:
  type: incident_update
  conditions:
    field: status
actions:
  - notify:
      channel: "#[service]-incidents"
      message: "ℹ️ Status changed to [status] for [service]"
  - update_timeline:
      type: status_change
      description: "Status updated to [status]"

Resolution Workflow​

name: "Incident Resolution"
trigger:
  type: incident_update
  conditions:
    status: resolved
actions:
  - trigger_runbook:
      name: "Post-Incident Tasks"
      variables:
        incident_id: [incident.id]
  - schedule_review:
      title: "[service] Incident Review"
      team: [team]
      due: "+3d"
  - notify:
      channel: "#[service]-incidents"
      message: "✅ Incident resolved: [title]"

Workflow Components​

Triggers​

triggers:
  alert_trigger:
    type: alert
    conditions:
      service: [service]
      severity: [severity]
  incident_trigger:
    type: incident
    conditions:
      status: [status]
      team: [team]
  update_trigger:
    type: incident_update
    conditions:
      field: [field]
      value: [value]

Actions​

actions:
  notification:
    type: notify
    channel: "#[channel]"
    message: "[message]"
  runbook:
    type: trigger_runbook
    name: [runbook]
    variables: [variables]
  meeting:
    type: create_meeting
    platform: [zoom, teams]
    title: "[title]"
  ticket:
    type: create_ticket
    type: [jira, servicenow]
    template: [template]

Conditions​

conditions:
  severity_check:
    field: severity
    operator: in
    values: [P1, P2]
  team_check:
    field: team
    operator: equals
    value: [team]
  status_check:
    field: status
    operator: changed_to
    value: [status]

Integration Examples​

Slack Integration​

slack_workflow:
  name: "Slack Incident Management"
  actions:
    - create_channel:
        name: "inc-[incident.id]-[service]"
        template: incident_channel
    - invite_users:
        teams: ["[team]", "sre"]
    - pin_message:
        content: |
          🚨 *Incident Summary*
          *Service:* [service]
          *Severity:* [severity]
          *Status:* [status]
          *Owner:* [owner]

Jira Integration​

jira_workflow:
  name: "Jira Incident Tracking"
  actions:
    - create_issue:
        project: [project]
        type: Incident
        title: "[title]"
        description: |
          h2. Incident Details
          * Service: [service]
          * Severity: [severity]
          * Status: [status]
          * Owner: [owner]
    - create_subtasks:
        - title: "Investigation"
          assignee: [owner]
        - title: "Communication"
          assignee: [comms_lead]

PagerDuty Integration​

pagerduty_workflow:
  name: "PagerDuty Escalation"
  actions:
    - create_incident:
        severity: [severity]
        service: [service]
        title: "[title]"
    - escalate_to:
        policy: [escalation_policy]
        message: "P1 incident requires immediate attention"

Best Practices​

Workflow Design​

  • Start simple
  • Add automation gradually
  • Test thoroughly
  • Monitor effectiveness
  • Document clearly

Action Configuration​

  • Set timeouts
  • Handle failures
  • Add retries
  • Log actions
  • Verify results

Integration Management​

  • Use service accounts
  • Secure credentials
  • Monitor usage
  • Update regularly
  • Test connections

Next Steps​

Documentation​