{"_id":"5d23e7bdae8464005cd66fe6","project":"564e5930c3553e0d003e53d0","version":{"_id":"564e5a9b1560880d008d30dc","project":"564e5930c3553e0d003e53d0","__v":27,"createdAt":"2015-11-19T23:26:19.166Z","releaseDate":"2015-11-19T23:26:19.166Z","categories":["564e5a9b1560880d008d30dd","566318e1f5ca460d00f41896","56631d08cd54d50d005015fa","56631d2a81ad7417006a202c","5668ba19fbd7680d009375f4","5668cb8b10bda80d00797ed9","5668cb9d10bda80d00797eda","56830d8a3f94e00d004e2a7a","56830d9072bb720d0091f594","56830d94cb4d190d0027698e","56830dc44aecbd0d00a464c5","569e90f3c9b43e0d00c4bab1","56a96d338791090d00113bab","56b12d8336d2580d00247877","56c36bf0a869d017002ea55b","56c36bf93d30210d00ea84bb","56c77749b935671700ff0304","56c7ab9e5652c217008e091a","56cb8bdad5c6241d00ef5e61","58aefce02470660f00b54539","58aefd0bebd7370f0078b954","59ca65ca4337830026edf24f","5c33cd9eb47ba20051ac8d64","5c33df728bec1d0063431c34","5c4783ef523219027055513a","5c4f35033400f3010203a999","5d1d0c9f19c3a0003aeb525a"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"Foundation","version_clean":"2.0.0","version":"2"},"category":{"_id":"56c7ab9e5652c217008e091a","pages":["56c7ace6606ee717003c475d","56d3d6390b39260b008da477"],"project":"564e5930c3553e0d003e53d0","version":"564e5a9b1560880d008d30dc","__v":2,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-02-19T23:56:14.582Z","from_sync":false,"order":7,"slug":"webhooks","title":"Webhooks"},"user":"5c2ff7abcef0c602316e8234","__v":0,"parentDoc":null,"metadata":{"title":"","description":"","image":[]},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2019-07-09T01:02:53.329Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":2,"body":"Last Major Revision: October 29 2019\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"body\": \"Webhook behavior described on this page may change in the future. If you encounter unexpected behavior, please check this page for documentation updates.\",\n  \"title\": \"Note\"\n}\n[/block]\n\n[block:api-header]\n{\n  \"title\": \"Setting Up Webhooks\"\n}\n[/block]\n[See our Knowledge Base article](https://support.pagerduty.com/docs/webhooks) for information on setting up webhooks on your account.This article contains technical documentation on our webhook delivery behavior.\n[block:api-header]\n{\n  \"title\": \"Ports and Connections\"\n}\n[/block]\nWebhooks can be sent to any publicly accessible web server, on any port, with or without encryption. \n\n## Webhook Ports\n\nIf you specify `http://` in your Endpoint URL, we will initiate a standard HTTP connection on port 80. Likewise, entering `https://` means we will attempt to make a secure connection on port 443. To override the default port, add a colon (:) after the host address with the desired port number.\n\n**Example:** `https://app.acmemonitoring.com:8443/pagerduty.php`\n\n[block:api-header]\n{\n  \"title\": \"Error Handling and Retries\"\n}\n[/block]\n## Timeouts\n\nPagerDuty expects a 2xx response within 5 seconds for Generic Webhooks and within 16 seconds for webhooks generated from Custom Incident Actions. \n\n## Retries (Server and Network Errors)\n\nPagerDuty will retry sending a webhook in these cases:\n  * No response / timeout\n  * 5XX (server error) response\n  * 429 (rate limit error) response\n  * Connection cannot be established (except in the case of TLS errors)\n  * DNS errors / host name cannot be resolved\n\nWebhooks will be retried up to 4 times (total of 5 attempts) with exponential backoff over the course of about 20 minutes. While attempting to retry, subsequent webhooks for this extension will be enqueued.\n\n*Max tries per webhook:* 5\n\n## Permanent Errors\n\nPagerDuty will drop a webhook *without* a retry in these non-transient cases:\n  * 4XX response from the server (except for 429)\n  * TLS errors when establishing a connection \n[block:api-header]\n{\n  \"title\": \"Temporary Disablement\"\n}\n[/block]\nAfter 3 consecutive webhooks are dropped due to delivery failure, the extension will be disabled for 24 hours and any other webhooks in the extension’s queue will be dropped. While an extension is disabled, no new webhooks will be enqueued for delivery. These webhooks will be dropped.\n\n## Re-enable An Extension\n\nWhen an extension is disabled, it will appear this way on a Service’s integrations tab (under Extensions) or on the Configuration > Extensions page in the PagerDuty web app and can be manually re-enabled there.\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/58d4118-Webhooks_2-webhook-disabled.jpg\",\n        \"Webhooks 2-webhook-disabled.jpg\",\n        1394,\n        221,\n        \"#ededed\"\n      ]\n    }\n  ]\n}\n[/block]\n\n[block:api-header]\n{\n  \"title\": \"Security\"\n}\n[/block]\nPagerDuty supports multiple methods for securing your webhooks.\n\n## Mutual TLS Authentication (Recommended)\n\nWe send a client TLS certificate with webhooks on request. You can use this to verify that the webhook was sent from PagerDuty. If you specify an HTTPS endpoint, we will also verify your server certificate before sending webhooks to that endpoint.\n\n[Learn how to secure PagerDuty's webhooks using TLS](doc:webhooks-mutual-tls) \n\n\n## IP Safelists\n\nYou can configure your network or server to only accept inbound connections from certain IP addresses. See our [knowledge base article](https://support.pagerduty.com/docs/whitelisting-ips#section-webhooks) for information on obtaining the list of IPs webhooks may come from.  (Previously called IP whitelisting, [see why we changed the name here.](https://twitter.com/secnerdette/status/898314208097427457))\n\n## Basic Authentication\n\nIf your web server uses basic authentication, you can add the username and password to the URL before the host address. Special characters such as :::at::: in the username or password should be percent-encoded. We recommend sending these requests over `HTTPS` to protect credentials in the URL.\n\n**Examples:**\n`https://username:password@app.acmemonitoring.com`\n`https://username:long%20password@example.com`\n\n[block:api-header]\n{\n  \"title\": \"Ordering\"\n}\n[/block]\nPagerDuty sends webhooks for a given extension in the order that they were generated.\n[block:api-header]\n{\n  \"title\": \"At-Least-Once Delivery\"\n}\n[/block]\nWhen a webhook endpoint is behaving as specified in this documentation, PagerDuty delivers webhooks at least once. This means that duplicate webhooks may be delivered to the target endpoint in certain circumstances.\n\nCustomers wishing to de-duplicate webhooks may do so by using the `X-Webhook-Id` header provided with each webhook request. The value of this header is unique to the webhook but is repeated for each delivery attempt, so it may be used to ignore subsequent delivery attempts after an initial success.\n[block:api-header]\n{\n  \"title\": \"Batching\"\n}\n[/block]\nPagerDuty may send webhooks in batches. Your code should be prepared to process a collection of webhooks in one POST. \n[block:api-header]\n{\n  \"title\": \"Processing Webhooks\"\n}\n[/block]\nWe recommend these best practices when processing webhooks:\n  * Return a `202 Accepted` once you receive a payload and then process the batch of webhooks. Asynchronous processing will help prevent the connection from timing out.\n  * Catch and handle errors when processing a webhook. This will isolate problems with a specific webhook and allow you to continue processing the rest of the batch.\n[block:api-header]\n{\n  \"title\": \"Size Limit\"\n}\n[/block]\nWebhook size is affected by a number of factors, including the size of the incoming event, the number of responders on an incident, the number of log entries for an incident, etc. PagerDuty will deliver webhooks up to 55 KB (56320 bytes) in size.\n\nIn the very rare case that a generated webhook is larger than 55 KB, PagerDuty will attempt to omit event details in the webhook to decrease the size below the maximum. The fields affected by this omission are present on the first `log_entry` in the `channel` object: `details`, `cef_details.details`, and `body`. The `body` field is only present on events received by email, and the `cef_details.details` field is only present on [PD-CEF](https://support.pagerduty.com/docs/pd-cef) events.\n\nWhen event details are omitted, this field is replaced with a message noting the omission. In addition, `channel` fields `details_omitted`, `cef_details.details_omitted`, and `body_omitted` will be updated, as appropriate, from `false` to `true` to indicate which fields have been changed.\n\nIn the extremely rare case that omitting event details does not bring the webhook under the max size, the webhook will be dropped.","excerpt":"","slug":"webhook-behavior","type":"basic","title":"Webhook Behavior"}
Last Major Revision: October 29 2019 [block:callout] { "type": "warning", "body": "Webhook behavior described on this page may change in the future. If you encounter unexpected behavior, please check this page for documentation updates.", "title": "Note" } [/block] [block:api-header] { "title": "Setting Up Webhooks" } [/block] [See our Knowledge Base article](https://support.pagerduty.com/docs/webhooks) for information on setting up webhooks on your account.This article contains technical documentation on our webhook delivery behavior. [block:api-header] { "title": "Ports and Connections" } [/block] Webhooks can be sent to any publicly accessible web server, on any port, with or without encryption. ## Webhook Ports If you specify `http://` in your Endpoint URL, we will initiate a standard HTTP connection on port 80. Likewise, entering `https://` means we will attempt to make a secure connection on port 443. To override the default port, add a colon (:) after the host address with the desired port number. **Example:** `https://app.acmemonitoring.com:8443/pagerduty.php` [block:api-header] { "title": "Error Handling and Retries" } [/block] ## Timeouts PagerDuty expects a 2xx response within 5 seconds for Generic Webhooks and within 16 seconds for webhooks generated from Custom Incident Actions. ## Retries (Server and Network Errors) PagerDuty will retry sending a webhook in these cases: * No response / timeout * 5XX (server error) response * 429 (rate limit error) response * Connection cannot be established (except in the case of TLS errors) * DNS errors / host name cannot be resolved Webhooks will be retried up to 4 times (total of 5 attempts) with exponential backoff over the course of about 20 minutes. While attempting to retry, subsequent webhooks for this extension will be enqueued. *Max tries per webhook:* 5 ## Permanent Errors PagerDuty will drop a webhook *without* a retry in these non-transient cases: * 4XX response from the server (except for 429) * TLS errors when establishing a connection [block:api-header] { "title": "Temporary Disablement" } [/block] After 3 consecutive webhooks are dropped due to delivery failure, the extension will be disabled for 24 hours and any other webhooks in the extension’s queue will be dropped. While an extension is disabled, no new webhooks will be enqueued for delivery. These webhooks will be dropped. ## Re-enable An Extension When an extension is disabled, it will appear this way on a Service’s integrations tab (under Extensions) or on the Configuration > Extensions page in the PagerDuty web app and can be manually re-enabled there. [block:image] { "images": [ { "image": [ "https://files.readme.io/58d4118-Webhooks_2-webhook-disabled.jpg", "Webhooks 2-webhook-disabled.jpg", 1394, 221, "#ededed" ] } ] } [/block] [block:api-header] { "title": "Security" } [/block] PagerDuty supports multiple methods for securing your webhooks. ## Mutual TLS Authentication (Recommended) We send a client TLS certificate with webhooks on request. You can use this to verify that the webhook was sent from PagerDuty. If you specify an HTTPS endpoint, we will also verify your server certificate before sending webhooks to that endpoint. [Learn how to secure PagerDuty's webhooks using TLS](doc:webhooks-mutual-tls) ## IP Safelists You can configure your network or server to only accept inbound connections from certain IP addresses. See our [knowledge base article](https://support.pagerduty.com/docs/whitelisting-ips#section-webhooks) for information on obtaining the list of IPs webhooks may come from. (Previously called IP whitelisting, [see why we changed the name here.](https://twitter.com/secnerdette/status/898314208097427457)) ## Basic Authentication If your web server uses basic authentication, you can add the username and password to the URL before the host address. Special characters such as @ in the username or password should be percent-encoded. We recommend sending these requests over `HTTPS` to protect credentials in the URL. **Examples:** `https://username:password@app.acmemonitoring.com` `https://username:long%20password@example.com` [block:api-header] { "title": "Ordering" } [/block] PagerDuty sends webhooks for a given extension in the order that they were generated. [block:api-header] { "title": "At-Least-Once Delivery" } [/block] When a webhook endpoint is behaving as specified in this documentation, PagerDuty delivers webhooks at least once. This means that duplicate webhooks may be delivered to the target endpoint in certain circumstances. Customers wishing to de-duplicate webhooks may do so by using the `X-Webhook-Id` header provided with each webhook request. The value of this header is unique to the webhook but is repeated for each delivery attempt, so it may be used to ignore subsequent delivery attempts after an initial success. [block:api-header] { "title": "Batching" } [/block] PagerDuty may send webhooks in batches. Your code should be prepared to process a collection of webhooks in one POST. [block:api-header] { "title": "Processing Webhooks" } [/block] We recommend these best practices when processing webhooks: * Return a `202 Accepted` once you receive a payload and then process the batch of webhooks. Asynchronous processing will help prevent the connection from timing out. * Catch and handle errors when processing a webhook. This will isolate problems with a specific webhook and allow you to continue processing the rest of the batch. [block:api-header] { "title": "Size Limit" } [/block] Webhook size is affected by a number of factors, including the size of the incoming event, the number of responders on an incident, the number of log entries for an incident, etc. PagerDuty will deliver webhooks up to 55 KB (56320 bytes) in size. In the very rare case that a generated webhook is larger than 55 KB, PagerDuty will attempt to omit event details in the webhook to decrease the size below the maximum. The fields affected by this omission are present on the first `log_entry` in the `channel` object: `details`, `cef_details.details`, and `body`. The `body` field is only present on events received by email, and the `cef_details.details` field is only present on [PD-CEF](https://support.pagerduty.com/docs/pd-cef) events. When event details are omitted, this field is replaced with a message noting the omission. In addition, `channel` fields `details_omitted`, `cef_details.details_omitted`, and `body_omitted` will be updated, as appropriate, from `false` to `true` to indicate which fields have been changed. In the extremely rare case that omitting event details does not bring the webhook under the max size, the webhook will be dropped.