Skip to content

/ bugs.python.org

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timeout delivering webhook #23

Open
Mariatta opened this issue Dec 27, 2018 · 2 comments
Open

Timeout delivering webhook #23

Mariatta opened this issue Dec 27, 2018 · 2 comments

Comments

@Mariatta
Copy link
Member

@Mariatta Mariatta commented Dec 27, 2018

Webhook to bpo pull_request with this payload keep resulting in Service timeout error.

I think it was taking too long to process this particular delivery? I've tried re-delivering the same payload three times, all returning Service timeout.

delivery id 6efba906-09dd-11e9-9b45-e06201fc4ec9
timestamp: 2018-12-27 05:43:46

{
  "ref": "refs/heads/master",
  "before": "34ae04f74dcf4ac97d07c3e82eaf8f619d80cedb",
  "after": "c6c7237272499b2c528acb5f62601421f329e92a",
  "created": false,
  "deleted": false,
  "forced": false,
  "base_ref": null,
  "compare": "https://github.com/python/cpython/compare/34ae04f74dcf...c6c723727249",
  "commits": [
    {
      "id": "c6c7237272499b2c528acb5f62601421f329e92a",
      "tree_id": "a1f00c8cec2833a64b45ce63d3c161ba2aec7441",
      "distinct": true,
      "message": "bpo-20182: AC convert remaining functions/methods in _hashopenssl.c (GH-9213)",
      "timestamp": "2018-12-27T15:43:43+02:00",
      "url": "https://github.com/python/cpython/commit/c6c7237272499b2c528acb5f62601421f329e92a",
      "author": {
        "name": "Tal Einat",
        "email": "taleinat+github@gmail.com",
        "username": "taleinat"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "username": "web-flow"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        "Modules/_hashopenssl.c",
        "Modules/clinic/_hashopenssl.c.h"
      ]
    }
  ],
  "head_commit": {
    "id": "c6c7237272499b2c528acb5f62601421f329e92a",
    "tree_id": "a1f00c8cec2833a64b45ce63d3c161ba2aec7441",
    "distinct": true,
    "message": "bpo-20182: AC convert remaining functions/methods in _hashopenssl.c (GH-9213)",
    "timestamp": "2018-12-27T15:43:43+02:00",
    "url": "https://github.com/python/cpython/commit/c6c7237272499b2c528acb5f62601421f329e92a",
    "author": {
      "name": "Tal Einat",
      "email": "taleinat+github@gmail.com",
      "username": "taleinat"
    },
    "committer": {
      "name": "GitHub",
      "email": "noreply@github.com",
      "username": "web-flow"
    },
    "added": [

    ],
    "removed": [

    ],
    "modified": [
      "Modules/_hashopenssl.c",
      "Modules/clinic/_hashopenssl.c.h"
    ]
  },
  "repository": {
    "id": 81598961,
    "node_id": "MDEwOlJlcG9zaXRvcnk4MTU5ODk2MQ==",
    "name": "cpython",
    "full_name": "python/cpython",
    "private": false,
    "owner": {
      "name": "python",
      "email": "",
      "login": "python",
      "id": 1525981,
      "node_id": "MDEyOk9yZ2FuaXphdGlvbjE1MjU5ODE=",
      "avatar_url": "https://avatars3.githubusercontent.com/u/1525981?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/python",
      "html_url": "https://github.com/python",
      "followers_url": "https://api.github.com/users/python/followers",
      "following_url": "https://api.github.com/users/python/following{/other_user}",
      "gists_url": "https://api.github.com/users/python/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/python/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/python/subscriptions",
      "organizations_url": "https://api.github.com/users/python/orgs",
      "repos_url": "https://api.github.com/users/python/repos",
      "events_url": "https://api.github.com/users/python/events{/privacy}",
      "received_events_url": "https://api.github.com/users/python/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "html_url": "https://github.com/python/cpython",
    "description": "The Python programming language",
    "fork": false,
    "url": "https://github.com/python/cpython",
    "forks_url": "https://api.github.com/repos/python/cpython/forks",
    "keys_url": "https://api.github.com/repos/python/cpython/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/python/cpython/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/python/cpython/teams",
    "hooks_url": "https://api.github.com/repos/python/cpython/hooks",
    "issue_events_url": "https://api.github.com/repos/python/cpython/issues/events{/number}",
    "events_url": "https://api.github.com/repos/python/cpython/events",
    "assignees_url": "https://api.github.com/repos/python/cpython/assignees{/user}",
    "branches_url": "https://api.github.com/repos/python/cpython/branches{/branch}",
    "tags_url": "https://api.github.com/repos/python/cpython/tags",
    "blobs_url": "https://api.github.com/repos/python/cpython/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/python/cpython/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/python/cpython/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/python/cpython/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/python/cpython/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/python/cpython/languages",
    "stargazers_url": "https://api.github.com/repos/python/cpython/stargazers",
    "contributors_url": "https://api.github.com/repos/python/cpython/contributors",
    "subscribers_url": "https://api.github.com/repos/python/cpython/subscribers",
    "subscription_url": "https://api.github.com/repos/python/cpython/subscription",
    "commits_url": "https://api.github.com/repos/python/cpython/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/python/cpython/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/python/cpython/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/python/cpython/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/python/cpython/contents/{+path}",
    "compare_url": "https://api.github.com/repos/python/cpython/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/python/cpython/merges",
    "archive_url": "https://api.github.com/repos/python/cpython/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/python/cpython/downloads",
    "issues_url": "https://api.github.com/repos/python/cpython/issues{/number}",
    "pulls_url": "https://api.github.com/repos/python/cpython/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/python/cpython/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/python/cpython/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/python/cpython/labels{/name}",
    "releases_url": "https://api.github.com/repos/python/cpython/releases{/id}",
    "deployments_url": "https://api.github.com/repos/python/cpython/deployments",
    "created_at": 1486754631,
    "updated_at": "2018-12-27T09:13:20Z",
    "pushed_at": 1545918224,
    "git_url": "git://github.com/python/cpython.git",
    "ssh_url": "git@github.com:python/cpython.git",
    "clone_url": "https://github.com/python/cpython.git",
    "svn_url": "https://github.com/python/cpython",
    "homepage": "https://www.python.org/",
    "size": 262946,
    "stargazers_count": 21608,
    "watchers_count": 21608,
    "language": "Python",
    "has_issues": false,
    "has_projects": false,
    "has_downloads": true,
    "has_wiki": false,
    "has_pages": false,
    "forks_count": 8094,
    "mirror_url": null,
    "archived": false,
    "open_issues_count": 951,
    "license": {
      "key": "other",
      "name": "Other",
      "spdx_id": "NOASSERTION",
      "url": null,
      "node_id": "MDc6TGljZW5zZTA="
    },
    "forks": 8094,
    "open_issues": 951,
    "watchers": 21608,
    "default_branch": "master",
    "stargazers": 21608,
    "master_branch": "master",
    "organization": "python"
  },
  "pusher": {
    "name": "taleinat",
    "email": "taleinat+github@gmail.com"
  },
  "organization": {
    "login": "python",
    "id": 1525981,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjE1MjU5ODE=",
    "url": "https://api.github.com/orgs/python",
    "repos_url": "https://api.github.com/orgs/python/repos",
    "events_url": "https://api.github.com/orgs/python/events",
    "hooks_url": "https://api.github.com/orgs/python/hooks",
    "issues_url": "https://api.github.com/orgs/python/issues",
    "members_url": "https://api.github.com/orgs/python/members{/member}",
    "public_members_url": "https://api.github.com/orgs/python/public_members{/member}",
    "avatar_url": "https://avatars3.githubusercontent.com/u/1525981?v=4",
    "description": "Repositories related to the Python Programming language"
  },
  "sender": {
    "login": "taleinat",
    "id": 532281,
    "node_id": "MDQ6VXNlcjUzMjI4MQ==",
    "avatar_url": "https://avatars3.githubusercontent.com/u/532281?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/taleinat",
    "html_url": "https://github.com/taleinat",
    "followers_url": "https://api.github.com/users/taleinat/followers",
    "following_url": "https://api.github.com/users/taleinat/following{/other_user}",
    "gists_url": "https://api.github.com/users/taleinat/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/taleinat/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/taleinat/subscriptions",
    "organizations_url": "https://api.github.com/users/taleinat/orgs",
    "repos_url": "https://api.github.com/users/taleinat/repos",
    "events_url": "https://api.github.com/users/taleinat/events{/privacy}",
    "received_events_url": "https://api.github.com/users/taleinat/received_events",
    "type": "User",
    "site_admin": false
  },
  "distinct_commits": [
    {
      "id": "c6c7237272499b2c528acb5f62601421f329e92a",
      "tree_id": "a1f00c8cec2833a64b45ce63d3c161ba2aec7441",
      "distinct": true,
      "message": "bpo-20182: AC convert remaining functions/methods in _hashopenssl.c (GH-9213)",
      "timestamp": "2018-12-27T15:43:43+02:00",
      "url": "https://github.com/python/cpython/commit/c6c7237272499b2c528acb5f62601421f329e92a",
      "author": {
        "name": "Tal Einat",
        "email": "taleinat+github@gmail.com",
        "username": "taleinat"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "username": "web-flow"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        "Modules/_hashopenssl.c",
        "Modules/clinic/_hashopenssl.c.h"
      ]
    }
  ],
  "ref_name": "master"
}
@ewdurbin
Copy link
Member

@ewdurbin ewdurbin commented Dec 28, 2018

GitHub has a timeout for web hook delivery, at 10s.

My best guess is that something on the new server is causing the handler to take longer than that. It's totally possible that the switch from the built in bespoke roundup-server to gunicorn is causing floods of requests to push GitHub past it's breaking point... or the handler is just slow as it's performed synchronously.

I've provisioned more worker processes for the cpython tracker's gunicorn stack, but we should probably investigate the performance of the handler itself to see if there's anything we can do to speed it up.

Aside from that... finding something to asynchronously process the hooks might be a better course of action, but roundup doesn't have any asynchronous task queue available.
@ewdurbin
Copy link
Member

@ewdurbin ewdurbin commented Dec 28, 2018

Also worth noting that each of the deliveries probably succeeded in doing whatever the handler would do with them... GitHub just gave up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@ewdurbin @Mariatta
You can’t perform that action at this time.