Skip to content

Fix CookieJar + OpenerDirector interactions and UnhandledExceptionWarning #106925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

josiahcarlson
Copy link
Contributor

If you run the following code against an http server that returns cookies...

cookies = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(
    urllib.request.HTTPCookieProcessor(cookies)
)
resp = opener.open(url)
cookies.extract_cookies(resp, opener)

You may / will experience the following warning + unhandled exception:

/usr/local/lib/python3.11/http/cookiejar.py:1629: UserWarning: http.cookiejar bug! Traceback (most recent call last):
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 1626, in make_cookies
    ns_cookies = self._cookies_from_attrs_set(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 1583, in _cookies_from_attrs_set
    cookie = self._cookie_from_cookie_tuple(tup, request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 1532, in _cookie_from_cookie_tuple
    req_host, erhn = eff_request_host(request)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 642, in eff_request_host
    erhn = req_host = request_host(request)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 627, in request_host
    url = request.get_full_url()
          ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'OpenerDirector' object has no attribute 'get_full_url'

  _warn_unhandled_exception()

Attempting to play whack-a-mole will make you realize that the cookiejar really wants a BaseRequest, not an OpenerDirector, so we proxy the request data via __getattr__, removing the warning.

When I run this to verify the patch locally, I no longer experience the issue in Python 3.11 installs:


def patch():
    from urllib import request
    import socket

    def __init__(self):
        client_version = "Python-urllib/%s" % request.__version__
        self.addheaders = [('User-agent', client_version)]
        # self.handlers is retained only for backward compatibility
        self.handlers = []
        # manage the individual handlers
        self.handle_open = {}
        self.handle_error = {}
        self.process_response = {}
        self.process_request = {}
        self.req = None

    request.OpenerDirector.__init__ = __init__

    def __getattr__(self, attr):
        dct = self.__dict__
        if attr in dct:
            return dct[attr]

        return getattr(self.req, attr)
       
    request.OpenerDirector.__getattr__ = __getattr__

    request.OpenerDirector.get_full_url = lambda x: x.full_url

    def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
        # accept a URL or a Request object
        if isinstance(fullurl, str):
            req = request.Request(fullurl, data)
        else:
            req = fullurl
            if data is not None:
                req.data = data

        self.req = req
        req.timeout = timeout
        protocol = req.type

        # pre-process request
        meth_name = protocol+"_request"
        for processor in self.process_request.get(protocol, []):
            meth = getattr(processor, meth_name)
            req = meth(req)

        sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
        response = self._open(req, data)

        # post-process response
        meth_name = protocol+"_response"
        for processor in self.process_response.get(protocol, []):
            meth = getattr(processor, meth_name)
            response = meth(req, response)

        return response

    request.OpenerDirector.open = open

patch()

Fixes:
/usr/local/lib/python3.11/http/cookiejar.py:1629: UserWarning: http.cookiejar bug!
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 1626, in make_cookies
    ns_cookies = self._cookies_from_attrs_set(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 1583, in _cookies_from_attrs_set
    cookie = self._cookie_from_cookie_tuple(tup, request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 1532, in _cookie_from_cookie_tuple
    req_host, erhn = eff_request_host(request)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 642, in eff_request_host
    erhn = req_host = request_host(request)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/http/cookiejar.py", line 627, in request_host
    url = request.get_full_url()
          ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'OpenerDirector' object has no attribute 'get_full_url'

  _warn_unhandled_exception()
@josiahcarlson josiahcarlson changed the title Fix CookieJar + OpnerDirector interactions and UnhandledExceptionWarning Fix CookieJar + OpenerDirector interactions and UnhandledExceptionWarning Jul 20, 2023
@serhiy-storchaka
Copy link
Member

It is not trivial change like fixing a typo. Please open an issue for reference and possible discussion. Copy your report to the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants