bpo-37093: Allow http.client to parse non-ASCII header names #13788
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lib/email/parser.py
Outdated
| """Create a message structure from the data in a file. | ||
|
|
||
| Reads all the data from the file and returns the root of the message | ||
| structure. Optional headersonly is a flag specifying whether to stop | ||
| parsing after reading the headers or not. The default is False, | ||
| meaning it parses the entire contents of the file. | ||
| """ | ||
| feedparser = FeedParser(self._class, policy=self.policy) | ||
| feedparser = FeedParser(self._class, policy=self.policy, strictheaders=strictheaders) |
There was a problem hiding this comment.
Please limit lines to 79 characters (PEP 8).
There was a problem hiding this comment.
There's still a litany of line-length violations in Lib/http/client.py and Lib/test/test_httplib.py but I think now at least I'm not making things any worse.
| @@ -0,0 +1 @@ | |||
| http.client now parses non-ASCII header names. | |||
Previously, when http.client tried to parse a response from an out-of-spec server that sent a header with a non-ASCII name, email.feedparser would assume that the non-compliant header must be part of a message body and abort parsing. However, http.client already determined the boundary between headers and body and only passed the headers to the parser. As a result, any headers after the first non-compliant one would be silently (!) ignored. This could include headers important for message framing like Content-Length and Transfer-Encoding. In the long-long ago, this parsing was handled by the rfc822 module, which didn't care about which bytes were in the header as long as there was a colon in the line. Now, add an optional argument to the email parsers to decide whether to require strict RFC-compliant header names. Default this to True to minimize the possibility of breaking other callers. In http.client, which already knows where the headers end and body begins, use False. Note that the non-ASCII names will be decoded as ISO-8859-1 in keeping with how header values are decoded.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, when
http.clienttried to parse a response from an out-of-spec server that sent a header with a non-ASCII name,email.feedparserwould assume that the non-compliant header must be part of a message body and abort parsing. However,http.clientalready determined the boundary between headers and body and only passed the headers to the parser. As a result, any headers after the first non-compliant one would be silently (!) ignored. This could include headers important for message framing likeContent-LengthandTransfer-Encoding.In the long-long ago, this parsing was handled by the
rfc822module, which didn't care about which bytes were in the header as long as there was a colon in the line.Now, add an optional argument to the email parsers to decide whether to require strict RFC-compliant header names. Default this to
Trueto minimize the possibility of breaking other callers. Inhttp.client, which already knows where the headers end and body begins, useFalse.Note that the non-ASCII names will be decoded as ISO-8859-1 in keeping with how header values are decoded.
https://bugs.python.org/issue37093