Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR #17311

Merged

Conversation

@aeros
Copy link
Member

aeros commented Nov 21, 2019

Edit: This PR has been revised based on @pitrou's proposal, due to Windows compatibility concerns.

https://bugs.python.org/issue37228

Copy link
Member

1st1 left a comment

I actually think that Antoine solution is more preferable. @asvetlov?

@bedevere-bot

This comment has been minimized.

Copy link

bedevere-bot commented Nov 21, 2019

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Nov 21, 2019

@1st1

I actually think that Antoine solution is more preferable

I prefer Antoine's solution as well as far as elegance and API design goes, my only concern is that it would be fairly disruptive with regards to backwards compatibility (from raising an error upon passing reuse_address=True) to apply all the way back to 3.5+. I'll wait for @asvetlov's feedback though.

Edit: If the consensus is to use Antoine's proposal instead, I can modify this PR accordingly. As long as it's within the next couple of days, I should have the time to do so. If I don't have enough time, I'll close this PR and let someone else implement it.

@aeros aeros force-pushed the aeros:bpo37228-create_datagram_endpoint-reuse_address branch from aae05e6 to e3c895c Nov 21, 2019
@aeros aeros changed the title bpo-37228: Fix loop.create_datagram_endpoint() reuse_address bpo-37228: Fix loop.create_datagram_endpoint()'s usage of SO_REUSEADDR Nov 21, 2019
@aeros aeros force-pushed the aeros:bpo37228-create_datagram_endpoint-reuse_address branch from e3c895c to 397c9be Nov 21, 2019
@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Nov 21, 2019

Due to Windows compatibility concerns, I no longer think that changing SO_REUSEADDR -> SO_REUSEPORT is going to be a viable option. As a result, I will start working on implementing Antoine's proposal. For more details, see https://bugs.python.org/msg357143.

@aeros aeros force-pushed the aeros:bpo37228-create_datagram_endpoint-reuse_address branch from 397c9be to d748ca1 Nov 21, 2019
@@ -527,6 +529,10 @@ Opening network connections
The *family*, *proto*, *flags*, *reuse_address*, *reuse_port,
*allow_broadcast*, and *sock* parameters were added.

.. versionchanged:: 3.5.10

This comment has been minimized.

Copy link
@aeros

aeros Nov 21, 2019

Author Member

I'm assuming that version 3.5.10 would be the next released 3.5.x, and the earliest version that this change will apply to. Is this correct?

This comment has been minimized.

Copy link
@ned-deily

ned-deily Dec 7, 2019

Member

Since this is being added to multiple releases mid-stream, I think the right approach is for each branch to have its own versionchanged value, except perhaps master which would also have the 3.8 version (since 3.9 isn't released yet). So perhaps change this to the right 3.8.x version and then either manually cherry-pick for 3.7 and earlier (which might need to be done anyway if there are merge conflicts) or submit separate post-merge PRs for them to change the versionadded value.

This comment has been minimized.

Copy link
@aeros

aeros Dec 7, 2019

Author Member

@ned-deily

So perhaps change this to the right 3.8.x version and then either manually cherry-pick for 3.7 and earlier (which might need to be done anyway if there are merge conflicts) or submit separate post-merge PRs for them to change the versionadded value.

Sounds good, I can do that. I'll set master and the 3.8 branch to 3.8.1; 3.7 to 3.7.6; 3.6 to 3.6.10; and 3.5 to 3.5.10. The easiest way to handle this would probably be through using cherry-pick to open the backport PRs manually. As a result, I'll remove the ``needs backport to x` labels, so that @miss-islington doesn't automatically open the backport PRs.

Should we check with @larryhastings regarding the 3.5 backport? I believe it would be appropriate since this is a significant security vulnerability.

@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Nov 21, 2019

@1st1 I finished changing the PR based on @pitrou's proposal, due to the Windows compatibility issues with the previous implementation.

I have made the requested changes; please review again.

@bedevere-bot

This comment has been minimized.

Copy link

bedevere-bot commented Nov 21, 2019

Thanks for making the requested changes!

@1st1: please review the changes made to this pull request.

@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Nov 21, 2019

I'll add a Misc/NEWS entry.

aeros and others added 3 commits Dec 3, 2019
Co-authored-by: Yury Selivanov <yury@edgedb.com>
@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Dec 3, 2019

The latest round of test failures is being caused by explicitly passing a default value for the reuse_address parameter (_not_set/_unset from @1st1's suggestion) specifying a socket through the sock parameter:

ValueError: socket modifier keyword arguments can not be used when sock is specified. (reuse_address=<object object at 0x7f9335b97be0>)

This occurs from:

        if sock is not None:
            ...
            if (local_addr or remote_addr or
                    family or proto or flags or
                    reuse_address or reuse_port or allow_broadcast):
                        ...
                        raise ValueError(
                            f'socket modifier keyword arguments can not be used '
                            f'when sock is specified. ({problems})')

My suggested solution is to remove or reuse_address from the above conditional, which I will apply in the next commit. This should be fine, since reuse_address will no longer have any functionality after the changes made in this PR.

Edit: Fixed.

@aeros aeros requested a review from 1st1 Dec 3, 2019
@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Dec 8, 2019

Based on @ned-deily's suggestion to manually set the versionchanged to separate values for the 3.7, 3.6, and 3.5 branches, I'll remove the needs backport to 3.7 and needs backport to 3.6 labels so that @miss-islington doesn't open the backport PRs automatically.

If this PR is accepted and merged, I'll use cherry-picker to manually open the backport PRs for 3.7, 3.6, and 3.5.

Regarding the "What's New" entries, I would prefer to do those separately to simplify the merge conflicts. I'll have definitely time to take care of this on Monday, but for today and tomorrow I'll likely be busy with finishing up my final exams (last week of the Fall semester).

I should have time to make very small changes or reply to feedback, but I will likely not have time to do anything substantial until Monday (Dec 9th), unless I'm able to finish my work early on Sunday.

Edit: If any larger changes are needed before Monday, feel free to open a PR to my branch at https://github.com/aeros/cpython/tree/bpo37228-create_datagram_endpoint-reuse_address and I should be able to merge it. I've also ticked allow edits from maintainers, which allows minor changes to the contents of the PR through the GitHub UI.

Copy link
Member

gvanrossum left a comment

LGTM

@ambv ambv merged commit ab513a3 into python:master Dec 9, 2019
4 checks passed
4 checks passed
Azure Pipelines PR #20191207.21 succeeded
Details
bedevere/issue-number Issue number 37228 found
Details
bedevere/news News entry found in Misc/NEWS.d
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@bedevere-bot

This comment has been minimized.

Copy link

bedevere-bot commented Dec 9, 2019

@ambv: Please replace # with GH- in the commit message next time. Thanks!

@miss-islington

This comment has been minimized.

Copy link

miss-islington commented Dec 9, 2019

Thanks @aeros for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒🤖

miss-islington added a commit to miss-islington/cpython that referenced this pull request Dec 9, 2019
pythonGH-17311)

(cherry picked from commit ab513a3)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
@bedevere-bot

This comment has been minimized.

Copy link

bedevere-bot commented Dec 9, 2019

GH-17529 is a backport of this pull request to the 3.8 branch.

ambv added a commit that referenced this pull request Dec 9, 2019
GH-17311) (#17529)

(cherry picked from commit ab513a3)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
@ambv

This comment has been minimized.

Copy link
Contributor

ambv commented Dec 9, 2019

Note: merged for 3.8 and 3.9. @aeros removed backport labels to 3.6 and 3.7 here. The issue on BPO still lists those versions as affected. Please fix either way.

@aeros

This comment has been minimized.

Copy link
Member Author

aeros commented Dec 10, 2019

@ambv

@aeros removed backport labels to 3.6 and 3.7 here. The issue on BPO still lists those versions as affected. Please fix either way.

This is because they need to be manually backported, due to a difference in the versionchanged note. I'll take care of this process and attach the backport PRs to the bpo issue.

ned-deily added a commit to ned-deily/cpython that referenced this pull request Dec 11, 2019
pythonGH-17311) (pythonGH-17529)

(cherry picked from commit ab513a3)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
ned-deily added a commit that referenced this pull request Dec 11, 2019
GH-17311) (GH-17570)

(cherry picked from commit ab513a3)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
aeros added a commit to aeros/cpython that referenced this pull request Dec 11, 2019
aeros added a commit to aeros/cpython that referenced this pull request Dec 11, 2019
…USEADDR (pythonGH-17311).

(cherry picked from commit ab513a3)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
ned-deily added a commit that referenced this pull request Dec 11, 2019
…USEADDR (GH-17311). (GH-17571)

(cherry picked from commit ab513a3)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.