Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[doc] multiprocessing Proxy docs need locking semantics explained #64063

Open
maxpolk mannequin opened this issue Dec 2, 2013 · 2 comments
Open

[doc] multiprocessing Proxy docs need locking semantics explained #64063

maxpolk mannequin opened this issue Dec 2, 2013 · 2 comments
Labels
3.11 docs Documentation in the Doc dir easy expert-multiprocessing type-feature A feature request or enhancement

Comments

@maxpolk
Copy link
Mannequin

maxpolk mannequin commented Dec 2, 2013

BPO 19864

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-12-02.19:00:54.395>
labels = ['easy', '3.11', 'type-feature', 'docs']
title = '[doc] multiprocessing Proxy docs need locking semantics explained'
updated_at = <Date 2021-12-06.18:43:18.089>
user = 'https://bugs.python.org/maxpolk'

bugs.python.org fields:

activity = <Date 2021-12-06.18:43:18.089>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2013-12-02.19:00:54.395>
creator = 'maxpolk'
dependencies = []
files = []
hgrepos = []
issue_num = 19864
keywords = ['easy']
message_count = 2.0
messages = ['205039', '205122']
nosy_count = 3.0
nosy_names = ['docs@python', 'sbt', 'maxpolk']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19864'
versions = ['Python 3.11']

@maxpolk
Copy link
Mannequin Author

maxpolk mannequin commented Dec 2, 2013

In the docs (both Python 2 and 3) for the multiprocessing module, the Proxy section needs to be explicit about the fact that is does NOT create locks around access to shared objects.

Why? Because on the same page, we read about multiprocessing.managers.SyncManager's Queue method to create a shared queue.Queue object and return a proxy for it, next to other methods like SyncManager.list to create a "process safe" list.

But later, a willful violation of these semantics exists in the "Using a remote manager" section where a Proxy is implicitly created (via the register method of multiprocessing.managers.BaseManager) surrounding a Queue.Queue object.

Note that it did not create a proxy around a SyncManager.Queue, it created it around a Queue.Queue. A user who copies this code and replaces Queue.Queue with a plain Python list gets the sense that all the needed locks will be created to protect the shared list.

However, due to lack of documentation in the Proxy section, the user will not know it's an unsafe list, and Proxy isn't helping them. I'm guessing that Queue.Queue has its own locks to protect it in a process-shared setting, and we "lucked out" in this example, but an unwary reader's luck runs out if they replace it with a plain Python list.

Therefore, may I suggest two changes: (1) replace Queue.Queue with SyncManager.Queue in the "Using a remote manager" section to avoid misleading readers; and (2) be explicit in Proxy class docs that "no locks are created" to protect against concurrent access, and maybe add that the user must go to the multiprocessing.managers.SyncManager methods (Queue, list, etc) to get "process safe" objects to place behind the Proxy.

@maxpolk maxpolk mannequin assigned docspython Dec 2, 2013
@maxpolk maxpolk mannequin added the docs Documentation in the Doc dir label Dec 2, 2013
@sbt
Copy link
Mannequin

sbt mannequin commented Dec 3, 2013

From what I remember a proxy method will be thread/process-safe if the referent's corresponding method is thread safe.

It should certainly be documented that the exposed methods of a proxied object should be thread-safe.

@iritkatriel iritkatriel changed the title multiprocessing Proxy docs need locking semantics explained [doc] multiprocessing Proxy docs need locking semantics explained Dec 6, 2021
@iritkatriel iritkatriel added the type-feature A feature request or enhancement label Dec 6, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 docs Documentation in the Doc dir easy expert-multiprocessing type-feature A feature request or enhancement
Projects
Status: No status
Development

No branches or pull requests

2 participants