Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pdb & multiprocessing.Pool: AttributeError: module '__main__' has no attribute '__spec__' #87115

Closed
macfreek mannequin opened this issue Jan 17, 2021 · 5 comments
Closed

pdb & multiprocessing.Pool: AttributeError: module '__main__' has no attribute '__spec__' #87115

macfreek mannequin opened this issue Jan 17, 2021 · 5 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir topic-importlib topic-multiprocessing

Comments

@macfreek
Copy link
Mannequin

macfreek mannequin commented Jan 17, 2021 •
edited by bedevere-app bot

BPO 42949
Nosy @macfreek

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-01-17.22:10:12.759>
labels = ['3.8', 'library', '3.9', '3.10']
title = "pdb & multiprocessing.Pool: AttributeError: module '__main__' has no attribute '__spec__'"
updated_at = <Date 2021-01-17.22:10:12.759>
user = 'https://github.com/macfreek'

bugs.python.org fields:

activity = <Date 2021-01-17.22:10:12.759>
actor = 'macfreek'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-01-17.22:10:12.759>
creator = 'macfreek'
dependencies = []
files = []
hgrepos = []
issue_num = 42949
keywords = []
message_count = 1.0
messages = ['385168']
nosy_count = 1.0
nosy_names = ['macfreek']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue42949'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs
@macfreek
Copy link
Mannequin Author

macfreek mannequin commented Jan 17, 2021

Summary:

multiprocessing.Pool contains a bug when the script is invoked with pdb.

Steps to reproduce:

Consider the following script:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

When called as python3.10 test.py, this works fine, and returns [1, 4, 9]

When called as python3.10 -m pdb test.py, it gives the following exception:

freek@minnie ~ » python3.9 -m pdb test.py
> /Users/freek/test.py(3)<module>()
-> from multiprocessing import Pool
(Pdb) c
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py", line 1704, in main
    pdb._runscript(mainpyfile)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/freek/test.py", line 3, in <module>
    from multiprocessing import Pool
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 183, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

I can reproduce this in Python 3.8.7, 3.9.1, 3.10.0a3. It works fine in Python 3.7.9.

A workaround is to define __spec__ in main:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    __spec__ = None
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

I'd say that multiprocessing/spawn.py", line 183, in get_preparation_data is flawed
main_mod_name = getattr(main_module.__spec__, "name", None)

@macfreek macfreek mannequin added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir labels Jan 17, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@qwertyuu
Copy link

This is very frustrating. Having the same issue

@bedevere-bot bedevere-bot mentioned this issue Jul 7, 2023
Closed
@gaogaotiantian
Copy link
Member

This does not reproduce in 3.11/3.13. Consider it as fixed.

@Soudeh-Jahanshahi Soudeh-Jahanshahi mentioned this issue Nov 29, 2023
Closed
@mx781
Copy link

mx781 commented Feb 29, 2024 •
edited

@gaogaotiantian
This still seems to occur in 3.11.6 as well as 3.13.0a4 if one uses "forkserver" as the start method.

If you run the following program with python -m pdb main.py, you still get AtributeError: module '__main__' has no attribute '__spec__':

import multiprocessing


def f(x):
    return x * x


if __name__ == "__main__":
    context = multiprocessing.get_context("forkserver")
    context.Process(target=f, args=(1,)).start()

Setting __spec__ = None also still circumvents the issue.

@bedevere-app bedevere-app bot mentioned this issue Feb 29, 2024
Merged
iritkatriel pushed a commit that referenced this issue Feb 29, 2024
@gaogaotiantian
gh-87115: Set __main__.__spec__ to None in pdb (#116141)
ccfc042
@gaogaotiantian
Copy link
Member

Okay now this is really fixed :)

gaogaotiantian added a commit to gaogaotiantian/cpython that referenced this issue Feb 29, 2024
@gaogaotiantian
pythongh-87115: Set __main__.__spec__ to None in pdb (python#116141)
8898070 
(cherry picked from commit ccfc042)
gaogaotiantian added a commit to gaogaotiantian/cpython that referenced this issue Feb 29, 2024
@gaogaotiantian
[3.12] pythongh-87115: Set __main__.__spec__ to None in pdb (pyth…
7e32e59 
…onGH-116141)

(cherry picked from commit ccfc042)

Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
This was referenced Feb 29, 2024
Merged
Merged
gaogaotiantian added a commit to gaogaotiantian/cpython that referenced this issue Feb 29, 2024
@gaogaotiantian
pythongh-87115: Set __main__.__spec__ to None in pdb (python#116141)
d258bd0 
(cherry picked from commit ccfc042)
gaogaotiantian added a commit to gaogaotiantian/cpython that referenced this issue Feb 29, 2024
@gaogaotiantian
[3.11] pythongh-87115: Set __main__.__spec__ to None in pdb (pyth…
02c005c 
…onGH-116141)

(cherry picked from commit ccfc042)

Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
iritkatriel pushed a commit that referenced this issue Feb 29, 2024
@gaogaotiantian
[3.12] gh-87115: Set __main__.__spec__ to None in pdb (GH-116141) (…
40e9295 
…#116154)

* gh-87115: Set `__main__.__spec__` to `None` in pdb (#116141)

(cherry picked from commit ccfc042)

* [3.12] gh-87115: Set `__main__.__spec__` to `None` in pdb (GH-116141)
(cherry picked from commit ccfc042)

Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
iritkatriel pushed a commit that referenced this issue Feb 29, 2024
@gaogaotiantian
[3.11] gh-87115: Set __main__.__spec__ to None in pdb (GH-116141) (…
d3282f5 
…#116155)

* gh-87115: Set `__main__.__spec__` to `None` in pdb (#116141)

(cherry picked from commit ccfc042)

* [3.11] gh-87115: Set `__main__.__spec__` to `None` in pdb (GH-116141)
(cherry picked from commit ccfc042)

Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
@gaogaotiantian @woodruffw
pythongh-87115: Set __main__.__spec__ to None in pdb (python#116141)
f38537f
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@gaogaotiantian @adorilson
pythongh-87115: Set __main__.__spec__ to None in pdb (python#116141)
e04595d
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@gaogaotiantian @diegorusso
pythongh-87115: Set __main__.__spec__ to None in pdb (python#116141)
589e05b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir topic-importlib topic-multiprocessing
Projects
Multiprocessing issues
Status: Done
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vstinner @mx781 @qwertyuu @serhiy-storchaka @gaogaotiantian