Skip to content

bpo-9263: _PyObject_Dump() detects freed memory #10061

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 23, 2018
Merged

bpo-9263: _PyObject_Dump() detects freed memory #10061

merged 2 commits into from
Oct 23, 2018

Conversation

vstinner
Copy link
Member

@vstinner vstinner commented Oct 23, 2018

_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "" in that case.

The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.

https://bugs.python.org/issue9263

_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "<freed object>" in that case.

The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.
Copy link
Member

@matrixise matrixise left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it indicated in the doc ?

@vstinner
Copy link
Member Author

Is it indicated in the doc ?

The heuristic is not reliable and it shouldn't be used directly. I chose to not expose it.

It's very new, I would prefer to use it during one release to see if it's reliable enough or not, to decide to make it more visible.

Note: Private functions are not documented (currently).

@vstinner vstinner merged commit 82af0b6 into python:master Oct 23, 2018
@vstinner vstinner deleted the freed branch October 23, 2018 15:39
vstinner added a commit that referenced this pull request Nov 22, 2018
…10662)

* bpo-9263: _PyObject_Dump() detects freed memory (GH-10061)

_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "<freed object>" in that case.

The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.

(cherry picked from commit 82af0b6)

* bpo-9263: Fix _PyObject_Dump() for freed object (#10661)

If _PyObject_Dump() detects that the object is freed, don't try to
dump it (exit immediately).

Enhance also _PyObject_IsFreed(): it now detects if the pointer
itself looks like freed memory.

(cherry picked from commit 2cf5d32)
vstinner added a commit that referenced this pull request Nov 22, 2018
…GH-10663)

* bpo-9263: _PyObject_Dump() detects freed memory (GH-10061)

_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "<freed object>" in that case.

The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.

(cherry picked from commit 82af0b6)

* bpo-9263: Fix _PyObject_Dump() for freed object (#10661)

If _PyObject_Dump() detects that the object is freed, don't try to
dump it (exit immediately).

Enhance also _PyObject_IsFreed(): it now detects if the pointer
itself looks like freed memory.

(cherry picked from commit 2cf5d32)
(cherry picked from commit 95036ea)
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86 Windows7 3.6 has failed when building commit c9b3fc6.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/90/builds/678) and take a look at the build logs.
  4. Check if the failure is related to this commit (c9b3fc6) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/90/builds/678

Click to see traceback logs
Reset branch '3.6'

The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, functio
n, script file, or operable program. Check the spelling of the name, or if a pa
th was included, verify that the path is correct and try again.
At line:1 char:18
+ Invoke-WebRequest <<<<  https://aka.ms/nugetclidl -OutFile 'D:\cygwin\home\db
3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe'
    + CategoryInfo          : ObjectNotFound: (Invoke-WebRequest:String) [], C 
   ommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
'py' is not recognized as an internal or external command,
operable program or batch file.
'"D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe"' is not recognized as an internal or external command,
operable program or batch file.
The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, functio
n, script file, or operable program. Check the spelling of the name, or if a pa
th was included, verify that the path is correct and try again.
At line:1 char:18
+ Invoke-WebRequest <<<<  https://aka.ms/nugetclidl -OutFile 'D:\cygwin\home\db
3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe'
    + CategoryInfo          : ObjectNotFound: (Invoke-WebRequest:String) [], C 
   ommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
'py' is not recognized as an internal or external command,
operable program or batch file.
'"D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe"' is not recognized as an internal or external command,
operable program or batch file.
Could Not Find D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\python*.zip
The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, functio
n, script file, or operable program. Check the spelling of the name, or if a pa
th was included, verify that the path is correct and try again.
At line:1 char:18
+ Invoke-WebRequest <<<<  https://aka.ms/nugetclidl -OutFile 'D:\cygwin\home\db
3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe'
    + CategoryInfo          : ObjectNotFound: (Invoke-WebRequest:String) [], C 
   ommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
'py' is not recognized as an internal or external command,
operable program or batch file.
'"D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe"' is not recognized as an internal or external command,
operable program or batch file.

test_fork1 skipped -- object <module 'os' from 'D:\\cygwin\\home\\db3l\\buildarea\\3.6.bolen-windows7\\build\\lib\\os.py'> has no attribute 'fork'
test_wait4 skipped -- object <module 'os' from 'D:\\cygwin\\home\\db3l\\buildarea\\3.6.bolen-windows7\\build\\lib\\os.py'> has no attribute 'fork'
test_fcntl skipped -- No module named 'fcntl'
minkernel\crts\ucrt\src\appcrt\lowio\write.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)
minkernel\crts\ucrt\src\appcrt\lowio\close.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)
minkernel\crts\ucrt\src\appcrt\lowio\close.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)
test_pipes skipped -- pipes module only works on posix
test_poll skipped -- select.poll not defined
test_ossaudiodev skipped -- No module named 'ossaudiodev'
test_epoll skipped -- test works only on Linux 2.6
test_crypt skipped -- No module named '_crypt'
test_gdb skipped -- Couldn't find gdb on the path
test_nis skipped -- No module named 'nis'
test_wait3 skipped -- os.fork not defined
test_resource skipped -- No module named 'resource'
test_ioctl skipped -- No module named 'fcntl'
test_dbm_gnu skipped -- No module named '_gdbm'
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_dbm_ndbm skipped -- No module named '_dbm'
test_spwd skipped -- No module named 'spwd'
stty: standard input: Inappropriate ioctl for device
test_kqueue skipped -- test works only on BSD
test_threadsignals skipped -- Can't test signal on win32
minkernel\crts\ucrt\src\appcrt\lowio\write.cpp(49) : Assertion failed: (_osfile(fh) & FOPEN)
test_pty skipped -- No module named 'termios'
test_devpoll skipped -- test works only on Solaris OS family
test_posix skipped -- No module named 'posix'
test_openpty skipped -- os.openpty() not available.
test_curses skipped -- No module named '_curses'
test_pwd skipped -- No module named 'pwd'
test_readline skipped -- No module named 'readline'
Timeout (0:15:00)!
Thread 0x00000608 (most recent call first):
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\connection.py", line 306 in _recv_bytes
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\connection.py", line 250 in recv
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\managers.py", line 517 in start
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\_test_multiprocessing.py", line 2558 in test_rapid_restart
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\case.py", line 605 in run
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\case.py", line 653 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\unittest\runner.py", line 176 in run
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\support\__init__.py", line 1914 in _run_suite
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\support\__init__.py", line 2008 in run_unittest
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 179 in runtest_inner
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 124 in runtest
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 72 in run_tests_slave
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\main.py", line 587 in _main
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\main.py", line 571 in main
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\libregrtest\main.py", line 627 in main
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\test\regrtest.py", line 50 in <module>
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\runpy.py", line 85 in _run_code
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\connection.py", line 942, in rebuild_pipe_connection
    handle = dh.detach()
  File "D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\lib\multiprocessing\reduction.py", line 129, in detach
    self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied
test_syslog skipped -- No module named 'syslog'
test_grp skipped -- No module named 'grp'
test test_multiprocessing_spawn failed

The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, functio
n, script file, or operable program. Check the spelling of the name, or if a pa
th was included, verify that the path is correct and try again.
At line:1 char:18
+ Invoke-WebRequest <<<<  https://aka.ms/nugetclidl -OutFile 'D:\cygwin\home\db
3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe'
    + CategoryInfo          : ObjectNotFound: (Invoke-WebRequest:String) [], C 
   ommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
'py' is not recognized as an internal or external command,
operable program or batch file.
'"D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe"' is not recognized as an internal or external command,
operable program or batch file.
The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, functio
n, script file, or operable program. Check the spelling of the name, or if a pa
th was included, verify that the path is correct and try again.
At line:1 char:18
+ Invoke-WebRequest <<<<  https://aka.ms/nugetclidl -OutFile 'D:\cygwin\home\db
3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe'
    + CategoryInfo          : ObjectNotFound: (Invoke-WebRequest:String) [], C 
   ommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
 
'py' is not recognized as an internal or external command,
operable program or batch file.
'"D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\\..\externals\nuget.exe"' is not recognized as an internal or external command,
operable program or batch file.
Could Not Find D:\cygwin\home\db3l\buildarea\3.6.bolen-windows7\build\PCbuild\python*.zip

@vstinner
Copy link
Member Author

Hi! The buildbot x86 Windows7 3.6 has failed when building commit c9b3fc6.

That's my old friend https://bugs.python.org/issue34513

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants