Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python.

Many Repo users have encountered resource leak warnings with Python 3.9.
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )

import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()

Tested on macOS with Python 3.9.7

python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.

This bug will 100% reproduce when then main module uses named resources as global variables and uses spawn context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in spawn context.

1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.

And I uploaded a MR.

I added core devs related to multiprocessing into a nosy list so they got a notification and the PR will be evaluated and merged faster. FYI, the devs are Davin Potts and Antoine Pitrou (as per <https://devguide.python.org/experts/\>).

loop file owner.

update,

Confirmed to affect all versions above 3.8+.

It's really annoying.

Command line tools like Repo get a warning after every (success) command.
And mess up with shell.

-------------------------------------------------------------------

$ repo status .
project ksbox/                                  branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo sync .
Fetching: 100% (1/1), done in 1.184s
Garbage collecting: 100% (1/1), done in 0.013s
repo sync has finished successfully.
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo status .
project ksbox/ branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

Also hitting this problem. Would be grateful for fixing it, since there is already PR created

@pitrou @benjaminp @1st1 @applio

@gpshead

any news on this issue?

Strangely, I see this error only when running inside PyCharm in debug mode, and it happens non-deterministically. When I run in cmd line it does not happen.

This problem is related to spawn method to create a new process (which is the default way in windows and macos since python 3.8). Example below works fine without warning⬇️

import multiprocessing as mp

mp.set_start_method('fork')  # switch to fork
global_resource = mp.Semaphore()


def submain():
    pass


if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()

To avoid this warning in another way, create global_resource inside if __name__ == '__main__' to avoid re-creation in subprocess. This question may be helpful

import multiprocessing as mp


def submain():
    pass


if __name__ == '__main__':
    global_resource = mp.Semaphore()
    p = mp.Process(target=submain)
    p.start()
    p.join()

Test environment is macos python 3.9, example above can be a temp solution, hoping MR fix this soon.

@HFrost0

Thanks for your workaround and tracking down.

Unfortunately, in the case of Google Repo. The global resource is not at the main module. Instead a module which the main module imports creates a global_resource. So, the workaround won't help in this case. : (

cc @pitrou @gpanders
Any idea or workaround?

It would be nice to have this fixed....this warning appears on most of the repo commands I use :-(

It would be nice to have this fixed....this warning appears on most of the repo commands I use :-(

me, too :-(

Okay, this is fixed now. Thanks @jxdabc for the contribution!