14,825 questions
1
vote
1
answer
119
views
Why doesn't multiprocessing.Process.start() in Python guarantee that the process has started?
Here is a code to demo my question:
from multiprocessing import Process
def worker():
print("Worker running")
if __name__ == "__main__":
p = Process(target=worker)
p....
- 387
-2
votes
0
answers
35
views
How to Implement an Asynchronous, Distributed Task Queue with Dynamic Worker Scaling Using Only Python’s Standard Library? [closed]
I’m looking for a way to implement a distributed task queue in Python that allows tasks to be added to a queue and processed asynchronously by workers. The system should support dynamic scaling of ...
0
votes
0
answers
64
views
Why does my first test run timeout (but second run does not) when running multiple Python scripts with ThreadPoolExecutor or ProcessPoolExecutor?
I am working on an automated grading tool for student programming submissions.
The process is:
Students submit their code (Python projects).
I clean and organise the submissions.
I set up a separate ...
0
votes
0
answers
61
views
Python - stress test all 24 CPU cores
I want to understand better how to work with multiprocessing in Python, and as a test, I wanted to set all cores in the CPU at 100%. First, I tested it with a CPU that has 8 cores, and all cores went ...
- 557
1
vote
1
answer
79
views
Multi-Process in deepface (face recognition)
Is there a way to send faces to the find function in parallel in face recognition?
(I want to create a batch processing or multi-processing mode so that I can process multiple faces on the GPU at the ...
- 11
-1
votes
2
answers
52
views
How to add a progress bar to Python multiprocessing.Pool without slowing it down (TQDM is 7x slower)?
I'm running a long computation using multiprocessing.Pool in Python, and I wanted to add a progress bar to track it. Naturally, I tried tqdm with pool.imap, but surprisingly, it's ~7.3x slower than ...
- 660
-1
votes
2
answers
75
views
Does multiprocessing Pool force the use of all cores?
The problem I experienced was that Pool used all of the processes, even though I had obviously passed an integer into the processes argument of Pool that was much smaller than the total number of ...
2
votes
2
answers
79
views
Cannot pickle local function when sending callable Filter objects via multiprocessing Queue
Problem Description
I'm developing a FilterBroker class that manages callable filters for subscriber processes. The broker receives functions wrapped in a Filter object via a message queue. However, I'...
- 61
0
votes
1
answer
58
views
Parallel processing with arguments to modify
I need to parallelize a function that modifies values of one of the arguments. For example modify this simple code:
def Test(i, a, length):
if i % 2 == 0:
for x in range(0, length, 2):
...
- 5,908
1
vote
1
answer
60
views
Dataframe can't multiprocess and reference in functions
I have a Pandas dataframe with columns that need to be sequentially calculated. Column A helps calculate Column B which helps calculate Column F, etc. Processing can be slow as Python is using only 1 ...
- 11
1
vote
1
answer
38
views
Can pthread_barrier_t be used across multiple processes with shared memory?
I'm working on a C++ application that launches multiple processes. All of them load the same executable and share a memory region using shm_open + mmap.
In that shared memory, we store a ...
- 111
1
vote
1
answer
66
views
Is with concurrent.futures.Executor blocking?
Why
when using separate with, the 2nd executor is blocked until all tasks from 1st executor is done
when using a single compound with, the 2nd executor can proceed while the 1st executor is still ...
- 517
0
votes
3
answers
184
views
Convert multiple-page PDF files to PNG quickly
I have a folder containing 600 PDF files, and each PDF has 20 pages. I need to convert each page into a high-quality PNG as quickly as possible.
I wrote the following script for this task:
import os
...
- 771
0
votes
1
answer
48
views
Deadlock in Multiprocessing Queue
I am developing a program to simulate a P2P network using the multiprocessing package. Specifically, I have created a Node class that inherits from multiprocessing.Process and contains all the basic ...
- 529
1
vote
0
answers
52
views
Logic behind using per cpu variable
I'm going through some linux kernel documentation on per cpu variables. I understand that having separate variable for each CPU help prevent cache line invalidation and make things faster.
But in ...
- 344