All Questions
Tagged with parallel-processing python
3,547 questions
0
votes
0
answers
85
views
Pathos multiprocessing map not completing on Linux machine, works on Mac M2
I am using the pathos multiprocessing to parallelize a gradient calculation that is embarrassingly parallel, using finite difference.
Below is a high level example of how it is set up,
`
...
class ...
- 1
0
votes
1
answer
58
views
Parallel processing with arguments to modify
I need to parallelize a function that modifies values of one of the arguments. For example modify this simple code:
def Test(i, a, length):
if i % 2 == 0:
for x in range(0, length, 2):
...
- 5,908
1
vote
0
answers
39
views
How to make flask asynchronous to handle multiple users at the same time?
I spun up a cloud server and put my flask web app code plus a frontend in there. When a user runs an operation on the frontend this gets sent to the backend endpoint as a JSON, then processed and ...
- 11
0
votes
3
answers
184
views
Convert multiple-page PDF files to PNG quickly
I have a folder containing 600 PDF files, and each PDF has 20 pages. I need to convert each page into a high-quality PNG as quickly as possible.
I wrote the following script for this task:
import os
...
- 771
1
vote
0
answers
52
views
Paralellizing a Genetic Algorithm in Python is slower
I've been coding a Genetic Algorithm to solve TSP using python's DEAP library. I've implemented my own Graph class, and an Algorithm class that runs the GA on a Graph instance. However, when adding ...
- 51
-2
votes
1
answer
45
views
Is there any parallel computing involved in scipy.linalg.solve? [closed]
I tried to understand how scipy.linalg.solve works and if it involves parallel computing. If yes, can someone give me more information?
Also, I would like to know: if we want to compute the inverse ...
0
votes
0
answers
54
views
Slurm parallelization using python?
There is a job I want to run on slurm that needs parallelization because it is otherwise too slow. Even though the computations are independent of each other, they don't happen at the beginning of the ...
- 113
0
votes
1
answer
49
views
The thread running under multiprocessing.Process does not update its instance attributes [closed]
I want to run several class instances in parallel and update the instance attributes using Redis. On each class instance, I run a thread in the background to listen to redis for changes. When I run ...
- 185
1
vote
0
answers
36
views
Slurm: how to wait for scripts without affecting parallelism
I have a problem running my slurm code (I am quite a newbie).
I am trying to run slurm with tasks in parallel in the same job, lets call them task1, task2, ..., task28, with jobs in parallel.
The srun ...
0
votes
0
answers
88
views
Running pyscf parallelly using mpi4py
I am trying to use PySCF for electronic structure calculations and was curious about the parallelization schemes available to get the best performance.
I've installed PySCF via Spack, which included ...
- 101
0
votes
1
answer
70
views
Parallelize a list of subsequent api calls in Python
I have some code that cycles through some files and then makes subsequent calls to a "chain" of multiple APIs using contents from each file, where each API needs the output of the previous ...
- 125
-1
votes
1
answer
84
views
How to avoid repeated CSV reading in each parallel process in Python?
I am working on a Python project where I need to process a large CSV file using Polars. Currently, I am reading the CSV file inside each parallel process, which works but is inefficient because the ...
0
votes
0
answers
80
views
Some CSV outputs are empty in parallel Python processing, even though the code path should be triggered
I have a Python script that processes a batch of CSV files in parallel. Each input CSV has exactly 200 rows, so I use an if num_videos == 200: check to run a function like:
if num_videos == 200:
...
- 11
1
vote
2
answers
128
views
How to properly run Python multiprocessing pool inside larger loop and shut it down before next loop starts
I have a large script where I am processing terabytes of weather/climate data that comes in gridded format. I have a script that uses an outer loop (over years - 1979 to 2024), and for each year, ...
- 1,162
0
votes
0
answers
23
views
How to fix time.process_time() not working when wrapped around a function run with Numba jit, with parallelisation=True
I have a function that has a numba @njit wrapper around it to make it faster, I've set parallel=True to make it run faster. And now want to measure the time it takes, using time.process_time(), ...
