This issue comes from the discussion started here.

🚀 Feature

A GPU deterministic mode for scatter_add and similar operations

Motivation

I know that there is a general idea that non-deterministic algorithms do not have a lot of influence on the results, but, in some cases, there is a huge impact. It is a little bit frustrating not be able to reproduce an experiment even though you have all seeds, random states, init weights and so on saved. Moreover, this behavior sometimes can lead to wrong conclusions, when you are trying new methodologies. For instance, I started this thread because in pytorch_geometric some of the operations are based on non-deterministic implementations of pytorch. As you can see in this github issue this behavior is impacting the results. At the end this non-deterministic behavior has an impact in other libraries like pytorch geometric, causing the lack of reproducibility.

Pitch

I would like to be able to have a deterministic mode for the previously mentioned operations.

Alternatives

I know that the deterministic mode for these kinds of operations will come with a huge runtime trade-off. However, I found this article that can help with the deterministic mode of scatter_add.