Execute C++ based GTest unit tests from a python wrapper #64067

malfet · 2021-08-26T23:47:20Z

Motivation

Currently lots of C++ based unit tests are executed directly from test.sh/win-test.sh for example:

Line 317 in 0bd8d09

    
           "$TORCH_BIN_DIR"/FileStoreTest --gtest_output=xml:$TEST_REPORTS_DIR/FileStoreTest.xml

Which have following drawbacks:

It excluded those test runtime from auto-sharding/auto-categorization
Make them subject of running on only particular platform (as win-test.sh and test.sh are two different files)
Make them abort abort test execution on the first failure despite existing policy
Make them susceptible of not reporting test execution results in standard XML format (if someone forgets to add --gtest_output) argument

Pitch

Add a .json file with all C++ GTest based tests or just add them directly to run_test.py akin to

pytorch/test/run_test.py

Line 53 in 0bd8d09

TESTS = [

This would allow our auto-sharding system to balance them correctly across the shards, continue execution over failed tests as well as using re-using some of standard decorators for excluding test runs on certain configurations

cc @ezyang @seemethere @malfet @walterddr @lg20987 @pytorch/pytorch-dev-infra

The text was updated successfully, but these errors were encountered:

Prashant723 · 2021-08-27T09:28:08Z

can I contribute to this? this will be my first, can anybody help me!

Akshat1331 · 2021-08-27T10:08:35Z

can i also work on this? it will be mine first too, soo can we contribute together @Prashant723 cause i am a begineer too.

Prashant723 · 2021-08-27T12:14:22Z

@Akshat1331 ok mail me at gargprashant1@yahoo.com

nikhilsajwan · 2021-08-28T11:39:30Z

hi, i would also like to contribute to this

Akshat1331 · 2021-08-28T11:50:38Z

Yeah sure we together will contribute on this issue on Monday as I have some work on hand now. Thanks & Regards

…

On Sat, 28 Aug 2021, 17:09 nikhilsajwan, ***@***.***> wrote: hi, i would also like to contribute to this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64067 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQLMPVQSBTR5GUO5LCMXF7TT7DDH5ANCNFSM5C4MRQHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

S-T-A-R-L-O-R-D · 2021-08-28T14:22:28Z

I cant find c++ Gtest, can anyone tell me where are they stored.

JaiJoshi123 · 2021-08-29T19:04:07Z

I would love to contribute too!

…pper Summary: This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 783d908a2d3a97b5e65e8b5d3b28b79e715b1cca

FlashStar09 · 2021-08-31T10:20:57Z

I would like to contribute here.

carolyngundimi · 2021-08-31T10:59:05Z

would like to contribute

Mansi2908 · 2021-08-31T11:03:18Z

I would like to contribute

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 5ba0d2993e2980c9272f8d627c45ed0965d0483b

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: ea3cb5de079b2916fd3515df9057454223f7c24f

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: fb34f398a6d5de26c11a0cf494991aeafea6b088

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 49b4ab43353d14dcdf43aaa8d9ae6901bbff04b9

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 128ab6d932591b6487ce430b5df0df651f4e7792

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 327f77fa7a2d25a1cfbdc24f358436ca2e197538

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 184214ffa35ed764f1c8b5fca9d7d4d33a2cc2ee

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 7566ba6c7d4dc47011368003b96be1f1ee41f778

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 7eaa189c7fcd6f09b1b4ffdaf1683bd9cd59f445

…pper (pytorch#64252) Summary: Pull Request resolved: pytorch#64252 This is the beginning of transitioning away from test.sh: pytorch#64067 Moved execution of FileStoreTest, HashStoreTest, TCPStoreTest from test.sh to run_test.py. Since these are compiled tests, the logic of test.sh was just migrated and the tests are run through subprocess. Test Plan: ``` python3 test/run_test.py --include cpp/c10d/FileStoreTest cpp/c10d/HashStoreTest cpp/c10d/TCPStoreTest ``` Wait for CI Differential Revision: D30662844 fbshipit-source-id: 3807399df05a0f5a08e06f11eba68f602b2bf014

This adds a Python-based wrapper for running C++ tests intended to be run in CI as part of #64067. Eventually this should replace all manual invocations of running a C++ test in CI. For local development running the binaries manually is still the best UX since you have full control of gtest flags and can avoid Python's startup time. For simple tests that can just be run out-of-the-box, add them to the [allowlist](https://github.com/pytorch/pytorch/pull/65514/files#diff-c9f1a162beb062b5905b0b8ae2a7d39ba453b1a821b62094687dbb8ce67763dbR39). This PR has an example of a more complicated invocation for `test_jit` as well which can be implemented in Python with some custom logic. Differential Revision: [D31296508](https://our.internmc.facebook.com/intern/diff/D31296508) [ghstack-poisoned]

malfet added module: ci triaged enhancement labels Aug 26, 2021

malfet added this to Needs Triage in PyTorch Dev Infra via automation Aug 26, 2021

zhouzhuojie added the good first issue label Aug 26, 2021

haovic self-assigned this Aug 30, 2021

janeyx99 moved this from Needs Triage to Sprint TODO in PyTorch Dev Infra Aug 30, 2021

haovic mentioned this issue Aug 31, 2021

Move some distributed C++ GTest unit tests from test.sh to Python wrapper #64252

Closed

haovic linked a pull request that will close this issue Aug 31, 2021

Move some distributed C++ GTest unit tests from test.sh to Python wrapper #64252

Closed

malfet moved this from Needs Triage to Backlog in PyTorch Dev Infra Oct 11, 2021

malfet moved this from Backlog to Sprint TODO in PyTorch Dev Infra Oct 11, 2021

driazati mentioned this issue Oct 18, 2021

Use a Python wrapper for C++ tests #65514

Open

driazati mentioned this issue Oct 18, 2021

Migrate C++ tests to Python runner #66824

Open

13 tasks

pytorch / pytorch Public

Execute C++ based GTest unit tests from a python wrapper #64067

Execute C++ based GTest unit tests from a python wrapper #64067

malfet commented Aug 26, 2021 •

edited

Prashant723 commented Aug 27, 2021

Akshat1331 commented Aug 27, 2021

Prashant723 commented Aug 27, 2021

nikhilsajwan commented Aug 28, 2021

Akshat1331 commented Aug 28, 2021

S-T-A-R-L-O-R-D commented Aug 28, 2021

JaiJoshi123 commented Aug 29, 2021

FlashStar09 commented Aug 31, 2021

carolyngundimi commented Aug 31, 2021

Mansi2908 commented Aug 31, 2021

pytorch / pytorch Public

Execute C++ based GTest unit tests from a python wrapper #64067

Execute C++ based GTest unit tests from a python wrapper #64067

Comments

malfet commented Aug 26, 2021 • edited

Motivation

Pitch

Prashant723 commented Aug 27, 2021

Akshat1331 commented Aug 27, 2021

Prashant723 commented Aug 27, 2021

nikhilsajwan commented Aug 28, 2021

Akshat1331 commented Aug 28, 2021

S-T-A-R-L-O-R-D commented Aug 28, 2021

JaiJoshi123 commented Aug 29, 2021

FlashStar09 commented Aug 31, 2021

carolyngundimi commented Aug 31, 2021

Mansi2908 commented Aug 31, 2021

malfet commented Aug 26, 2021 •

edited