@palaviv is it possible to generate and online dashboard that shows which tests pass and which need more help?

I am sure it is possible... Would you like to create such a dashboard? I think you can use testgrid for that.
My recommendation would be to run the test suite as described above and pick any of the exprectedFailure or skipped test marked as RustPython change. I am not sure a dashboard would be effective here.

@palaviv is it possible to get the report in a file in machine readable format, such as https://testanything.org ?

The test use test\libregtest and run the underline unittests tests. I am not sure if there is a builtin support in either for the format you want...

Would it help having the dashboard somewhere on the current website?

If you are looking for a dashboard, we can do two pages:

• one for tests passed/failed status
• another for the "what is left" script.

The goal would be for people to know what they can help on in one place.

I can probably get the output of both scripts to push to github repos and then use that as part of jekyll page or just as something people can read.

I think that would be cool, and it would probably require getting the test results in a machine readable format like TAP. It seems like tappy works perfectly fine on RustPython, the only issue is getting regrtest to run a custom unittest test runner.

@mireille-raad is it be possible to have single dashboard for both passed/failed and "what is left" by color coding them with green, red and yellow?

I don't know if it makes sense to have them in the same dashboard, since the regrtests are tests that get run and are well structured, while the whats_left script just shows standard library items that aren't present in RustPython; they don't really show the same kind of data.

@coolreader18 regrtests and standard library items are different test sets?

Well yeah, sort of; all whats_left.sh does is diff the list of standard library modules and the dir() outputs of them, and see what CPython has that RustPython doesn't. It doesn't actually run any code; you could "fix" something to remove it from the whats_left list by just doing def missingthing(): raise NotImplementedError.

@coolreader18 but do testnames match between what is available in RustPython and upstream? If they do, then it might be possible to get the test results from both test servers in TAP format and diff them.

That's not really necessary, since every test that's in CPython is passing (or should be, at least), so we can assume that any test that fails on RustPython should be passing, no diffing required.

whats_left.sh isn't for regrtests, it's only for the standard library.

@coolreader18 there should be platform dependent tests that are skipped in CPython. Therefore the system configuration of test environment for CPython and RustPython should be the same.

Ah, I see what you mean. My idea for that was to check if the skip message contains the string "RUSTPYTHON", since so far (just by convention) all the tests that we've marked as skip do.

@abitrolly do you think you can figure out how to generate one or a couple of machine readable files for the test results? (my preference would be json, but i can manage if xml or just plain text)... it looks like you have a specific idea of what need to be done. i can then put them together on a page.

I've created a PR for JSON output for CPython tests, #1834, you just need to run tests/jsontests.py with rustpython and it'll output all the results in a json file.

yay... i will take a look at the output, do a sketch. share here. get your feedback on gitter. if things look ok, we should be ready to roll soon!

I am excited to see how the JSON output looks like as well. Need to figure out how to provide it as a part of CI build result.

@coolreader18 can you automate running the tests and putting the .json file in the other repo https://github.com/RustPython/rustpython.github.io? under the _data/ directory?
https://github.com/RustPython/rustpython.github.io/tree/master/_data

Yeah, I think that would be possible, maybe running it once a week would be good?

@mireille-raad I've set up a cron job in #1866, and already tested it a few times, so the results file is already pushed to the git repo.

thank you! I saw it yesterday night and already created a draft page... I'll use the other repo for discussion on design, page layout and such.

@coolreader18 very nice. ) It would be easier to review changes in test coverage if the produced cpython_tests_results.json was indented.

@abitrolly for readiablity, you can always format the document in your code editor

or soon enough, go to the web page on rustpython.github.io ... adding indentation and spacing will make the file even bigger. usually you compress json files, not inflate them.

@abitrolly, yeah it's already 720kb, you can feel free to look through it and format it yourself if you'd like to review it manually, but I think the viewer that @mireille-raad is working on would be best, or just running rustpython -m test -v would be pretty readable as well.

@coolreader18 I meant reviewing diffs on GitHub. ) If git stores deltas in text format, it may reduce the storage size as well,