New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Metabug: Improving C-level coverage #94808
Comments
@mdboom: I think a better approach would be to create sub-checklists or a single issue per file with another checklist for the paths that need coverage, instead of opening dozen of issues for each individual path. Even having an issue per file will result in 548 new issues though, so I'm not sure if we want to do that preemptively for each file. I'd say it's better to keep the checklist with the files and paths here, and then directly create PRs for each (or multiple) paths. Another option is to create a project, and handle it there. You can add a new custom field to specify the file, and create draft issues for each path without creating actual issues here (let me know if you need more help with that). |
I thought about that, but most of the uncovered areas will have independent fixes, and bug-per-area sets up that work to happen. I'm starting with files that have seen a lot of changes lately, so we're seeing quite a few issues in them. I suspect most files will not be that way -- most will probably have no issues, and we can just check the box here and not create a flood of issues where they aren't needed.
Agreed.
I'm happy to use a project instead if you'd prefer. IIUC, it's pretty easy to move the existing issues already created into it. I know I can't create a project, but once it's created, I don't know what my limited permissions will allow me to do. |
Maybe it would be better to start by trimming down the list to remove files that have no issues, and see how many files are left first. The remaining files and their paths could be listed here, and if someone starts working on them and wants to discuss the approach they could create the issues lazily or directly create PRs that refers to this meta issues if the fix is straightforward enough. I don't think having lot of almost empty issues (like #94817) will help this effort. |
I programmatically removed files that have 100% coverage, and then also (manually) removed anything platform specific and all of
My thinking was this checklist was to say "someone has read through the file and identified all of the potential issues". Some of the issues will have simple tests that can be written, some are dead code, some might reveal bugs, but we need a place to have those discussions and deal with them individually (not here, ideally). For really simple ones, if it's ok to just reference this bug, that's fine by me, but we still need a way to keep track of which files have been vetted to track progress.
Maybe not in general. The ones I've filed so far are directly tied to the faster CPython work, and the ability to move with more confidence there. So there are motivated people who want to close these bugs. But I empathize with the concern of just creating bugs for the sake of creating them. |
This is much better, thanks for doing this!
My idea was to do something like:
Once people start working on these items, there are different options. They can:
The first two options work for straightforward cases, the third works if the problems are similar, and the last for more complex cases that require discussions. You can also do this incrementally, i.e. start with a list like the one above, then convert some to PRs and some to issues as needed. In addition, if you hover with the mouse over a checklist item, a ⨀ icon will appear to the right which will allow you to quickly convert the item to an actual issue if/when you decide to open a discussion about a specific file or problem.
That's great to hear, I just wanted to avoid creating a bunch of issues that might end up sitting there indefinitely :) |
Thanks. The revised plan you suggested should work just fine. I didn't know about the feature to automatically open an issue from a checklist item. |
@pablogsal, should we be backporting these coverage-improving tests to 3.11? I think we should be consistent in our handling of all of them. I don't think we usually backport test-only PRs, but perhaps these are sort of a special case. |
`bool_new` had no coverage. Automerge-Triggered-By: GH:brandtbucher
Test only PRs are generally ok to backport |
…gits is provided (pythonGH-94860) (cherry picked from commit 625ba9b) Co-authored-by: Michael Droettboom <mdboom@gmail.com>
`bool_new` had no coverage. Automerge-Triggered-By: GH:brandtbucher (cherry picked from commit df4d53a) Co-authored-by: Michael Droettboom <mdboom@gmail.com>
* main: (2069 commits) pythongh-96512: Move int_max_str_digits setting to PyConfig (python#96944) pythongh-94808: Coverage: Check picklablability of calliter (python#95923) pythongh-94808: Add test coverage for PyObject_HasAttrString (python#96627) pythongh-94732: Fix KeyboardInterrupt race in asyncio run_forever() (python#97765) Fix typos in `bltinmodule.c`. (pythonGH-97766) pythongh-94808: `_PyLineTable_StartsLine` was not used (pythonGH-96609) pythongh-97681: Remove Tools/demo/ directory (python#97682) Fix typo in unittest docs (python#97742) pythongh-97728: Argument Clinic: Fix uninitialized variable in the Py_UNICODE converter (pythonGH-97729) pythongh-95913: Fix PEP number in PEP 678 What's New ref label (python#97739) pythongh-95913: Copyedit/improve New Modules What's New section (python#97721) pythongh-97740: Fix bang in Sphinx C domain ref target syntax (python#97741) pythongh-96819: multiprocessing.resource_tracker: check if length of pipe write <= 512 (python#96890) pythongh-97706: multiprocessing tests: Delete unused variable `rand` (python#97707) pythonGH-85447: Clarify docs about awaiting future multiple times (python#97738) [docs] Update logging cookbook with recipe for using a logger like an output… (pythonGH-97730) pythongh-97607: Fix content parsing in the impl-detail reST directive (python#97652) pythongh-95975: Move except/*/finally ref labels to more precise locations (python#95976) pythongh-97591: In `Exception.__setstate__()` acquire strong references before calling `tp_hash` slot (python#97700) pythongh-95588: Drop the safety claim from `ast.literal_eval` docs. (python#95919) ...
#96610) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
… method (pythonGH-96610) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit e39ae6bef2c357a88e232dcab2e4b4c0f367544b) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
* main: pythonGH-88050: fix race in closing subprocess pipe in asyncio (python#97951) pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962) pythongh-95986: Fix the example using match keyword (python#95989) pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944) pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929) pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610) pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703) pythonGH-88968: Add notes about socket ownership transfers (python#97936) pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528)
…led (pythonGH-95926) * pythongh-94808: Coverage: Test that maximum indentation level is handled * Use "compile" rather than "exec" (cherry picked from commit 23e83a8) Co-authored-by: Michael Droettboom <mdboom@gmail.com>
…led (python#95926) * pythongh-94808: Coverage: Test that maximum indentation level is handled * Use "compile" rather than "exec"
…dled (GH-95926) (#97979) gh-94808: Coverage: Test that maximum indentation level is handled (GH-95926) * gh-94808: Coverage: Test that maximum indentation level is handled * Use "compile" rather than "exec" (cherry picked from commit 23e83a8) Co-authored-by: Michael Droettboom <mdboom@gmail.com> Co-authored-by: Michael Droettboom <mdboom@gmail.com>
* main: fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965) pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990) pythongh-97850: Remove the open issues section from the import reference (python#97935) Docs: pin sphinx-lint (pythonGH-97992) pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591) Add Pynche's move to the What's new in 3.11 (python#97974) pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785) pythongh-86482: Document assignment expression need for ()s (python#23291) pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949) pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit 72c166add89a0cd992d66f75ce94eee5eb675a99) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> (cherry picked from commit 72c166a) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
The test failed on a buildbot because the pointer was only 7 hex characters. To be safe, I bumped it down to 3: 4 in case we have 32-bit platforms, and 3 in case the pointer is very small.
The test failed on a buildbot because the pointer was only 7 hex characters. To be safe, I bumped it down to 3: 4 in case we have 32-bit platforms, and 3 in case the pointer is very small.
* main: (53 commits) pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926) pythonGH-88050: fix race in closing subprocess pipe in asyncio (python#97951) pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962) pythongh-95986: Fix the example using match keyword (python#95989) pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944) pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929) pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610) pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703) pythonGH-88968: Add notes about socket ownership transfers (python#97936) pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) ...
* main: (38 commits) pythongh-92886: make test_ast pass with -O (assertions off) (pythonGH-98058) pythongh-92886: make test_coroutines pass with -O (assertions off) (pythonGH-98060) pythongh-57179: Add note on symlinks for os.walk (python#94799) pythongh-94808: Fix regex on exotic platforms (python#98036) pythongh-90085: Remove vestigial -t and -c timeit options (python#94941) pythonGH-83901: Improve Signature.bind error message for missing keyword-only params (python#95347) pythongh-61105: Add default param, note on using cookiejar subclass (python#95427) pythongh-96288: Add a sentence to `os.mkdir`'s docstring. (python#96271) pythongh-96073: fix backticks in NEWS entry (pythonGH-98056) pythongh-92886: [clinic.py] raise exception on invalid input instead of assertion (pythonGH-98051) pythongh-97997: Add col_offset field to tokenizer and use that for AST nodes (python#98000) pythonGH-88968: Reject socket that is already used as a transport (python#98010) pythongh-96346: Use double caching for re._compile() (python#96347) pythongh-91708: Revert params note in urllib.parse.urlparse table (python#96699) pythongh-96265: Fix some formatting in faq/design.rst (python#96924) pythongh-73196: Add namespace/scope clarification for inheritance section (python#92840) pythongh-97646: Change `.js` and `.mjs` files mimetype to conform to RFC 9239 (python#97934) pythongh-97923: Always run Ubuntu SSL tests with others in CI (python#97940) pythongh-97956: Mention `generate_global_objects.py` in `AC How-To` (python#97957) pythongh-96959: Update HTTP links which are redirected to HTTPS (python#98039) ...
mdboom commentedJul 13, 2022
•
edited by orsenthil
This bug is going to be used to track work in a other bugs to improve the C-level coverage of the CPython test suite.
There is a set of baseline coverage results on main that can be used to find coverage gaps.
The plan, discussed on discuss.python.org is as follows:
Related work:
There is related work to publish coverage results from CPython on a regular basis, but this issue is concerned with using those results to actually reduce our gaps in coverage.
List of source files:
Include/internal/pycore_asdl.h
Include/internal/pycore_bitutils.h
Include/internal/pycore_call.h
Include/internal/pycore_code.h
Include/internal/pycore_frame.h
Include/internal/pycore_moduleobject.h
Include/internal/pycore_object.h
Include/internal/pycore_pymath.h
Include/internal/pycore_pymem.h
Include/internal/pycore_pystate.h
Include/object.h
Include/pydtrace.h
Objects/abstract.c
PyBuffer_FromContiguous
,PyObject_CopyData
,PyBuffer_FillContiguousStrides
PyNumber_Check
doesn't testcomplex
PySequence_Repeat
andPySequence_InPlaceRepeat
have no coveragePySequence_SetItem
with a negative index is untestedPySequence_SetSlice
andPySequence_DelSlice
are untestedPyMapping_HasKey
andPyMapping_HasKeyString
are untestedObjects/accu.c
Objects/boolobject.c
Objects/bytearrayobject.c
Objects/bytes_methods.c
Objects/bytesobject.c
InIt's only the case where the underlying libc is broken that isn't tested.PyBytes_FromFormatV
, the special handling of%p
isn't tested.Objects/call.c
PyEval_CallObjectWithKeywords
has no coverage_PyObject_CallMethodId_SizeT
has no coverageObjects/capsule.c
Objects/cellobject.c
Objects/classobject.c
Objects/codeobject.c
_PyLineTable_StartsLine
was not used #96609Objects/complexobject.c
Objects/descrobject.c
Objects/dictobject.c
dictresize
convert split table into new combined table" is uncovered._PyDict_GetItemHint
has no coverageObjects/enumobject.c
Objects/exceptions.c
Objects/fileobject.c
PyFile_FromFd
has no coveragePyFile_GetLine
overbytes
input has no coverageObjects/floatobject.c
Objects/frameobject.c
Objects/funcobject.c
PyFunction_GetCode
,PyFunction_GetGlobals
,PyFunction_GetModule
,PyFunction_GetDefaults
,PyFunction_SetDefaults
,PyFunction_GetKwDefaults
,PyFunction_SetKwDefaults
,PyFunction_GetClosure
,PyFunction_SetClosure
,PyFunction_GetAnnotations
,PyFunction_SetAnnotations
Objects/genericaliasobject.c
Objects/genobject.c
gen_new_with_qualname
and APIPyGen_NewWithQualName
andPyGen_New
have no coverage.PyCoro_New
has no coveragePyAsyncGen_New
has no coverageasync_gen_athrow_send
has poor coverageObjects/interpreteridobject.c
Objects/iterobject.c
Objects/listobject.c
Objects/longobject.c
_PyLong_Sing_t_Converter
has no coveragelong_format_binary
doesn't test outputting to UCS2 or UCS4int_bit_length_impl
andint_bit_count_impl
doesn't cover the case where expression overflowsObjects/memoryobject.c
init_slice
is not well-coveredObjects/methodobject.c
Objects/moduleobject.c
PyModule_GetFilename
has no coverageObjects/namespaceobject.c
Objects/object.c
PyObject_Print
has no coveragePyObject_Bytes
does not test the case where there is a__bytes__
PyObject_SetAttrString
doesn't test when object has atp_setattr
PyObject_GetAttrString
doesn't test when object has atp_getattr
_PyObject_LookupAttr
doesn't test when object has atp_getattr
Objects/obmalloc.c
Objects/odictobject.c
Objects/picklebufobject.c
PyPickleBuffer_FromObject
,PyPickleBuffer_Release
has no coverageObjects/rangeobject.c
Objects/setobject.c
Objects/sliceobject.c
PySlice_GetIndices
/PySlice_GetIndicesEx
has no coverageObjects/stringlib/codecs.h
Objects/stringlib/count.h
Objects/stringlib/ctype.h
Objects/stringlib/eq.h
Objects/stringlib/fastsearch.h
Objects/stringlib/find.h
Objects/stringlib/find_max_char.h
Objects/stringlib/join.h
Objects/stringlib/localeutil.h
Objects/stringlib/partition.h
Objects/stringlib/replace.h
Objects/stringlib/split.h
Objects/stringlib/transmogrify.h
Objects/stringlib/undef.h
Objects/stringlib/unicode_format.h
Objects/structseq.c
Objects/tupleobject.c
Objects/typeobject.c
wrap_sq_setitem
has no coverageObjects/unicodectype.c
Objects/unicodeobject.c
xmlcharrefreplace
doesn't test for codepoints < 100 (This seems almost impossible to occur).resize_inplace
has no coverageunicode_kind_name
when!PyUnicode_IS_COMPACT
isn't covered -- low priority used by consistency check onlyunicode_write_cstr
doesn't test writing into UCS2 or UCS4%p
inPyUnicode_FromFormat
#96677PyUnicode_AsDecodedObject
,PyUnicode_AsDecodedUnicode
,PyUnicode_AsEncodedObject
,PyUnicode_AsEncodedUnicode
has no coverage_Py_DecodeUTF8Ex
and_Py_EncodeUTF8Ex
has no coverage forerror == surrogateescape
PyUnicode_BuildEncodingMap
doesn't handle theneed_dict
caseucs1lib_find_slice
anducs1lib_rfind_slice
aren't covered.PyUnicode_Count
has no coveragesplit
has no coverage for UCS4,rsplit
has no coverage for UCS1, UCS2 or UCS4PyUnicode_CompareWithASCIIString
has no coverage for comparing with UCS2 or UCS4_PyUnicode_EqualToASCIIId
has no coverageObjects/unicodetype_db.h
Objects/unionobject.c
Objects/weakrefobject.c
Parser/action_helpers.c
_PyPegen_set_expr_context
doesn't cover "starred kind"_PyPegen_get_expr_name
switch statement coverage is non-exhaustiveParser/myreadline.c
(N/A Windows-only)Parser/parser.c
Parser/peg_api.c
Parser/pegen.c
Parser/pegen.h
Parser/pegen_errors.c
mock.Mock()._is_coroutine = False
workarounds #94926Parser/string_parser.c
Parser/tokenizer.c
Generated codePython/Python-ast.c
Python/Python-tokenize.c
Python/_warnings.c
show_warning
doesn't cover the case where there is asourceline
.PyErr_WarnExplicit
has no coveragePython/asdl.c
Python/ast.c
ensure_literal_*
functions aren't coveredvalidate_pattern_match_value
doesn't cover all elements of switchPython/ast_opt.c
check_complexity
doesn't cover thefrozenset
caseast_foldbody
isn't coveredPython/ast_unparse.c
Python/bltinmodule.c
Python/bootstrap_hash.c
Python/ceval.c
PyEval_AquireLock
andPyEval_ReleaseLock
are uncoveredSTORE_ATTR_WITH_HINT
doesn't cover the case where the dictionary doesn't have Unicode keysCALL_FUNCTION_EX
doesn't cover the case where kwargs is not an exact dictPyEval_EvalCodeEx
doesn't cover the case where kwargs are passed inPyEval_GetFrame
has no coveragePyEval_GetFuncName
andPyEval_GetFuncDesc
have no coveragePython/ceval_gil.h
Python/codecs.c
Python/compile.c
Python/condvar.h
Python/context.c
PyContext_Copy
,PyContext_Enter
,PyContext_Exit
have no coveragePython/deepfreeze/deepfreeze.c
Python/dtoa.c
Python/dup2.c
Python/dynamic_annotations.c
Python/errors.c
Python/fileutils.c
is_valid_wide_char
doesn't test error branchesencode_ascii
/decode_ascii
has no coverage (probably very low priority -- comment says only for platforms with a broken mbstowcs (FreeBSD, OpenIndiana)_Py_stat
has no coveragePython/formatter_unicode.c
Python/frame.c
Python/frozenmain.c
Python/future.c
Python/getargs.c
Python/getopt.c
Python/hamt.c
Python/hashtable.c
Python/import.c
Python/importdl.c
Python/initconfig.c
Python/marshal.c
Python/modsupport.c
Python/mysnprintf.c
Python/mystrtoul.c
Python/pathconfig.c
Python/preconfig.c
Python/pyarena.c
Python/pyfpe.c
Python/pyhash.c
Python/pylifecycle.c
Python/pystate.c
Python/pystrcmp.c
Python/pystrhex.c
Python/pystrtod.c
Python/pythonrun.c
Python/pytime.c
Python/specialize.c
Python/structmember.c
Python/suggestions.c
Python/symtable.c
Python/sysmodule.c
Python/thread.c
Python/traceback.c
depth > limit
_PyTraceBack_Print_Indented
with overflowingtracebacklimit
_Py_DumpDecimal
,_Py_DumpHexadecimal
,_Py_DumpASCII
,dump_frame
,dump_traceback
,_Py_DumpTraceback
,write_thread_id
,_Py_DumpTracebackThreads
-- possibly they have tests which are disabled under some circumstances.The text was updated successfully, but these errors were encountered: