This is a revert of the revert (GH-15826).  Having a tp_clear for
functions should be safe (and helpful) now that bpo-38006 has been
fixed.

…size (GH-16491) (#16506)

(cherry picked from commit 94e1650)

Co-authored-by: Giampaolo Rodola <g.rodola@gmail.com>

PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
(cherry picked from commit 8462a49)

Co-authored-by: Victor Stinner <vstinner@redhat.com>

* Windows: Fix counter name in WindowsLoadTracker. Counter names are
  localized: use the registry to get the counter name. Original
  change written by Lorenz Mende.
* Regrtest.main() now ensures that the Windows load tracker is also
  killed if an exception is raised
* TestWorkerProcess now ensures that worker processes are no longer
  running before exiting: kill also worker processes when an
  exception is raised.
* Enhance regrtest messages and warnings: include test name,
  duration, add a worker identifier, etc.
* Rename MultiprocessRunner to TestWorkerProcess
* Use print_warning() to display warnings.

Co-Authored-By: Lorenz Mende <Lorenz.mende@gmail.com>
(cherry picked from commit 982bfa4)

Co-authored-by: Victor Stinner <vstinner@redhat.com>

* bpo-38321: Fix _asynciomodule.c compiler warning (GH-16493)

bpo-38248, bpo-38321: Fix warning:

    modules\_asynciomodule.c(2667):
    warning C4102: 'set_exception': unreferenced label

The related goto has been removed by
commit edad4d8.

(cherry picked from commit efe74b6)

* bpo-38321: Fix PyCStructUnionType_update_stgdict() warning (GH-16492)

bpo-22273, bpo-38321: Fix following warning:

    modules\_ctypes\stgdict.c(704):
    warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'int', possible loss of data

(cherry picked from commit c9a413e)

(cherry picked from commit 891e9e3)

Co-authored-by: Géry Ogam <gery.ogam@gmail.com>

(cherry picked from commit b9a8b82)

Co-authored-by: Steve Dower <steve.dower@python.org>

* Fix TestWorkerProcess.__repr__(): start_time is only valid
  if _popen is not None.
* Fix _kill(): don't set _killed to True if _popen is None.
* _run_process(): only set _killed to False after calling
  run_test_in_subprocess().
(cherry picked from commit 2ea71a0)

Co-authored-by: Victor Stinner <vstinner@python.org>

test.pythoninfo now logs environment variables used by OpenSSL and
Python ssl modules, and logs attributes of 3 SSL contexts
(SSLContext, default HTTPS context, stdlib context).
(cherry picked from commit b3e7045)

Co-authored-by: Victor Stinner <vstinner@python.org>

…H-16550) (GH-16560)

* bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)

WindowsLoadTracker.read_output() now uses a short buffer for
incomplete line.

(cherry picked from commit 3e04cd2)

* bpo-36670: Enhance regrtest WindowsLoadTracker (GH-16553)

The last line is now passed to the parser even if it does not end
with a newline, but only if it's a valid value.

(cherry picked from commit c65119d)

* bpo-36670: Enhance regrtest (GH-16556)

* Add log() method: add timestamp and load average prefixes
  to main messages.
* WindowsLoadTracker:

  * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
  * Initialize the load to the arithmetic mean of the first 5 values
    of the Processor Queue Length value (so over 5 seconds), rather
    than 0.0.
  * Handle BrokenPipeError and when typeperf exit.

* format_duration(1.5) now returns '1.5 sec', rather than
  '1 sec 500 ms'

(cherry picked from commit 098e256)

(cherry picked from commit a0e3d27)

Co-authored-by: Steve Dower <steve.dower@python.org>

)

(cherry picked from commit 353fb1e)

Co-authored-by: Steve Dower <steve.dower@python.org>

)

(cherry picked from commit b23a842)

Co-authored-by: idomic <michael.ido@gmail.com>

…16571) (GH-16576)

(cherry picked from commit 3142c66)

Co-authored-by: Ashley Whetter <AWhetter@users.noreply.github.com>

…cl. (GH-16545)

On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.

On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.

Converting strings from Tcl to Python and back now never fails
(except MemoryError).
(cherry picked from commit 06cb94b)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

… bytearray. (GH-16603)

(cherry picked from commit 24ddd9c)

Co-authored-by: Hai Shi <shihai1992@gmail.com>

…16588)

The `required` argument to `argparse.add_subparsers` was added in GH-3027. This PR specifies the earliest version of Python where it is available.

https://bugs.python.org/issue26510

Automerge-Triggered-By: @merwok
(cherry picked from commit 9e71917)

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

(cherry picked from commit 038503e)

Co-authored-by: Krishna Oza <krishoza15sep@gmail.com>

(cherry picked from commit 321def8)

Co-authored-by: Ben Harper <btharper1221@gmail.com>

Always define _PyUnicode_CheckConsistency() in the CPython C API.

…tion. (GH-16442)

Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
(cherry picked from commit 15ae75d)

Co-authored-by: Ricardo Bánffy <rbanffy@gmail.com>

(cherry picked from commit e310af9)

Co-authored-by: James Abel <j@abel.co>

(cherry picked from commit 5dfbb4d)

Co-authored-by: Hansraj Das <raj.das.136@gmail.com>

…o stderr. (GH-16583)

It now escapes them with a backslash, as the regular Python interpreter.
Added the "errors" field to the standard streams.
(cherry picked from commit b690a27)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

Valgrind emits "Conditional jump or move depends on uninitialised
value(s)" false alarms on GCC builtin strcmp() function. The GCC code
is correct.

Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
(cherry picked from commit 03ab6b4)

Co-authored-by: Victor Stinner <vstinner@python.org>

…H-16641)

(cherry picked from commit 13abda4)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

(cherry picked from commit 8252c52)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…H-16662)

The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.

To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods..
(cherry picked from commit 10cd00a)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

(cherry picked from commit 01171eb)

Co-authored-by: Hansraj Das <raj.das.136@gmail.com>

…/unions. (GH-16589) (GH-16671)

(cherry picked from commit e8bedbd)

bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.

Reenable test_regrtest.test_multiprocessing_timeout().

bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
(cherry picked from commit 0ec618a)

Co-authored-by: Victor Stinner <vstinner@python.org>

…H-16658) (GH-16683)

* [bpo-38379](https://bugs.python.org/issue38379):  when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth.
(cherry picked from commit ecbf35f)

Co-authored-by: Tim Peters <tim.peters@gmail.com>





https://bugs.python.org/issue38379



Automerge-Triggered-By: @pablogsal

(cherry picked from commit 09895c2)

(cherry picked from commit a05fcd3)

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

PR GH-4906 changed the typing.Generic class hierarchy, leaving an
outdated comment in the library reference. User-defined Generic ABCs now
must get a abc.ABCMeta metaclass from something other than typing.Generic
inheritance.
(cherry picked from commit d47f0dd)

Co-authored-by: M. Eric Irrgang <mei2n@virginia.edu>

Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
(cherry picked from commit 7bb1431)

Co-authored-by: Ronan Lamy <ronan.lamy@gmail.com>

…-16591)

(cherry picked from commit 1dbe537)

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

(cherry picked from commit f900064)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>

(cherry picked from commit a8e0d31)

Co-authored-by: Hansraj Das <raj.das.136@gmail.com>

…H-16536)

 pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
(cherry picked from commit 2b7dc40)

Co-authored-by: Ruediger Pluem <r.pluem@gmx.de>

We no longer use AppVeyor on 3.x, and others were still pointed at master.

(cherry picked from commit dd6117c)

Co-authored-by: bariod <35639254+bariod@users.noreply.github.com>

)

Add SMTPNotSupportedError in the exports of smtplib

Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
(cherry picked from commit 3faf826)

Co-authored-by: nde <denayer.norman@gmail.com>

…low oper parsing of URLs" (GH-16724) (GH-16728)

This reverts commit 87bd207.

The reason for revert is a regression caused by the change in 3.8.0rc1, see bpo-38449 for more details.

https://bugs.python.org/issue38449
(cherry picked from commit 19a3d87)

Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>

KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
(cherry picked from commit 65dcc8a)

Co-authored-by: Andrei Troie <andreitroie90@gmail.com>

This is a fairly noticeable change that requires adjustments in
existing asyncio code. It should therefore be announced.
(cherry picked from commit e634da2)

Co-authored-by: Phil Jones <philip.graham.jones@googlemail.com>

…IX_C_SOURCE (GH-16733)

(cherry picked from commit 8177404)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

GH-16702)

(cherry picked from commit 793cb85)

Co-authored-by: Samuel Colvin <samcolvin@gmail.com>

(cherry picked from commit e3babbd)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

…ocs (GH-16743)

Metaclass was removed in Python 3.7 (there is already a `versionchanged` item about this).

https://bugs.python.org/issue28556
(cherry picked from commit 8144095)

Co-authored-by: Ivan Levkivskyi <levkivskyi@gmail.com>

(cherry picked from commit fdfe283)

Co-authored-by: Sebastian Rittau <srittau@rittau.biz>

* Use Unicode character for accent
* Various grammar fixes
* Sort library modules alphabetically; remove duplicated idlelib/IDLE section
(cherry picked from commit bb78f6c)

Co-authored-by: Andrew Kuchling <amk@amk.ca>

(cherry picked from commit c39d1dd)

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

…H-16758)

(cherry picked from commit 8a6cbf8)

Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>

…6762)

* bpo-38461: ncurses misspelled as curses

* bpo-38463: Fix broken link
(cherry picked from commit 61a6db5)

Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>

)

(cherry picked from commit a329153)

Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>

GH-16755) (#16760)

The symbol table handing of PEP572's assignment expressions is not resolving correctly the scope of some variables in presence of global/nonlocal keywords in conjunction with comprehensions.
(cherry picked from commit fd5c414)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

(cherry picked from commit 19d6842)

Co-authored-by: Ned Deily <nad@python.org>

(cherry picked from commit 298439c)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

(cherry picked from commit d83fc27)

Co-authored-by: Steve Dower <steve.dower@python.org>

(cherry picked from commit 274bd01)

Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>

Python 3.8.0

* [3.8] bpo-38456: Use /bin/true in test_subprocess (GH-16736)

Instead of sys.executable, "-c", "pass" or "import sys; sys.exit(0)"
use /bin/true when it is available.  On a reasonable machine this
shaves up to two seconds wall time off the otherwise ~40sec execution
on a --with-pydebug build.  It should be more notable on many
buildbots or overloaded slower I/O systems (CI, etc)..
(cherry picked from commit 67b93f8)

* Handle when there is no 'true' command

backport of 46113e0 by Pablo Galindo.

(cherry picked from commit 4504b45)

Co-authored-by: Julien Palard <julien@palard.fr>

(cherry picked from commit 4d20228)

Co-authored-by: Steve Dower <steve.dower@python.org>

* bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612)

bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.

Modify the following functions to also work in release mode:

* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()

Other changes:

* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
  (equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
  with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
  attempt to render repr() at the end.

(cherry picked from commit 6876257)

* bpo-36389: Fix _PyBytesWriter in release mode (GH-16624)

Fix _PyBytesWriter API when Python is built in release mode with
assertions.

(cherry picked from commit 60ec6ef)

* bpo-38070: Enhance visit_decref() debug trace (GH-16631)

subtract_refs() now pass the parent object to visit_decref() which
pass it to _PyObject_ASSERT(). So if the "is freed" assertion fails,
the parent is used in debug trace, rather than the freed object. The
parent object is more likely to contain useful information. Freed
objects cannot be inspected are are displayed as "<object at xxx is
freed>" with no other detail.

(cherry picked from commit 4d5f94b)

* Fix also a typo in PYMEM_DEADBYTE macro comment

* bpo-36389: Add newline to _PyObject_AssertFailed() (GH-16629)

Add a newline between the verbose object dump and the Py_FatalError()
logs for readability.

(cherry picked from commit 7775349)

…onal-only parameter (GH-16800)

(cherry picked from commit f3ef06a)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

(cherry picked from commit 2798b60)

Co-authored-by: Hansraj Das <raj.das.136@gmail.com>

(cherry picked from commit 3f36043)

Co-authored-by: Ned Deily <nad@python.org>

main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.

Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.

TLSv1.3 was disabled for the test: reenable it.
(cherry picked from commit fab4ef2)

Co-authored-by: Victor Stinner <vstinner@python.org>

…GH-16824)

(cherry picked from commit 7aebbd1)

Co-authored-by: Steve Dower <steve.dower@python.org>

Also updates the documentation to clarify the situation surrounding
the digestmod parameter that is required despite its position in the
argument list as of 3.8.0 as well as removing old python2 era
references to "binary strings".

We indavertently had this raise ValueError in 3.8.0 for the missing
arg.  This is not considered an API change as no reasonable code would
be catching this missing argument error in order to handle it.
(cherry picked from commit f33c57d)

Co-authored-by: Gregory P. Smith <greg@krypto.org>

(cherry picked from commit 93b81e1)

Co-authored-by: Stéphane Wirtel <stephane@wirtel.be>

Added periods at the end of the sentences.
(cherry picked from commit b1fa72a)

Co-authored-by: Jero Bado <tokidokitalkyou@gmail.com>

* bpo-27657: Fix urlparse() with numeric paths

Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.

* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.
(cherry picked from commit 5a88d50)

Co-authored-by: Tim Graham <timograham@gmail.com>

…H-16670)

https://bugs.python.org/issue38418

Automerge-Triggered-By: @zooba
(cherry picked from commit fbe3c76)

Co-authored-by: Steve Dower <steve.dower@python.org>

(cherry picked from commit 5eabec0)

Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>

The Nose package is no longer maintained.
(cherry picked from commit 88eeda6)

Co-authored-by: Jon Dufresne <jon.dufresne@gmail.com>

…mpty dict (GH-16846)

The reverse iterator for empty dictionaries was not handling correctly shared-key dictionaries.
(cherry picked from commit 24dc2f8)

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>

* math.perm() and math.comb()

* math.isqrt()

* Add singledispatchmethod()

* itertools.accumulate()

* Optional headers for xmlrpc.client.ServerProxy

* IDLE non-BMP characters

* import collections.abc directly

* @coroutine is deprecated

* pprint.pp()

* New options for object.__reduce__()

* DictReader no longer returns OrderedDicts

* "force" option for logging.basicConfig()

* Fix spelling

* cProfile context manager

* Various markup/grammar fixes from Kyle Stanley.
Other minor fixes as well.
Also, dedup the __reduce__ entry.

* Fix markup

* Fix grammar nits found by MS Word
(cherry picked from commit c93883c)

Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>

(cherry picked from commit 7414207)

Co-authored-by: Batuhan Taşkaya <47358913+isidentical@users.noreply.github.com>

…H-16869)

(cherry picked from commit 5bc6a7c)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

…H-16860)

(cherry picked from commit 10ecbad)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.

Test:

* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants

But don't test ATOMIC_VAR_INIT(): it's not used in Python.
(cherry picked from commit 028f734)

Co-authored-by: Victor Stinner <vstinner@python.org>

(cherry picked from commit 91528f4)

Co-authored-by: Daniel Baskal <MagnificentUrukHai@users.noreply.github.com>

(cherry picked from commit dfe726b)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>

(cherry picked from commit 2e3d873)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

(cherry picked from commit d34ac30)

Co-authored-by: Peter Bittner <django@bittner.it>

…GH-16901)

bpo-34679, bpo-38563: asyncio.ProactorEventLoop.close() now only calls
signal.set_wakeup_fd() in the main thread.
(cherry picked from commit 1b53a24)

Co-authored-by: Victor Stinner <vstinner@python.org>

(cherry picked from commit 01659ca)

Co-authored-by: Ned Deily <nad@python.org>

(cherry picked from commit 3bbb6db)

Co-authored-by: Kyle Stanley <aeros167@gmail.com>

These are valid even in python 2.7

https://bugs.python.org/issue33348

Automerge-Triggered-By: @gpshead
(cherry picked from commit 96b06ae)

Co-authored-by: Zsolt Dollenstein <zsol.zsol@gmail.com>

numbers's -> number's
(cherry picked from commit 7320ec0)

Co-authored-by: Hansraj Das <raj.das.136@gmail.com>

(cherry picked from commit cb2cf06)

Co-authored-by: Ammar Askar <ammar@ammaraskar.com>

… decorators. (GH-16861)

(cherry picked from commit 26ae9f6)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

(cherry picked from commit d898d20)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

(cherry picked from commit 894e30c)

Co-authored-by: Steve Dower <steve.dower@python.org>

(cherry picked from commit e3f90b2)

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>

(cherry picked from commit e31a79a)

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>

…H-16937)

Test by Daniel Hillier.
(cherry picked from commit 5c32af7)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

)


(cherry picked from commit 85c6f8c)
Co-authored-by: Marco Rougeth <marco@rougeth.com>

(cherry picked from commit edb172a)

Co-authored-by: Zackery Spytz <zspytz@gmail.com>