[MR+3] CI dropping python-3.5 #15106

adrinjalali · 2019-09-28T10:18:25Z

Related to #14449.

This PR drops support for python=3.5.

Please note that there are a few consequences:

Python 3.5 was/is the default python in ubuntu 16.04. Moving to python=3.6 means we either move the CI entirely to conda envs, or move the ubuntu images in the CI to ubuntu 18.04, which has a python=3.6 as the default python3.

This PR uses Ubuntu 18.04 as the baseline, and the corresponding packages:
numpy=1.13
scipy=0.19
matplotlib=2.1.1

adrinjalali · 2019-09-30T09:38:41Z

Azure still hasn't added the 18.04 image (https://github.com/Microsoft/azure-pipelines-image-generation/issues/506). Do we want to use a docker image like numpy does:

https://github.com/numpy/numpy/blob/68bd6e359a6b0863acf39cad637e1444d78eabd0/azure-pipelines.yml#L16-L34

    vmImage: 'ubuntu-16.04'
  steps:
  - script: |
           docker pull i386/ubuntu:bionic
           docker run -v $(pwd):/numpy i386/ubuntu:bionic /bin/bash -c "cd numpy && \
           apt-get -y update && \
           apt-get -y install python3.6-dev python3-pip locales python3-certifi && \
           locale-gen fr_FR && update-locale && \
           apt-get -y install gfortran-5 wget && \
           target=\$(python3 tools/openblas_support.py) && \
           cp -r \$target/usr/local/lib/* /usr/lib && \
           cp \$target/usr/local/include/* /usr/include && \
           python3 -m pip install --user --upgrade pip setuptools && \
           python3 -m pip install --user -r test_requirements.txt && \
           python3 -m pip install . && \
           F77=gfortran-5 F90=gfortran-5 \
           CFLAGS='-UNDEBUG -std=c99' python3 runtests.py -n --debug-configure --mode=full -- -rsx --junitxml=junit/test-results.xml && \
           python3 tools/openblas_support.py --check_version $(OpenBLAS_version)"
    displayName: 'Run 32-bit Ubuntu Docker Build / Tests'

Or do we want to use only conda?

(I also really would like to just have docker images for each test case instead of installing all those packages on each commit for all those machines. It hurts me to waste so much time/energy on those machines).

WDYT @thomasjpfan

thomasjpfan · 2019-10-01T17:46:04Z

The numpy azure config uses a docker image to test 32bit linux, which is what we currently do. They use TravisCI for their linux tests.

From my understanding of #14449, we have not decided to drop 3.5 yet.

adrinjalali · 2019-10-01T17:55:50Z

@thomasjpfan I thought trying it out may reveal things which may influence our decision. Since the NEP is kinda accepted, and pandas is also dropping 3.5 for the November release, I thought it'd be nice to see at least how it works.

Also, the NamedArray for feature_names_out_ requires numpy>=1.13, which is satisfied as a happy side effect of dropping 3.5

rth · 2019-10-07T16:40:23Z

From my understanding of #14449, we have not decided to drop 3.5 yet.

Well Python 3.8 is released next Monday. If we support 3 latest versions, we could drop 3.5 then? Also maybe this would be a lazy fix for #14168 and all the issues with Python 3.5 CI we had in conda lately.

Otherwise this means that we would need to keep it until the last minor release is done in the 0.22.X series which could take another 6 months.

adrinjalali · 2020-01-03T14:12:21Z

I think this is ready for a review. The failing tests have their corresponding issues.

thomasjpfan · 2020-01-10T19:36:59Z

For reference, the remaining issue is #16013

rth · 2020-01-12T09:59:13Z

For reference, the remaining issue is #16013

I don't think it should be a blocker to merge this. I am fine with marking that test as a known failure for for CI with Py36 32bit, since it's not a new failure, but we are just not testing for it at the moment in those conditions as far as I understand.

rth

Thanks for continuing working in this @adrinjalali ! LGTM, once the failing test with 32bit is marked as a known failure.

NicolasHug · 2020-01-12T16:16:14Z

Has numpy dropped 3.5 support? I can't find the PR.

We should probably sync with the other packages of the ecosystem to make sure we're not the only one actually applying the NEP?

rth · 2020-01-12T16:29:56Z

Has numpy dropped 3.5 support? I can't find the PR.

Pandas did I think and from pandas-dev/pandas#29034 (comment)

numpy.org/neps/nep-0029-deprecation_policy.html has September 2019 as the cutoff for 3.5.

NicolasHug · 2020-01-21T17:24:01Z

we can, but why?

To avoid having to move to ubuntu 18.04 which would avoid having to bump our minimal requirements

those old versions of our dependencies don't support 3.6 AFAIK

I don't see how a library can support 3.5 but not 3.6?

rth · 2020-01-21T20:52:24Z

I don't see how a library can support 3.5 but not 3.6?

numpy 1.11.3 was released a few days before Python 3.6 on Dec. 23, 2016 and explicitly supports 2.6 - 2.7 and 3.2 - 3.5. While 3.6 would probably work maybe with a few warnings, there are no wheels for it on PyPi for Windows. If the tar.gz was cythonized with an older cython that didn't support 3.6 it may also fail to compile from PyPi.

For comparison, the upcoming

pandas 1.0 will support Python 3.6+ and NumPy 1.13.3+.
matplotlib 3.2 will support Python 3.6+ and Numpy 1.15+

so numpy 1.13.3+ for the next release would be consistent with the rest of the ecosystem, unless we have a reason support older numpy versions. scipy=0.19 and matplotlib=2.1.1 were also released around that time.

adrinjalali · 2020-02-06T16:00:58Z

Should we merge then? @NicolasHug ? ;)

NicolasHug

Thanks Adrin, LGTM. I guess we can start using f-strings now \o/

pinging @scikit-learn/core-devs since this is an important change

NicolasHug · 2020-02-06T16:41:51Z

build_tools/azure/install.sh

@@ -1,6 +1,7 @@
 #!/bin/bash

 set -e
+set -x


was this just for debugging or do we still need it?

It was for debugging, but we have it in a few other scripts and I find it easier to debug whenever an issue happens, instead of putting it there every time we need it.

rth · 2020-02-06T16:48:24Z

BTW, projects that migrated to pyproject.toml (#16244) can get a nice correspondence between Python versions and minimal supported numpy versions, with https://pypi.org/project/oldest-supported-numpy/ as mentioned by @jeremiedbb in that PR,

    "numpy==1.13.3; python_version=='3.5',
    "numpy==1.13.3; python_version=='3.6',
    "numpy==1.14.5; python_version=='3.7',
    "numpy==1.17.3; python_version>='3.8'

so when we drop 3.6, it should indeed be numpy 1.13.3+

adrinjalali

also removed a bit of code which was there for numpy<1.13

adrinjalali · 2020-02-07T13:43:13Z

build_tools/azure/install.sh

@@ -1,6 +1,7 @@
 #!/bin/bash

 set -e
+set -x


It was for debugging, but we have it in a few other scripts and I find it easier to debug whenever an issue happens, instead of putting it there every time we need it.

GaelVaroquaux

LGTM. Thank you!

GaelVaroquaux · 2020-02-07T13:49:24Z

I approved, but did not merge (though we have a lot of approval on this PR), to give time for others to voice concerns, if any.

sklearn/utils/_mask.py

sklearn/utils/fixes.py

This reverts commit c6e867e.

ogrisel

LGTM besides the following:

README.rst

doc/install.rst

ogrisel · 2020-02-07T21:32:12Z

Yeah for f-strings!

ogrisel · 2020-02-07T21:34:38Z

Also what about type hints?

GaelVaroquaux · 2020-02-07T21:35:53Z

Also what about type hints?

+1

rth · 2020-02-10T10:41:25Z

Also what about type hints?

Some light type hints would be great. See earlier discussion in #11170

* dropping python-3.5 * fix install guide * fix setup.py * fix circleci * advanced_installation * index.html * readme * pyparsing.py * remove clean_warning_registry * 1.13.1 and 1.19.1 * don't use 16.04 * 18.04 libatlas-dev -> libatlas-base-dev * min pillow version for 3.6 is 4.2.1 * echo commands * fix conflict, and 32 bit * fix conflict for circle * fix scikit-image version dep * mark tests as xfail on 32bit py3.6 * move to 1.13.3 min numpy version, and simplify old code * remove the rest of _object_dtype_isnan usages * Revert "remove the rest of _object_dtype_isnan usages" This reverts commit c6e867e. * fix issues raised by jeremy * minor fix * minor fixes mentioned by Olivier

adrinjalali added 10 commits September 28, 2019 12:09

dropping python-3.5

8d03a7c

fix install guide

b13ba08

fix setup.py

1caac21

fix circleci

08a64a6

advanced_installation

e9d7550

index.html

5383ef9

readme

ad77077

pyparsing.py

455bd6a

remove clean_warning_registry

b208612

1.13.1 and 1.19.1

5ea9f0c

rth mentioned this pull request Oct 7, 2019

test_fit_csr_matrix failing on Linux py35_conda_openblas #14168

Closed

adrinjalali added 9 commits November 22, 2019 13:27

merge upstream/master

b9edf95

don't use 16.04

234583b

18.04 libatlas-dev -> libatlas-base-dev

4286d5b

min pillow version for 3.6 is 4.2.1

bc2debb

merge upstream/master

fc5418c

echo commands

923b537

fix conflict, and 32 bit

1406677

fix conflict for circle

f57cacb

fix scikit-image version dep

23d5f97

rth approved these changes Jan 12, 2020

View changes

thomasjpfan mentioned this pull request Jan 27, 2020

Use np.empty_like in PolynomialFeatures for NEP18 support #16196

Open

NicolasHug approved these changes Feb 6, 2020

View changes

move to 1.13.3 min numpy version, and simplify old code

6408b73

adrinjalali commented Feb 7, 2020

View changes

GaelVaroquaux approved these changes Feb 7, 2020

View changes

GaelVaroquaux changed the title ~~CI dropping python-3.5~~ [MR+3] CI dropping python-3.5 Feb 7, 2020

agramfort approved these changes Feb 7, 2020

View changes

adrinjalali added 2 commits February 7, 2020 15:12

remove the rest of _object_dtype_isnan usages

c6e867e

merge upstream/master

9e6c159

jeremiedbb reviewed Feb 7, 2020

View changes

sklearn/utils/_mask.py Outdated Show resolved Hide resolved

sklearn/utils/_mask.py Outdated Show resolved Hide resolved

sklearn/utils/fixes.py Outdated Show resolved Hide resolved

adrinjalali added 3 commits February 7, 2020 16:26

Revert "remove the rest of _object_dtype_isnan usages"

49f8ee1

This reverts commit c6e867e.

fix issues raised by jeremy

bf23aa8

minor fix

aa2915a

ogrisel approved these changes Feb 7, 2020

View changes

README.rst Outdated Show resolved Hide resolved

doc/install.rst Show resolved Hide resolved

ogrisel mentioned this pull request Feb 7, 2020

[MRG] BLD Specify build time dependencies via pyproject.toml #16244

Merged

minor fixes mentioned by Olivier

783eeac

adrinjalali merged commit 42706eb into scikit-learn:master Feb 10, 2020

adrinjalali deleted the ci/drop-3.5 branch February 10, 2020 11:28

cmarmo mentioned this pull request Feb 19, 2020

Examples require matplotlib 3.1.x #14925

Closed

[MR+3] CI dropping python-3.5 #15106

[MR+3] CI dropping python-3.5 #15106

adrinjalali commented Sep 28, 2019 •

edited

adrinjalali commented Sep 30, 2019

thomasjpfan commented Oct 1, 2019

adrinjalali commented Oct 1, 2019

rth commented Oct 7, 2019

adrinjalali commented Jan 3, 2020

thomasjpfan commented Jan 10, 2020

rth commented Jan 12, 2020

rth left a comment

NicolasHug commented Jan 12, 2020

rth commented Jan 12, 2020

NicolasHug commented Jan 21, 2020

rth commented Jan 21, 2020 •

edited

adrinjalali commented Feb 6, 2020

NicolasHug left a comment

NicolasHug Feb 6, 2020

adrinjalali Feb 7, 2020

rth commented Feb 6, 2020

adrinjalali left a comment

adrinjalali Feb 7, 2020

GaelVaroquaux left a comment

GaelVaroquaux commented Feb 7, 2020

ogrisel left a comment

ogrisel commented Feb 7, 2020

ogrisel commented Feb 7, 2020

GaelVaroquaux commented Feb 7, 2020 via email

rth commented Feb 10, 2020

[MR+3] CI dropping python-3.5 #15106

[MR+3] CI dropping python-3.5 #15106

Conversation

adrinjalali commented Sep 28, 2019 • edited

adrinjalali commented Sep 30, 2019

thomasjpfan commented Oct 1, 2019

adrinjalali commented Oct 1, 2019

rth commented Oct 7, 2019

adrinjalali commented Jan 3, 2020

thomasjpfan commented Jan 10, 2020

rth commented Jan 12, 2020

rth left a comment

Choose a reason for hiding this comment

NicolasHug commented Jan 12, 2020

rth commented Jan 12, 2020

NicolasHug commented Jan 21, 2020

rth commented Jan 21, 2020 • edited

adrinjalali commented Feb 6, 2020

NicolasHug left a comment

Choose a reason for hiding this comment

NicolasHug Feb 6, 2020

Choose a reason for hiding this comment

adrinjalali Feb 7, 2020

Choose a reason for hiding this comment

rth commented Feb 6, 2020

adrinjalali left a comment

Choose a reason for hiding this comment

adrinjalali Feb 7, 2020

Choose a reason for hiding this comment

GaelVaroquaux left a comment

Choose a reason for hiding this comment

GaelVaroquaux commented Feb 7, 2020

ogrisel left a comment

Choose a reason for hiding this comment

ogrisel commented Feb 7, 2020

ogrisel commented Feb 7, 2020

GaelVaroquaux commented Feb 7, 2020 via email

rth commented Feb 10, 2020

adrinjalali commented Sep 28, 2019 •

edited

rth commented Jan 21, 2020 •

edited