bpo-30450: Pull Windows dependencies from GitHub rather than svn #1783

zware · 2017-05-24T01:35:35Z

This is not an ideal solution, but may be enough to get us away from svn.python.org. The dependencies have not actually been pushed to python/cpython-source-deps or python/cpython-bin-deps yet, but you can use --organization zware for testing until we're agreed on the layout of everything.

mention-bot · 2017-05-24T01:35:36Z

@zware, thanks for your PR! By analyzing the history of the files in this pull request, we identified @methane and @ned-deily to be potential reviewers.

zooba

Looks good, though incomplete right now. And obviously AppVeyor is going to block until it's all fleshed out.

zooba · 2017-05-24T15:55:25Z

PCbuild/get_external.py

+                               os.path.abspath(__file__)    # '-> here
+                            )
+                        ),
+                        'externals'))


You're using f-strings but not pathlib? Shame ;)

zooba · 2017-05-24T15:55:25Z

PCbuild/get_external.py

+        verbose=args.verbose,
+    )
+    final_name = os.path.join(args.externals_dir, args.tag)
+    os.rename(extract_zip(args.externals_dir, zip_path), final_name)


Since we don't clean up the existing directory, should we fail early if it already exists? That would save downloading the zip just to fail at this point.

The other option is to either move the existing directory out of the way, or just overwrite it. I'm fine with any of those three options; opinions?

zooba · 2017-05-24T15:55:25Z

PCbuild/get_externals.bat

 )

+if %DO_FETCH%==false goto end


Good practice to use quotes around each side of the == in case one evaluates to empty.

Good point, will fix.

zooba · 2017-05-24T15:55:25Z

PCbuild/get_externals.bat

+    )
+    echo Installing Python via nuget...
+    %NUGET% install python -OutputDirectory %EXTERNALS_DIR%python
+    set PYTHON_FOR_BUILD=%EXTERNALS_DIR%python\python.3.6.1\tools\python.exe


If you use -ExcludeVersion with nuget and remove the extra python then the path should reliably be %EXTERNALS_DIR%python\tools\python.exe.

Alternatively, specify -Version 3.6.1 when installing to ensure that the path doesn't change unexpectedly.

Hadn't seen -ExcludeVersion; went with it. Just a little bit less hard-coding of variables.

zooba · 2017-05-24T15:55:25Z

PCbuild/get_externals.bat

+        powershell.exe -Command Invoke-WebRequest %NUGET_URL% -OutFile %NUGET%
+    )
+    echo Installing Python via nuget...
+    %NUGET% install python -OutputDirectory %EXTERNALS_DIR%python


Maybe we should actually install pythonx86 for both of our users building on 32-bit Windows? :)

zooba · 2017-05-24T15:55:25Z

PCbuild/get_external.py

+
+def fetch_zip(commit_hash, zip_dir, *, org='python', binary=False, verbose):
+    repo = f'cpython-{"bin" if binary else "source"}-deps'
+    url = f'https://github.com/{org}/{repo}/archive/{commit_hash}.zip'


This will download the entire repo at this commit - is your plan to use a branch per project? That's fine by me, just can't see the full picture yet.

Yes, that's the current layout. I don't think there's a way to get just a specific directory, and I like being able to not get OpenSSL or Tcl/Tk as desired.

zware · 2017-05-24T16:42:33Z

@zooba If the scheme in my source and bin repos looks good enough to you, I'll go ahead and to python/* for easier (and AppVeyor) testing.

zware · 2017-05-24T21:52:18Z

@zooba I did go ahead and push to the real repos, which makes AppVeyor happy. I've also successfully run Tools/msi/buildrealease.bat --test <dir> (a first).

Also fixes an outdated note about _lzma

zooba · 2017-05-31T17:29:11Z

@zware Paths with spaces in them are fine, but you'll need to be including quotes basically everywhere you use the variable. For build paths, I'd suggest leaving quotes out of the variable and using them when used, but for tools (e.g. the path to python.exe) I'd put the paths in the variable (so that py -3.6 is also a valid value and does not get quoted when invoked).

Essentially the same quoting rules apply to PowerShell as well.

zware · 2017-05-31T18:00:07Z

Right, I eventually stopped messing with it on here and started working on it in my VM, got it mostly fixed, and then ran out of time before pushing and haven't gotten back to it...

zware · 2017-06-09T03:30:37Z

Finally got this updated.

zooba · 2017-06-12T18:17:05Z

Tools/msi/get_externals.bat

+echo.Fetching external binaries...
+
+set binaries=
+set binaries=%binaries%     binutils


I feel like this deserves a comment saying that we always build with the latest tools and so the versions are not part of the CPython sdist, unlike source dependencies. That will save me having to figure out why we don't have versions marked on these every time :)

Maybe using "tools" as the term rather than "binaries" would make more sense?

…honGH-1783) The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio. Also fixes an outdated note about _lzma in PCbuild/readme.txt

…1783) (GH-2237) The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio. Also fixes an outdated note about _lzma in PCbuild/readme.txt (cherry-picked from commit 51599e2)

…honGH-1783) The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio. (cherry-picked from 51599e2)

bedevere-bot · 2017-09-04T20:58:25Z

GH-3306 is a backport of this pull request to the 2.7 branch.

GH-1783) (GH-3306) The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio. Cherry-picked from 51599e2, parts of 40a23e8, parts of 68d663c, d5cd21d, and possibly others that I've missed. Also: * Rename db -> bsddb for disambiguity * Update sqlite3 to 3.14.2.0 since it's the version we use on 3.x, and it's simpler to just use it than to also upload the old version to cpython-source-deps * Add PCbuild/*.ilk to .gitignore

Pull Windows dependencies from GitHub rather than svn

49e6717

zware added type-feature A feature request or enhancement needs backport to 2.7 OS-windows labels May 24, 2017

zware requested a review from zooba May 24, 2017

the-knights-who-say-ni added the CLA signed label May 24, 2017

zooba requested changes May 24, 2017

View changes

Use quotes around an envvar

8e3c238

zware added 4 commits May 24, 2017

Use 32-bit python from Nuget; exclude version

b9993bb

Use pathlib, leave directory creation to Python if possible

e95433a

Rewrite Tools/msi/get_externals.bat

e0bd20d

Fix Tools/msi/get_externals.bat

c304ea6

zware added 8 commits May 28, 2017

Be more consistent in quoting and capitalization

ffb3f71

Be more consistent in quoting and capitalization

99063c0

Remove SVN mentions from readme.txt

b7766fa

Also fixes an outdated note about _lzma

Fix capitalization of NuGet

2bc26ce

Hope this isn't what broke AppVeyor...

3d2545e

Can NuGet handle dirs with spaces in them?

27441cb

Fix support for dirs with spaces

e18a852

Allow a bit more environmental configurability

1c820dd

zware added 2 commits Jun 9, 2017

Don't add trailing slash to EXTERNALS_DIR

9f6f471

Update Tools/msi/get_externals.bat

26308cf

Merge branch 'master' into goodbye_svn

e6830ae

zware requested a review from zooba Jun 11, 2017

zooba approved these changes Jun 12, 2017

View changes

66s/binaries/tools/, add comment

db974fd

zware merged commit 51599e2 into python:master Jun 16, 2017

zware deleted the goodbye_svn branch Jun 16, 2017

bedevere-bot removed the needs backport to 2.7 label Sep 4, 2017

zware removed the needs backport to 3.6 label Sep 7, 2017

bpo-30450: Pull Windows dependencies from GitHub rather than svn #1783

bpo-30450: Pull Windows dependencies from GitHub rather than svn #1783

zware commented May 24, 2017

mention-bot commented May 24, 2017

zooba left a comment

zooba May 24, 2017

zooba May 24, 2017

zware May 24, 2017

zooba May 24, 2017

zware May 24, 2017

zooba May 24, 2017

zooba May 24, 2017

zware May 24, 2017

zooba May 24, 2017

zware May 24, 2017

zooba May 24, 2017

zware May 24, 2017

zware commented May 24, 2017

zware commented May 24, 2017

zooba commented May 31, 2017

zware commented May 31, 2017

zware commented Jun 9, 2017

zooba Jun 12, 2017

bedevere-bot commented Sep 4, 2017

bpo-30450: Pull Windows dependencies from GitHub rather than svn #1783

bpo-30450: Pull Windows dependencies from GitHub rather than svn #1783

Conversation

zware commented May 24, 2017

mention-bot commented May 24, 2017

zooba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zware commented May 24, 2017

zware commented May 24, 2017

zooba commented May 31, 2017

zware commented May 31, 2017

zware commented Jun 9, 2017

Choose a reason for hiding this comment

bedevere-bot commented Sep 4, 2017