bpo-28015: Support LTO build with clang #9908

serge-sans-paille · 2018-10-16T09:15:50Z

.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
to build autoconf test case, and some are not compatible with clang LTO
(they assume binary in the .o, not bitcode)
force llvm-ar instead of ar, as ar is not aware of .o files generated
by clang -flto

https://bugs.python.org/issue28015


        Support LTO build with clang

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto

the-knights-who-say-ni · 2018-10-16T09:15:53Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

serge-sans-paille · 2018-10-16T09:27:11Z

CLA just signed, that should be ok now.

I did not commit the autoreconf-generated file, don't know what's the procedure there.

vstinner

Python source code includes configure for practical reasons, you must run autoconf and include the modified configure in your PR.

bedevere-bot · 2018-10-16T09:45:35Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.


        Include generated auto-* scripts

vstinner

It would be convenient if configure fails with an error and an explanation how to fix it if llvm-ar is missing:

AC_PATH_TARGET_TOOL(LLVM_AR, llvm-ar, '', ${llvm_path})
(...)
AR="${LLVM_AR}"


        Better llvm-ar detection, ad proper error message if detection failed

vstinner · 2018-10-16T10:13:57Z

I tested the 3rd version. It seems like -flto is passed during the compilation of object files and to the final linker step:

$ grep lto Makefile
BASECFLAGS=	 -flto -g -Wno-unused-result -Wsign-compare
CONFIGURE_LDFLAGS=	 -flto -g
CONFIG_ARGS=	 '--with-pydebug' 'CC=clang' '--with-lto'

$ make
...
clang -pthread -fPIC -flto (...)
...
/usr/bin/llvm-ar rcs libpython3.8dm.a Modules/getbuildinfo.o (...) Python/frozen.o
clang -pthread -flto -g  -Xlinker -export-dynamic -o python Programs/python.o libpython3.8dm.a -lpthread -ldl  -lutil -lm   -lm

vstinner

LGTM. It works as expected (see my comment above) and it fix an obvious bug https://bugs.python.org/issue28015#msg327817

vstinner · 2018-10-16T10:18:01Z

You may add a NEWS entry using the tool "blurb".

@serge-sans-paille signed the CLA, but I have to wait until someone review his signature.

vstinner · 2018-10-16T10:20:22Z

I will remove the "CLA not signed" label once "Contrib Form Received" will become Yes on https://bugs.python.org/user22883


        Add NEWS entry

miss-islington · 2018-10-25T00:00:14Z

Thanks @serge-sans-paille for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

bedevere-bot · 2018-10-25T00:00:28Z

GH-10089 is a backport of this pull request to the 3.7 branch.


        bpo-28015: Support LTO build with clang (pythonGH-9908)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto (cherry picked from commit 5ad36f9) Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>

vstinner · 2018-10-25T00:01:54Z

@serge-sans-paille: The backport to 3.7 was straighfoward and could be automated, but backport to 3.6 creates a conflict on configure.ac. Do you want to try the backport? I'm not sure if it's worth it to backport the change up to 3.6. It's up to you.


        bpo-28015: Support LTO build with clang (GH-9908)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto (cherry picked from commit 5ad36f9) Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>

serge-sans-paille · 2018-10-29T10:23:30Z

@vstinner I think I have the 3.6 patch, should I create a new PR on the 3.6 branch?

serge-sans-paille · 2018-10-29T12:44:42Z

Finally, there a re many complex interactions with RANLIB and AR, dropping the backport.

vstinner · 2018-10-29T12:47:52Z

Finally, there a re many complex interactions with RANLIB and AR, dropping the backport.

The Python 3.6 backport? It's too complex? It's ok to only fix Python 3.7 and newer. (I just need your confirmation to close the issue.)

serge-sans-paille · 2018-10-29T12:56:03Z

Yeah, the 3.6 backport would require some more thoughts, better drop it and close the issue. Thanks!


        [3.6] bpo-28015: Support LTO build with clang (pythonGH-9908)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto (cherry picked from commit 5ad36f9) Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>


        [3.6] bpo-28015: Support LTO build with clang (pythonGH-9908)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto

stratakis · 2018-10-31T17:43:38Z

The 3.6 backport works fine if https://bugs.python.org/issue31625 is applied for 3.6 as well.


        bpo-28015: Support LTO build with clang (pythonGH-9908)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto


        bpo-28015: Support LTO build with clang (pythonGH-9908)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto (cherry picked from commit 5ad36f9) Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>


        bpo-28015: Support LTO build with clang (GH-9908) (GH-10922)

.o generated by clang in LTO mode actually are LLVM bitcode files, which leads to a few errors during configure/build step: - add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used to build autoconf test case, and some are not compatible with clang LTO (they assume binary in the .o, not bitcode) - force llvm-ar instead of ar, as ar is not aware of .o files generated by clang -flto (cherry picked from commit 5ad36f9) Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>

the-knights-who-say-ni added the CLA not signed label Oct 16, 2018

bedevere-bot added the awaiting review label Oct 16, 2018

vstinner requested changes Oct 16, 2018

View changes

bedevere-bot removed the awaiting review label Oct 16, 2018

bedevere-bot added the awaiting changes label Oct 16, 2018

Include generated auto-* scripts

Loading status checks…

00205f7

vstinner requested changes Oct 16, 2018

View changes

Better llvm-ar detection, ad proper error message if detection failed

Loading status checks…

45cc56e

vstinner approved these changes Oct 16, 2018

View changes

bedevere-bot added awaiting merge and removed awaiting changes labels Oct 16, 2018

Add NEWS entry

Loading status checks…

f19eb57

pablogsal requested a review from gpshead Oct 17, 2018

vstinner removed the CLA not signed label Oct 24, 2018

the-knights-who-say-ni added the CLA signed label Oct 24, 2018

bedevere-bot removed the awaiting merge label Oct 24, 2018

vstinner added the needs backport to 3.7 label Oct 25, 2018

bedevere-bot removed the needs backport to 3.7 label Oct 25, 2018

stratakis mentioned this pull request Oct 31, 2018

[3.6] bpo-31354: Let configure --with-lto work on all builds #10261

Merged

stratakis mentioned this pull request Nov 8, 2018

[3.6] bpo-31625: Stop using ranlib #10417

Merged

jjhelmus mentioned this pull request Oct 15, 2019

python v3.8.0 #274

Merged

1 of 3 tasks complete

python / cpython

bpo-28015: Support LTO build with clang #9908

bpo-28015: Support LTO build with clang #9908

serge-sans-paille commented Oct 16, 2018 •

edited by bedevere-bot

This comment has been minimized.

the-knights-who-say-ni commented Oct 16, 2018

This comment has been minimized.

serge-sans-paille commented Oct 16, 2018

vstinner left a comment

This comment has been minimized.

bedevere-bot commented Oct 16, 2018

vstinner left a comment

This comment has been minimized.

vstinner commented Oct 16, 2018

vstinner left a comment

This comment has been minimized.

vstinner commented Oct 16, 2018

This comment has been minimized.

vstinner commented Oct 16, 2018

This comment has been minimized.

miss-islington commented Oct 25, 2018

This comment has been minimized.

bedevere-bot commented Oct 25, 2018

This comment has been minimized.

vstinner commented Oct 25, 2018

This comment has been minimized.

serge-sans-paille commented Oct 29, 2018

This comment has been minimized.

serge-sans-paille commented Oct 29, 2018

This comment has been minimized.

vstinner commented Oct 29, 2018

This comment has been minimized.

serge-sans-paille commented Oct 29, 2018

This comment has been minimized.

stratakis commented Oct 31, 2018

python / cpython

Sponsor python/cpython

Join GitHub today

bpo-28015: Support LTO build with clang #9908

bpo-28015: Support LTO build with clang #9908

Conversation

serge-sans-paille commented Oct 16, 2018 • edited by bedevere-bot

This comment has been minimized.

the-knights-who-say-ni commented Oct 16, 2018

This comment has been minimized.

serge-sans-paille commented Oct 16, 2018

vstinner left a comment

This comment has been minimized.

bedevere-bot commented Oct 16, 2018

vstinner left a comment

This comment has been minimized.

vstinner commented Oct 16, 2018

vstinner left a comment

This comment has been minimized.

vstinner commented Oct 16, 2018

This comment has been minimized.

vstinner commented Oct 16, 2018

This comment has been minimized.

miss-islington commented Oct 25, 2018

This comment has been minimized.

bedevere-bot commented Oct 25, 2018

This comment has been minimized.

vstinner commented Oct 25, 2018

This comment has been minimized.

serge-sans-paille commented Oct 29, 2018

This comment has been minimized.

serge-sans-paille commented Oct 29, 2018

This comment has been minimized.

vstinner commented Oct 29, 2018

This comment has been minimized.

serge-sans-paille commented Oct 29, 2018

This comment has been minimized.

stratakis commented Oct 31, 2018

serge-sans-paille commented Oct 16, 2018 •

edited by bedevere-bot