Does that have any impacts on https://www.python.org/dev/peps/* ?(which already renders the peps well)

Indeed, since the main purpose is to be rendered on www.python.org I don't think this would change much. The added markup also isn't really used for PEPs.

So I envision this tying into #3 so that we can host things on Read the Docs and have a sub-domain for the PEPs.

Making that match the design and style currently used on python.org would be tedious, though.

So I envision this tying into #3 so that we can host things on Read the Docs and have a sub-domain for the PEPs.

So would that mean pepe would no longer be displayed on Python.org ?

Also from other project i'm contributing too, multiple domains for different section of a website are painful, (especially on readthedocs) because the navigation does not work well across site.

It would though most likely be feasible to build this on travis and cross-push on the python/pythondotorg repository to trigger a rebuild of the main website. Or of you don't like that, push a separate branch and auto submit a PR.

I don't care if the style changes. I just think having the PEPs buried in www.python.org and having such a separate build process is odd, especially when the contents have no direct connection to what's on www.python.org (who directly links to the PEPs from www.python.org anyway beyond navigation links?)

As for setting up some way to have pythondotorg automatically re-build the PEPs when a commit occurs, I'm quite happy to see that happen.

I don't care if the style changes. I just think having the PEPs buried in www.python.org and having such a separate build process is odd, especially when the contents have no direct connection to what's on www.python.org (who directly links to the PEPs from www.python.org anyway beyond navigation links?)

As for setting up some way to have pythondotorg automatically re-build the PEPs when a commit occurs, I'm quite happy to see that happen.

Thanks, that's a much simpler problem if we have less backward compatibility.

I'm combining #3 with this issue as there's no point in making the PEPs build with Sphinx if we don't create a subdomain of peps.python.org that is hosted on Read the Docs.

thanks for your effort

for fun i have created a PEP server that can be used to iterate on PEPs while seeing them in their final style.

it could be useful to get something close to the way PEPs look on python.org (since i already did the work to create a HTML template managing that does that):

https://gist.github.com/flying-sheep/c296482b785662d505f06c8bfe32f16e

it includes a server that serves HTML renders of PEPs without writing files. the pages contain live.js for auto-reloading and link the python.org stylesheet, so you can just edit away.

should i do a PR with it?

if yes, i’m interested in the scope for the provided scripts: are they all just for development? why the RSS stuff then?

The scripts in the project are actually used to generate the final files served at python.org/dev/peps.

As for a PR, feel free to propose it but I don't know what to think about it. I personally have never been bothered by the lack of styling when viewing generated PEPs (but then again I don't even bother with that anymore since we moved to GitHub when making edits to an existing PEP). If you want to discover this further, please open a new issue since this doesn't directly tie into using Sphinx for building.

I just learned that using Sphinx would also fix #165 since Sphinx converts typewriter quotes to typographically correct ones, very nice 😄

Noting what I'd personally consider the main advantage of switching to Sphinx for PEP generation: we'd be able to set up intersphinx inventory support referencing the development version of the CPython documentation, so we'd gain access to all our usual :term: glossary entries and other cross references to modules and API entries.

Of course, the downside of that is that changes to the dev docs could break references from old PEPs.

Switching to Sphinx also gets us Pygments support for code examples.

As for broken links, that's called the internet. 😉 I would assume the text in the PEP would be unambiguous enough to make it obvious what the PEP was referring to.

#701 is another problem that would be resolved by switching to Sphinx: it turns out SVGs don't currently work because the HTML 4.1 docutils writer that we're using emits them as an object tag instead of an img tag (for IE6 compatibility!), and the rendering code on the python.org side of things doesn't fix up the link correctly the way it does for image tags.

Clicking in the links do not work. Is that expected? For instance:

https://peps-aaturner.readthedocs.io/dev/peps/pep-0438

Good catch, sorry - this is as the links are still based on python.org's format (e.g. https://www.python.org/dev/peps/pep-0628/). I'll update to get it working on here, one moment please.

Does this mean that the URL format will change? I think that would be unwanted (except automatic redirection with/without trailing slash).

I noticed that the surname extraction does not handle Ernest W. Durbin III properly 🙂

@ericholscher no requirement ATM. This is an old issue where we considered trying to get the PEPs built using Sphinx instead of our custom scripts. My assumption was this would eventually end up on Read the Docs, but that discussions hasn't been had anywhere yet.

Sorry for delay - turns out Read the Docs runs sphinx 1.8.5 so had to back-port some classes - I wondered why my builds kept failing!

Clicking in the links do not work. Is that expected? For instance:

I've made it work with internal links for this project (https://peps-aaturner.readthedocs.io/en/latest/pep-0000.html) - including in-text references to PEPs, and built a seperate directory version here (https://peps-dir-aaturner.readthedocs.io/en/latest/pep-0000/).

Does this mean that the URL format will change? I think that would be unwanted (except automatic redirection with/without trailing slash).

RtD's current build process (at least from the standard end-user side) doesn't appear to allow for custom base directories, but I suspect a general redirect from python.org/dev/peps/* to peps.python.org/* (or equivalent) could be set up. Also of course the custom domain pointing to RtD (or the bare HTML files) could have that proxy inbuilt.

I noticed that the surname extraction does not handle Ernest W. Durbin III properly 🙂

To be fair, Ernest currently isn't parsed by python.org correctly ;-) - I think this is as the logic expects a comma (e.g. Adam Smith, Jr.) - I'll see what I can do to update this to fix

@AA-Turner hi

turns out Read the Docs runs sphinx 1.8.5 so had to back-port some classes

You can use any version you want, you just need to specified it in a requirements file or similar https://docs.readthedocs.io/en/stable/guides/specifying-dependencies.html

You can use any version you want, you just need to specified it in a requirements file or similar https://docs.readthedocs.io/en/stable/guides/specifying-dependencies.html

Brilliant, thanks!

Ah you are right! Then your PR should not concern itself with fixing the name extraction, it’s not a problem it introduces.

Ah you are right! Then your PR should not concern itself with fixing the name extraction, it’s not a problem it introduces.

Should I do a new one then? I have a solution pretty-much (including for special cases e.g. PEP 13)

I think it depends on the timing! I don’t know how much work is left on your PR for the sphinx build, and how much time will be needed to have it accepted. Probably a fix for the problem in the existing codebase would be merged really quick. But if it’s too much work and you would prefer to do it in your new code, then we could wait. (Same PR or new PR would depend on how much code is needed I think)

Does that make sense?

@merwok - see PR #1386 for the single-issue PEP 0 parsing change, though I have included it into my other branch so that it doesn't bother me now that you've pointed it out! I will keep both versions of the patch in sync if changes are requested

You can use any version you want, you just need to specified it in a requirements file or similar docs.readthedocs.io/en/stable/guides/specifying-dependencies.html

Just updated with a requirements file and it works perfectly, thanks @stsewd!

I've added fail-on-warning tests (as in #5, to continuous integration/Travis but there are currently 200+ warnings so I've had to set it to be an allowed failure within travis.

In a similar vein I've set up a linkcheck test on Travis, but it currently times out at 50 minutes as there is are a significant number of dead/moved links within the PEPs (probably to be expected). I'm less sure if this second one is useful, but it might be.

Link to the latest Travis job, if of interest: https://travis-ci.org/github/python/peps/builds/681275206

I worry that the warnings about footnotes mean that the footnotes are broken.

Is there a build preview somewhere to compare before/after?

Another question: to replace the current build system, the PEPs will need Sphinx theme customization (HTML includes to get the top menu, side menu, footer + CSS that blends with pydotorg style). Should that be another ticket? (this one: make sphinx build possible at all, another: integrate with pydotorg and do the switch)

I don't think a separate one is as this work can't even be considered for integration until it can work with the website as-is or a much larger plan to switch to a different hosting situation is worked out.

Is there a build preview somewhere to compare before/after?

I couldn't get the current (pep2html.py) build process to emit footnote refernce warnings, but checking a few manually in the warning list given by Sphinx against both py.org and pep2html (and the RST source) I believe the warnings are correct. E.g. Footnote 1 in PEP 3, FNs 1-6 in PEP 467, FNs 1-22 in PEP 538, any of the citations in PEP 520.

I think this gives a conclusion of living with the warnings as this work didn't create them, and potentially resolving them as a later piece of work.

Another question: to replace the current build system, the PEPs will need Sphinx theme customization

So to drop-in replace the current build system (i.e. still use the py.org converters) I don't think this is needed, as the current HTML output doesn't have this information. However checking all the classes are right etc. is something I've started and I'm considering writing a Sphinx HTML translator to mock as close as possible the docutils native pep/html4/css1 writer class.

To function as a standalone/self-contained system however, this is needed of course. I'm not sure which option is prefered (@brettcannon ?) so at the moment I'm targeting both equally.

Thanks,
Adam

Ah I didn’t know that the website did post-processing! In that case, it seems fine to have the default theme here in the sphinx build, and I assume some beta validation with the website will be needed.

Ah I didn’t know that the website did post-processing! In that case, it seems fine to have the default theme here in the sphinx build, and I assume some beta validation with the website will be needed.

I've now added a number of transforms to my pydotorg compatability layer (package.py in the PR), and having checked a number (~20) of PEPs near-visually-identical output is now being produced. (Sources are not entirely identical as Sphinx adds some extra classes that are transparent to pydot org, and sphinx also transforms '--' into an en dash '–' & straight quotes into typewriter-style quotes). This would satisfy the first option of a drop-in replacement, hopefully.

Work continues on creating a custom sphinx style to mirror what currently is shown on pydotorg, but I'm not the best in the world at CSS so that is a bit slower - apologies for the dearth of updates lately.

P.S. I'm also looking into visual regression testing for a more formal and automated look into if this method changes edge-case styling, but that is on the back burner at the moment.

Work continues on creating a custom sphinx style to mirror what currently is shown on pydotorg, but I'm not the best in the world at CSS so that is a bit slower - apologies for the dearth of updates lately.

I am a bit confused — is that necessary? We can check the sphinx results with default theme for proper rendering and functionality (footnotes etc), and run your transformer to see the pydotorg style version. Don’t do more work than required, it’s already a lot!

I am a bit confused — is that necessary? We can check the sphinx results with default theme for proper rendering and functionality (footnotes etc), and run your transformer to see the pydotorg style version. Don’t do more work than required, it’s already a lot!

Very happy if it's not needed! 😀 My thinking was that for the second option of a full peps.python.org *thing*, having the styling from the current PEPs location would be a boon and aid transition (and the current default scheme is a little ugly) -- if the PEPs were to be hosted on Sphinx the styling from pydotorg wouldn't pull across.

Broadly as Brett Cannon noted above I'm not sure this work would be deployed/beta'd in a partially complete state, so I'd like to get it to as close to complete as I can, if possible!

I've now finished styling (I think!) - I've checked 15-20 manually scrolling along with pydotorg, though I imagine that there are small things I won't have caught.

Also a couple of things explicitly don't want to be styled back to the same as pydotorg - e.g. code highlighting, quoting, typographic adjustments. (python/pythondotorg#1063, #165, #701)

Latest rendered files are here: https://aa-turner.github.io/peps/, and I've set up GH actions to deploy these automatically etc.

I will now work on chunking the PR into smaller blocks as per Victor's request.

Chunked into smaller PRs here - note these are mainly for code seperation and will not build individually - #1385 remains the 'master' PR.

#1565 - core PR
#1566 - docutils core, transforms/parsing/writing
#1567 - PEP 0 generation & hooks
#1568 - theming (both CSS and templating)
#1569 - pythondotorg compatability - very much a temporary layer.

I've also added support for last-modified links at the footer of all pages in a similar implementation to Mariatta's (python/pythondotorg#1564, python/pythondotorg#1565)

Thanks,
Adam

P.S. apologies for the delay; life got in the way!