Drop the FT2Font intermediate buffer. #30059

anntzer · 2025-05-16T12:07:50Z

Directly render FT glyphs to the Agg buffer. In particular, this naturally provides, with no extra work, subpixel positioning of glyphs (closing #29551) (this could also have been implemented in the old framework, but would have required careful tracking of subpixel offets).

Note that rendering of mathtext boxes (e.g., fraction bars) is currently missing, but should be "easy" (the main question being whether/how to ensure proper hinting/alignment to pixel edges in the horizontal or vertical output cases). Likewise, all baseline images should be regenerated. Finally, the new APIs added to FT2Font are likely up to bikeshedding (but they are all private).

In practice this should get included (if we agree to go this way) into the FreeType update (#29816). This would also supersede the patch advertised at #29816 (comment).

test modified from #29551 to also include mathtext:

import matplotlib.pyplot as plt
import matplotlib.animation as manim

fig = plt.figure(figsize=(2, 2))
ts = [
    fig.text(0, 0, '$ABC123$'),
    fig.text(0, 0.2, 'ABC123'),
]
change = 0.001

def update(*args, **kwargs):
    global change
    for t in ts:
        x, y = t.get_position()
        if not (0 <= x <= 1 and 0 <= y <= 1):
            change *= -1
        t.set_x(x + change)
        t.set_y(y + change)
    return ts

ani = manim.FuncAnimation(fig, update, interval=20, frames=int(2/change),
                          cache_frame_data=False)
ani.save('text.gif')

PR summary

PR checklist

"closes #0000" is in the body of the PR description to link the related issue
new and changed code is tested
Plotting related features are demonstrated in an example
New Features and API Changes are noted with a directive and release note
Documentation complies with general and docstring guidelines

story645 · 2025-05-16T14:39:09Z

Would you be open to using a tracking issue or project board for the font work (sample categories: draft, waiting on other PRs, ready for review, please review first)?

I think breaking everything up into small PRs is fantastic but I'm finding it a bit overwhelming to keep track of the order for reviewing things (and figure I'm not alone here) and how these PRs relate to each other.

anntzer · 2025-05-16T15:14:53Z

Sure, I'll do that. I'm sorry for the mess, but this is basically me trying to remember all the patches or even just ideas I have accumulated over the years that went nowhere because they would thrash all the baseline images, and that I now try to present given that I see an opportunity with the FreeType upgrade. Also, some of them are not fully complete patches but things that I believe can be made complete with relatively little effort (for some value of "little effort"...) simply because I do not have the time now to finalize them but want to put them up for discussion as the underlying idea should be clear.

Edit: see the wiki; attn @QuLogic as well too.

story645 · 2025-05-16T16:11:04Z

I'm sorry for the mess, but

No apologies needed, I commend you for taking advantage of the freetype upgrade. And thanks for the wiki, though some reason can't link cleanly ☹️

anntzer · 2025-05-16T17:05:32Z

link fixed

anntzer · 2025-05-17T09:55:20Z

Edit: fraction bar rendering has been added too.

QuLogic · 2025-05-22T01:22:56Z

A few warnings from compiling locally:

[2/4] Compiling C++ object src/ft2font.cpython-313-x86_64-linux-gnu.so.p/ft2font_wrapper.cpp.o
../../src/ft2font_wrapper.cpp: In lambda function:
../../src/ft2font_wrapper.cpp:1785:22: warning: unused variable ‘error’ [-Wunused-variable]
 1785 |             if (auto error = FT_Load_Glyph(face, idx, static_cast<FT_Int32>(flags))) {
      |                      ^~~~~
../../src/ft2font_wrapper.cpp:1788:22: warning: unused variable ‘error’ [-Wunused-variable]
 1788 |             if (auto error = FT_Render_Glyph(face->glyph, FT_RENDER_MODE_NORMAL)) {
      |                      ^~~~~
../../src/ft2font_wrapper.cpp: In lambda function:
../../src/ft2font_wrapper.cpp:1804:26: warning: unused variable ‘error’ [-Wunused-variable]
 1804 |                 if (auto error = FT_Glyph_To_Bitmap(&g, FT_RENDER_MODE_NORMAL, &origin, 1)) {
      |                          ^~~~~

anntzer · 2025-05-22T06:19:21Z

This is semi-intentional: if these checks fail (error > 0) then the error should really be raised using throw_ft_error, not with a plain raise as I currently do, but throw_ft_error is not visible from ft2font_wrapper.cpp so some code needs to be moved around, e.g. make throw_ft_error an inline function defined in ft2font.h, or move the new implementations into real C++ FT2Font methods in ft2font.cpp and add wrapper lambdas in ft2font_wrapper.cpp, or (would be my preferred choice) switch error checking to the FT_CHECK macro used in mplcairo (see mplcairo's macros.h).
I left the warning as is just to keep this as a side discussion point, but it can trivially be suppressed by not capturing the error value too.

QuLogic · 2025-05-24T09:34:57Z

On a side note, I'm in favour of this in general as removing the intermediate buffer will make emoji/colour glyphs easier since they won't need to go through it.

QuLogic · 2025-05-30T02:19:22Z

On the call, we were discussing whether it made sense to target all these font-related (and requiring full image rebuild) PRs to a separate feature branch, which would contain the final test image updates. Do you have any thoughts on retargetting this PR (instead of folding it in to #29816)?

anntzer · 2025-05-30T06:18:32Z

On the call, we were discussing whether it made sense to target all these font-related (and requiring full image rebuild) PRs to a separate feature branch, which would contain the final test image updates. Do you have any thoughts on retargetting this PR (instead of folding it in to #29816)?

Sure, I don't really mind either way? Should the branch live on this repo and be ultimately pruned out, or on yours?

anntzer · 2025-05-30T08:38:18Z

Fixed #30059 (comment) by using FT_CHECK (thus also needs #30123).

QuLogic · 2025-06-06T22:37:23Z

lib/matplotlib/backends/backend_agg.py

+            self._renderer.draw_text_image(
+                bitmap["buffer"],
+                bitmap["left"],
+                int(self.height) - bitmap["top"] + bitmap["buffer"].shape[0],


Just checking whether we should round here or not?

I think self.height is already always a integer-valued float, as it's the size of the buffer in pixels (I have a bit of doubt about hiDPI though). In fact, it would probably be nicer if self.height (and likewise self.width) was made an int (in fact, get_width_height, which is a FigureCanvasBase API, unlike self.{width,height}, which are specific to the Agg canvas, returns ints).

QuLogic · 2025-06-06T23:12:29Z

lib/matplotlib/backends/backend_agg.py

+                    np.full((round(h), round(w)), np.uint8(0xff)),
+                    round(x + dx), round(y - dy - h),
+                    0, gc)
+        else:


This latter half is seemingly untested.

I guess we don't test rotated mathtext fractions? Something like figtext(.5, .5, r"$\frac{3}{4}$", rotation=45) would do.

QuLogic · 2025-06-06T23:14:29Z

src/ft2font_wrapper.cpp

+        // TODO: Return a nicer structure than dicts.
+        // NOTE: The lifetime of the buffers is limited and could get invalidated...
+        // TODO: Real antialiasing flag.
+        // TODO: throw_ft_error.


Are these TODO nice-to-have, or to-be-done here? Also, throw_ft_error is done now?

Removed the mention of throw_ft_error.
We should definitely add support for antialiasing, but that should be easy, just a matter of threading the parameter down to the right place.
Returning a nicer structure than a plain dict would be nice but not really an obligation as long as _render_glyph stays private. Likewise, we can probably get away with not caring about the buffer lifetime as long as _render_glyph is private, but properly handling the buffer lifetime once that becomes a public API seems trickier? I basically wrote the minimum API that works for privately communicating between the C++ and Python layers, but didn't put a lot of thought into it.

QuLogic · 2025-06-06T23:15:08Z

src/ft2font_wrapper.cpp

+            auto matrix = FT_Matrix{
+            std::lround(c / hf), std::lround(-s), std::lround(s / hf), std::lround(c)};


Suggested change

auto matrix = FT_Matrix{

std::lround(c / hf), std::lround(-s), std::lround(s / hf), std::lround(c)};

auto matrix = FT_Matrix{

std::lround(c / hf), std::lround(-s), std::lround(s / hf), std::lround(c)};

or maybe, since it's 2D:

Suggested change

auto matrix = FT_Matrix{

std::lround(c / hf), std::lround(-s), std::lround(s / hf), std::lround(c)};

auto matrix = FT_Matrix{

std::lround(c / hf), std::lround(-s),

std::lround(s / hf), std::lround(c)};

Directly render FT glyphs to the Agg buffer. In particular, this naturally provides, with no extra work, subpixel positioning of glyphs (which could also have been implemented in the old framework, but would have required careful tracking of subpixel offets). Note that all baseline images should be regenerated. The new APIs added to FT2Font are also up to bikeshedding (but they are all private).

QuLogic · 2025-06-10T20:55:29Z

Sorry for the conflict; I realize that #29199 was going to be thrown away here anyway, but I wanted to acknowledge the (new) contributor's work, and I knew you wouldn't have any trouble rebasing.

anntzer · 2025-06-10T22:46:39Z

I didn't look carefully but I believe I can just get rid of the changes in #29199 when rebasing because my PR essentially already does the same thing, correct? (no problem with doing the rebase, of course)

anntzer added topic: text topic: text/mathtext labels May 16, 2025

github-actions bot added backend: agg topic: text/fonts and removed topic: text topic: text/mathtext labels May 16, 2025

This was referenced May 16, 2025

Update FreeType to 2.13.3 #29816

Open

Fix alpha compositing in ft2font's draw_bitmap. #30043

Closed

QuLogic mentioned this pull request May 17, 2025

Replace FT2Image by plain numpy arrays. #30044

Merged

5 tasks

anntzer force-pushed the ft-direct-render branch 3 times, most recently from d44360f to 66fdae0 Compare May 17, 2025 09:55

github-actions bot added the status: needs rebase label May 24, 2025

anntzer force-pushed the ft-direct-render branch from 66fdae0 to e980d97 Compare May 24, 2025 09:30

github-actions bot removed the status: needs rebase label May 24, 2025

anntzer mentioned this pull request May 24, 2025

Simplify/improve error reporting from ft2font. #30102

Merged

5 tasks

anntzer force-pushed the ft-direct-render branch from e980d97 to 383e594 Compare May 30, 2025 08:37

QuLogic changed the base branch from main to text-overhaul June 5, 2025 01:47

QuLogic added this to the v3.11.0 milestone Jun 5, 2025

QuLogic added this to Font and text overhaul Jun 5, 2025

QuLogic moved this to Ready for Review in Font and text overhaul Jun 5, 2025

QuLogic reviewed Jun 6, 2025

View reviewed changes

anntzer force-pushed the ft-direct-render branch from 383e594 to 4942861 Compare June 7, 2025 21:51

github-actions bot added the status: needs rebase label Jun 10, 2025

		auto matrix = FT_Matrix{
		std::lround(c / hf), std::lround(-s), std::lround(s / hf), std::lround(c)};

Uh oh!

Drop the FT2Font intermediate buffer. #30059

Are you sure you want to change the base?

Drop the FT2Font intermediate buffer. #30059

Uh oh!

Conversation

anntzer commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

PR checklist

Uh oh!

story645 commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anntzer commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

story645 commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anntzer commented May 16, 2025

Uh oh!

anntzer commented May 17, 2025

Uh oh!

QuLogic commented May 22, 2025

Uh oh!

anntzer commented May 22, 2025

Uh oh!

QuLogic commented May 24, 2025

Uh oh!

QuLogic commented May 30, 2025

Uh oh!

anntzer commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anntzer commented May 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QuLogic commented Jun 10, 2025

Uh oh!

anntzer commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

anntzer commented May 16, 2025 •

edited

Loading

story645 commented May 16, 2025 •

edited

Loading

anntzer commented May 16, 2025 •

edited

Loading

story645 commented May 16, 2025 •

edited

Loading

anntzer commented May 30, 2025 •

edited

Loading

anntzer commented Jun 10, 2025 •

edited

Loading