See https://matplotlib.org/stable/tutorials/text/annotations.html#annotating-with-text-with-box , I think you can update the path to get what ever shape you want now (but at the cost of writing the path your self) and has a number of (non-directional) shapes.

The tricky part of this generally is going to be merging the notion of the bounding box and the notion of arrow, but given that both are currently updated very late in the draw process, I strongly suspect that a bounding box that updates the spur based on where the annotated point is.