I won't consider this as a "bug", but it's something that we can definitely improve. Breaking on max_lines is nice, but I think iterating by characters(and creating new strings on the way) is a huge performance hit as well. I think we can just use internal string manipulations here to improve performance. I can work on some benchmark on this, I assume it would be faster.

I tried to replicate the exact behavior of the current _splitlines_no_ff function because I don't know if there's any specific concerns the author had for new line breakers, which is:

• Consider \n, \r\n and \r as line breakers
• Convert \r\n to \n
• Keep the line breaker as it is (do not convert \r)

This is not trivial in pure string manipulation, so I used re, which provided useful iterating matching.

Core code is

_line_pattern = re.compile(r"(.*?(?:\r\n|\n|\r|$))")
def _splitlines_no_ff(source, maxlines=-1):
    lines = []
    for lineno, match in enumerate(_line_pattern.finditer(source), 1):
        if maxlines > 0 and lineno > maxlines:
            break
        lines.append(match[0].replace("\r\n", "\n"))
    return lines

Pre-compile regex helped a bit in benchmarks, we can switch that to inline compile if that's considered ugly. The regex + replace almost replicated the behavior exactly except for the fact that it will end up with an empty string at the end when the source is not terminated by \n. I think that's completely acceptable - with maxlines input that line should not be used at all anyway.

>>> re.findall(r"(.*?(?:\r\n|\r|\n|$))", "a\f\rb\n\nc\r\ndd")
['a\x0c\r', 'b\n', '\n', 'c\r\n', 'dd', '']

Benchmark code as below:

import timeit

short_setup = f"""
import ast
code = \"\"\"def fib(x):
    if x < 2:
        return 1
    return fib(x - 1) + fib(x - 2)
\"\"\"
module_node = ast.parse(code)
function_node = module_node.body[0]
"""

long_setup_start = f"""
import ast
with open("Lib/inspect.py") as f:
    code = f.read()
module_node = ast.parse(code)
function_node = module_node.body[2]
"""

long_setup_end = f"""
import ast
with open("Lib/inspect.py") as f:
    code = f.read()
module_node = ast.parse(code)
function_node = module_node.body[-2]
"""

test = """
ast.get_source_segment(code, function_node)
"""

print(f"short:      {timeit.timeit(test, setup=short_setup, number=10000)}")
print(f"long+start: {timeit.timeit(test, setup=long_setup_start, number=10)}")
print(f"long+end:   {timeit.timeit(test, setup=long_setup_end, number=10)}")

We tested on a very short source code and a long one(inspect.py has about 3k lines). For the long one, we tested an ast that's at the very beginning, which should benefit from the maxlines a lot, and an ast that's at the end of the file.

The result of current implementation with no optimization:

short:      0.12191030000394676
long+start: 0.15196170000126585
long+end:   0.15174370000022464

The result of improved method using re and maxlines:

short:      0.041647400001238566
long+start: 0.0010708999980124645
long+end:   0.03278889999637613

As we can tell from the result, even on very short source code, the new implementation has a ~3x speed up, which is due to the elimination of character-level loop. For the long source code, the speed up is even more obvious ~4x-5x.

maxlines provided huge performance improvement as expected - much less loops when we can stop early. This improvement should be linear to the ratio of source code length and our target end line number.

Overall, I believe this is a promising improvement to ast.get_source_segment().

Thanks again!