Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
df = pd.DataFrame({'non_numeric': 10000*[pd.Timedelta(10)], 'numeric': 10000*[1]})
# Summing over axis 1 with numeric_only=True works as expected
assert_series_equal(df.sum(axis=1, numeric_only=True), df.numeric, check_names=False)
# Summing with the default numeric_only=None only works if df has <= 8192 = 2**13 rows
sample = df.sample(8192)
assert_series_equal(sample.sum(axis=1), sample.numeric, check_names=False)
# Summing over the whole dataframe we get a SystemError
df.sum(axis=1)
# The following also produces a SystemError
sample['numeric_2'] = 1
sample.sum(axis=1)
# The following does not produce a SystemError
sample.sample(1).sum(axis=1)Problem description
In previous pandas versions the last line does not throw a TypeError. The default numeric_only=None will only sum over the numeric columns of the dataframe.
In pandas 1.2.4 this is the case if the length of the dataframe is <= 8192 = 2**13. As soon as the length exceeds 8192 we get a SystemErrorbecause it tries to sum a Timedelta column and an int column. We also get a SystemError when df has more than one numeric column. Then the sum only works if the length of the dataframe is 1
The behaviour of sum is the same for other aggregation methods like mean.
Expected Output
I expect df.sum(axis=1) to return a series whose entries are the sum of the numeric columns of the dataframe along the axis 1 as is the case for dataframes of length <= 8192.
Output of pd.show_versions()
INSTALLED VERSIONS
commit : 2cb9652
python : 3.9.4.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-693.5.2.el7.x86_64
Version : #1 SMP Fri Oct 20 20:32:50 UTC 2017
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en.UTF-8
LOCALE : None.None
pandas : 1.2.4
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 52.0.0.post20210125
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None