df.rank() has different behavior for python2 and python3

[11rohans]

Code Sample, a copy-pastable example if possible

python3

In [1]: import pandas as pd

In [2]: pandas_df = pd.DataFrame({
   ...:     'col1': [1, 2, 3, 4],
   ...:     'col2': [4, 5, 6, 7],
   ...:     'col3': [8.0, 9.4, 10.1, 11.3],
   ...:     'col4': ['a', 'b', 'c', 'd']})
   ...:

In [3]: pandas_df.rank(axis=1)
Out[3]:
   col1  col2  col3
0   1.0   2.0   3.0
1   1.0   2.0   3.0
2   1.0   2.0   3.0
3   1.0   2.0   3.0

python2

In [1]: import pandas as pd

In [2]: pandas_df = pd.DataFrame({
   ...:     'col1': [1, 2, 3, 4],
   ...:     'col2': [4, 5, 6, 7],
   ...:     'col3': [8.0, 9.4, 10.1, 11.3],
   ...:     'col4': ['a', 'b', 'c', 'd']})
   ...:

In [3]: pandas_df.rank(axis=1)
Out[3]:
  col1  col2  col3  col4
0   1.0   2.0   3.0   4.0
1   1.0   2.0   3.0   4.0
2   1.0   2.0   3.0   4.0
3   1.0   2.0   3.0   4.0

Problem description

Running pandas on python2 produces different output from running on python3 for df.rank on mixed dtype dataframes. The expected output should be to receive the python3 version of the output, as strings and integers should not be automatically compared, similarly to how pandas handles quantile/describe.

Expected Output

In [1]: import pandas as pd

In [2]: pandas_df = pd.DataFrame({
   ...:     'col1': [1, 2, 3, 4],
   ...:     'col2': [4, 5, 6, 7],
   ...:     'col3': [8.0, 9.4, 10.1, 11.3],
   ...:     'col4': ['a', 'b', 'c', 'd']})
   ...:

In [3]: pandas_df.rank(axis=1)
Out[3]:
   col1  col2  col3
0   1.0   2.0   3.0
1   1.0   2.0   3.0
2   1.0   2.0   3.0
3   1.0   2.0   3.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-43-Microsoft
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.2.1
pip: 10.0.1
setuptools: 36.5.0.post20170921
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[WillAyd]

Thanks for the report it appears to be happening on master as well

[toobaz]

I'm not convinced this is a bug. The python2 output is consistent with the fact that strings and integers can be compared in python2.

@11rohans Can you provide an example of what you mean by "how pandas handles quantile/describe"?

[11rohans]

I meant that for when pandas handles quantile/describe, pandas drops all non-numeric columns for the sake of comparison. I would think that assumption of cutting non-numeric columns for operations across rows/columns would be consistent throughout pandas, especially since that's the behavior for python3 in this case and I can't think of any other case where comparing strings/integers would make sense. However, if this is intended behavior since python2 can compare/sort strings and integers then this isn't an issue.

[toobaz]

I meant that for when pandas handles quantile/describe, pandas drops all non-numeric columns for the sake of comparison

True, pd.DataFrame([['a', 1, 2]]).quantile(.2, axis=1) is the same as pd.DataFrame([[1, 2]]).quantile(.2, axis=1). However, df.sortvalues(., axis=1) sorts columns of different types just fine, so I'm not sure consistency is a strong argument for changing df.rank.

(sure, we could discuss changing both sortvalues and rank, but the deprecation cycle would presumably not end before we drop python 2 support, in January 2019...)

[topper-123]

This works in Python3 and is tested in pandas.tests.frame.test_rank::TestRank::test_rank_mixed_frame. the Python2 issue is irrelevant as Python2 support is being dropped now.

Closing.