@ttpro1995 thank you for the detailed repro. Will take a look at this when I get a chance. Can you please post how you read the "data" variable just so I'm not missing any of the parts that reproduces the issue? My first guess is there might be some missing values in the dataset.

code that work in python sklearn (so, data is not broken)

from sklearn.metrics import  accuracy_score
import pandas as pd
data = pd.read_csv("/data/public/HIGGS/higgs.test.predictioncsv/part-00000-2cf6c2c8-173c-49c1-ae78-9dfe697f3a0f-c000.csv")
data.head()
accuracy_score(data["label"], data["prediction"])

Another case on Spark Scala with error using different data

Notebook on zepl.com
https://www.zepl.com/viewer/notebooks/bm90ZTovL2hhaGF0dHByby9iMzBiYjJiNzZkMGI0YWVlYTQwMjMwY2JiMzNkYWE0MC9ub3RlLmpzb24

**Export to zeppelin (0.8.x): **
https://gist.github.com/ttpro1995/8e6227039b2593041267180a1d81efa7

Data using in notebook:

binary.train

binary.test

@ttpro1995 oh sorry, I just saw the problem now, you are setting the scores column but not the scored labels column, which is required if you are trying to compute all metrics:
https://github.com/Azure/mmlspark/blob/master/src/main/scala/com/microsoft/ml/spark/train/ComputeModelStatistics.scala#L99
I think the error message needs to be improved

I don't know what each function, each set col do.
i.e: what setLabelCol, setScoredLabelsCol, setScoresCol ... these name are similar and confusing, especially setScoredLabelsCol and setScoresCol.

@ttpro1995 oh sorry, I just saw the problem now, you are setting the scores column but not the scored labels column, which is required if you are trying to compute all metrics:
https://github.com/Azure/mmlspark/blob/master/src/main/scala/com/microsoft/ml/spark/train/ComputeModelStatistics.scala#L99

What should be setScoredLabelsCol ?

I suggest there should be more detail in documentation . Including: meaning of each parameter, default value (I have to new a object in code, then use get function to know the default value),

@ttpro1995 I think the column name convention is similar, but maybe not exactly, in most ML platforms, at least in scikit-learn, apache spark and ML.NET. The labelCol is the true labels or original label from the dataset, the ScoredLabelsCol is the label assigned by the classifier, and the ScoresCol is the raw prediction, for example the distance from the separating hyperplane for the support vector machine classifier. There's also usually a probability col, which is not used in ComputeModelStatistics, although you could pass it as the input to the ScoresCol too (it's used for the AUC metric).

@ttpro1995 "I suggest there should be more detail in documentation . Including: meaning of each parameter, default value (I have to new a object in code, then use get function to know the default value),"
totally agree, the documentation could definitely be improved here. thanks for pointing that out.