Spark UDF error - schema for type Any is not supported

Question

Im trying to create udf that will replace negative values ​​in a column with 0.

My dataframe is called df and contains one column called avg_x. This is my code to create udf

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else "avg_acc_x"}

I get this error

java.lang.UnsupportedOperationException: Schema for type Any is not supported

df.printSchema returns

|-- avg_acc_x: double (nullable = false) 

so I don’t understand why this error occurs?

Answer 1

This is due to elsereturn String: "avg_acc_x". Remove quotes:

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else avg_acc_x}

Answer 2

This error mainly occurs when the analyzer cannot correctly determine the udf type. That is, the types are mixed (Int and String).

Hence,

if(avg_acc_x < 0) 0 else avg_acc_x (integer type)

or

if(avg_acc_x < 0) "0" else "avg_acc_x" ( )

.