Geek Answers Handbook

Converting a value based on type in SparkSQL using type type matching

Is it possible to map a parametric type in Scala? Suppose I have a function that receives two parameters: a value and a type . I would like to use pattern matching for type conversion.

Something like that:

 datatype match { case IntegerType => return value.toInt case FloatType => return value.toFloat case StringType => return value case DecimalType(_,_) => return BigDecimal(value) // this is not working case _ => return strrepr }

Here, DecimalType takes two parameters to indicate the accuracy of the required accuracy. It could be, for example:

 org.apache.spark.sql.types.DecimalType = DecimalType(10,2)

I tried several options and nothing works:

  • For case DecimalType => return BigDecimal(value) I get:

     error: pattern type is incompatible with expected type; found : org.apache.spark.sql.types.DecimalType.type required: org.apache.spark.sql.types.DataType Note: if you intended to match against the class, try `case DecimalType(_,_)`

  • For case DecimalType(_,_) => return BigDecimal(value) I get:

     error: result type Boolean of unapply defined in method unapply in object DecimalType does not conform to Option[_] or Boolean

  • For case DecimalType[_,_] => return BigDecimal(value) I get:

     error: org.apache.spark.sql.types.DecimalType does not take type parameters
+6
source share
2 answers

The problem is using return in your code. You said you use this piece of code in some function. What is the return type of this function? Obviously, you assume that sometimes it is Integer, sometimes String, sometimes BigDecimal; but if you use return , it will look for the type of the returned object to determine the return type of the function. In general, you should avoid using return in Scala code. The last evaluated value in the function body is returned. The only use case for return is when you want to force a return somewhere else to the function body. Nevertheless, the best way is to save the returned object in a variable and simply evaluate this variable in the last line of your function body. And never use a refund!

Without return it works

 scala> val datatype = DecimalType(10, 2) datatype: org.apache.spark.sql.types.DecimalType = DecimalType(10,2) scala> val value = BigDecimal(10) value: scala.math.BigDecimal = 10 scala> datatype match {case DecimalType(_,_) => value} res150: scala.math.BigDecimal = 10

** Return Issues **

 scala> def test = {datatype match {case DecimalType(_,_) => return value}} <console>:138: error: method test has return statement; needs result type def test = {datatype match {case DecimalType(_,_) => return value}} scala> def test:BigDecimal = {datatype match {case DecimalType(_,_) => return value}} test: BigDecimal scala> def test:DataType = {datatype match {case DecimalType(_,_) => return value}} <console>:138: error: type mismatch; found : scala.math.BigDecimal required: org.apache.spark.sql.types.DataType def test:DataType = {datatype match {case DecimalType(_,_) => return value}} scala> def test3 = {datatype match {case DecimalType(_,_) => value}} test3: scala.math.BigDecimal
+4
source

It turns out that only the decimal type pattern matches the null arguments:

  case DecimalType() => ...

If you need accuracy and scale, you must determine the type of enclosure and manually remove them:

 datatype match { case dt: DecimalType => val precision = dt.precision val scale = dt.scale ...
+2
source

More articles:


All Articles