Spark SQL converts a string to a timestamp

Question

I am new to Spark SQL and trying to convert a string to a timestamp in a frame with spark data. I have a row that looks like '2017-08-01T02:26:59.000Z'in the time_string column

My code to convert this string to a timestamp

CAST (time_string AS Timestamp)

But it gives me a timestamp 2017-07-31 19:26:59

Why is he changing the time? Is there a way to do this without changing the time?

Thanks for any help!

Answer 1

You can use unix_timestamp to convert utc formatting date to timestamp

val df2 = Seq(("a3fac", "2017-08-01T02:26:59.000Z")).toDF("id", "eventTime")

df2.withColumn("eventTime1", unix_timestamp($"eventTime", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").cast(TimestampType))

Output:

+-------------+---------------------+
|userid       |eventTime            |
+-------------+---------------------+
|a3fac        |2017-08-01 02:26:59.0|
+-------------+---------------------+

Hope this helps!

Answer 2

Java

Spark SQL, .

: 20181224091530 → 2018-12-24 09:15:30

( Spark SQL):

SELECT
 ...
 to_timestamp(cast(DECIMAL_DATE as string),'yyyyMMddHHmmss') as 'TIME STAMP DATE',
 ...
FROM some_table

SQL, org.apache.spark.sql.SparkSession. , sql, Spark :

...
// You have to create an instance of SparkSession
sparkSession.sql(sqlStatement); 
...

:

• ,
• . , ...