Write data lines of curvature lines as elements in dynamoDB table

Question

Is there a way to write each row of my spark dataframe as a new element in the dynamoDB table? (in pySpark )

I used this code with the boto3 library, but I wonder if there is another way, avoiding the pandas and for loop steps:

 sparkDF_dict = sparkDF.toPandas().to_dict('records') for item in sparkDF_dict : table.put_item(Item = item) 

Answer 1

DynamoDB offers the BatchWriteItem API . This is available in boto3 , so you can call it after creating sparkDF_dict fragments of 25 elements in length. Please note: The BatchWriteItem API only supports writing 25 elements at a time , and not all entries can succeed at first (since they can be throttled to and get back to you in the UnprocessedItems part of the answer). Your application will need to look at UnprocessedItems in the response and try again.