Using psycopg2 with Lambda to update Redshift (Python)

Question

I am trying to update Redshift from a Lambda function using python. For this, I am trying to combine 2 pieces of code. Both fragments are functional when I run them separately.

• PyDev Redshift Update for Eclipse

  import psycopg2
  
  conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"
  conn = psycopg2.connect(conn_string)
  
  cursor = conn.cursor()
  
  cursor.execute("UPDATE table SET attribute='new'")
  conn.commit()
  cursor.close()
  

• Retrieving content loaded into S3 bucket (pre-built template available on lambda)

  from __future__ import print_function
  
  import json
  import urllib
  import boto3
  
  print('Loading function')
  
  s3 = boto3.client('s3')
  
  
  def lambda_handler(event, context):
      #print("Received event: " + json.dumps(event, indent=2))
  
      # Get the object from the event and show its content type
      bucket = event['Records'][0]['s3']['bucket']['name']
      key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
  
      try:
          response = s3.get_object(Bucket=bucket, Key=key)
          print("CONTENT TYPE: " + response['ContentType'])
          return response['ContentType']
  
      except Exception as e:
          print(e)
          print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
          raise e
  

Since both of these segments worked, I tried to combine them to update Redshift when uploading a file to s3:

from __future__ import print_function

import json
import urllib
import boto3
import psycopg2

print('Loading function')

s3 = boto3.client('s3')


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')

    conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"

    conn = psycopg2.connect(conn_string)

    cursor = conn.cursor()

    cursor.execute("UPDATE table SET attribute='new'")
    conn.commit()
    cursor.close()

    try:
        response = s3.get_object(Bucket=bucket, Key=key)
        print("CONTENT TYPE: " + response['Body'].read())
        return response['Body'].read()
    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
        raise e

Since I use an external library, I need to create a deployment package. I created a new folder (lambda_function1) and moved my .py file (lambda_function1.py) to this folder. I ran the following command to install psycopg2 in this folder:

pip install psycopg2 -t \lambda_function1

I get the following feedback:

Collecting psycopg2
  Using cached psycopg2-2.6.1-cp34-none-win_amd64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.6.1 

. zip . , , :

Unable to import module 'lambda_function1': No module named _psycopg 

, , "_psycopg", "_psycopg.pyd".

? , Lambda Python 2.7, 3.4? , Windows? - Redshift ?

Answer 1

psycopg2 libpq.so. repo https://github.com/jkehler/awslambda-psycopg2. psycopg2 , .

:

?

psycopg2 Linux.

, Lambda Python 2.7, 3.4?

, , 2.7. .

, Windows?

, , Linux,

- Redshift ?

.

Answer 2

. GitHub, , :

- , AWS Lambda PostgreSQL AMI, psycopg2 libpq.so PostgreSQL .

, , psycopg2 PostgreSQL. , . github psycopg2 :

, , psycopg2-binary PyPI:

$ pip install psycopg2-binary

, , .

pip- psycopg2-binary require.txt, postgresql -. , . , psycopg2 , . -, , , .

Answer 3

, , " no module named psycopg2 , psycopg2 Python.

Lambdas , psycopg2 ( manylinux_x86_64). segfault - libssl, . ( +1 jshammon )

The “right solution” probably lies in the fact that jkehlers are recompiled specifically for Lambda, which is missing only lib_pq.so, but currently it does not support ssl + py3.7, and we are too Windows to recompile it ourselves.