Switch AWS Lambda to Redshift - Timeout in 60 Seconds

Question

I created an AWS Lambda function that:

• register on Redshift via JDBC URL
• runs a request

Locally, using Node, I can successfully connect to the Redshift instance via JDBC and execute the request.

var conString = "postgresql://USER_NAME: PASSWORD@JDBC _URL"; var client = new pg.Client(conString); client.connect(function(err) { if(err) { console.log('could not connect to redshift', err); }  // omitted due to above error 

However, when I execute the function on AWS Lambda (where it is wrapped in an async # waterfall block), AWS Cloudwatch logs tell me that AWS Lambda was turned off after 60 seconds.

Any ideas on why my function cannot connect?

Answer 1

I find this either you open the Redshift public security group for all sources or not. Since the Lambda function does not work at a fixed address or even a fixed range of IP addresses, which is completely transparent to users (without an AKA server).

I just saw that Amazon announced a new Lambda feature to support VPC yesterday. I think if we can start the Redshift cluster in VPC, this may solve the problem.

Answer 2

If you are using serverless-framework v1.5.0, you should add:

iamRoleStatements: - Effect: Allow Action: - ec2:CreateNetworkInterface Resource: '*' - Effect: Allow Action: - ec2:DeleteNetworkInterface - ec2:DescribeNetworkInterfaces Resource: 'arn:aws:ec2:${self:provider.region}:*:network-interface/*'

You must also add all security rules to Inbounds rules, as shown below:

Additional information: https://serverless.com/framework/docs/providers/aws/guide/functions/#vpc-configuration

Answer 3

Going back, I would recommend using Kinesis [1] firehose to connect lambda and redshift. This is the best approach proposed in the documents [2].

Kinesis can use s3 as an intermediate storage to automatically switch data to redshift using a copy command.

"The COPY command is the most efficient way to load a table. Also add data to your tables using INSERT commands, although this is much less efficient than using COPY."

Footnote: [1] http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html

[2] http://docs.aws.amazon.com/redshift/latest/dg/t_Loading_data.html .