How can I package or install the entire program to work in AWS Lambda function

Question

If this is a case of using Lambda in a completely wrong way, please let me know.

I want to set Scrapy to a Lambda function and call the function to start the crawl. My first problem is how to install it so that all paths are correct. I installed the program using a directory that will be archived as its root, so zip contains all the source files and the executable. I base my efforts on this article. On the line mentioned at the beginning of my function, where does the "process" variable come from? I tried,

var process = require('child_process'); var exec = process.exec; process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT'] 

but I get an error

 "errorMessage": "Cannot read property 'PATH' of undefined", "errorType": "TypeError", 

Do I need to include all the library files or only the executable from / usr / lib? How to include this line of code in the article I need?

Edit: I tried moving the code to child_process.exec and got an error

 "errorMessage": "Command failed: /bin/sh: process.env[PATH]: command not found\n/bin/sh: scrapy: command not found\n" 

Here is my current whole function

 console.log("STARTING"); var process = require('child_process'); var exec = process.exec; exports.handler = function(event, context) { //Run a fixed Python command. exec("process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT']; scrapy crawl backpage2", function(error, stdout) { console.log('Scrapy returned: ' + stdout + '.'); context.done(error, stdout); }); }; 

Answer 1

You can execute arbitrary processes from your Node code in Lambda. We do this to launch a game server, which, for example, processes players. I'm not sure exactly what you are trying to accomplish with Scrapy, but remember that your entire Lambda call can only live for 60 seconds right now on AWS! But if this is good for you, here is a complete working example of how we execute our own custom Linux process from Lambda. (In our case, this is a compiled binary - it really doesn't matter if you have something that can work on the Linux image that they use).

 var child_process = require('child_process'); var path = require('path'); exports.handler = function (event, context) { // If timeout is provided in context, get it. Otherwise, assume 60 seconds var timeout = (context.getRemainingTimeInMillis && (context.getRemainingTimeInMillis() - 1000)) || 60000; // The task root is the directory with the code package. var taskRoot = process.env['LAMBDA_TASK_ROOT'] || __dirname; // The command to execute. var command; // Set up environment variables process.env.HOME = '/tmp'; // <-- for naive processes that assume $HOME always works! You might not need this. // On linux the executable is in task root / __dirname, whichever was defined process.env.PATH += ':' + taskRoot; command = 'bash -c "cp -R /var/task/YOUR_THING /tmp/; cd /tmp; ./YOUR_THING ARG1 ARG2 ETC"' child_process.exec(command, { timeout: timeout, env: process.env }, function (error, stdout, stderr) { console.log(stdout); console.log(stderr); context.done(null, {exit: true, stdout: stdout, stderr: stderr}); }); }; 

Answer 2

The problem with your example changes the node global variable process to

var process = require('child_process');

That way you cannot change the PATH environment variable and therefore you get the Cannot read property 'PATH' of undefined .

Just use a different name for the loaded child_process library, for example.

 //this gets updated to child_process var child_process = require('child_process'); var exec = child_process.exec; //global process variable is still accessible process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT'] //start new process with your binary exec('path/to/your/binary',...... 

Answer 3

Here is a Lambda function that runs a python script, setting the current working directory to the same directory as the Lambda function. You can use this with some changes in the relative location of your python script.

 var child_process = require("child_process"); exports.handler = function(event, context) { var execOptions = { cwd: __dirname }; child_process.exec("python hello.py", execOptions, function (error, stdout, stderr) { if (error) { context.fail(error); } else { console.log("stdout:\n", stdout); console.log("stderr:\n", stderr); context.succeed(); } }); };