Geek Answers Handbook

How to stop hive / pig installation in Amazon Data Pipeline?

I do not need Hive or Pig, and Amazon Data Pipeline installs them by default in any EMR cluster that it spins. This makes testing longer than necessary. Any ideas on how to disable the installation?

+8
amazon-data-pipeline emr
source share
1 answer

This is not possible today.

The only workaround is to start a small EMR cluster that you use for testing (for example, with a single wizard - m1.small). Then use it with a "workgroup" rather than "runOn".

Depending on the type of activity you want to use, the workgroup field may or may not be supported. But you can always wrap everything in a script (python, shell or blah) and use it with ShellCommandActivity.

Refresh (Christopher B. correctly recalled):

From version 3.x, AMI, Hive and Pig merge into AMI. Thus, the steps do not pull out new packages from S3, but only daemons on the master node activate. Therefore, if you are not worried that they are consuming the resources of your instance (processor, memory, etc.), everything should be in order. They will not take noticeable time to launch.

+1
source share

More articles:


All Articles