Save pig result in local file

Question

I run a pig script, perform some aggressive operation, and the output size is very small.

Now i run

hadoop fs -getmerge ... 

separately.

Any way to let the pig script directly output the result to a local file?

Answer 1

If you are not worried about merging with just one file, then you can use the copyToLocal command in grunt (http://wiki.apache.org/pig/Grunt):

 grunt> copyToLocal <src> <dest> 

Answer 2

Another possible way is to include Pig in Python or JavaScript. You can do something like this (in Python):

 import os from org.apache.pig.scripting import Pig P = Pig.compile("PUT YOUR PIG CODE HERE") hdfs_input = "YOUR HDFS INPUT" hdfs_output = "YOUR HDFS OUTPUT" local_output = "YOUR LOCAL OUTPUT" result = P.bind({'in': input, 'out': hdfs_output}).runSingle() os.system("hadoop fs -getmerge " + hdfs_output + " " + local_output) 

and run Python code (for example)

 pig -useHCatalog python_code.py