Saving UNION results in PIG in one file

Question

I have a PIG Script that gives four results. I want to save them all in one file. I try to use UNION , however, when I use UNION , I get four files part-m-00000, part-m-00001, part-m-00002, part-m-00003. Can't get one file?

Here is a PIG script

 A = UNION Message_1,Message_2,Message_3,Message_4 into 'AA'; 

In the AA folder, I get 4 files, as mentioned above. Can't I get one file with all its elements?

Answer 1

The pig does the right thing here and combines the data sets. All being one file does not mean one dataset in Hadoop ... one dataset in Hadoop is usually a folder. Since there is no need to start the reduction, this will not happen.

You need to trick the Pig in order to run the card and reduce it. I usually do this:

 set default_parallel 1 ... A = UNION Message_1,Message_2,Message_3,Message_4; B = GROUP A BY 1; -- group ALL of the records together C = FOREACH B GENERATE FLATTEN(A); ... 

GROUP BY groups all the records together, and then FLATTEN blows up this list.

---

It should be noted here that this is not much different from that:

 $ hadoop fs -cat msg1.txt msg2.txt msg3.txt msg4.txt | hadoop fs -put - union.txt 

(this is the concatenation of all the text, and then writing it back to HDFS as a new file)

This is not parallel at all, but not one of them redirects all your data through one reducer.

Answer 2

Have you tried setting the default_parallel property?

 grunt> set default_parallel 1 grunt> A = UNION Message_1,Message_2,Message_3,Message_4;