Geek Answers Handbook

Combining sorted files with fifos

I have sorted, gzipped files in a directory. How to combine some of them into another sorted, gzip file? I'm using explicit fifa right now. Is there any way to do this in bash without? I am a little bash noob, so please excuse my style.

#!/bin/bash
# Invocation ./merge [files ... ]
# Turns an arbitrary set of sorted, gzipped files into a single sorted, gzipped file,
# printed to stdout. Redirect this script output!
for f in $@
do
    mkfifo $f.raw
    gzcat $f > $f.raw &
    # sort -C $f.raw
done
sort -mu *.raw | gzip -c # prints to stdout.
rm -f *.raw

I want to convert this to something like ...

sort -mu <(gzcat $1) <(gzcat $2) <(gzcat $3) ... | gzip -9c # prints to stdout.

... but I don’t know how to do it. Do I need a loop building parameters for a string? Is there any magic shortcut for this? Maybe map gzcat $@?

NOTE. Each of the files exceeds 10 GB (and 100 GB unzipped). I have a 2 volt drive, so this is not a problem. In addition, this program MUST run in O (n), or it becomes impracticable.

+5
3

eval " " Bash. , (, , $@ "$@", , ), - :

cmd="sort -mu"
for file in "$@"
do cmd="$cmd <(gzip -cd $file)"
done
eval $cmd | gzip -c9 > outputfile.gz

bash -c "$cmd" eval $cmd . , . , :

cmd="sort -mu"
for file in "$@"
do cmd="$cmd <(gzip -cd '$file')"
done
eval $cmd | gzip -c9 > outputfile.gz

, .

+3

, , :

gunzip -c file1 file2 .... | sort | gzip -9 > mergedFile.gz

1 , file*.type gunzip, .

-c " stdout", , sort, stdout, pipe gzip, stdout . -9 - , ( gzip), . -1 -9 / .

, .

+1

, .

( ), eval ' , .

(
esc="'\''"
file="/Applications/iWork '09/Pages.app"
file="${file//\'/${esc}}"
#echo "'${file}'"; ls -bdl "'${file}'"
evalstr="echo '${file}'; ls -bdl '${file}'"
#set -xv
eval "${evalstr}"
)
+1
source

More articles:


All Articles