Bash script to view the folder

Question

I have the following situation:

There is a Windows folder that was installed on the Linux machine. There may be several folders (setting before hand) in this Windows mount. I have to do something (preferably a script to start with) to look at these folders.

These are the following steps: Keep track of any incoming files. Make sure they are fully transferred. Move it to another folder. I have no control over the file transfer program on a Windows machine. I believe this is secure FTP. Therefore, I cannot ask this process to send me a trailer file to ensure the completion of the file transfer.

I wrote a bash script. I would like to know about any potential obstacles with this approach. The reason is, there is the possibility of multi-sheeted copies of this script for multiple directories like this.

At the moment, there may be up to 100 directories that may need to be tracked.

The following is a list of script. I apologize for inserting a very long one here. Do not waste time browsing and commenting / criticizing. :-)

3 parameters are required: the folder that should be viewed, the folder into which the file should be moved, and the time interval, which was explained below.

Sorry, there seems to be a problem with alignment. Markdown doesn't seem to like it. I tried to organize it correctly, but could not do it.

Linux servername 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux

 #!/bin/bash log_this() { message="$1" now=`date "+%D-%T"` echo $$": "$now ": " $message } usage() { cat << EOF Usage: $0 <Directory to be watched> <Directory to transfer> <time interval> Time interval is the amount of time after which the modification time of a file will be monitored. EOF `exit 1` } if [ $# -lt 2 ] then usage fi WATCH_DIR=$1 APP_DIR=$2 if [ ! -d "$WATCH_DIR" ] then log_this "FATAL: WATCH_DIR, $WATCH_DIR does not exist. Exiting" exit 1 fi if [ ! -d "$APP_DIR" ] then log_this "APP_DIR: $APP_DIR does not exist. Exiting" exit 1 fi # This needs to be set after considering the rate of file transfer. # Represents the seconds elapsed after the last modification to the file. # If not supplied as parameter, defaults to 3. seconds_between_mods=$3 if ! [[ "$seconds_between_mods" =~ ^[0-9]+$ ]]; then if [ ${#seconds_between_mods} -eq 0 ]; then log_this "No value supplied for elapse time. Defaulting to 3." seconds_between_mods=3 else log_this "Invalid value provided for elapse time" exit 1 fi fi log_this "Start Monitor." while true do ls -1 $WATCH_DIR | while read file_name do log_this "Start Monitoring for $file_name" # Refer only the modification with reference to the mount folder. # If there is a diff in time between servers, we are in trouble. token_file=$WATCH_DIR/foo.$$ current_time=`touch $token_file && stat -c "%Y" $token_file` rm -f $token_file 2>/dev/null log_this "Current Time: $current_time" last_mod_time=`stat -c "%Y" $WATCH_DIR/$file_name` elapsed_time=`expr $current_time - $last_mod_time` log_this "Elapsed time ==> $elapsed_time" if [ $elapsed_time -ge $seconds_between_mods ] then log_this "Moving $file_name to $APP_DIR" # In case if there is no space left on the target mount, hide the file # in the mount itself and remove the incomplete file from APP_DIR. mv $WATCH_DIR/$file_name $APP_DIR if [ $? -ne 0 ] then log_this "FATAL: mv failed!! Hiding $file_name" rm $APP_DIR/$file_name mv $WATCH_DIR/$file_name $WATCH_DIR/.$file_name log_this "Removed $APP_DIR/$file_name. Look for $WATCH_DIR/.$file_name and submit later." fi log_this "End Monitoring for $file_name" else log_this "$file_name: Transfer seems to be in progress" fi done log_this "Nothing more to monitor." echo sleep 5 done 

Answer 1

This will not work for a while. During production, you will have network problems and other errors that may leave a partial file in the download directory. I also don't like the idea of ​​a trailer file. The usual approach is to download the file under a temporary name and then rename it after the download is complete.

Thus, you just need to specify the directory, filter out temporary names and, if something remains, use it.

If you cannot make this change, contact your boss for written permission to implement something that could lead to arbitrary data corruption. This is done for two purposes: 1) to make them understand that this is a real problem, and not what you make up, and 2) to protect yourself when it breaks down ... because she will guess who will get all the blame?

Answer 2

I believe a more reasonable approach would be to use notification of the elements of the kernel-level file system. For example inotify . Get also the tools here .

Answer 3

First make sure inotify-tools installed.

Then use them as follows:

 logOfChanges="/tmp/changes.log.csv" # Set your file name here. # Lock and load inotifywait -mrcq $DIR > "$logOfChanges" & # monitor, recursively, output CSV, be quiet. IN_PID=$$ # Do your stuff here ... # Kill and analyze kill $IN_PID cat "$logOfChanges" | while read entry; do # Split your CSV, but beware that file names may contain spaces too. # Just look up how to parse CSV with bash. :) path=... event=... ... # Other stuff like time stamps # Depending on the event… case "$event" in SOME_EVENT) myHandlingCode path ;; ... *) myDefaultHandlingCode path ;; done 

Alternatively, using --format instead of -c on inotifywait would be an idea.

Just man inotifywait and man inotifywatch for more information.

Answer 4

To be honest, a python application configured to run at startup will do this quickly and efficiently. Python has great OS support and is pretty comprehensive.

Running the script will most likely work, but it will be troublesome to take care and manage. I suppose you run them as frequent cron jobs?

Answer 5

incron is an inotify cron system. It consists of a demon and a table manipulator. You can use it just like a regular cron. The difference is that inotify cron handles file system events, not time periods.

Answer 6

To take you down, here is a small application that I wrote that takes a path and looks at the binary output of jpeg files. I never finished it, but it will get you started and see the python structure, as well as some use of os ..

I would not spend much time caring for my code.

 import time, os, sys #analyze() takes in a path and moves into the output_files folder, to then analyze files def analyze(path): list_outputfiles = os.listdir(path + "/output_files") print list_outputfiles for i in range(len(list_outputfiles)): #print list_outputfiles[i] f = open(list_outputfiles[i], 'r') f.readlines() #txtmaker reads the media file and writes its binary contents to a text file. def txtmaker(c_file): print c_file os.system("cat" + " " + c_file + ">" + " " + c_file +".txt") os.system("mv *.txt output_files") #parser() takes in the inputed path, reads and lists all files, creates a directory, then calls txtmaker. def parser(path): os.chdir(path) os.mkdir(path + "/output_files", 0777) list_files = os.listdir(path) for i in range(len(list_files)): if os.path.isdir(list_files[i]) == True: print (list_files[i], "is a directory") else: txtmaker(list_files[i]) analyze(path) def main(): path = raw_input("Enter the full path to the media: ") parser(path) if __name__ == '__main__': main()