Reading content from a text file created in Windows on Linux bash

Question

I am trying to download files from a database using wget and url. For instance.

wget " http://www.rcsb.org/pdb/files/1BXS.pdb "

Thus, the format of the URL is: http://www.rcsb.org/pdb/files/ ($ idnumber) .pdb "

But I have many files to download; so I wrote a bash script that reads id_numbers from a text file, forms a url string and loads wget.

!/bin/bash while read line do url="http://www.rcsb.org/pdb/files/$line.pdb" echo -e $url wget $url done < id_numbers.txt 

However, the url string is formed as

 .pdb://www.rcsb.org/pdb/files/4H80 

So .pdb reloading http . I can’t understand why. Does anyone have an idea? How can I format it so that the URL is

 "http://www.rcsb.org/pdb/files/($idnumber).pdb" 

? Thank you very much.

Note. This question has been marked as a duplicate of "How to merge strings in bash?" but I really asked for something else. I read this question before asking this question, and it turns out that my problem was preparing the txt file on Windows, but not at the beginning of the line. I edited the title of the question. Hopefully this will become clearer now.

Answer 1

It looks like your id_numbers.txt file has a DOS / Windows-style line ending (carriage return followed by line feeds) instead of simple unix line endings (line feed only). As a result, read thinks that the line ends with a carriage return, $line actually has a carriage return at the end and which is embedded in the URL, causing various confusions.

There are several ways to solve this problem. You may have bash trim the carriage return from the variable when using it:

 url="http://www.rcsb.org/pdb/files/${line%$'\r'}.pdb" 

Or you could read trim it, indicating that the carriage return is considered a space ( read will trim the leading and trailing spaces from what it reads):

 while IFS=$'\r' read line 

Or you can use a command like dos2unix (or whatever the equivalent is on your OS) to convert the id_numbers.txt file.

Answer 2

The -e echo parameter is used to display the desired content without inserting a new line, here you do not need it.

I also suspect your file containing identifiers that were incorrect, on which OS did you create it?

In any case, you can simplify your script as follows:

 !/bin/bash while read line do wget "http://www.rcsb.org/pdb/files/$line.pdb" done < id_numbers.txt 

I was able to successfully test it using the id_numbers.txt file created as follows:

 for i in $(0 9) ; do echo "$i" >> id_numbers.txt ; done 

Answer 3

Try the following:

 url="http://www.rcsb.org/pdb/files/"$line $url=$url".pdb" 

For more information, check out How to combine string variables in Bash?