Linux script for automatic notification of failed download

Scenario:

We need a CentOS machine to periodically automatically download a file from a remote server on the internet. If this download fails for any reason, the network admin should be emailed to investigate further.

Resolution:

A simple cron job can be used which executes a download script. The following script was used:

#!/bin/bash

wget https://list.bluecoat.com/bcwf/activity/download/bcwf.db -o download_result.txt
results=$(cat download_result.txt | grep 200 | wc -l)

if [ "$results" = "0" ]; then

        …insert emailing code here…

fi

In the above code snippet you can see a simple wget request for a site, and uses the “-o” switch to output the logs to the “download_result.txt” file.

We then output that file, and search (using grep) for any lines containing “200”. We then count the number of lines containing the “200” pattern. This value is stored in the “results” variable. If the download was successful, wget receives a HTTP 200 OK response to it’s request. Therefore, if the download was successful there should be at least 1 line containing the “200” pattern. If there are no lines containing 200, then obviously the download failed. This is what the “if” statement controls. If there are no lines containing “200”, then email the administrator.

This is not a foolproof method, since you may encounter some URLs containing "200”, but for most purposes it should work.

The last step is inserting code to email the admin. There are two ways around this. One is an easy way using a 3rd party program, the other is sticking to more “classical” programs but is a bit more involved.

The easy way

It really is easy… just head over to http://caspian.dotconf.net/menu/Software/SendEmail/, download and install the program. A single line can be used to send the email, though it is important to manually specify the mail server using the –s switch. Remember that this mail server should be the mail server of the recipient domain. You can run a DNS MX record lookup to figure out the IP address of this server.

The “classic” way

The classic way isn’t much more involved, but the main catch is that most programs that send emails from the Linux terminal (such as “mutt” in this example), do not directly contact the SMTP server like sendemail does. Instead, they handover the email to a “Mail Transfer Agent” (MTA) that handles the actual SMTP transmission. So, a quick and dirty configuration of an MTA is necessary.

Personally, I prefer the “postfix” MTA over CentOS’ default “sendmail” MTA. So first order of the day is to remove sendmail and install postfix (mutt needs to be reinstalled since it gets deleted when removing sendmail):

yum remove sendmail
yum install postfix
yum install mutt

Second, we need to configure postfix to accept emails coming from the local host, i.e. from the CentOS machine itself. By default, “mutt” (our email client) will send emails with the “FROM” address set to : [email protected]. So first, make a note of the hostname:

hostname
lincent.davidv.local

Now, we configure postfix to accept mail from this domain, and from the localhost IP 127.0.0.1:

postconf -e "mydestination = lincent.davidv.local"
postconf -e "mynetworks = 127.0.0.0/8"

 

Now, if you were to try to send an email to a public mail server from an address like [email protected]”, it will fail spam or reverse DNS checks and it will not deliver the email. So we next configure postfix to “map” the address [email protected] to a valid, public email address like [email protected]” To do this, we first create a file with the mapping (/etc/postfix/generic):

nano /etc/postfix/generic

#add the following line:

[email protected]   [email protected]

We next instruct postfix to use this file, so we edit the postfix configuration file “main.cf” to point to the file containing the mapping:

nano /etc/postfix/main.cf

# add the line:
smtp_generic_maps = hash:/etc/postfix/generic

Lastly, we use the “postmap” program to build a has from our mapping file:

postmap /etc/postfix/generic

Make sure you have valid DNS servers configured in your /etc/resolv.conf, restart postifx and we should be good to go… As a test, run the mutt client via terminal. First, build the body of the email… a simple:

echo "download failed" > data.txt

should suffice. Then, use mutt to actually send the email:

mutt -s "Downloader alert" [email protected] < data.txt

You should receive an email with subject line" “Downloader alert” containing “download failed”. So the final script would be:

#!/bin/bash

wget https://list.bluecoat.com/bcwf/activity/download/bcwf.db -o download_result.txt
results=$(cat download_result.txt | grep 200 | wc -l)

if [ "$results" = "0" ]; then
        mutt -s "Downloader alert" [email protected] < data.txt
fi