Ubuntu - wget - Test Download URL

When you are going to do scheduled download, you should check whether download will happen fine or not at scheduled time. To do so, copy the line exactly from the schedule, and then add -–spider option to check.

wget --spider DOWNLOAD-URL

If the URL given is correct, it will say

Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
<strong>Remote file exists</strong> and could contain further links,
but recursion is disabled -- not retrieving.

This ensures that the downloading will get success at the scheduled time.

But when you had give a wrong URL, you will get the following error.

Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 404 Not Found
<strong>Remote file does not exist -- broken link!!!</strong>

You can use the spider option under following scenarios:

• Check before scheduling a download.
• Monitoring whether a website is available or not at certain intervals.
• Check a list of pages from your bookmark, and find out which pages are still exists.