Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Apache - Logs - Extract all user agents from Apache logs ======

<code bash>
cat test.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
</code>

  * where “test.log” is the access logfile to analyze.

Returns

<code>
51916 MetaURI API/2.0 +metauri.com
59899 Twitterbot/1.0
87819 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
111261 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
187812 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
189834 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
390477 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
</code>


<WRAP info>
**NOTE:**  The first number is the amount of times this spider/crawler/user agent/ has accessed the site.

  * Beware, these are not all crawlers, as the data is intermixed with actual human user traffic and other useful traffic.


In the example above, notice that the “Facebookexternalhit” user agent accessed the site 390,477 times per month.

  * That is roughly 541x per hour. Excessive!!!.
  * On the kill list, you go!
</WRAP>