Differences

This shows you the differences between two versions of the page.

--- pfsense:pfblockerng:block_ads [2020/03/24 11:19] – [Testing By Browsing] peter
+++ pfsense:pfblockerng:block_ads [2021/02/07 18:34] (current) – removed peter
@@ Line 1: / Line 1: @@
-====== PFSense - pfBlockingNG - Block Ads ======
-Use the **DNSBL** portion of pfBlockerNG to remove advertising.
-It essentially creates functionality similar to the pi-Hole project just using your pfSense + pfBlockerNG.
-<WRAP important>
-**WARNING:** DO NOT install the latest version of pfBlockerNG unless you are on the most up-to-date version of pfSense.
-This is especially important if you are on a pfSense before 2.4.4.
-Version 2.4.4 introduced PHP 7.2 and it broke a lot of packages, not just pfBlockerNG.
-</WRAP>
-===== Upgrading from a previous version =====
-If you installing pfBlockerNG for the first time, skip this step and go to installation.
-If you have quite a few custom settings such as rules, IPv4 lists, and DNSBL lists and you want to keep all of your settings, go to **Firewall -> pfBlockerNG (General)** and make sure **Keep Settings** is checked. If it's not, put a check there and click **'Save'** at the bottom.
-{{:pfsense:pfsense_pfblockerng_general.png?800|}}
-<WRAP info>
-**NOTE:**  Unless you have a very complex setup, it might be better to not use the 'Keep settings' options and setup pfBlockerNG from scratch.
-</WRAP>
-Remove the package:  Go to **System -> Package Manager** and delete the package.
-----
-===== Install pfBlockerNG =====
-Go to **System -> Package Manager -> Available Packages** and type **pfblocker** into the search criteria and then click **search**.
-<WRAP tip>
-To have the latest version, suggestion is to get the development version; which still works very well even in production environments.
-In this case click **install** on the version with **-devel** at the end of it or the package or you will be installing might be an older one!
-**NOTE:**  If you do not see **pfBlockerNG-devel** in the list of available packages, you can also try running **pkg update -f** from the command line.
-</WRAP>
-Click **Confirm** and let the package install.  This will take a bit of time as it has to download several files and databases.  Wait a few minutes and you should see "success" once the installation is complete.
-The package should be installed.
-----
-===== Configure pfBlockingNG =====
-Go to the configuration page **Firewall -> pfBlockerNG**.
-Assuming you didn’t upgrade, you will receive the config wizard; which is literally 4 steps and will get you started.
-<WRAP important>
-**WARNING:**  Pay special attention to the interface step/page if you have a non-standard setup or if you want to enable pfBlockerNG on multiple interfaces.
-</WRAP>
-Finish up the wizard and you will be automatically directed to the update page (**Firewall -> pfBlockingNG -> Update**).
-The update will likely take a little bit to complete as it is downloading the various IP and DNSBL feeds associated with the wizard setup.
-Once the feeds are downloaded, the text in the gray box will stop scrolling and you will see "UPDATE PROCESS ENDED" at the very bottom along with your current date and time.
-Now go back to the main pfBlockerNG page by going to **General** or by clicking **Firewall -> pfBlockerNG**.
-The **Enable** and **Keep Settings** checkboxes should already be checked.
-----
-===== Further configuring DNSBL =====
-Go to the **DNSBL** tab and it will take you to the main DNSBL landing page.
-You should also notice there is already a checkmark in **Enable** next to DNSBL.
-If you only have one internal interface such as LAN, then you shouldn't need to do anything else.
-If you have multiple internal interfaces and you would like to protect them with DNSBL, then you will need to pay attention to the **Permit Firewall Rules** section below.  You may have already selected the "extra" interfaces when you went through the wizard above.  Either way, keep this in mind should you ever add interfaces or VLANs in the future!
-If you do need to add interfaces, place a checkmark in the **Enable** box (Permit Firewall Rules).  Then, select the various interfaces (to the right) by holding down the **Ctrl** key and left-clicking.  Don’t forget to hit **Save DNSBL settings** and move to the DNSBL feeds section.
-----
-===== Configure TLD (Optional) =====
-If your pfSense has plenty of memory, another really amazing feature to consider is TLD (below the DNSBL option).
-This option is required for the TLD blacklists.
-What does the TLD feature provide?  Normally, DNSBL (and other DNS blackhole software) block the domains specified in the feeds and that's that.  What TLD does differently is it will block the domain specified in addition to all of a domain’s subdomains.  As a result, a bad guy can’t circumvent the blacklist by creating a random subdomain name.
-pfBlockingNG is one of the few DNS blackholing software that does this.
-You can get an idea on memory requirements by clicking on the blue **info** icon next to TLD.  If you have less than 2GB of memory on your pfSense, I would skip it.  If you’re unsure on your memory, this might be a feature to come back to after you get your feeds and everything else configured.
-TLD can definitely add several layers of protection.
-{{:pfsense:pfsense_pfblockerng_dnsbl_tld.png?800|}}
-----
-===== Configuring DNSBL feeds =====
-Before we go adding additional feeds, we should at least understand what the wizard provided us.
-Go to **DNSBL -> DNSBL Groups** to see the current (post-wizard) configuration.
-if you want to add more, go to **Feeds** (not DNSBL Groups) at the top menu.  Here you will see all of the pre-configured feeds for the IPv4, IPv6, and DNSBL categories.  And yes, there are a bunch of them!  You’ll also see custom, user defined feeds at the very bottom if you performed an upgrade and it was unable to match a feed to an existing feed.
-<WRAP info>
-**NOTE:**  If you don’t have a **Feeds** sub-menu, that most likely means you are still on the older version of pfBlockerNG.
-Another way to check is if you have **Alerts** instead of **Reports** along the top row of pfBlockerNG options…  That too means you are still on the old version.
-You can either follow the walkthrough for the older version of pfBlockerNG or better yet, delete the old pfBlockerNG and install the pfBlockerNG-devel package.
-</WRAP>
-Scroll down to the **DNSBL Category** header, which is *after* all of the IPv4 and IPv6 sections.  The first DNSBL sub-category you should see is labeled EasyList.
-Note that EasyList has a checkbox near the top left.  This means the alias/group or category already exists and is being used.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_feeds_dnsbl_category._easylist.png?800|}}
-If you look toward the right, you will see another checkbox.  This means the individual feed is enabled.
-This subtle distinction is extremely important to understanding how aliases and feeds work.  In addition, if a category ever has a problematic feed, you can always disable that feed instead of the entire category, i.e. we do not need to enable every feed for a particular category.
-For example, if you want to add the **EasyList Adware Filter** or one of the language specific feeds, you would click the **+** sign to the far right and that would add the individual feed to the already existing **EasyList** group.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_feeds_dnsbl_category_easylist_adware_plus.png?800|}}
-Ensure you switch **OFF** to **ON** and then click **Save** at the bottom of the screen.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_feeds_dnsbl_category_easylist_adware.png?800|}}
-<WRAP important>
-**WARNING:**  You can add as many feeds as you like, but keep in mind that too many feeds can potentially slow down your firewall.
-It’s quite possible just adding a few categories by themselves is too much for a resource starved firewall!
-This is because feeds are periodically downloaded and likewise, unbound is reloaded regularly.
-If you using a system with limited resources (mainly RAM), you need to be extra careful.
-When in doubt, add feeds slowly and keep an eye on memory, CPU, etc.
-</WRAP>
-----
-==== Add Feed hphosts ====
-If we go back to the Feeds, a category (group) recommend adding is hpHosts.  Click the **+** next to the hpHosts header (top left) to add all the feeds related to this category.
-After clicking the **+** next to the hpHosts category, you are taken to a DNSBL feeds page with all of the feeds under that category pre-populated.  All of the feeds in the list will initially be in the **OFF** state.  You can go through and enable each one individually or you can click **Enable All** at the bottom of the list.
-{{:pfsense:pfsense_pfblockerng_feeds_hphosts.png?800|}}
-Make sure you switch the **Action** from Disabled to Unbound (below).
-Click **Save DNSBL Settings** at the bottom of the page and you should receive a message at the top along the lines of **<nowiki>Saved [ Type:DNSBL, Name:hpHosts ] configuration</nowiki>**.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_dnsbl_source_definitions.png?800|}}
-Click on the **DNSBL Groups** tab and you will be taken to the DNSBL feeds summary.  Assuming everything went as planned, your feeds summary should include the hphosts.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_dnsbl_groups_summary.png?800|}}
-----
-==== Other items worth mentioning ====
-If you take a look at the **Malicious** category, you will notice that some feeds have selectable options, such as such as the SANS Internet Storm Center feeds (bullet points).  I personally recommend switching the feed from ISC_SDH (high) to ISC_SDL (low) as the high feed has under 20 entries and the low feed includes the high feed.
-I addition, I haven’t seen many false positives when using the expanded (low) list.
-Take note of the door-arrow graphic icons next to several feeds.  The door-arrow graphic means the feed is a subscription feed, which at the very least means you need to register for it.  Some subscription feeds also have a fee associated with them.  Subscription feeds can have a lower false positive rate and are typically updated on a more frequent basis.  You will see selectable options and subscription feeds throughout the DNSBL feeds so it is important to understand what these graphics mean.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_feeds_dnsbl_category_malicious.png?800|}}
-----
-===== Other recommended feeds =====
-  * hpHosts (all of them) – From MalwareBytes.
-  * BBcan177 – From the creator of pfBlockerNG.
-  * BBC (BBC_DGA_Agr) – From Bambenek Consulting <- This feed is extremely large.
-  * Cryptojackers (all of them) – This blocks cryptojacking software and in-browser miners, but it also blocks various coin exchanges.
-<WRAP alert>
-**ALERT:**  You can add as many feeds as you like, but keep in mind that too many feeds can potentially slow down your firewall!
-If you using a system with limited resources (mainly RAM), you need to be extra careful.
-When in doubt, add feeds slowly and keep an eye on memory, CPU, etc
-</WRAP>
-----
-===== Problem Solving a Feed =====
-If you ever experience issues with a particular feed, go to **DNSBL -> DNSBL Groups** and then click the pencil/edit icon next to that particular category.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_dnsbl_groups_summary_edit.png?800|}}
-Once in the category edit screen, simply switch those feeds to **OFF** and then click **save** at the bottom.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_feeds_dnsbl_category_malicious_feed_off.png?800|}}
-You could also delete those feeds.
-----
-===== Forcing DNSBL feed updates =====
-Anytime you make changes, you can either wait for the next update or you can force the changes yourself.
-To force the changes, go over to the **Update** tab within pfBlockerNG.
-Heed the warning and make sure you are not going to run the updates near the time your cron job would automatically run.  If the countdown timer is less than 5 minutes, I would not recommend running it and instead just wait for the system to run it automatically.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_dnsbl_update.png?800|}}
-Assuming you are good on the time, go ahead and click the **Run** button.  You will see progress updates in the gray window below including the number of domains downloaded for each list, when the list was last updated, etc.  Also note that pfBlockerNG is smart enough to check for and eliminate duplicate DNS (# Dups) entries between the lists.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_dnsbl_update_run_manually.png?800|}}
-----
-===== Testing By Browsing =====
-So what does the finished product look like?
-On many sites, you’ll see gray boxes where an ad normally would have been.
-A browser add-on like uBlock Origin further cleans this up by removing the gray box entirely and it also provides some secondary protections.
-Also keep in mind that some ads are still served such as video ads on YouTube.
-Those ads cannot be blocked via pfBlockerNG since the ad content is served from the same domain names (DNS) as the video content.
-----
-===== Testing from the command line =====
-Normally, pinging a site will return the sites actual IP address.
-However, with pfBlockerNG properly setup you may instead see a reply of **10.10.10.1**, which is the default virtual IP address DNSBL creates.
-{{:pfsense:pfblockerng:pfsense_pfblockerng_dnsbl_ping_block.png?800|}}
-----
-===== Statistics and graphs =====
-pfBlockerNG has some really fantastic graphs built-in as shown below.
-You can even see the top blocked domains, source IPs with the most blocks, blocked user agent strings, TLDs, and much more.
-Also helpful is you need to whittle down the number of feeds you are using, i.e. this feed accounts for 50% of your blocks and it’s a third the size of these other two feeds combined.  Just go to **Firewall -> pfBlockerNG -> Reports -> DNSBL Stats** to see all the DNSBL eye candy, aka graphs/stats.
-{{:pfsense:pfsense_pfblockerng_reports_dnsbl_stats.png?800|}}
-----
-===== Troubleshooting - Whitelisting =====
-You can remove the offending list entirely (DNSBL -> DNSBL Feeds -> Edit the list in question) or more preferably, you can just whitelist the domain.
-The absolute easiest way to do this is by going to the **Reports** tab and scrolling down to the DNSBL section.  Clicking on the red lock will temporarily unlock the domain so you can verify if it is indeed the domain that needs to be whitelisted.  Clicking the **+** will add the domain to the DNSBL whitelist.
-When clicking the **+** you will then receive a prompt about whether you want to perform a wildcard whitelist or just a whitelist.
-Read the explanation, but typically use whitelist because it is more exact and less prone to letting something past.
-<WRAP tip>
-Adding a description so you know what was broken and/or why you fixed it, i.e. today it makes perfect sense, but it might not 6 months from now.
-</WRAP>
-If you go back to the main DNSBL tab and expand the DNSBL Whitelist section toward the bottom, you should now see the domain you whitelisted.
-You might also notice that if the domain you are whitelisting has CNAME records, pfBlockerNG is smart enough to add those too.
-Simply type each domain in on a separate line and then click **Save** if you know which domains to whitelist.  If you want the whitelist additions/changes to occur sooner rather than later, you will also need to go back to the **Update** tab and click **Run**.  If you don’t want to do the trial and error on your own see some whitelist recommendations below.
-It's also worth mentioning that if a system already resolved the domain name on your system and it is previously resolved to 10.10.10.1, then you may need to clear your local DNS cache, your browser cache, or both.  To clear your machine’s cache, from a command line on Windows, type in **ipconfig /flushdns** and that should take care of it.  You can run a similar command on a Linux system, although the commands can vary from one installation to the next.  More often than not, simply restarting your network interface will work; on most distributions, **service networking restart** or **systemctl restart network** should take care of it for you.  Each browser has a slightly different way to clear the cache, however, all of them allow you to pull a new version of the website if you hold down **Shift** while clicking on the refresh/reload button.
-If ads are not getting blocked and the ping commands above don’t return the virtual IP address, it’s also possible your local machine is not using pfSense for its DNS settings. If you are using Windows, check your network settings and make sure it is set to your pfSense IP address. On Linux/*nix, check your /etc/resolv.conf or even Network Manager (if using a GUI).  If you are not using pfSense for your DHCP server, you may need to do some digging.
-Browsers can also get in the way especially with the advent of DNS over HTTPS.  If you find your ping tests work, but your browser doesn’t, then that is most likely your issue.  Although somewhat uncommon, some anti-virus packages and endpoint protection can mess with your DNS settings too.  Furthermore, those changes may not necessarily be reflected in your operating system’s DNS settings.  For example, Avast Premier has a Secure DNS feature that will force your browser to use Avast specified DNS servers in an effort to prevent DNS hijacking.  If you find that other devices on your network are blocking ads and one particular device doesn’t, then your anti-virus or endpoint protection very well may be the culprit.  When all else fails, you can always fire up Wireshark for a packet capture to ensure your system is querying the DNS server(s) you specify.
-----
-===== Whitelist Recommendations =====
-These are a few domains that cause issues if they end up on the various DNSBLs.
-You can easily copy and paste them into the "custom list" as described above.  If you ended up using the pfBlockerNG wizard, BBCan actually incorporated these recommendations already.  If you have no plans to use some of them (based off their name alone), you can and should omit them from your whitelist.
-<code>
-s3.amazonaws.com
-s3-1.amazonaws.com # CNAME for (s3.amazonaws.com)
-.github.com
-.githubusercontent.com
-github.map.fastly.net # CNAME for (raw.githubusercontent.com)
-.apple.com
-.sourceforge.net
-.fls-na.amazon.com # alexa
-.control.kochava.com # alexa 2
-.device-metrics-us-2.amazon.com # alexa 3
-.amazon-adsystem.com # amazon app ads
-.px.moatads.com # amazon app 2
-.wildcard.moatads.com.edgekey.net # CNAME for (px.moatads.com)
-.e13136.g.akamaiedge.net # CNAME for (px.moatads.com)
-.secure-gl.imrworldwide.com # amazon app 3
-.pixel.adsafeprotected.com # amazon app 4
-.anycast.pixel.adsafeprotected.com # CNAME for (pixel.adsafeprotected.com)
-.bs.serving-sys.com # amazon app 5
-.bs.eyeblaster.akadns.net # CNAME for (bs.serving-sys.com)
-.bsla.eyeblaster.akadns.net # CNAME for (bs.serving-sys.com)
-.adsafeprotected.com # amazon app 6
-.anycast.static.adsafeprotected.com # CNAME for (static.adsafeprotected.com)
-google.com
-www.google.com
-youtube.com
-www.youtube.com
-youtube-ui.l.google.com # CNAME for (youtube.com)
-stackoverflow.com
-www.stackoverflow.com
-dropbox.com
-www.dropbox.com
-www.dropbox-dns.com # CNAME for (dropbox.com)
-.adsafeprotected.com
-control.kochava.com
-secure-gl.imrworldwide.com
-pbs.twimg.com # twitter images
-www.pbs.twimg.com # twitter images
-cs196.wac.edgecastcdn.net # CNAME for (pbs.twimg.com)
-cs2-wac.apr-8315.edgecastdns.net # CNAME for (pbs.twimg.com)
-cs2-wac-us.8315.ecdns.net # CNAME for (pbs.twimg.com)
-cs45.wac.edgecastcdn.net # CNAME for (pbs.twimg.com)
-</code>
-----
-===== TLD Blacklisting =====
-TLD (top-level domain) blacklisting is another option in DNSBL.
-Don’t forget you need to **Enable** the TLD option at the top of the DNSBL configuration page to use the features discussed here.
-Static blacklisting is not normally advocated because the bad guys will simply move around it, TLD blacklisting is a rare instance where you can eliminate some potential attack vectors although its usefulness depends entirely on your situation.  TLDs are the characters after the last dot on a domain name, e.g. com, net, and biz are some common ones.  The number of TLDs has skyrocketed and there were well over 1,500.  Over time, some TLDs have become wastelands for nefarious activity such as command and control servers.  If you no plans to connect with a particular TLD and it has shown to be less than reputable, i.e. most sane companies wouldn’t bother trying to use it for legitimate businesses, you can just go to the main DNSBL tab and block it outright.
-Some TLDs are used extensively for typosquatting — Omitting the "o" in .com could Be costly.
-If you’re looking for a little more guidance of what is ‘bad’ then look no further than Spamhaus and the website link below.  Spamhaus is constantly updating this list and related statistics so check it directly for the most up-to-date information.
-https://www.spamhaus.org/statistics/tlds/
-Suggest adding the top 3 TLDs, as they are used often for
-<code>
-cm
-party
-click
-link
-</code>
-Adding these others would likely not cause too many issues, although keep in mind that you will see false positives:
-<code>
-technology
-gdn
-study
-men
-biz
-reise
-stream
-</code>
-----
-===== References =====
-https://www.linuxincluded.com/block-ads-malvertising-on-pfsense-using-pfblockerng-dnsbl/
-https://github.com/jmdugan/blocklists/blob/master/corporations/facebook/all