pantz.org banner
DNS over TLS. Secure the DNS!
Posted on 04-22-2017 21:59:17 UTC | Updated on 04-22-2017 22:08:25 UTC
Section: /software/unbound/ | Permanent Link

Secure the Web

A while back we had the "secure the web" initiative, where everyone was inspired to enable encryption (https) on their websites. This was so we could thwart things like eavesdropping and content hijacking. In 2016 about about half of website visits where https. This is great and things seem to be only getting better. ISP's can not see the content in https traffic. Not seeing your traffic content anymore makes them sad. What makes them happy? They can still see all of your DNS requests.

Your ISP can see the websites you visit

Every ISP assigns you some of their DNS servers for you to use when you connect to them for your internet connection. Every time you type in a website name in your browser bar, this request goes to their DNS servers to look up an number called an IP address. After this happens an IP address is returned to your computer, and the connection to the website is made. Your ISP now has a log of the website you requested attached to the rest of the information they have about you. Then they build profiles about you, and sell that info to 3rd parties to target advertising to you in many different ways. Think you'll be slick and switch out their DNS servers with someone else's like Google's free DNS servers (8.8.8.8)? Think again. Any request through your ISP to any DNS server on the internet is unencrypted. Your ISP can slurp up all same requests and get the same info they did just like when you were using their DNS servers. Just like when they could see all of your content with http before https. This also means that your DNS traffic could possibly be intercepted and hijacked by other people or entities. TLS prevents this.

Securing the DNS

The thing that secures https is Transport Layer Security (TLS). It is a set of cryptographic protocols that provides communications security over a computer network. Now that we are beginning to secure the websites I think it is high time we secure the DNS. Others seem to agree. In 2016 rfc7858 and rfc8094 was submitted to Internet Engineering Task Force (IETF) which describes the use of DNS over TLS and DNS over DTLS. Hopefully these will eventually become a standard and all DNS traffic will be more secure in transit. Can you have DNS over TLS today? Yes you can!

Trying DNS over TLS now

DNS over TLS is in its infancy currently, but there are ways to try it out now. You could try using Stubby, a program that acts as a local DNS Privacy stub resolver (using DNS-over-TLS). You will have compile Stubby on Linux or OS X to use it. You could also setup your own DNS server at home and point it to some upstream forwarders that support DNS over TLS. This is what I have done to start testing this. I use Unbound as my local DNS server on my lan for all of my client machines. I used the configuration settings provided by our friends over at Calomel.org to setup my Ubound server to use DNS over TLS. Here is a list of other open source DNS software that can use DNS over TLS. With my Unbound setup, all of my DNS traffic is secured from interception and modification. So how is my testing going?

The early days

Since this is not a IETF standard yet, there are not a lot of providers of DNS over TLS resolvers. I have had to rearrange my list of DNS over TLS providers a few times when some of the servers were just not resolving hostnames. The latency is also higher than using your local ISP's DNS servers or using someones like Googles DNS servers. This is not very noticeable since my local DNS server caches the lookups. I have a feeling the generous providers of these DNS over TLS services are being overwhelmed and can not handle the load of the requests. This is where bigger companies come into play.

Places like Google or OpenDNS do not support DNS over TLS yet, but I'm hoping that they will get on board with this. Google especially since they have been big proponents of making all websites https. They also have the infrastructure to pull this off. Even if someone like Google turned this on, that means they get your DNS traffic instead of your ISP. Will this ever end?

Uggh, people can still see my traffic

Let's face it, if your connected to the internet, at some point someone gets to see where your going. You just have to choose who you want/trust to give this information to. If you point your DNS servers to Google they get to see your DNS requests. If I point my DNS at these test DNS over TLS servers then they get to see my DNS traffic. It seems like the lesser of 2 evils to send your DNS to 2nd party DNS servers then to your ISP. If you use your ISP's DNS servers they know the exact name attached to the IP address they assigned you and the customer that is making the query. I have been holding off telling you the bad news. https SNI will still give up the domain names you visit.

Through all of this even if you point your DNS traffic to a DNS over TLS server your ISP can still see many of the sites you go to. This is thanks to something in https called Server Name Indication (SNI). When you make a connection to an https enabled website there is a process called a handshake. This is the exchange of information before the encryption starts. During this unencrypted handshake (ClientHello) one of the things that is sent by you is a remote host name. This allows the server on the other end to choose appropriate certificate based on the requested host name. This happens when multiple virtual hosts reside on the same server, and this a very common setup. Unfortunately, your ISP can see this, slurp it up, and log this to your account/profile. So now what?

Would a VPN help? Yes, but remember now your DNS queries go to your VPN provider. What is nice is your ISP will not see any of your traffic anymore. That pesky SNI issue mentioned above goes away when using a VPN. But now your trusted endpoint is your VPN provider. They now can log all the sites you go to. So choose wisely when picking a VPN provider. Read their policy on saving logs, choose one that will allow you to pay with Bitcoin so you will be anonymous as possible. With a VPN provider you also have to be careful about DNS leaking. If your VPN client is not configured right, or you forget to turn it on or, any other myriad of ways a VPN can fail, your traffic will go right back to your ISP.

Even VPN's don't make you anonymous

So you have encrypted your DNS and web traffic with TLS and your using a VPN. Good for you, now your privacy is a bit better, but not anonymity. Your still being tracked. This time it is AD networks and services you use. I'm not going to go into this as many other people have written on this topic. Just know that your being tracked one way or another.

I know this all seems hopeless, but securing the web's infrasturcture bit by bit helps improve privacy just a little more. DNS like http is unencrypted. There was a big push to get websites to encrypt their data, now there needs to be the same attention given to DNS.

Del.icio.us! | Digg Me! | Reddit!

Related stories

Gmail and their IPV6 spam filter
Posted on 08-14-2016 22:30:28 UTC | Updated on 08-14-2016 22:33:51 UTC
Section: /software/smtp/ | Permanent Link

Failing forwards

Earlier this week Gmail's servers decided that any email sent from Gmail and then forwarded from pantz.org back to Gmail was now, as their servers put it "likely unsolicited mail". People sending mail from Gmail to pantz.org were getting bounce messages, which looks bad. All other email from any other domain was coming in without issue. I have been forwarding email from Gmail accounts for many years now without issue.

Time to recheck settings

  1. DNS A,AAAA and PTR records for IPV4,6 setup and work correctly
  2. SPF record setup correctly, but this is a forward so it always shows fail. The bounce message passes SPF, so that's nice.
  3. SMTP w/TLS working and available
  4. pantz.org does not modify, remove or shuffle message headers or modify the body of the message in any way.
  5. No increase in spam getting through. pantz.org filters most mail before it is forwarded.
  6. Not using Sender Rewriting Scheme (SRS).
  7. No DKIM setup

The fix

After seeing that everything checked out, I hit up Google to see if anyone else was having this issue. From the results it seems that many people have had this same issue. Some people just started using SRS to fix their issue. Others had to fix their PTR records in DNS. The last group of people had to stop using IPV6 for mail delivery. Since all of the other pantz.org mail server settings were correct, the only thing I could try was implementing SRS or turn off IPV6. Turning off IPV6 delivery was the easiest test. After turning off IPV6 mail delivery, and just leaving IPV4, all mail from Gmail being forwarded through pantz.org was now being accepted again. How dumb is that?

What is up with IPV6?

It seems Gmail has changed a setting (or I hit some new threshold) on their side dealing with only IPV6. Since Google will not tell you why certain mail is considered "unsolicited mail" we can not figure out what was done to try to fix the issue. If I had to speculate on what is happening, my guess is they turned up the sensitivity on email coming from IPV6 as it is obvious that IPV4 filter is not as sensitive. It is not just my server as it is happening to many other people as well.

I had also noticed that mail coming in from a friend whose server delivers mail to my server via IPV6, and then was forwarded to Gmail via IPV6 was being marked as spam every time. According to Google if the user is in your contacts list (and his email address is) the email is not supposed to be marked as spam. That is straight broken. Now that I switched back to just IPV4 delivery, all of his mail is not being marked as spam anymore. I believe Google has an issue with IPV6 mail delivery and spam classification.

What now?

I hate that I had to turn off IPV6 for mail forwarding to Gmail. My next likely step is to implement SRS for forwarding, and see if I can turn IPV6 back on. The best article I found on setting this up on Postfix is here. It also shows how to setup DKIM which might be fun to do as well.

Del.icio.us! | Digg Me! | Reddit!

Related stories

SaltStack Minion communication and missing returns
Posted on 05-30-2016 04:42:57 UTC | Updated on 08-02-2016 00:27:12 UTC
Section: /software/saltstack/ | Permanent Link

Setting up SaltStack is a fairly easy task. There is plenty of documentation here. This is not an install tutorial, this is an explanation and trouble shooting of what is going on with SaltStack Master and Minion communication. Mostly when using the CLI to send commands from the Master to the Minions.

Basic Check List

After you have installed your Salt Master and your Salt Minions software the first thing to do after starting your Master is open your Minion's config file in /etc/salt/minion and fill out the line "master: " to tell the Minion where his Master is. Then start/restart your Salt Minion. Do this for all your Minions.

Go back to the Master and accept all of of the Minions keys. See here on how to do this. If you don't see a certain Minions key here are some things you should check.

  1. Is your Minion and Master running the same software version? The Master can usually work at a higher version. Try to keep them the same if possible.
  2. Is your salt-Minion service running? Make sure it is set to run on start as well.
  3. Has the Minions key been accepted by the Master? If you don't even see a key request from the Minion then the Minion is not even talking to the Master .
  4. Does the Minion have an unobstructed network path back to TCP port 4505 on the Master? The Minions initialize a TCP connection back to the Master so they don't need any ports open. Watch out for those Firewalls.
  5. Check your Minions log file in /var/log/salt/minion for key issues or any other issues.

Basic Communication

Now lets say you have all of basic network and and key issues worked out and would like to send some jobs to your Minions. You can do this via the Salt CLI. Something like salt \* cmd.run 'echo HI'. This is considered a job by Salt. The Minions get this request and run the command and return the job information to the Master. The CLI talks to the Master who is listening for the return messages as they are coming in on the ZMQ bus. The CLI then reports back that status and output of the job.

That is a basic view of this process. But, sometimes Minions don't return job information. Then you ask yourself what the heck happened. You know the Minion is running fine. Eventually you find out you don't really understand Minion Master job communication at all.

Detailed Breakdown of Master Minion CLI Communication

By default when the job information gets returned to the Master and is stored on disk in the job cache. We will assume this is the case below.

The Salt CLI is just an small bit of code that interfaces with the API SaltStack has written that allows anyone to send commands to the Minions programmatically. The CLI is not connected directly to the Minions when the job request is made. When the CLI makes a job request, is handed to the Master to fulfill.

There are 2 key timeout periods you need be aware of before we go into a explanation of how a job request is handled. They are "timeout" and "gather_job_timeout".

When the CLI command is issued, the Master gathers a list of Minions with valid keys so it knows which Minions are on the system. It validates and filters the targeting information from the given target list and sets that as its list (targets) of Minions for the job. Now the Master has a list of who should return information when queried. The Master takes the requested command, target list, job id, and a few pieces of info, and broadcasts a message on the ZeroMQ bus to all of Minions. When all Minions get the message, they look at the target list and decide if they should execute the job or not. If the Minion sees he is in the target list he executes the job. If a Minion sees he is not part of the target list, he just ignores the message. The Minion that decided to run the command creates an local job id for the job and then performs the work.

While the Minions are working their jobs the CLI is waiting for the first initial timeout period (-t or timeout:) to start. When that hits, the CLI sends sends the first "find_job" query. This kicks off the gather_job_timeout timer. The Minions receive the the first find_job request with the original job_id. If they are still running the job, the Minion responds to "find job" request with a status of "still working" or "Job Finished". If a Minion does not respond to the request within the gather_job_timeout time period (10 secs), the CLI marks the Minion as "non responsive" for the polling interval. All Minions will keep being queried on the gather_job_timeout interval. If the Minions do not reply within this timeout, or all report that they are no longer running the job in question, the CLI command will return. If one of more minions replies that they are still running the job, the initial timeout is triggered again and the cycle repeats.

The CLI will show the output from the Minions as they finish their jobs. For the Minions that did not respond, but are connected to the Master, you will see the message "Minion did not return". If a Minion does not even look like it has a TCP connection with the Master, you will see "Minion did not return. [Not connected]".

By this time the Master should have marked the job as finished. The jobs info should now be available in the job cache. The above explanation is a high level explanation of how Master and Minions communicate. There are more details to this process than the above info, but this should give you a basic idea of how it works.

Takeaways From This Info

  1. There is no defined period on how long a job will take. The job will finish when the last responsive Minion has said it is done.
  2. If a Minion is not up or connected when a job request it sent out, then the Minion just misses that job. It is _not_ queued by the Master, and sent at a later time.
  3. Currently there is no hard timeout to force the Master to stop listening after a certain amount of time.
  4. If you set your timeout (-t) to be something silly like 3600, then if even one Minion is not responding the CLI will wait the full 3600 seconds to return. Beware!

Missing Returns

Sometimes you know there are Minions up and working, but you get "Minion did not return" or you did not see any info from the Minion at all before the CLI timed out. It is frustrating, as you can send the same Minion that just failed a job and it finishes it with no problem. There can be many reasons for this. Try/check the following things.

Del.icio.us! | Digg Me! | Reddit!

Related stories


RSS Feed RSS feed logo
About


3com
3ware
alsa
alsactl
alsamixer
amd
android
apache
areca
arm
ati
auditd
awk
badblocks
bash
bind
bios
bonnie
cable
carp
cat5
cdrom
cellphone
centos
chart
chrome
cifs
cisco
cloudera
comcast
commands
comodo
compiz-fusion
corsair
cpufreq
cpufrequtils
cpuspeed
cron
crontab
crossover
cu
cups
cvs
database
dbus
dd
dd_rescue
ddclient
debian
decimal
dhclient
dhcp
diagnostic
diskexplorer
disks
dkim
dns
dos
dovecot
drac
dsniff
dvdauthor
e-mail
echo
editor
emerald
ethernet
expect
ext3
ext4
fat32
fedora
fetchmail
fiber
filesystems
firefox
firewall
flac
flexlm
floppy
flowtools
fonts
format
freebsd
ftp
gdm
gmail
gnome
greasemonkey
greylisting
growisofs
grub
hacking
hadoop
harddrive
hba
hex
hfsc
html
html5
http
https
idl
ie
ilo
intel
ios
iperf
ipmi
iptables
ipv6
irix
javascript
kde
kernel
kickstart
kmail
kprinter
krecord
kubuntu
kvm
lame
ldap
linux
logfile
lp
lpq
lpr
maradns
matlab
memory
mencoder
mhdd
mkinitrd
mkisofs
moinmoin
motherboard
mouse
movemail
mplayer
multitail
mutt
myodbc
mysql
mythtv
nagios
nameserver
netflix
netflow
nginx
nic
ntfs
ntp
nvidia
odbc
openbsd
openntpd
openoffice
openssh
openssl
opteron
parted
partimage
patch
perl
pf
pfflowd
pfsync
photorec
php
pop3
pop3s
ports
postfix
power
procmail
proftpd
proxy
pulseaudio
putty
pxe
python
qemu
r-studio
raid
recovery
redhat
router
rpc
rsync
ruby
saltstack
samba
schedule
screen
scsi
seagate
seatools
sed
sendmail
sgi
shell
siw
smtp
snort
solaris
soundcard
sox
spam
spamd
spf
sql
sqlite
squid
srs
ssh
ssh.com
ssl
su
subnet
subversion
sudo
sun
supermicro
switches
symbols
syslinux
syslog
systemrescuecd
t1
tcpip
tcpwrappers
telnet
terminal
testdisk
tftp
thttpd
thunderbird
timezone
ting
tls
tools
tr
trac
tuning
tunnel
ubuntu
unbound
vi
vpn
wget
wiki
windows
windowsxp
wireless
wpa_supplicant
x
xauth
xfree86
xfs
xinearama
xmms
youtube
zdump
zeromq
zic
zlib