In this post, we’d like to share our current plans for .. PowerDNS 4.x! We shared this first with the PowerDNS-development community, and after we gathered feedback, we’re now announcing it more broadly.

The tl;dr: For the next few months we will be spring cleaning git master, and stable code and releases can be found in the auth-3.4 and rec-3.7 branches. We’ll also be moving to C++ 2011. Please read on for the whole story.

First some background. PowerDNS is a 15 year old software project, and over these 1.5 decades, we have built up some ‘technical debt’, and it is time for a spring cleaning in our code.

Meanwhile, we are broadening what our code does, to include for example smart, DNS-native, load balancing and further denial of service mitigation. And of course, the major work of bringing carrier-grade DNSSEC to the recursor.

Finally, we’ve fallen in love with C++ 2011, and we would like to start taking advantage of this now 4 year old revision of C++.

All this means some important changes. For one, where it used to be the case that our git ‘master’ was usually fit to run in production (and people actually did this), for the coming few months please consider our master branch a ‘heavy development zone’. While we’ll try to keep things working, it might break for hours or even days at a time. Even though there will be somewhat of a wild-west aspect to development, major changes will be implemented as pull requests from separate branches that can be studied by the community.

Meanwhile, PowerDNS 3.x development and maintenance will continue on separate release branches. The latest 3.x releases will remain actively supported until 4.x is more powerful, more stable, and can be compiled on Debian Stable (more about this later). Active support means more than passive maintenance, if there are pressing things that need to happen, they will happen. But the focus for new things will shift to 4.x.

(as an example of what active support means, we are currently gathering the patches for auth-3.4.3, see here)

Things we will be addressing during our spring cleaning include:

• We treat DNS names as ASCII strings, which we escape and unescape repeatedly. DNS names are not ascii strings, and we keep finding issues related to us treating them like strings.
• The PowerDNS Authoritative Server distributes queries to multiple backends inefficiently
• The PowerDNS Recursor cache is both slower and less memory efficient than it could be
• DNSSEC in the PowerDNS Recursor
• Move our own atomic, locking and semaphore infrastructure to C++ 2011 native
• The Lua APIs use an ascii based interface for domain names and IP addresses, and this could be faster
• One thing we are probably not going to do is change the database format, by the way.

The somewhat bad news about the spring cleaning is that we’ll come out of it as a C++ 2011 project, which means that to compile PowerDNS, you’ll need GCC 4.8 (released in March 2013). Gcc 4.8 is not currently the default in Debian stable or RHEL/CentOS 6, but it is available. (Update May 2015: Debian 8.0 is now stable and 4.9 is default.)

It is the default in RHEL7 and in what will become the next Debian stable (May 2015: now the current Debian stable). GCC 4.8 also ships in Ubuntu 14.04. We will also be targeting clang 3.5. We have chosen C++ 2011 for a variety of reasons, many of which are described in an earlier blogpost.

NOTE: PowerDNS 4.x products WILL run on older distribution releases of course! However, on older distros, compiling with the system default compiler may not work.

To clarify, the 4.x branch will not fundamentally alter PowerDNS. This should not be compared to BIND 9 to BIND 10, for example (or even 8 to 9). Fundamentally we think the PowerDNS design is sound, it just needs a decent spring cleaning. This will come in especially handy when deploying our DNSSEC validation.

So how long will it take until 4.x is production ready? We’ll let you know once we get there, but we are hoping to finish the cleanup in several months, after which we expect further work to iron out remaining issues. In any case, 3.x will remain supported until gcc 4.8 is widely available on currently shipping distributions.

Thanks, and please again let us know your thoughts about this proposed plan. Although this is what we intend to do, we can be change our mind if there are good reasons to do so!

Over the years, PowerDNS users have frequently asked us about our preferred DNS load balancing solution, and we’ve never had a satisfying answer for that. Users of dedicated hardware often tell us that vendors spend most of their time and effort on balancing HTTP, and frequently deliver substandard or even buggy DNS functionality.

UPDATE: Dnsdist now has its own homepage on http://dnsdist.org/!

In terms of software, one big PowerDNS deployment is happy with OpenBSD relayd, and it indeed does look powerful. Other operators have deployed keepalived, which is very strong from a networking standpoint, but does not offer a lot of DNS specifics.

But all in all, we never found any load balancing solution for DNS that made people truly happy.

Simultaneously, there is now a lot of focus on providing the very best DNS performance, even in the face of the ongoing reflection attacks (one of which has been dubbed the “Chinese water torture attack“).

Putting these three things together (no really satisfying DNS aware load balancer, drive for the very best performance, ongoing attacks) led us to pollute the waters of the internet with yet another piece of software: dnsdist.

From its README:

“dnsdist is a highly DNS-, DoS- and abuse-aware loadbalancer. Its goal in life is to route traffic to the best server, delivering top performance to legitimate users while shunting or blocking abusive traffic.”

This is quite a mission statement, but we’ve tried to keep things simple. The simplest possible invocation:

# dnsdist --local 0.0.0.0:53 192.168.1.2 192.168.1.3 192.168.1.4

Delivers sensible and smart distribution of queries over three downstream servers (.2, .3 and .4). All three are polled for availability, and traffic is delivered to the server with the lowest amount of outstanding queries. If this is all you need, you should be done. However, much more power is available.

For example, the following configuration does something quite special:

newServer{address="192.168.1.2", qps=10000, order=1}
newServer{address="127.0.0.1:5300", qps=5000, order=2}
newServer{address="192.168.1.79:5300", qps=7000, order=3}
setServerPolicy(firstAvailable)

This configuration again defines three downstream servers, but it also configures the preferred QPS limit of each. Finally, it changes the default server selection policy to ‘firstAvailable’. In production, as viewed from the dnsdist console, this looks like this:

(This console is available either locally or remotely.)

It has frequently been observed that a busy nameserver is a happy nameserver, since this means that all caches are hot and that more cache hits are generated per TTL. The configuration above will send traffic to the first server that has not hit its configured QPS limit. If all have exceeded their limit, it decays to the standard ‘leastOutstanding’ policy.

Built in, two further load balancing policies are available:

• wrandom: weighted random, where traffic is distributed by the weight parameter
• roundrobin: as the name implies

In the configuration, custom policies can be written in Lua (see below).

Traffic inspection

Now, if all traffic were bona fide, and everything worked as planned, we’d be done with the feature set above. In the real world however, we frequently suffer degraded performance from malicious or clearly illegitimate traffic. Dnsdist offers quite some features to inspect live traffic:


This shows us the ‘top 5’ of incoming queries, showing a rather ‘flat’ profile for this server. Nothing really jumps out. Potentially however, this is because unwanted traffic consists of many unique names, so let’s see:

This shows traffic grouped by the last 2 components of query names, and now ‘t-ipnet.de.’ jumps out. Let’s say we determine these queries as unwanted (which would be HIGHLY unlikely in this case), dnsdist offers three options:

1. Block outright: addDomainBlock("t-ipnet.de.")
2. Rate limit to (say) 10/s: addQPSLimit("t-ipnet.de.", 10)
3. Shunt to a dedicated abuse pool:
   • newServer{address="192.168.1.30:5300", pool="abuse")
   • newPoolRule("t-ipnet.de.", "abuse")

All three options have their merits, but we’re especially excited about this last option. Operators usually face a stark choice with traffic – block it, and potentially upset lots of bona fide users, or allow and degrade performance for everyone. By dedicating a bunch of servers to dubious traffic, it is possible to isolate worrying traffic, and only have it impact the users of the impacted domain names.

Further analysis is possible based on response codes. This for example shows the top-5 servfail responses:

Analogous inspection based on IP addresses is possible via:

And the three options outlined above for blocking, limiting and shunting traffic also accept IP addresses as parameters.

Full flexibility

While dnsdist natively comes with a broad set of primitives to route, shape and block traffic, ultimate flexibility is possible with the built-in Lua support.

For example, to block all domain names with 5 consecutive numbers in them, try:

function blockFilter(remote, qname, qtype, dh)
return string.match(qname:tostring(), "%d%d%d%d%d") ~= nil
end


Similarly, blockFilter can implement for example returning TC=1 answers for all ANY queries, or if you are so inclined, rate limit all ANY queries. A example of this can be found in the sample configuration file.

Server load balancing policies can also be written entirely in Lua, and direct traffic to specific servers, or to a specific built-in load balancing policy, but based from a different pool. This last service is great for providing split-horizon service, for example.

Practical details

dnsdist is a daemon that can partially be configured from the commandline, but otherwise has a configuration file. It can be queried at runtime via an optionally encrypted TCP/IP connection. At this console, which is readline() based and offers history search etc, every setting can be displayed and changed.

The Lua parser is the console commandline and configuration parser. In practice, most configurations can run without involving Lua for every packet. However, even in very dynamic setups where this is required, the practical overhead is minimal, especially when compiled against luajit.

Note that dnsdist is not PowerDNS specific, and will happily balance traffic between nameserver implementations or even third party services!

What dnsdist is not

dnsdist tries to be high performance, but is fundamentally limited by the fact that it is a general purpose handler of DNS packets. Practically speaking, this means that dnsdist can’t be used to scale to many hundreds of thousands of forwarded packets per second, as this tends to stress out commodity servers and operating systems. However, care has been taken to allow dnsdist to itself be hooked up to network-based load balancing solutions, for example by allowing non-local binds, and proper behavior on binding to 0.0.0.0.

Similarly, dnsdist is not an alternative to many gigabit capable DoS scrubbing devices or services.

It is however a great way to deal with “the last gigabit” of an attack, or simply smaller scale attacks that algorithmically kill your performance, instead of simply flooding your pipe. And by distributing queries wisely, your users experience better performance.

Some further cool things

• showResponseLatency(): prints a histogram of response times
• showServers(): show statistics for all configured downstreams
• getServer(0):setDown(): force server 0 down administratively
• showPoolRules(): show configured pool rules
• showQPSLimiters(): show configured QPS limiters
• setServerPolicy(wrandom): set weighted random server selection policy
• setServerPolicyLua(): configure Lua-based server policies
• .. more in the README

Current status & how to get it

dnsdist is still very fresh software. However, we are actively seeking feedback, but not quite yet production users. Please let us know your thoughts on powerdns.ideas@powerdns.com!

Snapshot releases can be found on http://dnsdist.org/. Compiling dnsdist requires a comparatively recent compiler (gcc 4.8, clang 3.5) and Lua. More details can be found in the README.

Actual production releases should happen shortly.

Hi everybody,

Two announcements in one: First, like 7000 others, we’ll be visiting World Hosting Days in Rust, Germany next week. Peter, Pieter and I will be there, as will be two of our wonderful Certified Consultants (Kees Monshouwer and Christian Hofstaedtler).

If you want to meet up, please email any of us (or powerdns.ideas@powerdns.com) and we can coordinate. The PowerDNS team will in any case be available for drinks! We always like to  hear from users since you have more experience running PowerDNS than we do, and can help us guide new features.

Secondly, last year we made our ‘public graphing as a service’ available, as described on
http://blog.powerdns.com/2014/12/11/powerdns-graphing-as-a-service/

Today, we’re happy to announce that we now also have a private variant for supported customers and selected users. This means you can benefit from a one-line setup in PowerDNS (simply set the ‘carbon-server’ variable and you are done), and view all your PowerDNS instances from one single interface, and in private.

If you’d like to use our private graphing service, please contact us for details.

The public graphing instance is now receiving over 1 gigabyte of graphs every week, so we think we are fulfilling a need!

Cheers,

Bert

(Peter van Dijk, Bert Hubert, Rafael Laguna, Mikko Linnamäki, Markku Kenttä, Timo Sirainen)

Hi everybody,

We’re currently at World Hosting Days in Rust Germany, where we just announced that PowerDNS will be joining the Open-Xchange family of companies.  Last week it was also announced that the famous Dovecot IMAP server project is now a part of OX too.

Update: The Register and The WHIR covered the news!

We’ve been working with Timo and his team at Dovecot and with the OX Team in Email Security projects and are already sharing personnel and infrastructure with each other and the cooperation works really well for all of us.

From the Open-Xchange website: “With over a decade of developing open-source software, Open-Xchange believes that only by engineering ruthlessly open products and services can the next generation of innovation emerge on the web. “Stay Open” contains many aspects of how we develop, engineer and deploy our products together with and for client-partners.”

We fully believe in that mission, and are glad that PowerDNS will become part of the Open-Xchange family. It will be great to have Timo and friends from Dovecot as cousins!

We’ll share more details of what the merger will and will not mean, but rest assured PowerDNS will stay as open and as community friendly as it has ever been.

Meanwhile, if you are at WHD, please come meet us at the Open-Xchange booth!

Bert, Peter and Pieter

This weekend, PowerDNS attended the DNS-OARC Spring Workshop 2015 in force, with 100% attendance. I shamefully have to admit this is the first time I’ve gone to an OARC workshop, but I was well rewarded. Both the speakers and audience were stellar. Full video is available in four parts: Saturday morning, Saturday afternoon, Sunday morning, Sunday afternoon.

UPDATE: Geoff Huston also did a writeup with more and different details from mine. Very much worth your time!

In this post, I briefly want to summarise the big themes of the meeting. But I want to start off with describing the audience. We had people running the biggest (cc)TLDs on the planet, we had authors from all the big name servers. There were the people that run the root and plan the DNSSEC key transitions. The largest access providers on the planet. Folks instrumenting the whole internet to get statistics on DNS(SEC) performance. For fun. The people that protect our websites from denial of service attacks. In short, everyone was there. This workshop was easily the best DNS event I have ever attended.

The biggest theme of the meeting was the flood of ongoing reflection attacks. In short, bad people send random questions to open resolvers on the internet. These in turn often forward their queries to powerful recursive name servers over at providers. And these reach out frequently and insistently to numerous “authoritative servers” to find answers to these questions.

However, the actual goal is not to get answers to questions. The actual goal is to perform a powerful denial of service attack on these “authoritative servers”, which frequently don’t even run DNS. But they do get bombarded with DNS traffic from all over the world and go offline.

These attacks have been going on for the past year or so, and are the biggest thing in DNS for a long time. All large resolver implementations have had to implement changes to protect themselves from the attacks, and to attempt to no longer to take part in this malicious traffic. This has not been easy.

At the workshop, we had presentations describing how BIND and Nominum name servers implement their protection strategies, with ISC implementing various tuneable knobs that attempt to detect unresponsive servers, and Nominum doing (among other things) “threat lists” of domains currently known to be involved in attacks.

In addition, Kazunori FUJIWARA of JPRS presented how NSEC records from DNSSEC could be used to silence such attacks – an NSEC denial of existence range can be used to block many random queries, as long as we have a denying range for them. There was some discussion if this would work for NSEC3 too, and OARC attendees are now pondering that question.

Tangentially related, Microsoft presented research on how well the internet performs negative caching, specifically how long. I was very happy to see Microsoft open up and become a part of the DNS community. Microsoft has long had smart people working on DNS, but up till maybe a year ago, you’d never see anyone from Microsoft present at a workshop or working group. This has now changed, and the internet will surely be a better place with Microsoft at the table. Even if Microsoft legal still insists they carry a ‘Microsoft Confidential’ warning on every slide!

Moving on: dealing with random packet floods requires the best statistics, and John Dickinson presented work on Hedgehog which can present DSC data.

Another major theme of the meeting was ‘dealing with packet floods’, random or not. Part of this is writing smart name server software, but at some level of traffic, packets need to be processed or blocked at the network layer. Various folks presented on this, and I want to specifically thank Cloudflare for sharing their vast DoS-quenching knowledge. It is not common for the DoS protection people to open up on how they do their work, because these are of course the crown jewels. However, not everyone can be a Cloudflare customer, so it is great that we can learn from them.

In short, they had some key insights. Efficient name server software is mostly limited by the UDP stack in Linux and other operating systems, and this stack really hits a wall somewhere around 200kqps. This was repeated in several other presentations, and the reasons for this limitation appear to be pretty fundamental. With careful tuning and specific hardware configuration higher numbers can be achieved, but it is uphill work. But, you have to view this presentation, it is full of unexpected insights.

At the end of this post, I get back to both the ‘200kqps’ issue and possible new ways to deal with unrequested packet floods.

Cloudflare separately presented their gross DNSSEC hacks, which while clever don’t fill me with glee. In the words of Filippo “I’ve done stuff I ain’t proud of and the stuff I am proud of is disgusting”. Read all about their NSEC Shotgun in any case.

Verisign also opened up with various eye-popping statistics on denial of service attacks they have to weather all day, and what countermeasures they have in place. During one such attack last year, Verisign filtered a big PowerDNS user, causing mayhem for us. Piet Barber feared I would call him out on that during the presentation, but in fact we had no “hard feelings”. I mostly feel bad we were part of relaying that attack to the GTLD servers!

Measurements

Various people reported interesting measurements. Geoff Huston of APNIC did another one of his incredible presentations, this time on how well ECDSA signed domain names get resolved on the internet. Geoff really is an asset to the world, he truly has his finger on the pulse of the internet. In short, 80% of DNSSEC resolvers can validate ECDSA records. The ones that can’t behave oddly, or in the words of Geoff to the implementors present “you lot write a lot of crap”. I am sure this is true. Another key insight is that around 2000 resolver ‘pairs’ represent 95% of query load on authoritative nameservers. In another presentation, it was reported there are around 150k bona fide resolver IP addresses in the world, and this matches my own observations. In short, if you are under DoS as an auth, you could do worse than block everyone except the top 99% of resolvers.

Shumon Huque of Verisign presented measurements of the privacy enhancing qname minimisation idea, and in short, because of Akamai’s current nameserver implementation, it works very badly. Akamai is aware of the problem and has made a vague promise to do something about it one day.

Ralf Weber of Nominum did measurements how well several resolvers deal with random query attacks. Unsurprisingly, Nominum came out best, but this was a very good even-handed presentation, which showed that most modern nameservers have been updated to deal well with such attacks. NOTE: the current presentation shows unfavourable numbers for Unbound, but during the meeting Ralf and Wouter Wijngaards found out why this was the case, and Ralf will be redoing the tests. As with Microsoft, I’m very happy to see Nominum join the (public) DNS community and this can only be helpful in improving the state of DNS.

William Sotomayor of OARC presented remotely on how various countries and university networks use AS112 networks, but I have to admit most of this presentation went over my head as I’m not very good with large scale internet routing. Similarly his work on ‘RSSAC-02’ is undoubtedly very important, but outside my expertise.

Joao Luis Silva Damas worked hard to get access to actual customer DNS traffic to do statistics on that, and when he finally got the data he tore it apart and learned a lot. Recommended reading. After this presentation, various DNS trace anonymization strategies were discussed, including the PowerDNS ‘dnswasher’ and the NZ registry more sophisticated ‘keyed’ blinding solution. For both these programs however, keep in mind that data can frequently be de-anonymized with sufficient correlation!

Bruce Van Nice of Nominum showed statistics on the life of a resolver and popular domain names. I call fake on this one since not a single domain listed was of the ‘XXX’ variety ;-) I assume these were quietly filtered from the graphics so as not to upset anyone. The PowerDNS statistics I presented later had actual adult domain names in them, but we’re Dutch, so we get away with that!

Sebastien Castro of the NZ registry presented work on how to discover popular or important domain names using statistical measures, and showed how these change over the week. Similarly Francisco Cifuentes of CL NIC research presented on how to do realtime DNS analytics with Apache Storm and other technologies.

Root zone related

The root is, of course, at the root of all of DNS and thus the internet. Anything affecting the root affects us all. And there is enough stuff to think about. The root Key Signing Key is getting stale and the batteries on the Hardware Security Modules housing the KSK are similarly showing their age. I understand this is a problem. To change the KSK is a major effort however.

For one, during the change, root answer packets might get (a lot) bigger. Duane Wessels of Verisign presented numbers on what various changeover scenarios would mean for fragmentation and truncation. The good news appears to be that the sky isn’t falling.

Meanwhile, before the root and most important (cc)TLDs got signed, there was the DLV-registry, which made it possible to specify your DNSSEC keys in a parallel registry over at ISC and some other places. It is now high time to sunset this registry, and Jim Martin of ISC set out how they plan to do that. After the presentation a huge line formed at the mike to wholeheartedly support the rapid shutdown of the DLV registry.

Kazunori Fujiwara presented on the changing ratio between JP and ROOT queries on the JPRS infrastructure.

Finally at the end of the day, Edward Lewis (now of ICANN) presented about the process of changing the root KSK. This is fraught with difficulty and I for one doubt it will happen before circumstances force us to. Ed and his very capable friends, including Geoff Huston, are giving it their best however, and surprisingly (to me at least), during and after the presentation, some new ideas were raised to facilitate the transition. This involves having one root-server serve with the old keying material, perhaps giving people a chance to limp on during the transition.

Other presentations

Florian Maury of the French IT government security agency ANSSI presented on the iDNS attack they discovered, which felled PowerDNS, Bind and Unbound last year, but notably not DJBDNS since “1999” Dan Bernstein was smarter than all of us and failed to fall for that one. This provided the brief moment of drama of OARC with one audience member claiming the iDNS stuff was not news, not important, and that by publishing it, Florian had only helped potential attackers. Luckily sanity returned and this member of the audience apologized later in the day. Who says DNS is boring?

Florian also presented on the French government DNS guidelines (which are, of course, in French), but that interestingly (and correctly I think) do not propose to implement DNSSEC before a host of other best practices in DNS are implemented, including registry locks.

Patrik Wallström of .SE presented on Zonemaster, a Swedish-French collaborative zone checker, intended to supersede DNSCheck and Zonecheck.

And I did a presentation too on dnsdist, a highly DoS and DNS aware load-balancer, where I asked the audience if there is room for a ‘smart load balancer that has some features of a nameserver’. The feedback I got was overwhelmingly yes. Further discussions afterwards were instrumental in finding the limits of what dnsdist should and should not do.

Followup work

Two things stand out for me from this workshop. For one, Cloudflare and many others feel compelled to implement a ‘Super NXDOMAIN’ answer that allows an authoritative server to send a response to a bona fide resolver: “you can stop sending me queries for x.y.somedomain.com, or in fact anything within somedomain.com. It is not going to happen.”. We jokingly called this the Shut Up Packet. However, it appears this idea has merit and Olafur already wrote some text on it. We will also be working on this and studying the parallels with the (failed) ICMP Source Quench packet, or in fact even a real ‘NXDOMAIN’ response.

Secondly, too many people to mention lamented the suboptimal performance of the Linux (and UNIX in general, but specifically Linux) on dealing with UDP packets. Where you can now blast gigabits of TCP/IP, the supposedly more efficient UDP struggles to reach hundreds of thousands of packets per second. Part of this problem is neglect of UDP in the kernel. Part of this is an inefficient ‘one system call per packet’ interface (not usefully addressed by recvmmsg()).

Since we all feel the pain of this and have to buy special hardware to get better (filtering) performance, I feel it is time to liaise with various kernel folks to see what could be done. Individually, everyone I spoke to agreed, but nothing has coalesced yet. I’ll continue to agitate for something to happen, please let me know if you want to join in!

PowerDNS 2.x End of Life Statement

21st of May 2015

PowerDNS Authoritative Server 2.9.22 was released more than 6 years ago, in January 2009. Because of its immense and durable popularity, some patch releases have been provided, the last one of which (2.9.22.6) was made available over three years ago in January 2012.

The 2.9.22.x series contains a number of probable and actual violations of the DNS standards. In addition, some behaviours of 2.9.22.x are standards conforming but cause interoperability problems in 2015. Finally, 2.9.22.4 and earlier are impacted by PowerDNS Security Advisory 2012-01, which means PowerDNS can be used in a Denial of Service attack.

Although we have long been telling users that we can no longer support the use of 2.x, and urging upgrades to 3.x, with this statement we formally declare 2.x end of life.

This means that any 2.x issues will not be addressed. This has been the case for a long time, but with this statement we make it formal.

To upgrade to 3.x, please consult the instructions on how to upgrade the database. If you need help with upgrading, we provide migration services to our supported users. If you are currently running 2.9.22 and need help to tide you over, we can also provide that as part of a support agreement.

But we urge everyone to move on to PowerDNS Authoritative Server 3.4 or later – it is a faster, more standards conforming and more powerful nameserver!

Hi everybody,

As we’re working on PowerDNS 4.x, we are wondering: what are we missing?

The somewhat longer story is that as a software developer, a sort of feature-blindness creeps up on you. We try to make the software better, faster etc, but by focusing so much on the technology, one can lose sight of the actual use cases.

In this way it is possible that a software vendor neglects to implement something, even though many users desperately want it. If so, please speak up! The short version: please mail powerdns.ideas at powerdns.com your ideas!

As concrete examples, PowerDNS took its time to add an API, and once we had it, people immediately started using it, even before we had documented the API. Similarly, for many years, we did not deliver a proper graphing solution, and now that it is there it is highly popular.

But what more are we missing? Should we expand into IPAM and do DHCP and IP address management? Should we make an out of the box NAT64/DNS64 solution? Do we need to improve replication beyond “database native” and “AXFR-based” (so ‘super-duper-slave’)?

Should we start doing versioned databases so people can roll back changes?  IXFR?

Should we add a built-in DNS based load balancer where we poll if your IP addresses are up?

Or would it be wise to move on beyond the geographical versatile backends, and simply add ‘US’ and ‘Europe’, ‘Oceania’, ‘Asia’ IP address profiles?

Should the recursor gain cache sharing abilities? Or pre-fetching? Or even TTL-faking in case auths are down?

The list above is just to prime your imagination: if you have any ideas on what you are missing, please reach out to powerdns.ideas at powerdns.com, or use our contact form.

Thanks!

PowerDNS is a dynamic nameserver, with a ton of backends. If the supplied backends aren’t flexible enough, our architecture enables operators to write their own, or to use one of our forwarding backends (Pipe and Remote), which can send PowerDNS queries over a pipe, a UNIX domain socket, an HTTP connection or even a ZMQ link.

Over time, many operators have done just that, and this allows you to mix a ‘real’ nameserver, with everything you can expect from that, with a custom data source.

Very often however (weekly at this point!), we get questions from users confused about our backends:

1. Why does my backend get ANY queries, when no ANY queries are sent to the nameserver?
2. How do I generate an NXDOMAIN response from my backend?
3. Why do I get SOA queries, even for domains not in my backend?
4. Why does PowerDNS ignore the records my backend sent back to put in the packet?
5. Why do I get more backend queries than DNS queries (sometimes)?
6. Why do I get *way less* backend queries than DNS queries at other times?
7. Why are backends launched for AXFRs?

All of these questions stem from a misunderstanding what a PowerDNS backend is, possibly combined with the fact that people want things that aren’t easily achieved from a backend. We may not have been doing a sufficiently good job explaining PowerDNS backends.

To explain, we must first do away with what a PowerDNS backend is not. It is for example not this:

In this model, packets from the internet get handed over to a backend, and the backend drafts an answer packet for PowerDNS to send out. We do have functionality that does that, more about which later, but backends are not it. Backends get DNS queries and return data, or perhaps none if there isn’t any.

So why did we not build it as above? It turns out DNS needs a lot of logic. So for example, if a nameserver gets a question for ‘www.powerdns.com’, it has to figure out if it is even authoritative for that question, or in other words, does it have a zone with relevant data for that query? To figure that out, the RFC algorithm tells us we must find the best zone that matches the query. Initially this could be ‘www.powerdns.com’, and if we don’t have that zone, we should look for ‘powerdns.com’, then ‘com’ and finally perhaps a root zone.

Once we have the best zone ready, we must check if a ‘www’ record exists, if there is perhaps a delegation of ‘www’ to another server, if there is perhaps a CNAME for it, if there is a wildcard ‘*.powerdns.com’ record etc. DNSSEC brings further questions, should we do DNSSEC processing, and if so, are there keys we need to sign with?

Because of all this complication, and because back in 2000 we already predicted there would be many many backends, we decided to keep the ‘DNS logic’ central, where we would implement it once and leave the backends to being providers of data, pure and simple. It might have been better if we called the backends ‘DNS Data stores’ perhaps. But alas, we missed that chance.

So now that we’ve described how it doesn’t work, let’s check out the present day reality:

Packets come in from the Internet and get processed by our PacketHandler class. This class has all the logic on best zone selection, CNAMEs, wildcards, delegations, DNSSEC metadata etc. And to figure this all out, it sends many questions to the UeberBackend. The UeberBackend is subsequently responsible for distributing the queries over the multiple possible configured backends. In addition, it hosts a cache which optionally remembers answers (or lack thereof) a backend provided previously.

The best way to understand this procedure is that the PacketHandler turns DNS packets into data backend queries. This also means that backends should do no thinking. Just answer the question, with data or none. The PacketHandler can send many kinds of questions depending on the nature of your zone. For example, it may ask about SOA records, even for zones you do not host in your backend. This is because when a question comes in for ‘www.something.com’, PowerDNS must go hunt for a backend with relevant data.

As a further example of how things work, if PowerDNS asks a backend for ‘www.powerdns.com’, it can answer with a CNAME to ‘webserver1.cloudprovider.com’. PowerDNS itself then will attempt to follow the CNAME chain and check if there is data for the ‘cloudprovider.com’ zone. The backend does not need to think about this!

Why does my backend get ANY queries when no such queries are sent to the nameserver?

In addition, as a speedup, PowerDNS may ask ANY questions, which allow it to see if there is a CNAME for the DNS query name, and if there is no CNAME, get the right answer from the same question. This answers questions 1 and 3 from the list above.

How do I generate an NXDOMAIN answer from my backend?

Answering question 2, how to generate an NXDOMAIN from my backend also becomes easy: you need to convince PowerDNS that you do host the domain in question by delivering a SOA (aka Start of Authority) record when asked for your zone, and when you then get a question for ‘nosuchdomain.powerdns.com’, just return.. nothing. From this PowerDNS will conclude that 1) you host the zone 2) there is no record called ‘nosuchdomain.powerdns.com’. And this will lead to an NXDOMAIN.

So, the clarify even further, if you run a zone called ‘maik.de’, and PowerDNS receives a DNS query for ‘nosuchdomain.maik.de’, it will ask your backend if it knows about ‘nosuchdomain.maik.de’, and you should return no data. Next it will ask about ‘maik.de’, and if it asked for ANY or the SOA record, you should return the SOA record. From this PowerDNS will conclude that it should send out an NXDOMAIN.

Why does PowerDNS ignore the records my backend sends back to put in the packet?

Answering question 4, if a backend sends back records that PowerDNS did not ask for, these do indeed not end up in your packet. PowerDNS asked the backend if it had data for a certain name, and any replies not for that name are out of spec. Not only might PowerDNS ignore them, it could also decide your backend is not functioning correctly and drop the packet.

Why do I get way less/way more backend queries than DNS queries?

Answering questions 5 and 6: you may indeed get many more backend queries than DNS packets coming in. An individual DNS query may cause 4 or more backend queries. Each of these queries is necessary to figure out details about the domain. If your Pipe backend is flooded with questions it does not want to hear about, the ‘pipe-regex’ feature is available to quickly deny any knowledge about such domains.

You may also get a lot less questions, since the UberBackend has a cache that optionally stores previous answers from your backend. If this is not what you want, caches can be disabled, which may be useful if you want to provide fully dynamic answers.

Why does PowerDNS instantiate extra copies of my backend for AXFR?

Finally, the answer to question 7, why do new backends get spawned for zone transfers? An AXFR may last minutes, and since during those minutes a backend can do nothing else, a dedicated instance is created for each zone transfer (both incoming and outgoing).

Summarising 

We hope that this information has been helpful in clarifying what a PowerDNS backend is and isn’t.

Finally, if it turns out a PowerDNS backend is not actually what you want, you may be interested to hear about a currently undocumented feature in the PowerDNS Authoritative Server that we use for testing. This allows you to intercept queries sent to PowerDNS Authoritative Server, much like is possible (and documented) in the PowerDNS Recursor.

From the point of interception, you can mangle questions to your heart’s delight from Lua, and send back any answer you want. This feature will become documented and supported in PowerDNS 4.0, but for now, study the ‘lua-prequery-script’ configuration parameter. And if you have any questions, you can find us on the mailing lists or IRC.

Good luck!

Hi everybody!

Since we are now happily part of Open-Xchange, together with our friends at Dovecot IMAPd, we will also be present at the Open-Xchange summit in Berlin, October 8-9 this year. This is a free event, and you are invited!

Besides marketing presentations like ‘The Power of DNS to Engage More Customers’, we’ll also be having a serious technical presentation on Friday about PowerDNS. Feel free to invite your manager to the first presentation and join us for the second one ;-)

Also important, the summit includes a ‘Bier-Xchange’, which also serves other things than beer, plus a party at the end. More about the event can be read on http://summit.open-xchange.com/

All PowerDNS users are cordially invited to join the summit, which is free of charge: http://www.cvent.com/d/6rq9my/4W or using the ‘Register Now’ button at the bottom of http://summit.open-xchange.com/ page.

If you register, please drop us an email (powerdns.ideas @ powerdns.com) so we can invite you to the PowerDNS-specific gathering during the drinks and party.

Also: feel free to invite your manager to show that PowerDNS is a real company and more than some free software from the internet.

Thanks!

        Bert & Team

The recently exposed denial-of-service vulnerability in a popular DNS server implementation really highlights the benefits of software providers writing and owning the software that they deliver to internet service providers. The flaw in the software allowed hackers to potentially crash DNS servers, effectively allowing them to take huge sections of the internet offline. As we have highlighted previously, DNS is an absolutely critical part of the internet and it’s tremendously important to ensure its smooth running.

As a customer, the best way to secure your DNS servers is to make sure that your software provider is in full control of the product that they deliver. No software, including PowerDNS, is free from occasional security issues. However, companies that haven’t written the core software they deliver and who simply repackage other providers’ products with additional services, are on the back foot when it comes to security issues.

The problem is that when a security flaw emerges, these companies are unable to fix it themselves. Instead they have to wait for the original providers to issue a suitable and effective patch. With security issues, the speed with which you can fix vulnerabilities is crucial. Often these companies are not on friendly terms with their ‘upstream’ (which may view them as competition!) or they have no contractual/financial agreement with them to fix these bugs and so customers are left vulnerable for a prolonged period of time.

So by definition, if you rely on a DNS vendor or appliance that repackages a third party DNS server, you are getting second-tier, second-hand support – at best.

Security breaches are one of the biggest sources of reputational damage for organisations. Customers will be unimpressed by the loss of their data or the inability to access your services but moreover, it reflects badly on the management decisions of a business that fails to take proper measures to secure their service.

Here at PowerDNS we understand the benefit of authoring, supporting and distributing our own product: we can always act quickly to ensure our software stays free from performance impacting bugs and patched against the latest security vulnerabilities.

Make sure you ask your DNS vendor if they can do the same!

Hi everybody,

In this post we’d like to update you on what has been achieved in the development of the PowerDNS 4.x products: Authoritative, Recursor and dnsdist. First, I am very proud that we managed to do a “spring cleaning”. As mentioned earlier, over time any software project picks up so called “technical debt“. Things that looked good at the time, shortcuts to quickly get to a new feature, whole features that weren’t fully thought through: it all piles up.

It is very wise for any piece of software to periodically take a step back and do a cleanup, but it rarely happens. Some projects do go for the “grand redesign”, but this frequently does not lead to a product that is actually better in production (“Second system effect“).

We’re grateful for our customers and users that allowed us the time for the cleanup, and happy that the PowerDNS community itself too had the discipline to work on these invisible improvements. It is always more fun to add new features than to break down things that were working to make them better!

So what happened under the hood? Quite a lot of things.

C++ 2011

PowerDNS is mostly written in C++, and since 2011 this monumental language has been available in a new revision. C++ 2011 makes life a lot easier on programmers, which in turn means you can deliver more functionality in the same time, or perhaps the same functionality but then with higher quality.

C++ 2011 has merged a lot of functionality from the equally monumental Boost libraries, and we’ve taken the opportunity to move PowerDNS to the ‘native’ versions of these functionalities: range-for instead of BOOST_FOREACH, std::shared_ptr instead of boost::shared_ptr, std::to_string instead of boost::lexical_cast, for example.

Taking this step required revamping our entire build & regression testing infrastructure because the environments on which we test did not support the recent versions of the C++ compilers and libraries we required.

DNSName

Within PowerDNS hid a sin. DNS names frequently look like “ascii strings”. But they are anything but. DNS names compare case insensitively. Also, there is the issue of the trailing dot. “www.powerdns.com.” and “WwW.PoWeRdNs.com” are the same from a DNS perspective. Life becomes even more complicated when we realize that DNS names are ‘8-bit clean’. You can put any binary string in DNS and it should work. But how do we encode “some name.powerdns.com”? As some\032name.powerdns.com? Some\ name.powerdns.com?

There is only one worthy answer to these questions: we don’t. DNS is internally not stored as ascii but as a series of labels with specified lengths. So “www.powerdns.com” is stored as the value 3, followed by www, followed by the value 8, followed by powerdns, followed by 3, by com and finally the zero value. This is the right way to store DNS names.

To achieve this, we wrote the DNSName class which stores DNS values in this way, and also offers ways to parse DNSNames straight from packets, and to output them in “human friendly” form. Over the course of 4.0 development, DNSName got reimplemented a few times as we learned more, finally taking a shape where we could do canonical ordering very fast. This gave us the benefit of cleaning up a lot of ugly reversal code, and allowing all relevant caches to be purged not just name-by-name, but for whole zones in one go.

Finally, we’ve equipped DNSName with many methods that are useful in a DNS context like isPartOf() and chopOff(), removing lots of redundant code from PowerDNS.

Ridding PowerDNS from “DNS Names as ASCII” was a monumental undertaking that would not have been possible without extensive help from the community, specifically Kees Monshouwer and Aki Tuomi.

DNSResourceRecord

On a related note, for a long time, the PowerDNS Recursor showed its heritage as a spin-off from PowerDNS Authoritative Server. In the Authoritative Server, backends store DNS details as ASCII. So to encode the AAAA record 2001:888:2000:1d::2, we actually have that string “2001:888:2000:1d::2” of 19 characters in the database. This is not the most efficient thing to do, but for databases it is ok. They are good with strings.

However, INTERNALLY, it makes very little sense to drag these ASCII representations around and convert them into binary addresses and back again for processing. This is the sort of technical debt you build up over 15 years.

With great effort, we’ve been able to purge the PowerDNS Recursor of the DNSResourceRecord struct which carried those ASCII strings, and move everything to the DNSRecord class. We’ve checked, no bits of ASCII are hurt when answering questions in the Recursor anymore!

Netmask & Domain trees

Frequently, PowerDNS products need to check an IP address or domain name against a long list of possible matches. Based on the DNSName, ComboAddress and Netmask classes, Aki Tuomi and we have built generic structures that allow for high speed lookups of domain names & IP addresses against domain suffixes and netmasks, using Patricia Tries. These rapid lookups are now used within all three PowerDNS products, and are also available from Lua.

We can now safely disclose that our previous method for testing an IP address against many netmasks consisted of trying each one in order. Sorry for that.

Malloc Tracer

C++ is a wonderful language, we think, especially if you don’t turn it into a circus. A big problem of C++ however is the astounding amount of memory allocations that happen under the hood. And while memory allocators have gotten better over the years, we discovered we were doing hundreds of mallocs per packet in some circumstances.  We’ve found that Heaptrack was helpful in getting a statistical overview of where our allocations were coming from, but we got very high per-packet precision using a very simple built-in (optional, turned off by default) malloc tracer. Using this, we’ve been able to reduce the allocation traffic by over 60% so far.

As an example, in the common case, the PowerDNS Recursor will now only issue two small mallocs per packet.

Configuration State

We try and succeed in getting a performance boost out of using multiple CPUs. This is not straightforward. No two threads can alter the same memory simultaneously, or bad things would happen. The easy solution against that is to lock the data. It turns out that when you sprinkle locks all over your code, any performance boost is gone. Performance might in fact go down with more threads.

A common case however is where a piece of memory rarely if ever changes and we can spare the memory to give each thread its own copy to read from. Only if something changed in the ‘master’ should each thread get a new copy. This idea is roughly what is known as Read Copy Update within system programming, and is for example also used by the great Knot DNS server. We created a set of classes called ‘State Holders‘ to bring this technology to PowerDNS as well.

Package Builder, Repositories

Although we loved Jenkins and we are mostly happy with Travis, we have gotten a lot of power from our ‘buildbot’ building engines. We now build PowerDNS for more and more platforms automatically, and push out those packages to our repositories on https://repo.powerdns.com/. These repositories allow you to install the latest and greatest builds of PowerDNS using apt or yum, and get native packages. This makes testing 4.0 really easy.

So what did we do with all those improvements?

Once this better new infrastructure was in place, we’ve implemented many new things:

• RPZ aka Response Policy Zone, as outlined here
• IXFR slaving in the PowerDNS Recursor for RPZ
• DNSSEC processing in Recursor (authoritative has had this for years)
• DNSSEC validation
• EDNS Client Subnet support in PowerDNS Recursor (authoritative has had this for years)
• GEOIP backend supporting custom netmasks, “fields” in answers
• Newly revived ODBC backend for talking to Microsoft SQL Server & Azure
• Lua asynchronous queries for per-IP/per-domain status
• Caches that can now be wiped per whole zone instead of per name
• An astounding amount of dnsdist features (check out the movie & the presentation!)
• And much more

We’ll outline what is in 4.x more completely in an upcoming post, including details on when and how it will be released. For now, it may be good to know you can test these new features via the package builder and repo service as outlined above.

Good luck!

As an Open Source project, PowerDNS works hard to support its users and customers. We also love to work with our collaborators, our community, on projects inside and outside of PowerDNS. This way we’ve contributed to the development of ExaBGP, Luawrapper, Botan, incbin, MbedTLS, Unbound, NSD, BIND, OSX Homebrew, Ansible, buildbot and many other projects we depend on or interoperate with. We also work closely with our downstream distributors (such as Debian and Fedora) to make sure their needs are covered.

If you are a PowerDNS user and you have a question or a problem we’ll gladly help you with that. If there is a bug in PowerDNS, or a ‘surprising feature’, we’ll fix it for you, thus improving the product. In this way, the community contributes to QA and product management of PowerDNS. For this we are grateful.

Meanwhile, we also get a lot of people that simply need help with PowerDNS. There likely is no bug, the features work as intended, the documentation is there. But still PowerDNS does not do what the user wants it to do.

In such cases too, we and the PowerDNS community provide support for our products. And in fact, we should not forget how special that is. Free software with free support!

One thing we don’t offer however: free private support. By providing support in the open, other people can learn, search engines pick up our answers, the community can pitch in with solutions or suggestions. Doing free support this way provides a true public benefit.

Another reason is that PowerDNS needs to live and thrive. As noted before, it takes money to develop quality products. PowerDNS and the PowerDNS Certified Consultants can also help you privately as part of a PowerDNS support agreement, and this is how we pay the rent & our employees.

If you want to be part of the PowerDNS community and get free support from us and that community.. with only a few exceptions everything we need to know to help you needs to be out and in the open.

If you have a domain that does not resolve, we need the actual name of that domain. Not ‘example.com’. If we cannot query your nameservers because you won’t tell us their IP address, we can’t help you. If you wrote a custom script that does not do what you want, but you can’t post that script to the mailing list because it is proprietary, we cannot help you. And in fact we do not want to help you for the reasons outlined above.

(By the way, if you can’t get PowerDNS to compile, or can’t find out which package to install, there is of course no need to tell us your domain name or IP address etc. This post is about the things we need to know to solve your issue).

Here are three things you can do if you or your organization can’t post sufficient details of the issue on our mailing lists or on IRC or on our ticketing system:

Reproduce your problem using details you CAN share

Reproduce your issue with things you CAN publish online. So for example, minimize your problematic script so it still displays your issue, but you feel comfortable posting it to the mailing list. On Stack Overflow, this is known as a “Minimal, Complete, and Verifiable example” and helps us (and others) to quickly determine the cause of the issue.

If your issue affects ‘bigbank.com’ and you want to keep that secret, see if you can reproduce it with ‘mytestdomain.com’. Similarly, if you can’t post the IP address of your servers, rent a small VPS, reproduce the issue there and share that.

Be a contributing member of the PowerDNS community

We realize such reproduction as required above is a burden. So let’s say you or your company have contributed patches, worked on issues, helped improve the documentation etc, work on a project we depend on, we are likely to treat you as a customer and provide support even though you supplied some details privately.

Become a customer

As noted above, we are an actual company in the business of providing support for our software. If your company has a policy of not posting to mailing lists, know that many of the largest telecommunications companies and hosters in the world are already among our customers. More information is on https://www.powerdns.com/support.html.

Take away

PowerDNS provides free support for its free products.. out in the open. If you cannot share sufficient detail of your issue on our mailing lists, IRC or ticketing system, you will need to reproduce the problem using data you can share. If you’ve worked on PowerDNS in some way, we can help you more privately. Alternatively, consider becoming a customer.

Thanks!

As we gear up for the release of PowerDNS Recursor 4.0.0, we are doing a series of posts describing new cool features which you can try out today. Many deployments are already running with 4.x alphas or snapshots, and now is a great time to familiarize yourself with the new and upcoming possibilities.

In this post, we’ll explain how to efficiently use the PowerDNS Recursor to optionally block certain domains for some or all of your users. This could be to stop users being tracked, to block advertisements or to protect against malware. The simple scripts below scale to millions of domain names and millions of users, all with acceptable startup times (seconds).

It starts with getting a good data set, and the good people from Mozilla have us covered here. The Mozilla Focus project through their partner Disconnect have a list of trackers and advertising servers which we can retrieve like this:

(Note, the actual Focus project performs blocking more cleverly than can be achieved purely with DNS. Focus also takes into account the URL of the page containing the advertisement).

To make these lists usable by PowerDNS, we need a little bit of conversion using the most excellent ‘jq’ tool:

This delivers a file which looks like ‘return{“a.com”, “b.com”, “c.com”}’, which is how to rapidly import data into Lua. To also block trackers, rerun this script for ‘disconnect-analytics.json’ and store as ‘trackers.lua’.

Next up, we use a small script ‘adblock.lua’ to tell PowerDNS to use this list to deny the existence of the filtered domains:

Finally, put ‘lua-dns-script=adblock.lua’ in recursor.conf to tell PowerDNS to load this list. And for the basic functionality, we are done! A few words about the script. With ‘newDS()’ we defined a new Domain Set.  In the second line, we load the list of advertisement related domain names.

Next we define the function preresolve() which gets called by PowerDNS to determine what to do with a domain. If we see that a query name is not part of one of the advertisement domains, or the query is not for an IP(v6) address, we return false and the normal resolution process continues.

Otherwise if it is a domain to be blocked, we insert a SOA record that says “this domain name exists, but the type you queried for doesn’t”.

Making it optional

Not everyone will want their advertisements filtered like this. And some people just want to be tracked online. To make this optional, first we make a list of IP addresses that want their DNS to be filtered:

This delivers a file with around 650000 IP addresses, which we can import in under a second in our Lua script like this:

New line number 13 informs PowerDNS that this domain name will have variable responses depending on who asks. The following new lines check if this user wants to be filtered or not.

This script can easily be expanded to have different lists for users that want advertising or perhaps only trackers to be blocked.

Finally, to reload the script with new data, issue ‘rec_control reload-lua-script’.

Take away

Using the elementary scripts shown above, we can optionally provide very large userbases with optional advertisement or tracker filtering. To learn more about the PowerDNS Recursor dynamic abilities, head to the documentation, where you can also find how to retrieve user and domain status in real time from external servers.

This is the second post in a series that highlights interesting new features of the PowerDNS 4.x.x and dnsdist 1.x.x releases.

As noted previously, in some organizations it is desirable to filter access to some domains (malware, advertising, ‘adult’ etc) for some users or subscribers. In our previous post we showed how to use common DNS blocking lists to offer adblocking for users of your resolver.

A problem however for other kinds of filtering is that subscribers typically share a single IP address among all their devices. This makes it hard to provide ‘parental control’ to the kids’ tablets, but not to other computers in the household. (Note: we realize not everyone likes parental controls or other forms of DNS filtering. In the real world however, it is something people want, and PowerDNS is technology to make such things possible!)

Now, it would be easy if we could do selective filtering based on the device’s MAC address, but this MAC address is not visible from the nameserver. All we see is the MAC address of our router, not that of your devices.

It turns out however that various home routers and cable modems (‘CPEs’) have the option to embed requestor MAC addresses into DNS queries themselves. This allows a service provider to conditionally filter DNS based on this MAC address.

A quick survey discovered that different vendors embed the MAC address or other ‘user identifiers’ differently, and for this reason PowerDNS support for this feature is dynamic.

First, we’ll use dnsdist to embed the MAC address in queries. The natural and universally chosen mechanism for this is an EDNS option, and we can configure dnsdist to add the MAC address like this:

This makes dnsdist listen on port 53 and forward all queries to 192.168.5.123:5300. With the last line, we add the MAC address of the requestor to all queries using EDNS option code point 65001 (5 is also used sometimes). We could restrict this to certain IP addresses if we wanted, or hash or truncate the MAC address for increased privacy. In a typical setup, this configuration is how dnsdist would run on a subscriber’s CPE.

Next up we need to configure the PowerDNS Recursor with our filtering instructions:

These lines implement a set of bad domains, in this case everything within the TLD “xxx”. Secondly, for two IP addresses we list some MAC addresses that desire filtering. 192.168.5.24 is in this case for the dnsdist configuration listed above.

Now to make use of these filters:

This first defines macPrint() to pretty print the binary encoded MAC address. Secondly, we use the PowerDNS Recursor Lua hook ‘preresolve’ to intercept incoming queries. Then we check if there is an embedded MAC address in EDNS option 65001, and if there is, we look up if for that IP address, we have that MAC address listed.

If so, and the query is part of our ‘baddomains’ list, we return a CNAME to a server which would then inform the user their request got blocked:

The last two lines are from a host with a different MAC address which is not filtered.

Takeaway

With the configuration outlined above, it is easily possible to provide per-device DNS filtering instructions for each subscriber. As shown, this configuration is not very dynamic. PowerDNS however has multiple ways to retrieve such filtering instructions, either from remote servers or from local indexed files.

Please do not hesitate to contact us if you need help integrating PowerDNS with your existing domain categorization lists, customer database or CPE deployed base! We have solutions that make this work out of the box.

Enjoy!

This is the third post in a series that highlights interesting new features of the PowerDNS 4.x.x and dnsdist 1.x.x releases.

I (Bert) have to admit something. Even though I set out to write a database driven nameserver 16 years ago.. I have never actually liked maintaining domains in a database. Of course, most large scale PowerDNS users have web-interfaces and scripts to provision the database. But out of the box, you could either use the pretty cool ‘BIND’ backend in PowerDNS.. or edit SQL.

Today I’m proud to announce that for the first time I personally enjoy editing my domains with PowerDNS. For completeness, this post will outline the full process of setting up a redundant PowerDNS server setup.

Basic setup (master)

First, install a 4.x.x version of PowerDNS, most easily obtained as packages from our repositories as found on https://repo.powerdns.com/. For this demo, install packages ‘pdns-server’ (‘pdns’ on RHEL/CentOS) and ‘pdns-backend-sqlite3’ (although these examples work just as well for the MySQL, PostgreSQL and ODBC backends).

Next, we need to create a database, in this case sqlite3 :

Then we configure PowerDNS to listen on all IP addresses, and to run with the database we just generated. This file goes in /etc/powerdns/pdns.conf:

We can now start PowerDNS, for example with: service pdns start.

To verify that everything works, run: dig chaos txt version.bind @127.0.0.1 +short

If everything is ok, it will tell you the exact version of PowerDNS you are running (this can of course be disabled).

Up to here, this is how installing PowerDNS Authoritative Server has worked for years.

The new command-line based zone editing features

To create a new domain name, operators typically copy some known good zone file from somewhere, and load that up. We’ve automated this process like this:

These few commands create a well configured zone, and add a new record to it. Finally, the ‘dig’ line verifies that all is good.

We can further edit the zone with pdnsutil edit-zone:

Full size animation

Note how this command actually runs a check on your changes to see if your zone still makes sense, and prompts you to edit some more or start over if it doesn’t look right. This uses the built-in DNS knowledge of PowerDNS, something which was frequently lacking in earlier scripts to update zone contents.

The pdnsutil commands ‘create-zone’, ‘edit-zone’, ‘list-zone’, ‘add-record’, ‘replace-rrset’, ‘remove-rrset’ thus form a complete way to edit a zone safely – since PowerDNS actually checks if what you entered makes DNS sense before committing it to the database.

But wait, there’s more

In the ‘pdnsutil edit-zone’ session above, we added an A record for ns2.powerdns.com, but we did not add an NS record to the zone yet. To do this, execute: pdnsutil add-record example.com @ NS ns2.example.com'.

When that is done, we tell the PowerDNS master server to allow that slave to actually transfer the zone by executing: pdnsutil set-meta example.com ALLOW-AXFR-FROM AUTO-NS. This tells PowerDNS: allow any IP address listed as a nameserver for this domain to transfer it.

Finally, we tell PowerDNS this is a ‘master’ zone for which we should send out out notifications by executing: pdnsutil set-kind example.com master

To verify that it all worked, we use pdnsutil show-zone:

Now we need to install PowerDNS on the slave server, and this is just like for our master, but we add the word ‘slave’ to pdns.conf. Please see above.

Once PowerDNS is installed and setup, we run: pdnsutil create-slave-zone example.com 192.168.1.2, where 192.168.1.2 is the IP address of our master server ns1.powerdns.com.

PowerDNS periodically checks the database for new slave zones, and within a minute it will pick up the new zone:

If you are in a hurry, execute pdns_control retrieve example.com and the zone will be slaved immediately.

Making further changes on the master server

Meanwhile, if we make a change on the master server, we need to bump the SOA serial number to make the DNS master/slave ritual aware of it. Do so either in ‘edit-zone’, or like this: pdnsutil increase-serial example.com. And again, PowerDNS will pick this up shortly and send out notifies. And like on the slave, if you are in a hurry, execute: pdns_control notify example.com.

What about DNSSEC?

All of the above works just as well with DNSSEC. Simply issue pdnsutil secure-zone example.com on the master server, bump the serial number, and inform your registrar of your new DS record (as shown by pdnsutil show-zone example.com).

Takeaway

PowerDNS has always offered classic zonefiles for your editing delight. Most large deployments have used various (web)tools to fill out the database with DNS details. But if like me you like a solid set of command line tools, PowerDNS Authoritative Server 4.x now has you covered!

Since yesterday we have been following and studying CVE-2015-7547. More about which here.

In short, this is a vulnerability not in PowerDNS products but in the Linux C library. This vulnerability could be exploited if it would be possible to relay specifically crafted records to Linux clients.

It appears the PowerDNS Recursor out of the box makes it hard to transport such specifically crafted records.

However, at this point there is still uncertainty over how CVE-2015-7547 could be exploited exactly. It may be that there are still ways to get the PowerDNS Recursor to relay content that could exploit vulnerable clients.

(we have tweeted earlier that we thought this was not possible. It now appears not enough is known about CVE-2015-7547 to be sure).

To be on the safe side, we have published a Lua script that puts in place further restrictions in the recursor that should help block CVE-2015-7547, as far as we currently understand it.

We urge everyone to patch their Linux C libraries of course. But as long as this is in progress or not yet possible, this script may help you protect vulnerable systems:

NOTE: We will keep updating the version of the script on GitHub and on our blog. Please check back for updates.

Please let us know if you have further questions!

Hi everybody,

This is a heads-up on some announcements you will be seeing on powerdns.com relating to new PowerDNS products which (gasp) are not fully Open Source. We know this is a sensitive subject, so before we go live, we want to inform you fully of what we are doing. We’d also like to hear & incorporate your feedback.

The tl;dr: PowerDNS will remain enthusiastically Open Source, but we will be selling a ready-to use ‘Platform’ of PowerDNS Open Source & other technologies, without degrading our current products. For details, please read on.

As you may know, PowerDNS sells support on the core nameserver technologies: PowerDNS Authoritative Server, PowerDNS Recursor and dnsdist. And this is going well, well enough to fund four full-time developers & engineers. This delivers a lot of value to the Open Source world.

Over the past few years, as part of our paid support, we have also been delivering custom PowerDNS configurations based on our open source products. Such configurations integrate with Graphite, Ansible, exabgp, bird, iptables and loads of other products to deliver features like parental control, configuration management, governmental/judicial blacklists, DoS protection of (legacy) nameservers, malware filtering, quarantining, NXDOMAIN redirection, “customer communications”, monitoring, user-experience graphing, audit trail of configuration changes, (management) reporting, webbased control, BGP/OSPF/VRRP failover, ‘production’ DNS64 etc etc.

What we have also found is that many of our users (big hosters, large scale telecommunications service providers) need more from us than “/usr/sbin/pdns_recursor”. Although PowerDNS can easily be integrated with lots of things to deliver powerful functionalities and many of our users still love open source, they would prefer to get it packaged in a more ready to use way.

Putting it more strongly: we have learned that many organizations simply no longer have the time or desire to assemble all the technologies themselves around our Open Source products.

We will therefore be marketing the additional functionalities we have been delivering to our customers as a product tentatively called the “PowerDNS Platform”. I say tentatively because we want to inform you of this news first, even before we have settled on a name and updated our website with the new product.

The “PowerDNS Platform” as we ship it consists of our core unmodified Open Source products, plus loads of other open source technologies, combined with a management shell that is not an Open Source product that we’ll in fact sell.

Now, we understand this may be worrying to some of you. Some formerly truly Open Source products like MySQL are going down a path where you can see their products turning into a sales pitch for the commercially licensed version.

We would therefore like to clarify that we regard our core Open Source products as our crown jewels, jewels which only shine because we are an integral part of the DNS and PowerDNS Communities with whom we work together to create great software. We will continue to make sure that our nameserver software is a viable and hopefully even the best choice for the Internet at large. And in fact, there will not be “two versions” of the PowerDNS nameserver software: of the actual daemons there will be just one version – also because we would otherwise not get the advantages of scale we get from over 150000 deployments!

Simultaneously, we hope that by bringing PowerDNS in a more integrated fashion will enable more companies to benefit from running Open Source & open standards based software. Because this is what deeply believe in – that the future of the world is open, and that software can simultaneously be good Open Source and also work well in a commercial environment.

Thank you for reading this to the end! We would like to hear your feedback and perhaps worries. Please contact me on bert.hubert@powerdns.com to let us know your thoughts and concerns.

Bert

Today a rather epic journey ends. In this post, we describe how 4.0.0 came to be, what we did, what we added, but also answer the big question: should I deploy PowerDNS 4?  And enable DNSSEC validation? Finally.. to celebrate, we’ll be handing out vouchers for FREE PowerDNS 4.0.0 Coffee (or tea) mugs! 

But first, a round of thanks. PowerDNS Authoritative Server 4.0.0 and PowerDNS Recursor 4.0.0 are the biggest releases in our history. This would not have been possible without the help of a lot of people. The PowerDNS community continues to be the stuff of dreams.

We believe in being an open company and producing powerful technology as open source. We are extremely grateful to be part of such a wonderful community that enables us together to make the internet and our software even greater.  Thanks to you, this is the most powerful version of PowerDNS ever, and one we feel can be relied upon to serve your needs!

Secondly, we’d like to thank our supported users (customers) too. Through their efforts, we were able to cram even more features into PowerDNS 4.0.0 than originally anticipated. Specifically, RPZ, IXFR and DNSSEC validation have been fast-tracked and enabled by (sadly) anonymous but very large PowerDNS customers.

Additionally, a shout out to Spamhaus, Farsight and ThreatSTOP who all made their wonderful RPZ feeds freely available for interoperability testing.

Finally, we are grateful for your understanding. PowerDNS 4.0.0 was a major ‘spring cleaning‘ operation that took 16 months. It is rare for software projects to be granted the time to revisit and cleanup old code. We trust it was worth the wait!

The history

In February 2015 we announced our plans for the 4.x.x branch of PowerDNS. Late May of that year, we asked for your help determining the roadmap for 4.x.x, and we got a lot of feedback from that. Late June we published the outcome of that process.

At the end of 2015 we launched the 4.0.0 Technology Preview releases (including dnsdist), where we noted:

A few months into the development, various users and customers suddenly chimed in on absolutely mandatory features we had somehow missed. Because of that, 4.x both under- and over-delivers.

During the 4.0.0 release process, we have stayed in close touch with our users and customers. And although we would have liked to have stuck to our roadmap, inevitably, some absolutely mandatory requirements came up. We spent most of early 2016 working with large (future) deployments to ensure 4.0.0 delivered what they needed (and deployed!).

So what did we do? You can read the full details in the release notes (auth link, recursor link), but here in short:

Spring cleaning

Over time, most software projects keep adding features, but sadly also a lot of complexity and “cruft”. For us, 4.0.0 was a “spring cleaning” exercise. We removed a lot of ancient code, tons of workarounds, loads of no longer relevant optimisations, non-functional backends and otherwise outdated code. We switched to C++2011, which allowed us to benefit from its enhanced features to make our code briefer and better.

Things we added

• Full DNSSEC in the PowerDNS Recursor (Authoritative had this since 3.0)
• RPZ in Recursor, tested to work with Spamhaus, Farsight Security and ThreatSTOP.
• IXFR slaving in Authoritative and Recursor (for RPZ)
• ODBC (Microsoft SQL Server & Azure) and LDAP backends are fully supported again in Authoritative
• Vastly improved Lua modules in Recursor, including the ability to asynchronously query reputation servers or databases (!)
• EDNS Client Subnet support in Recursor (Authoritative supported this in 3.x.x too)
• GEOIP backend enhanced, for example to support countries but also direct subnets for source dependent answers
• All caches can now be wiped for whole subtrees
• Powerful new metrics that point out performance and operational problems (fd usage, memory usage, network responsiveness, kernel dropped packets)
• ALIAS records so you can “CNAME your domain”, including DNSSEC support (as used by search.whitehouse.gov!)
• New pdnsutil commands like ‘pdnsutil edit-zone‘, create-zone, add-record, replace-rrset
• Halved query load on most database backends

Should I deploy PowerDNS 4.0.0?

Definitely. PowerDNS Authoritative Server 4.x.x and PowerDNS Recursor pre-releases are already widely deployed. All of us over at PowerDNS rely fully on the 4.0.0 version, and in fact find 3.x.x somewhat painful to use in comparison. We trust the code in 4.0.0 more.

In terms of performance, both Authoritative and Recursor look to offer higher peak performance than 3.x.x. We have performed extensive benchmarking on the Recursor, and reliably achieve 400kqps on “actual customer traffic”. For Authoritative, we note that 4.0.0 halves the database backend query load in many circumstances.

Enabling DNSSEC processing in Recursor 4.0.0 (the default) means slightly higher CPU utilization than 3.x.x. Turning on validation roughly doubles the CPU load.

What about DNSSEC validation?

DNSSEC does not make DNS any easier. Many DNS and DNSSEC enabled domains are misconfigured. Our trials indicate PowerDNS Recursor 4.0.0 will successfully validate all correctly configured domains (that we have tested). The bad news is that many domains, some important ones even, are not correctly configured.

Our advice for now is: turn on DNSSEC validation if you are prepared to spend time monitoring the log files for validation failures. And even as we improve our resilience against badly configured domains and work out issues, this advice will remain in place. DNSSEC validation, regardless of software used for it, requires monitoring. A useful option at this time is ‘log-fail‘, which will do the validation but only log the failures, and not block the answers.

Enough of this, how do I get my hands on the glorious PowerDNS 4.0.0 release mug?

As a small token of our appreciation, we have teamed up with MugBug to ship free PowerDNS 4.0.0 release mugs to anyone who was in any way part of the process. Uniquely, this giveaway extends to anyone deploying PowerDNS Authoritative Server 4.0.0 or PowerDNS Recursor 4.0.0 in the coming months!

So, apply for a free mug or even a set of mugs (if you are in an office), if you:

• Opened an issue relevant for PowerDNS 4.0.0 on GitHub
• Contributed code or a pull request that ended up in 4.0.0
• Supplied testing data (PCAPs) now or in the past
• Deployed PowerDNS 4.0.0 betas, release candidates, alphas or the technology preview
• Authored one of our dependencies
• Feel in any other way that you contributed to 4.0.0!

If you are part of a team, feel free to apply for mugs for the whole team. There is no need to send us your address details (since MugBug will do the actual logistics), but we do need to know who you are and what you did to be part of the PowerDNS community! Please email to powerdns-4.0-contributors@powerdns.com with your details (which we absolutely promise not to use in any other way than to authorize MugBug to send you your mugs!).

We’ve allocated a generous budget for the free mug giveaway, but it is limited – but we expect to be able to ship hundreds of mugs.

Finally

Thank you for your interest in PowerDNS Authoritative Server 4.0.0 and Recursor 4.0.0! Other blog posts have the full details and download links for the Authoritative Server and the Recursor.

Hi everybody,

Like last year, this year PowerDNS will again be part of the OX/Dovecot/PowerDNS summit. This time round we visit Frankfurt on the 13th and 14th of October. This is already in a few weeks!

All information is on: http://summit.open-xchange.com/oxs16-frankfurt.html

Many users of Dovecot, PowerDNS and AppSuite will be there. Specifically for PowerDNS, on Friday we will be hosting a 90 minute long session on malware filtering and parental control with DNS, with per-user settings, opt-in, opt-out, all with a single set of nameserver IP addresses.

Attendance is free! Please register here. When you register, you can also sign up for our malware session, which might even allow you to sell this trip to your company as ‘work’. The summit also involves (free) lunch and drinks.

If you are a PowerDNS user, or want to be, we hope to meet you there!

Way way back in the history of PowerDNS, we bought the full suite of domain names for our new company: powerdns.com, powerdns.net, powerdns.org. Alternate names for the company we considered at the time were ‘SuperDNS’ and ‘UltraDNS’. We later found out that SuperDNS was the internal name over at Verisign for what is now the Atlas software that powers the COM and NET servers. And UltraDNS eventually became a DNS company in its own right!

Over time we no longer used powerdns.org and eventually the domain lapsed by accident and was quickly picked up by folks that held it ransom for a decade or so, without ever using it. I (Bert) personally always had issues with paying up to get the domain back, and over the years some very unsavoury parties ended up owning powerdns.org.

Recently we decided it was time to get the .org back anyhow and after negotiating for a few days we finally paid up, and shortly after that we were back in control of powerdns.org, at a cost of $1000.

This personally left me with a bad aftertaste since effectively we have paid a chain of people that specialise in taking over domains for ransom purposes.

To compensate for all this, we’ve decided to donate $1000 to the Doctors without Borders charity. On doing the currency conversion I felt bad about that too, so we turned it into a €1000 donation.

So welcome back powerdns.org and hopefully we’ve atoned for our mistake a decade ago!