Problems with .de domains: What is known so far

The evening of May 5th was not a pleasant one for many administrators who manage services and websites with .de domains: Shortly before 10 p.m., monitoring systems sounded the alarm, customers and employees triggered support cases and troubleshooting began – websites were not accessible, apps did not work and VPN connections failed. However, the cause was not with service operators, but rather at a central point: in the Domain Name System (DNS) of the .de zone, or more precisely in their DNSSEC configuration.

DENIC eG is responsible for the configuration and has so far only provided brief information about the incident. On Wednesday the dust settled, the disruption was eliminated and some details about the events became clearer. A look at the DNS data also shows that only DENIC can explain the exact cause; a statement is still pending. To reconstruct the events, we examined the historical DNS records recorded by the dnsviz.net service.

What happened

DNSSEC is tasked with protecting DNS responses against manipulation using digital signatures. Without DNSSEC, attackers could forge responses by intercepting and altering traffic between the client and resolver. DNSSEC works with asymmetric cryptography, i.e. with key pairs made up of public and private keys. The public keys are stored in a DNSKEY type entry in the DNS. There are shorter zone signing keys (ZSK) for signing replies, which in turn are signed with a longer key signing key (KSK).

The integrity of a DNS response is checked step by step, starting from the root zone, whose keys the requester must trust. The root zone, digitally signed, points to the responsible name servers of the top level domain – in this specific case .de. If they provide a valid signed reference to the name server responsible for this domain, it will be questioned. If a signature along the way is broken, the entire chain is considered broken and name resolution fails. This is the desired behavior that protects against manipulation.

The zone signing keys are exchanged at the top level domain level at regular intervals. Because this is a central step with far-reaching consequences, it happens in several steps: On May 2nd, DENIC, as the person responsible for the .de zone, announced a new public key with the ID 33834. This happened in time for the new entry to get around in the DNS. Signing was not done with the new key for the time being; the old key 32911 took over. 33834 first appeared as a signing key on May 5th at 9:43 p.m. (7:43 p.m. UTC) in a signature (RRSIG) for the SOA entry of the .de zone. SOA stands for “Start of Authority”, the entry contains information about the zone itself. This signature was invalid for reasons that are still unclear. The data from dnsviz.com shows: At this time, all 6 responsible name servers delivered this defective signature with the key 33834.

At around 9:59 p.m. the first countermeasures were apparent: One of the name servers, n.de.net, from then on delivered a new RRSIG entry for the SOA entry with a valid signature, signed with the new key 33834. The other five servers continued to spread the false signature.

At 10:27 p.m. a new picture appeared: n.de.net delivered an invalid signature again, now a.nic.de and z.nic.de had valid entries with the old key 33834 – but different ones via IPv4 and IPv6. At the same time, z.nic.de also showed a broken signature. At 10:31 p.m. the next situation occurred, now five servers were able to sign correctly with the new key 33834 and only n.de.net, which had already achieved this feat, was wrong with its entry. Just three minutes later, everyone gave incorrect answers for a change, and different combinations came from the servers at short intervals, all of which were invalid.

By 10:50 p.m. most servers had agreed on a common valid signature with the old one, only n.de.net was further off the mark. This changed for the first time at 1:15 on May 6th (23:15 UTC), when all servers again had correct answers – although not yet perfect: a.nic.de and z.nic.de delivered two signatures in parallel, at least both were valid. At 1:17 the desired situation finally arrived: six name servers were able to generate a valid signature. Because it was not possible to convince all six name servers to sign with 33834 at the same time, everyone had switched to key 32911 at this point, so the key exchange was reversed. Nothing had changed until the afternoon of May 6th, and so far there had been no further attempt to reinsert key 33834.

What follows from this?

A look at the historical DNS data shows: There was something wrong with key 33834, at least in conjunction with SOA entries. Other DNS entries could be successfully signed at any time. Countermeasures began 15 minutes after the first invalid signature. Despite some confusion, it was not possible to use the key on all six servers for SOA signatures. In order to get the failure under control, it was decided to sign them again with the key 32911.

Such an outage due to a DNSSEC error is so far unique for .de domains. In 2022, the .se domain in Sweden experienced problems due to DNSSEC. Russia had a DNSSEC problem with .ru domains in 2024. The cause at that time was a keytag ID that was assigned twice.

DENIC is now responsible for identifying the cause of the problem and explaining what countermeasures have been taken. The question of why the signature problems were not already noticed in a test environment is also unanswered.

(jam)