OpenBSD version: might have helped... or not.
Arch:            Any
NSFP:            ffs... -.-'

I just had a rather not so fun encounter with the joys of routing loops. Yesterday, already, I saw a bit too many packets running circles in V4LESS-AS:

...
22:38:37.659748 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:8ed9:c1fc:448a:114:7fb7:4a79: ICMP6, echo request, id 50599, seq 21853, length 8
22:38:37.659748 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:cd62:c1fc:3fe1:a9b:2ef0:f541: ICMP6, echo request, id 27682, seq 53451, length 8
22:38:37.659748 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:ab46:c1fc:27d7:6863:2686:949: ICMP6, echo request, id 37253, seq 46574, length 8
22:38:37.659786 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:41dd:c4fc:57b5:6e40:3f7:50e8: ICMP6, echo request, id 35591, seq 37275, length 8
22:38:37.659800 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:612:c4fc:5019:c1f0:10e4:cf2c: ICMP6, echo request, id 48464, seq 3565, length 8
22:38:37.659810 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:8ed9:c1fc:448a:114:7fb7:4a79: ICMP6, echo request, id 50599, seq 21853, length 8
22:38:37.659814 IP6 240e:c2:1800:84:0:1:1:2 > 2a06:d1c3:cd62:c1fc:3fe1:a9b:2ef0:f541: ICMP6, echo request, id 27682, seq 53451, length 8
...

The rather obvious root-cause here is a combination of ‘high TTL’ (255) and ‘somebody’ (read: me) doing stupid things. In this case: having a routing loop.

Making your own routing loop

Routing loops are surprisingly common, and even easier to make yourself. All you need is two routers that think the corresponding other one is the next hop for a packet they want to forward. A common way to make that happen would be, for example, to install the covering prefixes you announce without a discard or blackhole-like statement.

Drawing from your IGP to find prefixes to route to, your border routers will happily take any packet destined for prefixes not in the IGP on a scenic tour through your AS.

Why this is annoying

Now, as you see up there, there seems to be an address sending a lot of v6 echo requests; My assumption is that this is some form of research work testing a new target generation algorithm for v6.

Well, whatever it is, even with just 8b sized ICMPv6 payloads, this can quickly go to 20mbit on an interface. If your routers route a bit more, it can be a bit more… happily stacking up over time.

After having experienced a bit of looping yesterday (and getting rid of it by making loops go away), I was playing around with what this can do.

Turns out, a single linux box behind a standard end-user access connection can easily ship 5gbit+ on a looping link:

for i in {1..1024}; do sudo ping6 -f -t 255 -s 1452  2001:db8:: & done;

Even a single ping6 -f -t 255 -s 1452 2001:db8:: put ~100mbit on a single link.

This, of course, is long since known, and has the simple solution of ‘do not have loops in your network’.

While I was playing around, I got a visit from 240e:c2:1800:84:0:1:1:2 again; However, this time in another network, with a bit more ‘loop-capability’:

This was, at least, ‘mildly annoying’. Luckily, our friends at 240e:c2:1800:84:0:1:1:2 are just using an 8b payload; Otherwise, I would have had ‘a few’ gbit more on that link.

What to do

Well, obviously, the solution is ‘do not have loops in your network’; In practice, though, this is often easier said than done (entropy, bodies, basement… you know). What I ultimately ended up doing for AS59645 was installing a central iBGP peer running bird that just injects blackhole routes for each ‘announced prefix -1 bit’ I have (assuming those do not collide with any other prefix). That way, traffic always finds a way… if only into /dev/null.

Still, annoying, and probably something to watch out for. -.-‘

OpenBSD version: might have helped
Arch:            Any
NSFP:            That seems to be debatable...

Disclaimer: I am writing this article as an individual member of the RIPE Community, and do not represent my employer. Furthermore, I am a person prone to ‘doing stupid things’, and a repeated offender when it comes to personal ASes: I have three. Furthermore, I am on friendly terms with several LIRs mentioned in the original article.

Today, an article about personal ASNs was published on RIPE Labs that has been ‘critically acclaimed’ in the community. It follows two talks at RIPE88, during which there was already an ‘engaged’ discussion around the propositions in the corresponding talks.

As I had already been rather vocal during the talks at RIPE88, it is not overly surprising that I have ‘some thoughts’ regarding the article. Well, keeping thoughts to yourself may make you less non-friends, but it is also less fun, so here we go.

The article itself

The article titled “Driving the ASN Truck Without a Licence” essentially revisits the arguments of the presentations held at the last meeting given the increasing number of personal ASNs. To summarize them:

• Personal ASes are not run ‘better’ than ‘traditional’ networks, but likely worse, causing issues; If they seem to be run better, it is because it is easier to do so at smaller scale, and also, a tunnel-based ASN is simply not relevant, even if it does RPKI correctly.
• Hobby networks do not help with IPv6 deployment, but instead hinder it, as individuals no longer need to pressure their ISP into rolling out v6.
• Virtual Internet Exchanges are useless and–along with all the tunnel ASes–degrade the MTU below 1500, leading to further issues.
• You do not need a ‘real’ personal ASN to get experience; Read a book or join dn42.
• LIRs misrepresent the policy to implement ‘shady business practices’ to push end-users to request resources
• IPv6 PA is being abused as a PI-ish resource, especially by those ‘shady business practice’ LIRs
• Personal ASNs are regularly leveraged for policy abuse, and to pollute public databases and protocols

In conclusion, the article calls for stronger restrictions on handing out personal ASNs, curbing “the business practices of some LIRs” promoting “irresponsible behaviour”, and making resources less accessible, ideally by making them more expensive.

The tone of the article

The article, in several points, tends to use an authoritative language. Looking at the comments that already came in also suggest that the article creates an impression of trying to gate-keep the Internet from those, mostly defined by their nature as natural persons wanting to hold resources, possibly unworthy but certainly unqualified for actually doing so.

I personally know the author, and know that there are some good intention (and partially even very sensible points) behind the article. Hence, while I personally agree with the statements on the phrasing, I will focus on engaging the underlying arguments in this post.

But first, a message from…

… the reason we are talking about this. Kind of: Me. I personally hold three different ASes, all of them ‘personal ASes’. I do acknowledge that my little personal AS might be ‘a bit bigger’ than the usual personal AS, including a few more gbit of 9000MTU L2 connectivity between PoPs, and a couple more big-name-brand routers than usually seen in personal ASes. Also, following the article’s definition, these are not really personal ASes, as I am also an LIR. However, I also sponsor some end-users’ PI/ASN resources… so, I think I can confidently say that I am ‘part of the problem’.

Still… the question remains… why would I need three ASes?

Note: AS211286 and AS215250 are funded via the RIPE Community fund and supported by contributions of various other RIPE Community members, including LWLcom, OpenFactory, DE-CIX, VirtuaCloud, and WIIT AG, see measurement.network.

AS59645: Where it all begins

The ‘root cause’ of me being an LIR is a combination of ‘mild frustration’ with the state of the Internet, academia, and the general realization that I kind of picked a day job that is more on the management and less on the ‘doing things with my hands’ side than is generally good for my well-being.

So, AS59645 is there to make it ping; Learn and automate things, and get the practice necessary to, say, work on some documents about the Internet.

AS211286: Crisis, Ethics, Reliability & a measurement.network

The next issue I stumbled upon was the aforementioned academia thing; Researchers doing researcher things (speaking as a researcher myself…) can kind of be… difficult… So, I set out and started building something, in an attempt to get ahead of the bike-shed; And well… for reasons.

AS215250: V4LESS-AS

As we already went about me having a thing for doing stupid things… it should be no surprise that I can have very weird ideas. Like… why not build an AS that does not have IPv4 on any of its routers.

Obviously, this is a rather great idea, and RFC8950 simply the future. Surprisingly, even some of the serious not-so-personal-ASes seem to think so.

Naturally, the only thing to do then is making something that allows people to try out RFC8950; Without actually having to break production infrastructure. Well, this is what V4LESS-AS does.

On the arguments for an ASN-Driving-License

So, with the reasons for which I have some ASes out of the way, let’s delve into the arguments around personal ASes made in the original blog article.

Make the Internet worse by being badly run

This argument centers around an (implicit) idea that ‘companies know better’. They will, more likely, run a better service, and employees do have to answer to management, and the company to customers if something goes wrong.

For personal ASes, though, this is not the case. Instead, they can do whatever, and if the Internet breaks… well. They do not need to care.

Or, to put it into a nice quote from a related talk:

“OSPF is amazing, because it allows you to break your own network in ways you do not understand. BGP on the other hand allows you to break everyone’s network in ways nobody understands.”

Good thing that Facebook never head to deal with a global BGP outage due to a misconfiguration, and there is not a single actual company that does anything remotely shady with their AS.

I am skipping on examples for the latter to protect the guilty; But you know who you are.

Also, running your ISP from a couple of re-flashed 100G switches tends to be a thing as well… And the number of ISPs without any form of filtering I have seen…

Well, I guess the point is clear. And I did not even get into the point that all this tunnel stuff… Well, MPLS is not exactly not tunneled, is it? (Keeps starring in BGP-free core…)

Does not help but hinder IPv6 Deployment

The argument around IPv6 adoption is a bit… weird. While, yes, you should not just get an AS because you need v6, you could use the same argument around HE’s tunnel broker service.

In turn, you could also argue that HE’s tunnel broker service is a better option anyway.

Ultimately, this creates an impression of a bit of straw burning…

Virtual Internet Exchanges are useless

Well, besides the whole ‘tunnels are bad’ issue–given that basically everything these days goes through a lasagna of tunnels–there is the point of these things being practically pointless. This is something I very much agree with (even though I do connect to some of these Toaster-IXes); Still, there is a lot of self-inflicted foot-shooting going on on these… and I mean… better shoot yourself in the foot with a toaster than with a truck. Or something like that.

Besides that, there is of course the MTU argument; However… to be frank… it is not like actual big players aren’t doing things that may er… restrict the MTU to say… 1492b, no?

Just go for DN42

Again, the recommendation to, instead, go learning in a very much enclosed environment pretty much holds true, and is one I gave out myself (and will happily give out again). However, here, there is not much of an argument left beyond you also could do similar stuff elsewhere. And honestly, part of the appeal of the personal AS is that you can make yourself eat your own dogfood; Which usually works wonders on service quality and learning outcomes.

There are LIRs with ‘shady business practices’ pushing Personal ASes

This point is, in my opinion, ‘somewhat difficult and maybe not necessarily ideally phrased’, up to a point of ‘maybe attributing a bit too much malice to third parties’. Note, though, I am on rather friendly terms with mentioned entities, and they also support measurement.network.

The issue here is that a lot of things are going into the same bucket. To grab the reference to freetransit.ch as an example, the article suggests that freetransit.ch is “not respecting and enforcing current policies” of the RIPE region. Instead they execute “shady practices” like “offering ASNs and IPs to children (“Minors can still request resources!”)”, all in a bit to “help themselves”.

I would argue that the argumentation taken here is, at least, severely worrisome. First, the insinuation that ASNs and IPs are marketed to children is an arguable stretch for a checkbox in the contact form inquiring whether the sender is able to legally sign legal documents, i.e., not a minor:

The statement thereunder is also not something “not respecting and enforcing current policies”. In fact, it is pretty much exactly what RIPE-637 is saying about contractual relationships: There has to be one.

RIPE-812 also clarifies that a member can be “A natural person or a legal entity that has entered into the RIPE NCC Standard Service Agreement with the RIPE NCC.” There is no need for a member to be (at least depending on jurisdiction, but generally for the EU, IANAL) 18. A 17 year old can very much become a member, if they enter into a legal relationship with the RIPE NCC. Usually, this will require consent and signature from their legal guardian. At least that is my rather naive non-lawyer reading of the absence of any age restrictions in RIPE-812. (Also: How else would minors otherwise join the local football club of a small town in northern Friesland; It, after all, has the same legal structure as RIPE…) Hence, I see no reason why this could not be the case for an end user’s relationship to a sponsoring LIR.

This leaves the whole issue of framing a statement around minors to children out of the picture; I am not making a value statement regarding the contents, but do note that I find using the framing to be a discussion style I personally do not necessarily associate with content-directed argumentation.

Finally, there is the argument that this is being done for ‘own gains’ by these LIRs; (Note: I am not contesting that there also are some actors trying to leverage end-users to create a (quick) profit; In fact, I am pretty sure there are some; Just not necessarily the specific examples selected here.)

However, again going back to the explicitly chosen example, using an LIR invoicing customers a yearly re-occuring cost very close to the actual cost billed by the NCC (EUR145 YRC for ASN+PI, i.e., a ‘profit’ of EUR20… which still includes VAT and covering costs) is ‘somewhat a stretch’. Considering how much time usually goes into a registration process (KYC, contract, interaction with the NCC etc.), I would argue that this is, best case, covering the actual costs, even when considering the higher fee billed during the first year.

Hence, overall… especially this argument feels… difficult.

IPv6 PA is being abused for PI

Subsequently, the argument is made that, indeed, some LIRs are also offering end-users to use PA instead of applying for PA. In general, the argument vaguely alludes to this being missuse of resources.

My point here, first and foremost, would be that there are indeed some issues with the current IPv6 assignment policy that lead to not ideal effects.

However, I would also argue that it might make sense to, instead, participate in the policy-development-process in order to actually do something about the underlying operational issues.

Personal ASNs are used for Policy Abuse and to pollute databases/the GRT

The final argument revolves around an argument of PI/personal ASN resources being used for abusing the policy; It leads with the argument of a person requesting multiple ASNs for multiple projects, noting that it is rather unlikely that the person actually needs that many ASNs; I feel somewhat called-out. ;-)

However, argumentatively, this section feels a bit like a lot of straw burning again. After all, for any personal AS used for nefarious stuff, I can likely name an AS registered to a ‘real’ company or even LIR that is no less engaged in annoying or directly malivious activity.

Similarly, I think that the author of the initial article is insanely lucky that no Secret-WG does not even not not exist, that may or may not even have the audacity to pollute the database with limericks. Otherwise, rather black helicopters would already be circling overhead, given this straight out attack on such a non-existing organization.

And knowing the (PI and personal ASN holding) person who put in the XSS referenced (and was the one thereby finding an issue with the web edits tooling in the process, reporting it, and getting it fixed)… I am not sure whether it makes it more of an argument for personal ASNs.

Conclusion

In conclusion, the arguments presented in the RIPE Labs post are not able to convince this revie^H^H^H^H^Hmember. Despite acknowledging that there are serious issues around some of these developments, and that we certainly do need more care in the operation of the Internet, we also need a wide availability of operational experience currently often far too lacking even for professional companies. Picking on some of the LIRs actually rather engaged in poking the end users they sponsor to do (and learn to do) the right thing may also be… ‘not necessarily the ideal approach’ toward improving the situation. And–based on my engagement with the arguments–I, indeed, find myself seeing how commenters reached the conclusion that the article creates an imperession of ‘gate keeping’.

So, as a herder of cats, reading the conclusion, I kind of do not feel too bad about being the one to put the ‘Mau’ into (“silly smöl meow meow”[sic]) networks. And I think I will continue doing that; Responsibly, for the good of the Internet. For it to become again the open distributed end-to-end infrastructure ultimately owned by noone, enabling equitable participation, expression and learning which it was once envisioned to be; Making sure that those I take responsibility for do not break the Internet. Well, at least I will keep trying that (as well as not breaking it myself).

(Final Note: Yes, I know, AS59645 also sometimes does stupid things, like leaking routes because of an algorithmic error in community handling for exported prefixes leading to an overloaded router, which collided with FRR’s non-atomic config application; But hey, at least that motivated me ultimately to take a shot at updating BCP194. Thee who is without any weird configuration body in their basement, throweth the first depeering; Or something like that.)

OpenBSD version: might have helped
Arch:            Any
NSFP:            Well... obviously not.

“It has been 0 days since it was DNS.”

Last Monday I noticed some of my nagios checks going critical. Specifically, it was the checks for the anycasted DNS recursors testing Momoka’s draft on v4 resolution for v6 resolvers that would try to resolve tudelft.nl. Why tudelft.nl? Well, this zone is reliably free of any IPv6 support. At the same time, various projects I operate were unable to deliver mail to users @tudelft.nl. The reason here being that the domain was not resolvable.

I initially did not really want to write about this incident, but given that the number of people who asked me about this, given my background in a) operations, and b) research about “exactly this” has slowly reached “too many”, I figured this might be easier.

Disclaimer

I am writing this text as a system and network engineer and scientist working on the subject of digital infrastructure operations. Statements, suggestions, and conclusions presented in this document are based on my practical experience and scientific findings around running digital infrastructure in a higher-education/research network context. My statements do not necessarily represent the official position of my employer or any other affiliation I hold, and are, as such, my own.

I note that I was employed by TU Delft until 2022-03-31 and held a hospitation agreement with TU Delft up until 2024-03-31. However, I was never part of the operational staff involved with the digital infrastructure of TU Delft. At this point in time, I am in no way affiliated with TU Delft. As such, I do not have any privileged insight or information on the operational efforts in general or the incident at hand specifically.

All information provided and statements made in this document are either the result of analysis informed by public information, or conjecture based on common operational practices. This document does not contain confidential or proprietary information of TU Delft.

The purpose of this document is the open discussion of challenges in the operation of digital infrastructure for higher education institutions. Observed deviations from operational best practices are only stated when independently verifiable. Conjecture is marked as such.

Timeline of Events

At the moment, I see the following timeline of events in relation to the events at hand, all times UTC:

• 2024-05-28T02:19:24: monitoring for n64v6res01.ber01.as59645.net notified that ?IN A tudelft.nl cannot be resolved. n64v6res01.ber01.as59645.net first notified at 2024-05-28T02:28:32, and finally n64v6res01.ams01.as59645.net notified at 2024-05-28T03:18:50.
• 2024-05-28T11:41:10: Monitoring notes that tudelft.nl is resolvable again on n64v6res01.dus01.as59645.net, n64v6res01.ber01.as59645.net follows at 2024-05-28T11:44:06. n64v6res01.ams01.as59645.net was already able to resolve tudelft.nl at 2024-05-28T06:12:59.
• 2024-05-28: TU Delft ICT published a notification on https://meldingen-ict.tudelft.nl/en/, noting ongoing issues with DNS which are being investigated. This message has since been removed.
• 2024-05-30: Issues with resolving tudelft.nl persist intermittently. It appears that the DNSSEC configuration of tudelft.nl has been misconfigured, i.e., new keys were added on the authoritative servers, but the DS records in .nl were not updated.
• 2024-05-31: While intermittent reachability issues continue. Meanwhile, TU Delta reported that the attack from 2024-05-28 reached 2.8 trillion requests per half hour. A former colleague also informally noted that, apparently, during the attack external connectivity was briefly interrupted by ICT to be able to work on further mitigating the attack.
• 2024-06-01: tudelft.nl seems to have partially recovered.
• 2024-06-01: Around 13:00, DNS resolution seems to return to normal for most clients. However, DNSViz still reports the zone to be bogus. Based on that data, it appears that tudelft.nl currently responds with RRSIGs signed with keyid 135, while .nl only holds DS records for keyid 47965 and keyid 28945, with neither signing 135. Furthermore, responses from the tudelft.nl namesevers still have a high RTT, sometimes timing out.

What might be going on…

The question now is: What really happened, and why is it not yet fixed? Let’s delve into that, shall we?

The nature of the DoS

The root cause of these events seems to be the DoS attack from Monday. This has been claimed to have caused 2.8 trillion requests per 30 minutes. Now, this may mean 1,000,000,000,000 (10^12) or 1,000,000,000,000,000,000 (10^18) requests depending on the definition of trillion.

This allows us to gauge the bandwidth that must have come in per second, on average. A request for ?IN A tudelft.nl has, using dig A tudelft.nl @192.0.2.1, 79 bytes. For 10^12, this makes ca. 351.11 Gbit/s on average, for 10^18 it makes ca. 351.11 Pbit/s; Mildly unlikely.

Even the avg. 351.11 Gbit/s are somewhat unlikely to have reached the servers, given that it would require a campus uplink of at least 800 Gbit/s (considering that such small packets come with a lot of overhead) if the statement that the servers had to process this amount of requests also should hold. Then again, these stats might also have been collected upstream, i.e., in the SURF backbone.

In any case, this is obviously a lot of data, and nothing overly easy to handle.

The Authoritative DNS Servers of tudelft.nl

RFC2182 has some opinions on running authoritative nameservers; or rather, the second one for a zone, and says in 3.1:

   When selecting secondary servers, attention should be given to the
   various likely failure modes.  Servers should be placed so that it is
   likely that at least one server will be available to all significant
   parts of the Internet, for any likely failure.

Now, how does this look for tudelft.nl:

% dig NS tudelft.nl @ns1.dns.nl

; <<>> DiG 9.16.48 <<>> NS tudelft.nl @ns1.dns.nl
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17759
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;tudelft.nl.			IN	NS

;; AUTHORITY SECTION:
tudelft.nl.		3600	IN	NS	ns1.tudelft.nl.
tudelft.nl.		3600	IN	NS	ns2.tudelft.nl.

;; ADDITIONAL SECTION:
ns1.tudelft.nl.		3600	IN	A	130.161.180.1
ns2.tudelft.nl.		3600	IN	A	130.161.180.65

;; Query time: 30 msec
;; SERVER: 2001:678:2c:0:194:0:28:53#53(2001:678:2c:0:194:0:28:53)
;; WHEN: Sat Jun 01 17:31:06 CEST 2024
;; MSG SIZE  rcvd: 107

Well. Those two are certainly in the same /24. No matter how geographically distributed the setup is, this does not fulfill the requirements of RFC2182 or RFC3258. And this has been like that for… some… time. And, incidentally, still is like that.

How it all Comes Together

Based on the observations above, I make the following conjectures concerning the events that transpired in response to the DoS attack.

• I assume that the authoritative DNS servers of tudelft.nl were the primary target of the attack, given that TU Delft ICT first reported to be investigating an issue related to their DNS servers.
• As the nameservers of tudelft.nl are single-homed behind its campus network, disconecting or otherwise severing, i.e., via the DoS, the campus network from the rest of the Internet makes tudelft.nl unreachable, also making, e.g., emails undeliverable.
• In response to the observed attack, TU Delft ICT hopefully asked SURF to ACL, e.g., via Flowspec, certain routes into their network.
• As there was still notable inbound requests hitting the authoritative NS, ICT decided to either replace or provision an additional authoritative DNS worker. Alternatively, possibly a middle box or DNS filtering solution was brought in.

Together, the above steps restored operations, especially after the DoS attack subsided a bit. Issues then resurfaced after roughly 48 hours when the TTL of 172800 seconds for the DNSKEY records of tudelft.nl expired on more and more recursive resolvers.

Possibly, during the change (adding or upgrading a worker/traffic filter) a new ZSK DNSSEC key (keyid=135) was introduced. Based on the DNSViz resolutions, it seems like this DNSKEY is not always delivered with an RRSIG signed by one of the keys for which a DS record is installed in .nl, even though it seems like it should be signed by keyid=47965. Nevertheless, it not always is, and the keyid was not present earlier this year. which might have benign reasons, e.g., a ZSK rollover.

Similarly, it might be that those queries with those specific rrsigs/ DNSKEY records simply get eaten somewhere on-path or enjoy just being dropped. This may be the result of a misconfiguration, the result of ongoing DoS, or caused by the introduction of specific defense mechanisms, e.g., heavy per-client rate limits.

In any case, something is still broken, and this heavily impacts reachability for tudelft.nl in the DNS.

Update 2024-06-02: The more I think about this, the more this feels like the result of ratelimiting; Which will likely also impact large resolvers (q1/q8/q9) less, because they will re-resolve from different IPs. Also makes sense given 135 is a zsk, and the other two are ksk.

Absence of Monitoring

Given the prolongued nature of the DNSSEC issues, I would argue that TU Delft ICT might not be fully aware of the issue, may not yet have identified the root cause, or may face challenges in fixing the specific DNS server implementation they are running to present properly signed zones.

What could have been done

Now, it is always easy (and mildly fun) to shout at the TV when a football game is going on. So, being a person that likes easy things, I will go for that. What would I have done seeing a ton of packets coming to my authoritative NS all of a sudden?

Do not run NS in a single /25

The obvious first step is not having all NS in a single /25 (yes, ‘twenty-five’), and most certainly not all of them on campus. Ideally, also more than just two. Instead, I would have tried to find multiple secondaries, likely just ingesting AXFR, across different networks. Ideally also one anycasted.

This would already have sufficed to ensure that email delivery is not impacted, and all cloud hosted services remain accessible.

DoS Defense

Assuming that NS are also off-campus in other netblocks with unique routing policies would have made it a lot easier to very blanket-ly block, e.g., inbound packets with dport udp/53 and tcp/53; Flowspec comes in handy there, and SURF would likely have been happy to assist there.

There are also some options with the current setup of ‘one’ DNS server network. Depending on what else lives in those /25 (ideally not much but well… ), one could have drawn an AXFR from the authoritative and started serving the zone from the /24 now being anycasted.

That would have required surf to actually set a ROUTE object, and ideally configure a ROA in RPKI; Furthermore, one would have had to convince SURF to borrow one of their (spare) ASNs. Hence, in general, this is more of a fun thing to do with a bit more time than ‘everything burns’.

Monitor Stuff

I noted above that there is some monitoring likely… missing, given that the issue still persists. Which is ‘not really ideal’. So getting that in place would also be a top priority.

Now, something one could rather easily do would be setting up some RIPE Atlas measurements; Like these ones for mpg.de:

• IPv4 UDP: https://atlas.ripe.net/measurementdetail/72293568/
• IPv4 TCP: https://atlas.ripe.net/measurementdetail/72293620/
• IPv6 UDP: https://atlas.ripe.net/measurementdetail/72293583/
• IPv6 TCP: https://atlas.ripe.net/measurementdetail/72293641/

That can then be easily polled by some off-site monitoring setup. Such a monitor can then also check a lot of other services, especially end-to-end things like “Can a user send an email, does it arrive somewhere else, and can its DKIM signature be verified then?”

Conclusion

There can be broad speculation as to why this is now happening to TU Delft, and why DNS(SEC) does not really want to play nice at the moment. In my scientific work, there are some conjectures that an increasing introduction of a cloud first strategy can lead to a capability errosion, especially for basic services, due to an organization realizing cost savings by not maintaining or actively reducing in-house capacity for, especially, basic services. Still, even though TU Delft is following a very cloud focused approach, finding a conclusive answer there would require perspectives from inside the organization, which I do not have.

Hence, what remains is me being hopefull that ICT is aware of the underlying issue, and am convinced that the team members do their best to resolve the incident.

From a technical perspective, the issues should get a lot better by adding new authoritative servers, i.e., simple AXFR’ing secondaries. I am not sure why this is not being done (not only now, but in the years before). It might be, for example, the complexity of the current setup, or, simply that organizational knowledge on the setup was lost over time;

However, as a person who had to emergency migrate a production setup to a PowerDNS authoritative in a few hours… I am pretty sure that it is not too difficult to:

• Get a few additional servers in different providers
• Make sure that there exists one complete copy of the zone, i.e., an AXFR that is properly signed and contains all necesarry DNS key records
• Optional: Add a new DS to tudelft.nl via the registrar
• Throw NSD, Knot, Bind9, or PowerDNS on the servers with a configuration that can handle ~7k qps
• Make the registrar add those new NS (+glue) to .nl
• Bonus: Get additional NS under different TLDs for extra redundancy

But I guess things are just looking a bit too simple from the seat of a professional commentator: With the remote to the right, and a bowl of chips to the left, giving good advice from afar. And with that, I grab another beverage, and murmor at the TV: “Well, maybe you should have not tried to drive a golf car through a race track!”; Not realizing that I might be watching Golf instead.

OpenBSD version: doesn't really matter
Arch:            Any
NSFP:            More of it, please

Somewhen in 2022, is complained a bit about the impact DMARC can have on mailinglists. Specifically, I was struggling a bit with OpenBSD’s mailinglists stripping DKIM while–obviously–breaking SPF. With my p=reject DMARC policy, this meant that sending to OpenBSD mailinglists would ‘reduce’ the number of people who would actually get my mails.

Back then, i solved that by registering reads-this-mailinglist.com, which let me have a domain with a significantly relaxed policy (and the OpenBSD list hosts included in SPF), to make my mails deliver again when traversing through the OpenBSD mailinglists.

However, over the past two years, the number of mails i (or rather my mailserver) kept rejecting from OpenBSD kept constantly growing, also because of–in most cases–DMARC.

The list changing

I hence decided to complain again (well, you gotta do what you’re good at, no?) on misc@. However, complaining actually had an effect this time around, as Todd C. Miller was very quick to take action this time around, and implemented from-rewriting for addresses that have a DMARC policies for the OpenBSD majordomo.

And, what shall i say, it works! I of course (accidentally) tested that right away, because i slipped and sent my reply from the wrong (strict p=reject) address; However, with the change implemented, this now worked flawlessly! :-)

So, thanks Todd!

OpenBSD Version: -
Arch:            All
NSFP:            Uff...

With the recent announcement of RCE in Exim, and patches still being pending, it might be opportune to move existing Exim setups behind another frontend. Postfix, for example.

In this article, I will describe methods to a) add an additional inbound MX using postfix, and b) enabling authenticated email sending with postfix for existing Exim environments.

Transparent additional MX / Inbound Email

Requirements

- Clean VM with Debian 12
- IPv4/IPv6 Address for that box with inbound port tcp/25
- Existing Exim MXes

Initial Situation

DNS:
example.com. IN MX 50 mx01.example.com.
example.com. IN MX 75 mx02.example.com.
mx01.example.com. IN A 203.0.113.5
mx02.example.com. IN A 192.0.2.23

Furthermore, both mx01.example.com and mx02.example.com run a vulnerable Exim.

Configuring mx03.example.com

DNS:
example.com. IN MX 50 mx01.example.com.
example.com. IN MX 75 mx02.example.com.
example.com. IN MX 99 mx03.example.com. ; Add this entry
mx01.example.com. IN A 203.0.113.5
mx02.example.com. IN A 192.0.2.23
mx03.example.com. IN A 198.51.100.12 ; Add this entry

The system should be called mx03.example.com!

# hostname -f
mx03.example.com

Install required software

# apt-get install python3-certbot certbot postfix

Obtain TLS certificates

# certbot certonly --standalone --preferred-challenges http --agree-tos --email contact@example.com -d `hostname -f`

Configure postfix (edit /etc/postfix/main.cf)

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on
# fresh installs.
compatibility_level = 3.6



## TLS parameters
# IMPORTANT: Update the path to fit your environment/names!
smtpd_tls_cert_file=/etc/letsencrypt/live/mx03.example.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/mx03.example.com/privkey.pem
smtpd_tls_security_level=may
# This could be better; But well.
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1
smtpd_tls_loglevel = 1

## Enforce TLS to primary; We should have TLS setup, no? If not, set:
# smtp_tls_security_level=may
smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=verify
smtp_tls_verify_cert_match = hostname, nexthop, dot-nexthop
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_loglevel = 1

## Use reject_unverified_recipient to enforce validation of deliverability to prevent backscatter
# messages are denied with 450, i.e., non perm-error
smtpd_relay_restrictions = permit_mynetworks reject_unverified_recipient defer_unauth_destination
myhostname = mx03.example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname, mx03.example.com, localhost.example.com, localhost

## Relay configuration; Add your domains under relay domains
relay_domains = example.com, example.net
# You can optionally hard-code the relay host you want to use; Usually this should
# not be necessary, though.
# relayhost = [mx01.example.com]:25
# Enable soft bouncing, i.e., do not permanently reject mail to ensure no messages
# are lost.
soft_bounce = yes

mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all

Restart postfix

# service postfix restart

Test mail sending

In one terminal, run:

# journalctl -f|grep postfix

Then, from your host, use the following script; Please update sender/recipient addresses as appropriate:

#!/usr/bin/env python3
import sys
import time
import datetime
import hashlib
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

def send_mail(msg):
        srvr = 'mx03.example.com'
        port = 25
        sndr = 'My Sender <test@example.com>'

        s = smtplib.SMTP(host=srvr, port=port)
        # Uncomment this if you did not configure TLS
        s.starttls()
        s.send_message(msg)

def create_message(dst='dst@example.com'):
        msg = MIMEMultipart()
        msg['To']='<'+dst+'>'
        msg['From']='My Sender <test@example.com>'
        msg['Subject']="Test mail"
        msg_text = "Test mail"
        msg.attach(MIMEText(msg_text, 'plain'))

        return msg


msg = create_message()
send_mail(msg)

If mail sending works, you should see the mail accepted and relayed in the log, and ultimately find it in the right inbox wherever these are stored.

Firewalling and Filtering

Next, you must limit access to port tcp/25 on mx01.example.com and mx02.example.com, while still allowing access from mx03.example.com. How you can do this depends on your setup. The easiest way will likely be using iptables, or rules in your local firewall, see the picture below.

Furthermore, you should make sure that mx01.example.com and mx02.example.com accept emails forwarded by mx03.example.com regardless of their SPF status/spam filtering etc.; This is, of course, not ideal, but the purpose of this activity is mainly to survive until exim patches are out. “Readers are advised to conduct personal risk assessment at their own discretion.”

Caveats

There are some things to keep in mind:

• If you use DANE/TLSA, you will have to add a record for mx03.example.com
• If you use MTA-STS, you need to update your MTA-STS policy

Transparent Authenticated Mail Relaying

To get authenticated mail relaying on a temporary postfix proxy, we will use the IMAP plugin for saslauthd to seamlessly integrate authentication for our proxy.

Requirements

- Clean VM with Debian 12
- IPv4/IPv6 Address for that box with inbound port tcp/465 and tcp/587 in the network segment in which the authenticated relay resides!
- Existing Exim relays
- An IMAP server with the same(!) user/pass syntax as your mail servers!

Initial Situation

DNS:
smtp-auth01.example.com. IN A 198.51.100.10
imap.example.com. IN A 192.0.2.23

Here, smtp-auth01.example.com runs a vulnerable Exim.

Configuring smtp-auth02.example.com

DNS:
smtp-auth01.example.com. IN A 198.51.100.10
smtp-auth02.example.com. IN A 198.51.100.12
imap.example.com. IN A 192.0.2.23

The new system should be called smtp-auth02.example.com.

# hostname -f
smtp-auth02.example.com

Install required software

# apt-get install postfix libsasl2-modules sasl2-bin stunnel net-tools

Obtain TLS certificates

Copy the TLS certificates of smtp-auth01.example.com over to smtp-auth02.example.com.

Setup stunnel for IMAP

Sadly, the rimap feature of saslauthd does not support TLS. Hence, to use it, we must use stunnel to connect to the remote IMAP server and bind a non-TLS port to localhost.

Add the following to /etc/stunnel/imap_example_com.conf:

setuid = stunnel4
setgid = stunnel4

#pid = /var/run/stunnel.pid

[exmpl-imap]
client = yes
accept = 127.0.0.1:143
connect = imap.example.com:993
verifyChain = yes
CApath = /etc/ssl/certs
checkHost = imap.example.com
OCSPaia = yes

After adding the config file, restart stunnel:

service stunnel4 restart

Thereafter, stunnel should be listening on lo:

# netstat -an | grep 143
tcp        0      0 127.0.0.1:143           0.0.0.0:*               LISTEN

Setup SASL authentication

Next, we have to configure saslauthd. There is a good summary on the tool in the debian wiki;

To make it quick:

Create /etc/postfix/sasl/smtpd.conf (we are assuming you only support login and plain):

pwcheck_method: saslauthd
mech_list: LOGIN PLAIN

Create /etc/default/saslauthd-postfix:

DESC="Postfix reverse IMAP SASL Authentication Daemon"
NAME="saslauthd-postfix"
START=yes
MECHANISMS="rimap"
MECH_OPTIONS="127.0.0.1"
THREADS=5
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Setup SASL requirements

Create a missing directory and add postfix to the sasl group:

# dpkg-statoverride --add root sasl 710 /var/spool/postfix/var/run/saslauthd
# adduser postfix sasl

Restart and test SASL authd

# service saslauthd restart

After starting saslauthd, you should not only see it running in ps aux, but should also be able to authenticate.

# testsaslauthd -u validmailuser -p validpassword -f /var/spool/postfix/var/run/saslauthd/mux
0: OK "Success."

While an invalid password should of course not work:

# testsaslauthd -u validmailuser -p invalidpassword -f /var/spool/postfix/var/run/saslauthd/mux
0: NO "authentication failed"

Configure postfix (edit /etc/postfix/main.cf)

Be aware, we are making this look like it is smtp-auth01!

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on
# fresh installs.
compatibility_level = 3.6



## TLS parameters
smtpd_tls_cert_file=/etc/letsencrypt/live/smtp-auth01.example.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/smtp-auth01.example.com/privkey.pem
smtpd_tls_security_level=may
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1
smtpd_tls_loglevel = 1

## Enforce TLS to primary
smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=verify
smtp_tls_verify_cert_match = hostname, nexthop, dot-nexthop
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_loglevel = 1

## Use reject_unverified_recipient to enforce validation of deliverability to prevent backscatter
# messages are denied with 450, i.e., non perm-error
smtpd_relay_restrictions = permit_mynetworks reject_unverified_recipient defer_unauth_destination
myhostname = smtp-auth01.example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname, smtp-auth01.example.com, localhost.example.com, localhost

## SASL configuration
cyrus_sasl_config_path = /etc/postfix/sasl
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_auth_enable = no
broken_sasl_auth_clients = yes
smtpd_sasl_security_options = noanonymous

## Relay configuration
relayhost = smtp-auth01.example.com
# Enable soft bouncing, i.e., do not permanently reject mail to ensure no messages
# are lost.
soft_bounce = yes

## Restriction classes for submission/smtps
smtpd_restriction_classes = mua_sender_restrictions, mua_client_restrictions, mua_helo_restrictions
mua_client_restrictions = permit_sasl_authenticated, reject
mua_sender_restrictions = permit_sasl_authenticated, reject
mua_helo_restrictions = permit

mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all

We have multiple options for handling re-injecting authenticated emails. What we are using here is relaying to smtp-auth01.example.com on port tcp/25, assuming that this port is available from smtp-auth02.example.com. However, we could also use a dedicated system account for that.

For configuring postfix with authenticated relaying, please see other resources.

Configure postfix (edit /etc/postfix/master.cf)

In addition to configuring the basic postfix features outlined above, we also need to enable tcp/465 (smtps) and tcp/587 (submission). We can do that in /etc/postfix/master.cf, by adding the following lines:

submission     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=$mua_client_restrictions
  -o smtpd_helo_restrictions=$mua_helo_restrictions
  -o smtpd_sender_restrictions=$mua_sender_restrictions
  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=$mua_client_restrictions
  -o smtpd_helo_restrictions=$mua_helo_restrictions
  -o smtpd_sender_restrictions=$mua_sender_restrictions
  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject

Restart and Test postfix

With everything in place, we can restart postfix:

# service postfix restart

If everything comes up clean, we can redirect the mail submission ports to this host (see the figure in the beginning). Technically, you might want to test whether auth+relay actually works before doing so (with the script below); But we are just very trusty people around here, so we guess it works. Also, testing it now would be difficult, as the TLS certificate currently does not match (remember: we took the one from smtp-auth01.example.com). Of course, you should also make sure that there are no other open exim ports.

When the redirect has been put in place, and we are sure mails are successfully pushed to the ‘original’ mail relay (so our hopefully existing toolchain involving dkim etc. does not get bypassed), we can test it with the following script (again, please double check the mail addresses/credentials to make them fit your infra):

#!/usr/bin/env python3
import sys
import time
import datetime
import hashlib
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

def send_mail(msg):
        srvr = 'smtp-auth01.example.com'
        port = 465
        user = 'mailuser'
        pswd = 'securepassword'
        #pswd = 'awd'
        sndr = 'Mail Sender <external-recipient@example.net>'

        if port == 465:
                s = smtplib.SMTP_SSL(host=srvr, port=port)
                s.login(user, pswd)
        else:
                s = smtplib.SMTP(host=srvr, port=port)
                s.starttls()
                s.login(user, pswd)
        s.send_message(msg)

def create_message(dst='dst@example.com'):

        msg = MIMEMultipart()
        msg['To']='<'+dst+'>'
        msg['From']='Mail Sender <external-recipient@example.net>'
        msg['Subject']="Test mail"
        msg_text = "Test mail"
        msg.attach(MIMEText(msg_text, 'plain'))

        return msg


msg = create_message()
send_mail(msg)

If everything was setup correctly, you should see your mail arriving at the correct destination, and find it having traversed smtp-auth02.example.com and then smtp-auth01.example.com.

It of course always makes sense to test whether mail sending works with https://email-security-scans.org/.

Summary

So, that’s it; Two ways of hiding exim behind postfix.

OpenBSD Version: Erm... keine. o.O
Arch:            Alle
NSFP:            Ja, ne.

Disclaimer: Der Text ist mehr oder minder auto-uebersetzt, mit einigen manuellen aenderungen damit es nicht gaaaaanz so schlimm ist. Ich empfehle den englischsprachigen Artikel, falls du auch Englisch sprichst (oder zumindest liest).

Die Mastodon-Instanz von Digitalcourage e.V. hatte in den letzten paar Monaten ein paar Netzwerkproblemen. User konnten Dateien nur mit ca. 100KB/s von der Instanz downloaden, und alles war irgendwie langsam. Das war natürlich eher ‘nicht so gut’.

Ueber die Zeit zeichnete sich in den Rueckmeldungen von Usern ein klares Muster ab. Konkret schienen hauptsächlich User der Deutschen Telekom betroffen zu sein, während andere in der Lage waren, Dateien mit der erwarteten Geschwindigkeit abzurufen. Die Admins von Digital Courage hatten sogar versucht, die IP-Adresse des Systems zu ändern: Vergeblich. Interessanterweise schnitt ein anderes System direkt neben der Mastodon-Instanz bei digitalcourage.social ziemlich gut ab, selbst für ansonsten betroffene Deutsche Telekom-User. Letztendlich führte dies zu einer ziemlich ausführlichen Diskussion darüber, ob die Deutsche Telekom den Datenverkehr zu den Servern von Digitalcourage möglicherweise etwas ungleicher behandelt, als sie sollte.

Ich bin letzten Mittwoch (9. November 2022) auf die Diskussion aufmerksam geworden, und das Probleme hat es mir irgendwie angetan. Ueber einen Bekannten bei der Deutschen Telekom, der bereits an der Fehlersuche beteiligt war, kam ich mit den Digitalcourage-Leuten in Kontakt, und schlug vor zu schauen, ob ich eventuell helfen kann. Diese fanden die Idee nicht schlecht, und somit konnte der Debug-Spasz beginnen.

Da es sicher einige interessiert was nun wirklich passiert ist (und das wirklich habe ich auch erst heute nacht herausgefunden), dachte ich, ich schreib mal einen Blogartikel.

DEBUG: Start

Der erste Schritt beim Debuggen eines solchen Problems, ist natürlich das Problem zu reproduzieren. Für den vorliegenden Fall war dies überraschend einfach. Als ich versuchte, eine Testdatei an meinem Rechner herunterzuladen, sah das nicht so gut aus:

% wget -O /dev/null https://digitalcourage.social/1GB.bin
--2022-11-09 18:53:17--  https://digitalcourage.social/1GB.bin
Resolving digitalcourage.social (digitalcourage.social)... 217.197.90.87
Connecting to digitalcourage.social (digitalcourage.social)|217.197.90.87|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null             0%[                    ] 615.70K  98.8KB/s    eta 2h 53m 

Hier ist das ausnahmsweise gut.

• a) Es gibt mir ein System, auf dem ich Debuggen kann
• b) Mein Netz geht durch einen Tunnel, der letztendlich durch AS59645 zum Internet kommt; Und IN-BERLIN (der Hoster von Digitalcourage) ist einer meiner Upstreams.
• c) Es bedeutet, dass die Deutsche Telekom hier unschuldig ist. Daher sollte sich–was auch immer kaputt ist–auf den Systemen von Digitalcourage fixen lassen.

Also, die Telekom ist raus, ich kann auf den kompletten Pfad bis zur Uebergabe an IN-BERLIN schauen… los gehts.

Der nächste Schritt bestand also darin, zu sehen, ‘wo’ Dinge kaputt gehen.

• Das Abrufen der Datei von meiner Workstation war langsam.
• Das Abrufen der Datei vom ersten Router war langsam.
• Die Datei von meinem Router drüben bei IN-BERLIN abzurufen war schnell. Da die Verbindung zwischen meinem Router bei IN-BERLIN und dem vor meiner Workstation über eine weniger-als-1500b-MTU-Verbindung verläuft, beschlich mich doch recht schnell der Verdacht, dass wir hier ein MTU Problem haben.

MTU?

Die MTU oder Maximum Transmission Unit ist ein Wert in Bytes, der Rechnern im Internet mitteilt, wie groß Netzwerkpakete sein dürfen. Die MTU haengt mit der MSS (Maximum Segment Size) für TCP-Verbindungen zusammen. Üblicherweise ist die MSS ‘MTU - 20b’ (für IPv4) und ‘MTU - 40b’ (für IPv6), d.h. wenn die MTU 1500 ist (wie es für die meisten Ethernet-Verbindungen und im Wesentlichen die meisten Verbindungen im Internet ist), sollte die MSS 1480 für IPv4 und 1460 für IPv6 sein.

Leider haben nicht alle Links im Internet eine MTU von 1500. Meine (wireguard-basierte) Tunnelverbindung hat beispielsweise eine MTU von 1420. Daher ist die MSS für IPv4 1400 hier. Ebenso haben DSL-User der Deutschen Telekom eine MTU von 1492, da für PPPoE 8b benötigt werden. Kabelnetzbetreiber verwenden normalerweise kein PPPoE, und daher haben ihre User normalerweise eine MTU von 1500.

Da Links unter 1500b üblich sind, gibt es natürlich Mechanismen für Hosts, um die maximale MTU auf einem Pfad zu einem Server/Client zu bestimmen, damit sie sicherstellen können, dass keine größeren Pakete gesendet werden. Dies wird allgemein als Path MTU Discovery (PMTUD) bezeichnet, und ich hatte bereits meinen fairen Anteil an Kopfschmerzen damit. Die Idee hinter PMTUD ist im Wesentlichen, dass jeder Host auf dem Pfad, sobald er ein Paket nicht weiterleiten kann weil es zu groß ist, eine ICMP-Fehlermeldung vom Typ 3 Code 4 (Fragmentation Needed and Don’t Fragment was set) sendet. Wenn der sendende Host dieses Paket erhält, weiß er, dass er kleinere Pakete senden muss, wenn er möchte, dass sie ankommen.

Witzigerweise würde die MTU-Problematik auch erklären, warum Telekom-User betroffen sind, und andere nicht: Die 1492b-MTU wegen PPPoE.

MTU? Bist du’s wirklich?

Es war vergleichsweise einfach zu überprüfen, ob dieses Problem wirklich mit der MTU zusammenhängt. Ich habe tcpdump auf dem externen Interface meines Routers bei IN-BERLIN gestartet (wo Pakete auf einer 1500-MTU-Verbindung eingehen und über eine 1420-MTU-Verbindung hinausgehen) und einen Filter für ICMP-Pakete gesetzt, die für digitalcourage.social bestimmt sind; Dann habe ich versucht, eine Datei von meiner Workstation zu wgeten. Und tatsächlich:

% tcpdump -i vio0 -n host 217.197.90.87 and icmp
tcpdump: listening on vio0, link-type EN10MB
20:16:38.979978 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
20:16:39.984627 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
20:16:39.985268 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
20:16:40.984956 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
...

Das ist natuerlich etwas seltsam. Eigentlich sollte die Box die da mit zu groszen Packeten um sich wirft (217.197.90.87) recht schnell merken, dass die MTU zu grosz ist. So nach dem ersten oder zweiten ‘Packet too large’. Aber das ist der Box wohl egal…

Wo es bei der PMTUD harkt

Zusammen mit Christian von Digitalcourage (der mir beim Debuggen geholfen hat und meine /bin/instantmessangersh zur Digitalcourage-Infrastruktur war, da ich auf deren Servern natürlich keinen Login habe) habe ich nun angefangen zu graben. Irgendwo muessen die Paeckchen ja verloren gehen. Das Graben war relativ schnell erfolgreich, da sowohl der Router von Digitalcourage als auch digitalcourage.social nur ICMP-Typ 8 (Echo-Antworten, was gemeint ist wenn Menschen von ‘ping’ reden) zuließ, aber nicht Typ 3 (Code 4). Interessanterweise legte die Mastodon-Dokumentation dies nahe; Dankenswerterweise gibt es da aber schon einen pull-request bereits aktualisiert.

Zwei (in die Firewall-Frameworks der Systeme integrierte) iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT später kamen die ICMP-Pakete endlich auch dort an, wo sie hingehören. Also, wget and, schneller Download, oder? Nope. -.-'

Irgendwas is komisch…

Irgendwas ist also komisch. Genaugenommen: Eine ganze Menge. Die MSS wird einem entfernten Host mitgeteilt, wenn eine Verbindung hergestellt wird. Eigentlich nageln meine Router die MSS fuer Verbindungen ueber Links mit einer MTU unter 1500 auf 1320 fest. Also, PMTUD beiseite, die hätte eigentlich nichtmal gebraucht werden duerfen. Und da PMTUD nun eigentlich funktioniert… sollte wirklich nichts diese Hosts davon abhalten, so zu kommunizieren, wie sie sollten.

Wir haben also angefangen, das funktionierende (wo user schnell downloaden koennen) mit dem anderen Host zu vergleichen. Insbesondere haben wir uns sysctl -a (nicht auffaellig), iptables -L -v -n (wieder nichts) und route get $myip/route show $myip (nope) angesehen. Wir haben auch nochmal ein tcpdump an die Leitung gehalten um zu verifizieren, dass mein Client die richtige MSS übermittelte (1320, und ja, das kam beim Starten der TCP-Session an). Obwohl nichts an digitalcourage.social speziell zu sein schien ignorierte die box die existenz von MTUs unter 1500. Jedes Packet wurde erst mit einer Groesze von 1500b gesendet… und dann 40b Schrittweise bei den retransmissions verkleinert, bis es irgendwann durch kam. Das gibt natuerlich viel overhead und wenig Bandbreite.

Dinge zum Laufen bringen (vorerst)

Etwas frustriert (und mit etwas Input von anwlx), entschieden wir uns, einfach mal die MTU fuer meine IP mit einer statischen Route (vorsicht, Fachbegriff) festzudengeln.

ip route add 195.191.197.206 via 217.197.90.81 mtu lock 1320

Und siehe da… es funktioniert. 100Mbit+ beim Downloaden auf meiner Workstation.

Trotz erweitertem Rumprobierens konnten wir leider nicht herausfinden, warum der Host MTU und MSS ignorierte. An diesem Punkt kurz nach 00:00 Uhr schlug ich vor, einfach einen Hotfix auf das Problem zu werfen, der für die meisten User funktioniert:

ip r a 0.0.0.0/1 via 217.197.90.81 mtu lock 1320
ip r a 128.0.0.0/1 via 217.197.90.81 mtu lock 1320

Damit hatten wir die MTU fuer zwei spezifischere Routen als die default-Route (0.0.0.0/0) auf 1320 festgedengelt; Und da sie spezifischer als die default route sind, werden diese beiden dann auch dieser vorgezogen. Und im zweifel ist das einfach etwas entspannter, als die default Route auf einer Prod-Box zu aendern. Christian war ziemlich glücklich über diesen Vorschlag und warf ihn gleich auf die Box. Für den Moment war das Problem also erstmal weg, und Antworten auf die Ankündigung, dass Dinge jetzt funktionieren sollten, suggerieren, dass es tatsächlich funktionierte.

Muss das so?

Mir hat das natuerlich alles keine Ruhe gelassen (und ich hatte irgendwie am Anfange des Artikels eine richtige Erklaerung versprochen… ): Also habe ich mal versucht das ganze nachzubauen

• Erstmal eine VM mit der gleiche Software (Debian 10) aufsetzen (Debuggen in Prod ist immer etwas unlustig)
• Mastodon ohne Docker installieren (vielleicht macht das irgendwas kaputt?!)
• Die sysctl settings und geladenen Kernel-Module an die Einstellungen der prod-box anpassen

Ich habe das am Samstag eingerichtet und war eigentlich ziemlich hoffnungsvoll, dass das nicht funktioniert. Also er… der Fehler auftritt. Tat er aber nicht. Kein Problem fuer mich.

Womit wir bei heute wären. Mich beschlich die Frage, ob vielleicht irgendwas an der Virtualisierungsumgebung komisch ist. Ich habe mich wieder bei Christian gemeldet und gefragt, was fuer eine Virtualisierungsloesung laeuft (libvirt+kvm auf Debian). Mein Bauchgefuehl liesz mich dann auch nach einem lspci von der Mastodon Box fragen. Vielleicht ist da was dabei?

Und da fiel mir in der tat gleich was auf:

00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 20)

KVM VMs machen eigentlich nur mit virtio Netzwerkkarten so richtig spasz.

Also, schneller Wechsel auf meiner Test-VM, die NIC wird eine ‘rtl8139’ und nichtmehr ‘virtio’, uuuuuuuuuuuund… Karpot. Endlich. wget von meiner Workstation benimmt sich nun auch so wie es (nicht) soo, mit nur etwa 100 KB/s im Download. Die zweite Digitalcourage Box, von der alle User immer mit der zu erwartenden Geschwindigkeit downloaden konnten, nutzt auch erwartungsgemaesz eine virtio NIC:

00:03.0 Ethernet controller: Red Hat, Inc Virtio network device

Mit einer Idee wonach ich nun suchen koennte (ok, rtl8139, kaputt’ ist nicht wirklich ein einzigartiger Suchbegriff), fand ich schließlich diesen Thread auf den qemu-dev/netdev Mailinglisten. Es scheint, dass andere tatsächlich schonmal das gleiche Problem hatten. Vor sechs Jahren. Der Thread stellt ziemlich schnell fest, dass die MTU für das TCP-Segment-Offloading im Qemu rtl8139-Code auf 1500 festgedengelt zu sein scheint. Der Thread schlägt sogar einen Patch vor, der das Problem beheben sollte (aber danach wirds still im thread). Letztendlich kommt der Code mit diesem Fehler vermutlich aus dem Jahr 2006, ist also rund 16 Jahre alt (oder auch: war schon zehn Jahre alt, als das Thema auf der Mailingliste aufkam).

Wenn ich mir den ‘Modifications:’ Header in rtl8139.c ansehe, habe ich den starken Verdacht, dass dies letztendlich auf diese Änderung hinausläuft:

 *  2006-Jul-04                 :   Implemented TCP segmentation offloading
 *                                  Fixed MTU=1500 for produced ethernet frames

Dazu passt auch, dass das abschalten von TSO/GSO den gleichen positiven Effekt wie eine lock mtu 1320 route hat:

ethtool -K ens18 tx off sg off tso off

Aber da ich nicht wirklich eine C-artige Person oder übermäßig qualifiziert bin was Systemrogrammierung angeht bin, dachte ich, es wäre vielleicht besser, einfach ein Ticket zu erstellen und Feierabend zu machen..

Damit gibt das Problem mir ruhe, und die Leute von Digitalcourage haben einen Grund, den Netzwerkschnittstellentyp ihrer virtualisierten NIC zu ändern. ^^

Woran ich denken sollte

Zu zum Abschluss als ‘lessons learned’:

Beim Vergleichen zweier virtueller Maschinen auf dem selben Hypervisor ist es nicht moeglich davon auszugehen, dass diese die gleiche virtuelle ‘Hardware’ haben.

Oder davon auszugehen, dass VMs keine Hardwareprobleme haben…

OpenBSD version: Actually none
Arch:            Any
NSFP:            Please no...

So, for the last couple of months, the mastodon instance of Digitalcourage e.V. has been suffering from network issues. These manifested themselves in users having a very low (around 100KB/s) download speed from the instance. This, of course, was rather ‘not so good’.

Over these months, and with lots of user feedback, some clear patterns started to emerge. Specifically, it seemed like mostly customers of Deutsche Telekom were affected, while others were able to retrieve files at the speeds they were expecting. The admins over at digital courage had even tried to change the IP address of the system: To no avail. Interestingly, though, another system right next to the mastodon instance at digitalcourage.social performed pretty much fine, even for otherwise affected Deutsche Telekom users. Ultimately, this lead to a rather extensive discussion on whether Deutsche Telekom might be treating traffic to Digitalcourage’s servers a bit less equal than it should.

Last Wednesday (9 Nov 2022), i saw the ongoing thread and was kind of intruiged by the somewhat strange problem. I chatted up a contact over at Deutsche Telekom, who was already involved in debugging the issue, who was friendly enough to introduce me to the Digitalcourage people, suggesting i might be able to help. They agreed, and got into contact with me, and we could take a shot at debugging this.

As people might find it fun to read up on what really happened there (to which i only got tonight), i thought i drop this into a blog article.

Starting to debug

The first thing one needs when debugging such an issue is, naturally, a way to really look at what is happening, i.e., you have to be able to reproduce the issue. For the case at hand, this was surprisingly easy. When i tried to download the test-file from my home connection, things did not look good:

% wget -O /dev/null https://digitalcourage.social/1GB.bin
--2022-11-09 18:53:17--  https://digitalcourage.social/1GB.bin
Resolving digitalcourage.social (digitalcourage.social)... 217.197.90.87
Connecting to digitalcourage.social (digitalcourage.social)|217.197.90.87|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null             0%[                    ] 615.70K  98.8KB/s    eta 2h 53m 

This is good for multiple reasons.

• a) It gives me a system to look at while debugging.
• b) My home connection is through a tunnel ultimately leaving through AS59645, which (partially) draws upstream from IN-BERLIN, the hoster Digitalcourage uses as well.
• c) It essentially rules out Deutsche Telekom as a culprit, which means that whatever is broken can most likely be fixed over at the systems of Digitalcourage.

Hence, knowing DT is most likely out of the picture, and equipped with full controll on the (at least overlay) path from my machine up to the point where packets are handed to IN-BERLIN, i went to the next step.

So, the next steps was seeing ‘when’ things broke.

• Getting the file from my workstation was slow.
• Getting the file from the first router was slow.
• Getting the file from my router over at IN-BERLIN was fast. As the link between my router at IN-BERLIN and the one in front of my workstation goes via a less-than-1500 MTU link, this lead to a reall strong suspicion that this is an MTU related issue.

MTU?

The MTU, or Maximum Transmission Unit is a byte value, which tells networked systems how large network packets are allowed to be. It is strongly tied to the MSS (Maximum Segment Size) for TCP connections. Commonly, the MSS is MTU - 20b (for IPv4) and MTU - 40b (for IPv6), i.e., if the MTU is 1500 (as it is for most ethernet links, and essentially most links on the Internet), the MSS should be 1480 for IPv4 and 1460 for IPv6.

Sadly, not all links on the Internet have an MTU of 1500. My (wireguard based) tunnel connection, for example, has an MTU of 1420. Hence, the MSS for IPv4 is 1400. Similarly, DSL customers of Deutsche Telekom have an MTU of 1492, because 8b are needed for PPPoE. Cable network providers ususally do not use PPPoE, and hence their customers commonly have an MTU of 1500.

Of course, with links below 1500b being common, there are mechanisms for hosts to determine the maximum MTU on a path to a server/client, so they can make sure to not send packets larger than that. This is commonly called Path MTU Discovery (PMTUD), and i had my fair share of headeaches around that already. The idea behind PMTUD is, essentially, that each host on the path, as soon as it can not forward a packet because it is too large sends back am ICMP error message of ‘Type 3 Code 4’ (Fragmentation Needed and Don’t Fragment was Set). When the sending host gets that packet, it knows that it has to send smaller packets if it wants them to arrive.

Funnily enough, the issue being related to the MTU would also explain why Deutsche Telekom customers are affected, while others are not: The 1492b MTU due to PPPoE.

Making sure it’s the MTU

Verifying that this issue is MTU related was comparatively easy. I fired up tcpdump on the outbound interface of my router at IN-BERLIN (where packets come in on a 1500 MTU interface and go out via a 1420 MTU interface), and set a filter for ICMP packets destined to digitalcourage.social; Then, i tried to wget a file from my workstation. And sure enough i saw:

% tcpdump -i vio0 -n host 217.197.90.87 and icmp
tcpdump: listening on vio0, link-type EN10MB
20:16:38.979978 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
20:16:39.984627 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
20:16:39.985268 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
20:16:40.984956 IP 217.197.83.197 > 217.197.90.87: ICMP 195.191.197.217 unreachable - need to frag (mtu 1420), length 36
...

You might notice that this is odd. There should not be that many, the sender of large packets (217.197.90.87) should rather quickly get the message that the MTU is too large… But the box seems to be somewhat oblivious…

Where PMTUD is lost

Together with Christian from Digitalcourage (who helped me debugging this and was my /bin/instantmessangersh for commands on the Digitalcourage infrastructure, as i of course did not have a login on those machines) i now started to dig where those packets might get lost. Digging was complete relatively quickly, as both Digitalcourage’s router and the digitalcourage.social only allowed ICMP type 8 (echo-reply, what you usually know was ping), but not Type 3 (code 4). Interestingly, the mastodon documentation suggested this; Based on our findings, this was already updated, though.

Two (integrated in the systems’ firewall frameworks) iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT later, the icmp packets finally arrived where they should. Eagerly, i fired up a wget, looking forward to seeing the file rushing in with unseen speed and saw… well, nothing having changed, really.

Thinking about oddities

Now, with things not working now, there must be a bit more going wrong. First of all, technically my systems do MSS clamping. The MSS is communicated to a remote host when a connection is established, and my routers should set it to 1320 for packets traversing the tunnel. So, PMTUD aside, the remote server should not have needed it to begin with. With PMTUD now technically being able to work, there should really be nothing keeping these hosts from communicating as they should.

Hence, we started digging into what might be different between the working and non-working host. Specifically, we looked at sysctl -a (nothing special), iptables -L -v -n (again, nothing), and route get $myip/route show $myip (nothing out of the ordinary). We also held a tcpdump to the wire and verified that my client communicated the correct MSS (1320, and yes, it indicated that when starting the TCP session). There seemed to be nothing special about digitalcourage.social, yet it was still blissfully ignorant of the existence of any MTUs < 1500. This meant that packets would be resend, each time with a segmentsize 40b lower, until it would finally arrive. This, of course, causes a bit of overhead, making things… slow.

Making things work (for now)

Somewhat frustrated with how things were going (and thanks to some input from anwlx), we decided to install a route for my IP with a locked MTU on the mastodon host:

ip route add 195.191.197.206 via 217.197.90.81 mtu lock 1320

Lo and behold… things suddenly worked. I got an excess far beyond 100 Mbit from the host.

We continued to try around for a bit, but were unable to figure out why the host was ignoring MTU and MSS, except when explicitly locked. At this point–slightly after 12AM–I suggested to apply a hotfix, which would work for the majority of users, and call it a night:

ip r a 0.0.0.0/1 via 217.197.90.81 mtu lock 1320
ip r a 128.0.0.0/1 via 217.197.90.81 mtu lock 1320

This locked the MTU for two more-specifics (i.e., preferred) routes for ‘everyting’ to an MTU of 1320; Essentially the same thing as for my single IP, but for, well, everything. And, in case of doubt, this is a more resillient approach than trying to update the default route on a production box. Christian was actually quiet happy about that suggestion, and moved it in place. For now, this fixed the issue, and replies to the announcement that things should work now suggested, that it actually worked.

But WHY?!

The hotfix was not really satisfactory for me (and i promised you some input on the root cause). Hence, i tried to replicate the issue:

• Setup a system with the same software (Debian 10) under my control (debugging in prod is always a bit unfunny)
• Install Mastodon without docker (maybe this breaks something?!)
• Set all sysctl values and loaded kernel modules the same as on the live host

I set that up on Saturday, and was actually quiet hopeful that things would not work. Well, except they did. I could not replicate the issue.

Which brings us to today. I started to wonder whether maybe there was something about the virtualization environment going on. I checked in with Christian again, and asked them what they were using (plain libvirt+kvm on Debian). Then, on a whim, i asked for an lspci from the box. Maybe there is something there? Looking at the lspci output, one line immediately sprung into my eye:

00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 20)

Usually, when running kvm VMs, you want to have a virtio network card.

So, quick change over on my test-VM, making the NIC ‘rtl8139’ and not ‘virtio’ aaaaaand… things broke. Finally. Doing a wget behind a low MTU link finally gave me only around 100KB/s. Similarly, checking the second box from which thins worked all along, Christian found that it uses a virtio NIC:

00:03.0 Ethernet controller: Red Hat, Inc Virtio network device

Equipped with something to search for (to be fair ‘rtl8139, broken’ is not the most unique thing to look for), i ultimately found this thread on the qemu-dev/netdev mailinglists. It appears that others actually had the same problems before. The thread rather quickly notes that the MTU for TCP Segment Offloading seems to be fixed at 1500 in qemu’s rtl8139 code. The thread even suggests a patch which should fix the issue (but runs dry afterwards). Ultimately, the code with this bug must have been written in 2006, so round about 16 years ago (or rather: already ten years ago when the issue was discussed on the mailinglist).

Looking at the ‘Modifications:’ header in rtl8139.c i have a strong suspicion that this ultimately boils down to this change:

 *  2006-Jul-04                 :   Implemented TCP segmentation offloading
 *                                  Fixed MTU=1500 for produced ethernet frames

This also fits well with the fact that disabling TSO/GSO for the rtl8139 fixes the issue as well:

ethtool -K ens18 tx off sg off tso off

But, as i am not really a C-ish person, or overly qualified coder… i figured it might be better to just file a bug ticket and call it a day.

At least I have closure now.

And the people at Digitalcourage a reason to change the network interface type of their virtualized NIC. ^^

Lesson Learned

So, as a rather final-final thought; There is one big lesson learned.

If you compare two virtual machines on the same hypervisor, do not assume that the hardware is the same.

Or that virtual machines cannot have hardware bugs.

OpenBSD version: 7.1 (Yeah, no, probably not...)
Arch:            Any
NSFP:            Opinions are never... 

So, on Monday the 18th of October, the ‘Financieele Dagblad’ published an article based on an interview with Martina Lindorfer and me, on our joint work (arXiv preprint from 2021) together with Seda Gürses and further colleagues from TU Delft. This was then quickly picked up by NOS, the NL Times, and Tweakers.net, ultimately also finding it’s way to Reddit (and other politically relevant venues like the Dutch Parliament). Naturally, this lead to a lot of questions and comments raining into the forum sections of those publications. As there are sometimes some interesting questions and opinions in these comments–and a brief newspaper article tends to be too brief for some technical depth–i figured it might be nice to have a brief FAQ (or FCC? Frequently Commented Comments?) on some points raised frequrently. So, here we go, with a list in no particular order.

But those servers are actually in the EU

What might be the top comment among all of the hundreds is the point that the instances running on Amazon’s cloud are physically located in the EU, or the universities may have forgotten to select the right (EU) availability zone. Thing is, what we measured and claimed is whether specific infrastructure–and in this case the statement refers to Learning Management Systems–is hosted on systems that are part of the Amazon cloud, independent of the specific location of those systems. And, for Dutch Universities, by now, most have their LMS hosted with Amazon (note that Blackboard.com moved from Azure to AWS relatively recently, and the article references a perspective from before then). Of course–for functional reasons like latency alone already–these systems are either in Dublin (Amazon EU West) or Frankfurt (Amazon EU Central). Some of the IP adresses for these instances are even held by a very non Amazon-ish sounding A100 ROW Inc/GmbH.

Well, A100 ROW GmbH is, of course, a 100% Amazon subsidiary. (Note, the RIPE NCC registry id of ‘us.a100row’ for this entity; For comparison: When my main residence was in the Netherlands, my own LIR was ‘nl.tobias’; After relocating that to Germany it became ‘de.wybt’… ) And that is basically the point. Amazon’s cloud is not like Champagne. It doesn’t become ‘sparkling automated infrastructure with an API’, just because it is no longer from the ‘Silicon Valley region of tech’.

First of all, the Cloud Act applies. What this law basically says is ‘US authorities can subpoena US companies for data stored on their systems and their foreign subsidiaries regardless of where the data is physically located.’ This is well known, and hence has also been one of the major points in the Schrems rulings. See also this ruling of a German state court on whether subsidiaries of US cloud companies are even viable in public tenders. The court claims no, because the necessary guarrantees, especially with the end of Standard Contract Clauses in Schrems II, can not be provided.

So, in summary, as long as things are in infrastructure belonging to a US company, it does not matter whether the servers are physically located in the EU; The bad parts of US law still apply.

And, besides this, the main point we’re making is not that much about the US government, but the power of individual cloud companies can inflict on universities (and society as a whole).

Can you share your data?

Right up next are requests for our raw data. For the long-term survey, also looking at things over time, we used the Farsight SIE dataset of historic DNS requests seen by sensors all around the world. This, of course, is not really a dataset to share publicly.

However, what we measure can be quickly gathered from the public DNS by oneself. I hence wrote a small script that checks in on Dutch universities’ mail setups and learning management systems, and provides a detailed overview of the current status, as well as a brief interpretation, i.e., what is hosted where. You can find that script here: https://git.aperture-labs.org/Cloudheads/cloudheads_nl_scraper Please feel free to run it yourself to gather the data, or include any institutions, like HBOs, which we did not include.

Anyway, if you want to take a look at the data, get it from the repository, or–if you don’t trust me–grab the script there and run it for yourself.

Are LMS really all of students’ data?

Well, the statement made in the FD article is–for the majority of Dutch universities–about their Learning Management Systems being in the cloud. Those systems usually hold data on which courses a student registered for, depending on the setup (partial) as well as reported final grades, and a bunch of interaction in between students and with teachers. This is, of course not all data universities hold on students.

Of course, the logs of what students do via the local Wifi–if logged by some network security solution, that is–are most likely stored locally (or send off to an offsite SOC for threat analysis). Similarly, final bookkeeping on grades often is not in the LMS, but in a dedicated application. Still, some universities are already evaluating cloud-based replacements. Finance will run another system handling fee payments for students etc. Email will also be handled by a different bunch of systems, too. Here, though, we also find that Dutch universities regularly use the email offerings of major cloud providers, with Microsoft being the leading vendor there.

So, the LMS in the cloud is not all of students’ data.

But an important chunk.

How else should you run a service?

A common issue pointed out by commentators is that it is really hard to run a service without relying on cloud infrastructure. That, is very true, and i also wrote about this on a more general level; Running stuff well is hard. Like, really hard. And we are not even talking about things like the cancerous nature of Google’s font hosting. (I am still amazed how often i struggle to cut these out of self-hosted tools) And this while the caching advantages that lead to the rise of Google fonts are gone by now for security reasons.

Besides–and coming back to the previous point–one of our main arguments is that the continuous use of cloud infrastructure leads to an increasingly reducing ability to run stuff yourself. This comes from saving money on those expensive engineers… which leads to dependence, because you no longer have the teams in-house to move out of the cloud. Hence, with the point being that many organizations already can’t run infrastructure without the help of clouds–‘see also my work on the compelxity of email’–this essentially just highlights the point we are making regarding progressing dependence.

Hence, even though it is a difficult task, we have to think about how to retain our ability to host (research and teaching) infrastructure ourselves. In the Netherlands, there is apparently an ongoing pettition for this. I don’t doubt that this is a hard task, but it doesn’t get easier the longer we wait; Actually, on the contrary.

Of course universities might be dependent; But so are they on energy…

Continuing down the issue of dependence. In our work, we are claiming that universities being dependent on some entities makes it really easy for tech companies to–essentially–blackmail universities. Back at TU Delft, when listening to a presentation on this measurement work, a colleague brought this argument to the point by claiming: ‘Well, then universities are also dependent on their gas suppliers. Should we now produce our own energy?’ (You might notice that this argument was made before the 24th of February, and really didn’t age well… ) Thing is, the most accurate rebuttal of this point came from another colleague, a full professor on energy systems, essentially saying: “Well, of course that is an issue in the energy domain. That is why that market is *heavily* regulated.” I can hardly add to that; Hits the nail on the head.

Companies are rational actors, which will do what is best for them under a set of given rules. If the rules permit something, and it can make them profit, they will do it. This is how rational actors work.

Also, in general, the approach of companies like Google to get a foothold in new markets is kind’a known: Barge in with a cheap (or free) offer, start charging once you are ubiquous. This is rather well analyzed (PT; p. 27ff.), and currently also kind of in the find-out phase of ‘sign up for the cloud, and find out’ for several different universities.

Assuming tech companies would skip on this (legal) lever to increase their profits just due to their good heart is, to be honest, just a bit naive.

But why would they even care for making/interfering with curricula?!

So, one of our big points is that the tech companies might use their aforementioned blackmail ability to also influence what is taught and researched, leading to our point that clouds may threaten academic integrity. This regularly triggers the question of why tech comapnies would even care to meddle with curricula and research.

As for curricula, the answer is quite obviously because they already have curricula for teaching ‘the cloud’ in universities, as well as K-12 curricula on computer science. Similarly, there already were instances were large corporations used their market positions to influence research. Facebook (now called Meta), for example, pulled a rather interesting move by demonstrating that ‘canceling the private facebook accounts of people doing research they don’t like’ is very much in their toolbox. Given that most people have stashed away quite a big part of their social circles and memories in their Facebook and Instagram accounts, this is a serious threat. Similarly, we have the story of Timnit Gebru, hired to critically reflect on dangers of AI by google, only to be fired when that work was too critical. And, while all this is going on, we increasingly see tech companies putting faculty (indirectly) on their payroll.

Summary

So, i hope to have contextualized some of the most pressing comments and questions i saw coming up. If you think i missed to cover some important parts, please drop me a line, and i will work on another blog article. Also, please feel free to share any non-question-comments you may have. For both, please fee free to use my contact mail address for this blog: contact@as59645.net.

Until then, enjoy your day, and always remember:

Just because you hope it doesn’t happen, doesn’t mean it won’t.

OpenBSD versie: 7.1 (Ja, nee, waarschijnlijk niet...)
Arch:           Eender
NVVP:           Nooit voor meningen... (TM)
Vertaaling:     Stijn Pletinckx

Dus, op maandag 18 oktober verscheen er een artikel in het ‘Financieele Dagblad’ over een een interview met Martina Lindorfer en ik over ons gezamelijk werk met Seda Gürses en andere collega’s van TU Delft. Dit werd al snel opgepikt door NOS, NL Times, Tweakers.net, en uiteindelijk ook door Reddit (en andere politiek relevante kanalen zoals de Tweede Kamer). Zoals vaak gebeurt in de media ontstonden er al snel reacties met discussies en vragen over ons artikel (arXiv preprint). Daar er soms interessante vragen en meningen voortvloeien uit desbetreffende discussies - en een kort krantenartikel de neiging heeft vaak te kort te zijn voor technische diepgang - besloot ik om een samengevatte FAQ (of FCC? “Frequently Commented Comments”) op te stellen die ingaat op de meest voorkomende vragen in verband met mijn artikel. Dus, hier gaan we, met een lijst in willekeurige volgorde.

Maar eigenlijk bevinden die servers zich in de EU

Wat misschien wel de meest voorkomende opmerking is van alle honderden, is het argument dat de servers die op Amazon’s cloud draaien zich fysiek in de EU bevinden, of dat de universiteiten zijn vergeten de juiste (EU) beschikbaarheidszone te selecteren. Het punt is, onze metingen en claims gaan over het feit of specifieke infrastructuur - en in dit geval verwijst dit naar Learning Management Systems - wordt gehost op systemen die deel uitmaken van de Amazon-cloud, onafhankelijk van de specifieke locatie van die systemen. En, wat betreft de Nederlandse universiteiten hebben de meeste inmiddels hun LMS draaiende bij Amazon (merk op dat Blackboard.com relatief recentelijk van Azure naar AWS is verhuisd, en het artikel verwijst naar een perspectief van daarvoor). Natuurlijk - om functionele redenen zoals bijvoorbeeld “latency” - bevinden deze systemen zich in Dublin (Amazon EU West) of Frankfurt (Amazon EU Central). Sommige van de IP-adressen voor deze servers zijn zelfs in handen van een niet Amazon-klinkende A100 ROW Inc/GmbH.

Welnu, A100 ROW GmbH is natuurlijk een 100% Amazon-dochteronderneming. (Zie ook de RIPE NCC registry id van ‘us.a100row’ voor deze entiteit; Ter vergelijking: toen mijn hoofdverblijfplaats in Nederland was, was mijn eigen LIR ‘nl.tobias’; Toen ik naar Duitsland verhuisde werd het ‘de.wybt’… ) En dat is in wezen het punt. De “cloud” van Amazon is niet zoals champagne. Het wordt geen ‘sprankelende geautomatiseerde infrastructuur met een API’, alleen maar omdat het niet meer uit de ‘Silicon Valley region of tech’ komt.

Allereerst is de Cloud Act van toepassing. Wat deze wet in feite zegt is dat ‘Amerikaanse autoriteiten Amerikaanse bedrijven kunnen dagvaarden voor gegevens die zijn opgeslagen op hun systemen en hun buitenlandse dochterondernemingen, ongeacht waar de gegevens zich fysiek bevinden.’ Dit is algemeen bekend en is daarom ook een van de belangrijke punten in de Schrems-arresten geweest. Zie ook deze uitspraak van een Duitse staatsrechtbank over de vraag of dochterondernemingen van Amerikaanse cloudbedrijven zelf aanklaagbaar zijn bij openbare aanbestedingen. De rechtbank beweert van niet, omdat de nodige garanties, vooral met het einde van de modelcontractbepalingen in Schrems II, niet kunnen worden verstrekt.

Dus, kort samengevat, zolang servers zich in de infrastructuur van een Amerikaans bedrijf bevinden, maakt het niet uit of de servers zich fysiek in de EU bevinden; De slechte delen van de Amerikaanse wetgeving zijn nog steeds van toepassing.

En daarnaast, wat we voornamelijk aankaarten gaat niet per se over de Amerikaanse overheid, maar meer over het punt dat individuele “cloud” bedrijven te veel macht krijgen en als gevolg invloed kunnen heben op universiteiten (en de maatschappij in het algemeen).

Kun je de data delen?

Vervolgens de verzoeken om onze ruwe data te delen. Wat betreft de langetermijnstudie, die de resultaten in een tijdsperspectief plaatst, maken we gebruik van de “Farsight SIE” historische DNS data, gebasseerd op wereldwijde sensoren. Dit is uiteraard data die best niet openbaar wordt vrijgegeven.

Desalniettemin kan men vrij snel hetgeen we meten zelf verzamelen op basis van publieke DNS data. Om dit te demonstreren heb ik zelf een klein stukje code geschreven die informatie verzamelt over de mail server opstelllingen en LMS van Nederlandse universiteiten, en details weergeeft over de huidige status, alsook een korte interpretatie zoals wat waar gehost wordt. De code is hier te vinden: https://git.aperture-labs.org/Cloudheads/cloudheads_nl_scraper Voel je vrij om het zelf uit te voeren om de gegevens te verzamelen, of om andere instanties toe te voegen, zoals HBO’s, die we niet hebben opgenomen.

Hoe dan ook, als je de gegevens wilt bekijken, kun je ze vinden in de repository, alsook de code - voor als je me niet vertrouwt - om de experimenten zelf uit te voeren.

Bevat LMS echt alle gegevens van studenten?

Welnu, het punt dat we maken in FD is dat - voor de meerderheid van de Nederlandse universiteiten - hun “Learning Management Systems” in de cloud staan. Die systemen bevatten meestal gegevens over voor welke cursussen een student zich heeft ingeschreven, afhankelijk van de opzet (eind)cijfers, en een heleboel interactie tussen studenten en docenten. Dit is natuurlijk niet alle data die universiteiten over studenten hebben.

Uiteraard, data over wat studenten doen over de locale WiFi verbinding - indien gemonitord - staan meestal lokaal opgeslagen (of worden doorgezonden naar een externe SOC voor “threat analysis”). Tevens, het opslaan van officiele cijferlijsten gebeurt vaak niet via een LMS, maar eerder via specifieke software. Desalniettemin overwegen sommige universiteiten om alsnog over te stappen naar cloud applicaties ter vervanging van desbetreffende systemen. Financiele handelingen gebeuren ook via andere systemen, tevens ook als e-mail. Doch, hier zien we echter ook dat Nederlandse universiteiten regelmatig gebruik maken van het e-mailaanbod van grote cloudproviders, waarbij Microsoft daar de leidende leverancier is.

Het LMS in de cloud is dus niet alle gegevens van studenten.

Maar wel een belangrijk stuk.

Hoe moet je anders een dienst runnen?

Een vaak aangekaard punt in de reacties is dat het bijzonder moeilijk is om een infrastructuur te draaien zonder te berusten op een cloud opplossing. Dat klopt, en ik schreef hier ook over op een meer algemeen niveau; Een systeem goed draaiende houden is moeilijk. Maar dan ook echt moeilijk. En dan hebben we het nog niet eens over zaken als de rottende aard van Google’s font hosting. Ik sta er nog steeds van versteld hoe bijzonder lastig het is om deze uit zelf-gehoste tools te verwijderen. En dit terwijl de caching voordelen die hebben geleid tot de populariteit van Google fonts om veiligheidsredenen inmiddels verdwenen zijn.

Bovendien - en terugkomend op het vorige punt - is een van onze belangrijkste argumenten dat het continue gebruik van cloudinfrastructuur leidt tot een steeds kleiner wordende mogelijkheid om dingen zelf te draaien. Dit komt omdat we willen geld besparen op die dure ingenieurs… wat leidt to afhankelijkheid, omdat je niet langer het juiste personeel in huis hebt om uit de cloud te verhuizen. Vandaar dat, het punt makende dat organisaties nu al geen infrastuctuur kunnen draaien zonder de hulp van clouds–‘zie ook mijn werk over de complexiteit van e-mail’–dit in wezen alleen maar het argument van geleidelijke afhankelijkheid nog meer benadrukt.

Net daarom, ook al is het een moeilijke taak, dat we moeten nadenken over hoe we ons vermogen om zelf (onderzoeks- en onderwijs)infrastructuur te hosten, kunnen behouden. In Nederland is hier blijkbaar een lopend verzoek voor. Ik ontken niet dat dit een moeilijke taak is, maar het wordt niet makkelijker naarmate we langer wachten; Integendeel zelfs.

Natuurlijk kunnen universiteiten afhankelijk zijn; Maar dat zijn ze ook op zaken zoals energie…

Voortzetting van de kwestie over afhankelijkheid. In ons werk beweren we dat universiteiten, die afhankelijk zijn van sommige entiteiten, het voor technologiebedrijven heel gemakkelijk maken om - in wezen - universiteiten te chanteren.

Destijds op TU Delft, luisterend naar een presentatie over deze studie, bracht een collega het volgend argument ter zake: ‘Nou, dan zijn universiteiten ook afhankelijk van hun gasleveranciers. Moeten we nu onze eigen energie opwekken?’ (Het is je misschien opgevallen dat dit argument werd gemaakt vóór 24 februari, en echt niet beter geworden is met der tijd… ) De meest adequate weerlegging van dit punt kwam van een andere collega, een hoogleraar in energiesystemen, die in wezen zei: “Nou, dat is natuurlijk een kwestie in het energiedomein. Daarom dat die markt sterk gereguleerd is.” Ik kan daar nauwelijks iets aan toevoegen; Slaat de spijker op de kop.

Bedrijven zijn rationele actoren, die, gegeven de huidige omstandigheden, altijd zullen doen wat het meest in hun voordeel uitkomt. Als de regels iets toestaan, en het kan hen winst opleveren, zullen ze het doen. Dat is hoe rationele actoren werken.

Tevens, de strategie van Google om voet aan grond te krijgen in nieuwe markten is allom bekend. Stap binnen met een goedkope (of gratis) aanbieding, begin met extra kosten zodra je niet meer vervangebaar bent. Dit is goed bestudeerd (PT; p. 27ff.), en bevindt zich momenteel ook een beetje in de zoek-het-uit-fase van ‘meld je aan voor de cloud, en zoek het uit’ voor meerdere, verschillende, universiteiten.

Ervan uitgaan dat techbedrijven deze (juridische) hefboom om hun winst te verhogen zouden overslaan vanwege hun goede hart, is, om eerlijk te zijn, een beetje te naïef.

Maar waarom zouden ze zich zelfs bekommeren over het maken van/ingrijpen in curricula?!

Een van onze grote punten is dat de technologiebedrijven hun eerder genoemde chantagevermogen kunnen gebruiken om ook te beïnvloeden wat er wordt onderwezen en onderzocht, wat ertoe leidt dat clouds de academische integriteit in gedrang kunnen brengen. Dit roept regelmatig de vraag op waarom techbedrijven zich in de eerste plaats zouden willen bemoeien met curricula en onderzoek.

Wat curricula betreft, is het antwoord simpelweg omdat ze al curricula hebben voor het onderwijzen van ‘de cloud’ op universiteiten, evenals K-12 curricula over informatica. Evenzo waren er al gevallen waarin grote bedrijven hun marktposities gebruikten om onderzoek te beïnvloeden. Facebook (nu beter gekend als Meta) deed bijvoorbeeld een nogal interessante zet door aan te tonen dat ‘het annuleren van de private facebook-accounts van mensen die onderzoek doen dat ze niet leuk vinden’ wel degelijk binnen hun capaciteiten ligt. Aangezien velen een behoorlijk groot deel van hun sociale kringen en herinneringen opgeslagen hebben in hun Facebook- en Instagram-account, is dit een serieuze bedreiging. Eveneens hebben we het verhaal van Timnit Gebru, aangenomen door Gogle om kritisch na te denken over de gevaren van AI, echter later ontslagen toen dat werk te kritisch werd. En terwijl dit allemaal gaande is, zien we steeds vaker dat techbedrijven universitaire docenten (indirect) op hun loonlijst zetten.

Tot slot

Dus, ik hoop een aantal van de meest dringende opmerkingen en vragen die ik zag opkomen in hun context te hebben geplaatst. Indien ik een aantal belangrijke delen heb gemist, stuur me dan een bericht, en dan zorg ik voor een vervolg blogartikel. Voel je ook vrij om eventuele andere commentaren met mij te delen. Voor beide kun je me bereiken op het volgende e-mailadres voor deze blog: contact@as59645.net.

Tot die tijd wens ik je een fijne dag toe, en onthoud altijd:

Hopen dat iets niet gebeurt is geen garantie dat het nooit zal gebeuren.

OpenBSD version: 7.1 (and whatever nl-ams02a-rc2 runs on)
Arch:            Any
NSFP:            Well... found this in production... so... probably ok?

So, i had a problem; Had, because since yesterday, i actually know why i had this problem, and much more importantly, how to fix it. See, for quiet some time, i had been having issues with low transfer speeds of my OpenBSD boxes accross a rather long link. While, technically, i should be getting a clean downstream of 1gbit from my DOCSIS link from Ziggo at home, i would be stuck with anything between 10 and 50mbit, sustained, in the downlink. Well, not everywhere. Windows boxes were fine. Linux systems, too. Well, kind of fine. They also did not really excell. But i attributed the 300-400mbit i saw to the shady setup of different tunnels i am running to put my home router into the dfz.

What made this thing even more odd was that it–at first at least–seemed like only SSH/scp/rsync via SSH was affected. Just ripping open a socket and pouring in packets was fine, i would see my 1gbit down. Also, the boxes among each other locally would be fine. Furthermore, it would not matter if this whole thing went down via TCP directly, or via TCP over one of my UDP based tunnels. So, for some time i figured that it was _some_thing in the combination of SSH and the long(er) latency on my path to–back then–Hetzner. Furthermore, this whole situation also did not really improve when i moved from my tunnel based shady-AS setup to a rack over at AS24961.

Hit by a bit of frustration (and awlnx’ suggestion to look at TCP windows), i gave debugging this issue that tracked me over two different DCs another shot.

The Case Description(TM)

So, let’s first look at the case. What we have looks roughly like this (we will expand this graph in this article a bit):

      +-------+               +-------+
Ziggo |AS33915|<--------+-----+AS24961| MyLoc
      +-------+         |     +-------+
                        |
                        |     +-------+
                        +-----+AS59645| Me :-)
                        |     +-------+
                        |
                        |     +-------+
                        +-----+AS29670| IN-Berlin
                        |     +-------+
                        |
                        |     +-------+
                        +-----+AS24940| Hetzner
                              +-------+

I have endpoints at MyLoc, Hetzner, IN-Berlin, and my own AS (which, among others) gets upstream from AS24961. For yesterday’s debugging session, i also started testing using the OpenBSD base tool tcpbench; This helped me realize that this was not an SSH issue, but much more a TCP issue. Now, from an endpoint at MyLoc and my own AS, i would see the reduced SSH bandwidth.

A look out of the windows

So, first things first, i started a tcpbench server (tcpbench -s) on my node at AS33915, and another one at one of my machines which is solely routed through MyLoc (and not my own AS; Well, tunnel backbone on another routing table… ;-) ) As expected, transfer speeds were ‘meh’; The same for trying the same thing inside the tunnel. However, this time around, i ran a tcpdump on both sides.

Equipped with the file from my node at home (in AS33915), i fired up wireshark. Using the TCP window size scaling plot (Statistics -> TCP Stream Graphs -> Window Size Scaling), selecting the correct TCP stream, and looking at the correct direction, we end up with the following plot:

We see the window step-wise scaling up… and some weird drops relatively evenly spaced every 2-4 seconds. That… is odd. So, what happens then?

Well, a packet got confused, and decided to get lost. And now, all of a sudden, i knew why bandwidth was so horrendous. Packetloss. Occasional. With implications for the TCP window. And, of course, that packet loss does not mind TCP packets being in a tunnel. Drop is drop.

This leaves a lot of nice opportunities of where things could go amis:

• My “high quality CPE” could be at fault
• My pf config could be funny
• My little VM box behind the Ziggo link (or a switch on the way there) might be an issue
• My routing table toying could lead to issues
• DOCSIS could be DOCSIS
• The switch in my rack could be ‘funny’
• One of the cables to my upstream could be ‘funny’
• Something else could be off

The first things I ruled out were my pf config (which, to be fair, is not soooooo sophisticated), by simply disabling it, and my routing table stuff, simply by setting up a box that would route directly via AS24961 directly. No tables involved. Both did not change the result.

Something else changed

Interestingly, though, when i tested a bit further yesterday, i noticed something else that was odd: Now, both from Hetzner and IN-Berlin, traffic looked a lot closer to what i’d expect. While this adds a lot more oddity to this issue, it also rules out some possible causes:

• This can’t be DOCSIS being funny
• This can’t be my CPE being funny
• This actually can’t be anything around the systems behind the end-user connection in AS33915

Hence, our map now looks somewhat like this (With intermediate steps obviously skipped), as two ASes are certainly different than the other two:

      +-------+               +-------+
Ziggo |AS33915|<-------+------+AS24961| MyLoc
      +-------+        |      +-------+
          ^            |
          |            |      +-------+
          |            +------+AS59645| Me :-)
          |                   +-------+
          |
          |                   +-------+
          +-------------------+AS29670| IN-Berlin
          |                   +-------+
          |
          |                   +-------+
          +-------------------+AS24940| Hetzner
                              +-------+

Routing around

Still, this leaves ‘things’ around my rack as possible issues. So… how to exclude that. Well, technically, i should have multiple routes to the box in AS33915 via my various upstreams. This is where two of my other upstreams come in: AS50629 (LWLcom) and AS34927 (iFog). Playing around with my upstreams, i quickly noticed that, for iFog, things pretty much looked the same as when i was going via AS24961.

Going through AS50629, however, things looked a bit different. My TCP session steadily climbed to 1gbit throughput. The TCP window scaling plot also looked significantly more promising:

This combination of things (not) working has a beautiful side-effect: Both upstreams come in as tagged VLANs via the same cable. Effectively, this rules out the following causes:

• The switch in my rack could be ‘funny’
• One of the cables to my upstream could be ‘funny’

Effectively only leaving me with:

• Something else could be off

Great. Back to start.

Can’t you have issues elsewhere?

Having ruled out all of my stuff as the root cause, i set out to find other places where this might happen. Using the NLNOG Ring Node at MyLoc and with some help from AS34936, i could quickly establish that it really was not just me. Now, hunting for what was different between AS29670, AS24940, and AS50629 and all the others was on. Tool for that, of course, traceroutes, or rather, MTR.

The issue here, though, is that the packet drops do not really seem to be all that frequent, except in an established TCP session. Which makes it really hard to debug. Only when looking at smokeping over 10 days, i see a very modest drop rate around 0.01%, which i do not see towards my node at IN-Berlin. So, let’s play the exclusion game: What looks different (or the same?) between working and non-working traces?

Looking at traceroutes–here via LWLcom and MyLoc as an example–quickly showed that all traffic to AS33915 would go through the Liberty Global B.V. AS (AS6830) before passing through AS9143 (Vodafone/Ziggo) to the Ziggo AS.

nw-test (37.157.249.226) -> <AS33915> 2022-10-08T23:20:59+0000
   Host										Loss%	Snt	Last	 Avg	Best	Wrst	StDev
1. (waiting for reply)
2. lag19.core3-dus1.bb.as24961.net			 0.0%	7	 0.4	 0.3	 0.2	 0.4	0.1
3. lag8.core1-dus-ix.bb.as24961.net			 0.0%	7	 0.6	 0.6	 0.5	 0.7	0.1
4. 76.74.9.169								 0.0%	7	 0.8	 0.9	 0.6	 1.6	0.3
5. ae22.cr2-fra6.ip4.gtt.net				 0.0%	7	 4.8	 4.5	 3.9	 4.9	0.4
6. ip4.gtt.net								 0.0%	7	12.7	 8.8	 4.0	13.0	3.5
7. de-fra02a-rc1-ae-49-0.aorta.net			 0.0%	7	 7.5	 7.6	 7.5	 8.0	0.2
8. nl-ams02a-rc2-lag-11-0.aorta.net			83.3%	7	 7.4	 7.4	 7.4	 7.4	0.0
9. asd-tr0021-cr101-be60-2.core.as33915.net	 0.0%	7	 7.8	 8.0	 7.8	 8.5	0.3
10. gv-rc0052-cr102-et2-2.core.as33915.net	 0.0%	7	 9.6	 9.7	 9.5	10.0	0.2
11. (waiting for reply)
12. HOST.cable.dynamic.v4.ziggo.nl			 0.0%	7	21.3	20.6	16.4	23.1	2.2

And, for comparison, the working one via LWLcom:

gw01.dus01.as59645.net (195.191.197.254) -> <AS33915> 2022-10-08T23:18:43+0000
   Host										Loss%	Snt	Last	 Avg	Best	Wrst	StDev
1. gw02dus01.lwlcom.dus01.as59645.net		 0.0%	8	 0.7	 0.7	0.5		 1.2	0.2
2. lag-108.ear1.Dusseldorf1.Level3.net		33.3%	7	 0.6	 0.7	0.6		 0.8	0.1
3. ae1.3104.edge7.Amsterdam1.level3.net		 0.0%	7	 4.1	 4.4	3.8		 6.8	1.1
4. nl-srk03a-ri1-ae-8-0.aorta.net			 0.0%	7	 7.6	 7.6	7.5		 7.8	0.1
5. asd-tr0021-cr101-be65.core.as9143.net	14.3%	7	 8.6	 8.7	8.6		 8.8	0.1
6. gv-rc0052-cr102-et2-2.core.as33915.net	 0.0%	7	10.5	10.3	10.1	10.5	0.1
7. (waiting for reply)
8. HOST.cable.dynamic.v4.ziggo.nl			 0.0%	7	21.8	21.6	18.3	22.9	1.5

Looking again a bit closer at the traceroutes, we finally find one host that is the same for all traceroutes that show the occassional drops ruining our throughput, that is absent in all other traces: nl-ams02a-rc2 Sometimes it is nl-ams02a-rc2-lag-11-0.aorta.net. (84.116.130.150), sometimes it is nl-ams02a-rc2-lag-12-0.aorta.net (84.116.139.125). However, when packets drop it is always nl-ams02a-rc2, and if they don’t there is no nl-ams02a-rc2. Also, note that this box seems to be rather busy dropping control plane packets, rocking an 83.3% packet loss over 7 packets (yeahyeah, router not pinger; but let’s call this ‘circumstancial evidence’; Still, over more packets that box usually gets to ~93% loss). It must be busy with something.

Taking nl-ams02a-rc2 into account, this leaves us with this version of our graph:

  Ziggo                                   iFog
+-------+                               +-------+        +-------+
|AS33915|           +-------------------+AS34936|<---+---+AS59645| Me :-)
+-------+           |                   +-------+    |   +---+---+
    ^               |                                |       |
    |               |                   +-------+    |       |
    |               +-------------------+AS24961|<---+       |
    |               |                   +-------+            |
    |               |                     MyLoc              |
    |               | < nl-ams02a-rc2                        |
    |               v                                        v
+---+---+       +-------+                                +-------+
| AS9143|<------+ AS6830| Liberty Global                 +AS50629| LWLcom
+-------+       +-------+                                +-------+
Vodafone/           ^                                        |
  Ziggo             | < no nl-ams02a-rc2                     |
                    |                                        |
                    +----------------------------------------+
                    |
                    |
                    |                                    +-------+
                    +------------------------------------+AS29670| IN-Berlin
                    |                                    +-------+
                    |
                    |                                    +-------+
                    +------------------------------------+AS24940| Hetzner
                                                         +-------+

Making it ping^Wnot drop again

To test our hypothesis that nl-ams02a-rc2 has something to do with our bandwidth issues, the most straight-forward way is changing the path for one of the currently affected ASes to one no longer passing it. After filling a ticket with a write-up of the issue with MyLoc, they changed the path for their route to AS33915. Lo and behold: The loss is gone, my windows behave properly, and i can finally saturate my link.

  Ziggo                                   iFog
+-------+                               +-------+        +-------+
|AS33915|           +-------------------+AS34936|<---+---+AS59645| Me :-)
+-------+           |                   +-------+    |   +---+---+
    ^               |                                |       |
    |               |                   +-------+    |       |
    |               |             MyLoc +AS24961|<---+       |
    |               |                   +-------+            |
    |               |                       |                |
    |               | < nl-ams02a-rc2       |                |
    |               v                       |                v
+---+---+       +-------+                   |            +-------+
| AS9143|<------+ AS6830| Liberty Global    |            +AS50629| LWLcom
+-------+       +-------+                   |            +-------+
Vodafone/           ^                       |                |
  Ziggo             | < no nl-ams02a-rc2    |                |
                    |                       |                |
                    +-----------------------+                |
                    |                                        |
                    +----------------------------------------+
                    |
                    |
                    |                                    +-------+
                    +------------------------------------+AS29670| IN-Berlin
                    |                                    +-------+
                    |
                    |                                    +-------+
                    +------------------------------------+AS24940| Hetzner
                                                         +-------+

Conclusion and TODO

So, in conclusion: Sometimes it is really worth it to dig into something very strange that is happening to you. And, methodological debugging can be fun (especially the feeling when you FINALLY figure it out.

Also, there is still the oddity left of why seemingly only OpenBSD, andnot Windows/Linux. My personal guess is the specific implementation of TCP congestion control/window resizing in OpenBSD being just a tad more sensitive.

At the same time, this gives me the interesting TODO item of figuring out how to get AS6830 to fix this issue; Even though i can fix this myself for ASes where i can either change routes myself or motivate others to do so… there is a bit more ASes out there than those i can change routes for (which er… is one)… and i kind of like to fetch stuff via TCP. Let’s be honest, if i call the Ziggo customer support, they’ll probably have me reset my router a couple of times. Then again… probably worth trying. :-|