A Content Delivery Network (CDN) describes the following: distributing content to users via multiple edge servers, strategically positioned around the globe. The advantage for the end user is high availability and quick delivery, while the network providers profit through reduced bandwidth cost and routing effort - the edge node is closer to the user and sends traffic to lesser foreign systems, whereas “closer” refers to proximity in connectivity and often implicitly physical distance as well.

Anycast is an established technique used by some CDNs. With eBGP we can announce the same IP address prefix from multiple geographically distributed locations to neighboring autonomous systems (AS). The adjacent ASes are likely to choose the shortest, most convenient path to our IP address prefix.

Example: Cloudflare’s 1.1.1.1

Cloudflare’s well known primary DNS resolver is reachable via 1.1.1.1. I started a traceroute from my laptop to this address and reached Cloudflare’s AS13335 on the 12th hop. This observation from a network layer tells us which ASes and their hops are traversed on the communication path, what it does not reveal is which edge node we’re reaching - they all share the 1.1.1.1 address after all.

$ traceroute -A 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  10.181.223.252 (10.181.223.252) [*]  4.475 ms  5.136 ms  5.125 ms
 [...]
11  kr-bls6-0.x-win.dfn.de (188.1.246.17) [AS680]  17.653 ms  17.629 ms  18.189 ms
12  162.158.112.33 (162.158.112.33) [AS13335]  18.399 ms  19.150 ms  18.569 ms
13  one.one.one.one (1.1.1.1) [AS13335]  17.714 ms  16.092 ms  16.055 ms

Let’s move from network layer up to application layer for trying to identify where 1.1.1.1 is located. We can send a DNS query and include a request for the namerserver’s configured identifier (NSID):

$ dig +noall +comments +nsid @1.1.1.1
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60323
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; NSID: 74 78 6c 30 31 ("txl01")

… and get txl01 in our reply. TXL likely is the IATA airport code for Berlin Tegel Airport, which indicates this DNS server is located in Berlin, Germany.
I ran the same request from a VPS in Frankfurt, and the reply included ; NSID: 66 72 61 31 33 ("fra13") - an edge node in Frankfurt, Germany.

This example shows that we reach a different Cloudflare edge, depending on our vantage point. However, it also shows that proximity in connectivity does not strictly imply physical proximity: The Frankfurt VPS connects to an edge node in Frankfurt; My laptop, while sitting in the Bavarian State Library in Munich, connects to an edge node in Berlin. Why is this traffic routed through the whole of Germany? To answer that, some context information of the network is necessary. The library is connected to the internet via the German research network DFN X-WiN, which seems to have made the routing decision of reaching 1.1.1.1 via its exit router in Berlin. The rDNS name of the 11th hop contains bls (see traceroute above) - Berlin, Lützowstraße / Schöneberg - as the location of the X-WiN core router, which is one of the four points where X-WiN routes “externally”, according to their plenum presentation in October 2025. Cloudflare does have edge servers in Munich, however since the exit routers of X-WiN are located elsewhere, their respective neighbors are preferred.

The map above shows DFN X-WiN locations and their interconnection. External hosts are reached via Frankfurt, Berlin, Düsseldorf and Hamburg. Image source: DFN Plenum NACS-1

Acquiring an AS Number and IPv6 prefix

IP addresses are handed out by a regional Internet registry (RIR), whereof currently five exist worldwide. My region is covered by the RIR called RIPE (~“Europe IP Networks”), which then allocates blocks of IP addresses to local Internet registries (LIR). Those LIRs can lend their addresses to customer organizations.

I decided on the London-based LIR Lagrange Cloud Technologies Ltd as a sponsor for my AS Number (ASN) and IPv6 prefix. “Sponsor” does not imply that this service is free - both the LIR and the RIR charge fees, like a normal product subscription.

The application process for an AS at RIPE involves creating several objects in the RIPE Database. Some of the objects are then updated by RIPE and/or the sponsoring LIR after approval. Generally, the mnt-by tag signals who is maintaining an object.

The relation between those objects is rather intuitive: Both the AS and the IPv6 prefix are “owned” by my organization, while the prefix is maintained by my LIR. The contact points for both are directed to a “role” objects, which lists my email addresses. There exist more objects, e.g. “mntner” as above referenced by mnt-by.

Configuring BGP

bgp is a brilliant information hiding protocol. ~Randy Bush, 1. Nov 2009

Now that we have an ASN and IPv6 prefix, the next step is to send out announcements from our server. I decided to use bird3 as the software for BGP communication. We configure bird to announce an IP prefix to our neighbor. A neighbor can be another Internet router directly connected via layer 2, or some distant server which has a tunnel connection with us. Anyhow, in order for a BGP session to successfully establish between two routers, both parties need to have their import and export filters configured accordingly. In the context of the quote above, announcements can be used to offer routing paths for IP address prefixes. It is up the router which of it’s routable prefixes to include in the announcements. I pay my upstream to connect me to the rest of the Internet (via his peers and upstreams), which is why I can expect him to announce internet routes to me. If we did not have this paid contract, he would have no interest in providing me connectivity to the rest of the world, instead exclude those routes from his announcements. The same applies in the other direction: My upstream has no interest in propagating my announced address prefix to his peers thus providing them a path to reach me, unless I pay him to do so.

I ordered a VPS which explicitly supports BGP sessions, accepting my prefix announcement through the provider’s AS. The following is my (minimal) bird configuration to achieve exactly that.

log syslog all;

router id 118.91.187.179;  # should be unique within our AS,
                           # usually public IPv4 of the VPS

define MY_ASN = 218373;
define MY_PREFIX = 2a06:9801:15bc::/48;

protocol device {}

protocol kernel kernel6 {
    # for now let's not import routes from
    # linux to bird or vice versa.
    # The VPS already has a default route configured.
    ipv6 {
        import none;
        export none;
    };
}

protocol static origin6 {
    ipv6;
    route MY_PREFIX blackhole;
}

filter ifog_import6 {
    if net = ::/0 then {
        # accept a default route announce via
        # the neighbor
        bgp_local_pref = 80;
        accept;
    }
    reject;
}

filter ifog_export6 {
    if net = MY_PREFIX && proto = "origin6" then {
        # only export our prefix
        accept;
    }
    reject;
}

protocol bgp ifog6 {
    local as MY_ASN;

    source address 2a0c:9a40:2711:111::1022;
    neighbor 2a0c:9a40:2711:111::1 as 34927;

    ipv6 {
        import filter ifog_import6;
        export filter ifog_export6;
        import limit 10 action disable;
    };
}

After loading this configuration, we can observe the exported prefix:

$ birdc show route export ifog6
BIRD 3.4.0 ready.
Table master6:
2a06:9801:15bc::/48  blackhole [origin6 2026-11-01] * (200)

While upstreams react promptly to changes in announcements, some other actions like RIPE Database updates and RPKI propagation might take more time (hours/days) due to caching.

More PoPs

We want to build a CDN with multiple geographically distributed edge nodes. Therefor I’ve rented another VPS in Frankfurt, Germany and another one in Ashburn, USA. The configuration is mostly identical to the above, apart from router_id, source address and neighbor information.

With that we have multiple locations announcing the same IPv6 prefix. The following BGP connectivity diagram shows how my prefix reaches the Tier 1 ISPs. My AS AS218373 is connected via 3 upstreams (HYEHOST, iFog, Ouiheberg), each of them houses one of my rented VPS.

Image source: https://bgp.tools/as/218373#connectivity

DNS Nameserver using Anycast

Now that the IP address prefix is reachable, it is time to provide some services on this infrastructure. On the edge servers I have assigned the first two addresses of my prefix to the loopback interface. Incoming packets with those addresses as destination will be routed to the local machine.

$ ip -c -6 a show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet6 2a06:9801:15bc::2/128 scope global
       valid_lft forever preferred_lft forever
    inet6 2a06:9801:15bc::1/128 scope global
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever

Address 2a06:9801:15bc::1 should provide the authoritative DNS Server for s4nd.ro. I’ve decided to stick with the creators of the previously configured bird3 software, CZ.NIC. They offer the authoritative DNS server solution Knot DNS, which I’ve installed and configured on my edge nodes.

Knot can be set up in a primary-secondary cluster. The primary server is the source of truth and actively syncs the zone to the secondary servers. If incoming zone update requests - e.g. via rfc2136 - reach the secondary server, Knot will simply forward it to the primary and proxy the reply. I have connected the edge servers via wireguard tunnels and set up the Knot syncing on top of that.

My primary DNS server ams01 is configured in the following way:

server:
    rundir: "/run/knot"
    nsid: ams01  # note: we made use of this "location hint" in
                 # the cloudflare 1.1.1.1 example above
    user: knot:knot
    automatic-acl: on
    listen: [ 2a06:9801:15bc::1@53, 10.13.37.1@53 ]

log:
  - target: syslog
    any: info

database:
    storage: "/var/lib/knot"

key:
  - id: fra01-xfr-notify-key
    [hidden]
  - id: iad01-xfr-notify-key
    [hidden]

remote:
  - id: fra01
    address: 10.13.37.2@53
    key: fra01-xfr-notify-key
  - id: iad01
    address: 10.13.37.3@53
    key: iad01-xfr-notify-key

template:
  - id: default
    storage: "/var/lib/knot"
    file: "%s.zone"

zone:
  - domain: s4nd.ro
    notify: [fra01, iad01]

The zonefile for s4nd.ro is nothing special for now. It includes start of authority (SOA) record for the zone, a reference to the nameserver ns.s4nd.ro, which is reachable on 2a06:9801:15bc::1. Furthermore there’s a useless hello world txt record and a quad-A record for the web server on 2a06:9801:15bc::2 that is going to be set up in the next section.

s4nd.ro.            	3600	SOA	ns.s4nd.ro. hostmaster.s4nd.ro. 1792769127 21600 3600 604800 86400
s4nd.ro.            	3600	NS	ns.s4nd.ro.
s4nd.ro.            	3600	AAAA	2a06:9801:15bc::2
hello.s4nd.ro.      	3600	TXT	"hello world"
ns.s4nd.ro.         	3600	AAAA	2a06:9801:15bc::1

There more to experiment here, e.g. with the recently standardized SVCB and HTTPS Resource Records, or the seldom used LOC for querying coordinates.

Secondary Node

For completeness, here’s what the Knot DNS configuration looks like on one of the secondary servers.

server:
    rundir: "/run/knot"
    nsid: fra01
    user: knot:knot
    automatic-acl: on
    listen: [ 2a06:9801:15bc::1@53, 10.13.37.2@53 ]

log:
  - target: syslog
    any: info

database:
    storage: "/var/lib/knot"

key:
  - id: fra01-xfr-notify-key
    [hidden]

remote:
   - id: ams01
     address: 10.13.37.1@53
     key: fra01-xfr-notify-key

template:
  - id: default
    storage: "/var/lib/knot"
    file: "%s.zone"

zone:
  - domain: s4nd.ro
    master: ams01

Serving a webpage via Anycast

For serving a resource via HTTP I installed the nginx web server on each of the edge nodes. It is configured to listen to 2a06:9801:15bc::2 on port 80 and 443. However, every incoming HTTP requests unencrypted on port 80 is replied with a 301 permanent redirect to https://....

server {
    listen [::]:80;
    server_name s4nd.ro *.s4nd.ro;
    return 301 https://$host$request_uri;
}

server {
    listen [::]:443 ssl;

    server_name s4nd.ro *.s4nd.ro;

    ssl_certificate     /etc/nginx/tls/s4nd.ro/current/cert.pem;
    ssl_certificate_key /etc/nginx/tls/s4nd.ro/current/key.pem;

    add_header Strict-Transport-Security "max-age=31536000" always;

    root /var/www/html;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

TLS certificates required for serving TLS encrypted HTTP traffic can be obtained free of charge from the Let’s Encrypt certificate authority. This process better be automated, especially since the lifetime of issued certificates will be reduced, requiring more frequent rollovers.

I’ve decided to use uacme as an Automatic Certificate Management Environment (ACME) client for communication with Let’s Encrypt. Through hook scripts it will update my DNS zone to include the DNS-01 challenge payload as received by Let’s Encrypt. This update is done via rfc2136 by invoking the knsupdate command line utility. After successful reception of renewed certificates, another script will propagate them to the remaining edge nodes and reload nginx afterwards.

The CDN now serves a static web page.

RIPE Atlas & Traffic Engineering

RIPE Atlas is a global Internet measurement platform. Volunteers around the world contribute by operating “probes”, which can be orchestrated through the RIPE Atlas platform to run measurements like ping, DNS lookup, traceroute and more. This is a very effective method for us to assess which edge nodes are reached from different vantage points. We can use this information to fine-tune our BGP announcements, e.g. prepending our ASN to an edge router’s announced path multiple times such that it appears less attractive to our neighbors.

Image Source: RIPE Atlas. This exact measurement result is public, with id 230540595.

Before any traffic engineering, most hosts reached my edge node located in Amsterdam, Netherlands. This VPS’ upstream through iFog AS23927 is apparently very well connected such that most ISPs considered it the best and most convenient route. However, the ripe measurement shows us that non-european hosts suffer under this decision with high round-trip-times. Northamerican hosts could expect a much lower latency if their ISP chose my edge node located in Ashburn.

One technique for steering the other AS’s route decisions was mentioned before: ASN prepending. By prepending our ASN to the BGP announcement multiple times, we artificially lengthen the path. A common routing decision by AS routers is to choose the shortest path, which due to our prepending might now have changed.

Another technique for traffic engineering is to add specific BGP Communities to announcements. BGP Communities are like tags, which our neighboring ASes can interpret to do traffic engineering on their side (for our prefix).

In the following example I’ve modified the FRA01 edge node to attach two BGP communities. 174:972 is offered by Cognet (Tier 1 ISP) to decrease the local preference for announcements towards the region Northamerica. 5511:100X is offered by another Tier 1, namely Orange S.A.. The X can be set to {0, 1, 2} and instructs their router to add X ASN prepends towards US-American peers.

filter oui_export6 {
    if net = MY_PREFIX && proto = "origin6" then {
        bgp_community.add((174, 972)); # Cognet; localpref 10 upon entering NA Region
        bgp_community.add((5511, 1001)); # Orange S.A.; TUNE US peers 1x
        accept;
    }
    reject;
}

For my edge node in Amsterdam, I made similar changes. My upstream provider iFog offers communities allowing per-upstream prepends. Through trial and error with a lot of measurements through RIPE Atlas, I’ve settled on these values:

filter ifog_export6 {
    if net = MY_PREFIX && proto = "origin6" then {
    bgp_path.prepend(MY_ASN);
    bgp_path.prepend(MY_ASN);
    bgp_community.add((34927, 9131)); # Hurricane Electric
    bgp_community.add((34927, 9891)); # Exascale
    bgp_community.add((34927, 9561)); # Liberty Global
    bgp_community.add((34927, 9481)); # RETN
    accept;
    }
    reject;
}

A follow-up measurement confirms that many hosts in Northamerica are now routed to my local edge server, decreasing their round-trip-time. Image source: RIPE Atlas measurement 231290562

With that the DIY CDN is complete for now. For the future I plan to join an Internet Exchange, and experiment more with bird3’s configuration options, and also set up AS Provider Authorization (ASPA).