Link Preview System Design: Unfurling URLs Safely with SSRF Defences

Your server visits a web address that any user can type, from inside your own network. Most of the time that address is a news article. Sometimes it points at a private machine your server can reach but the public cannot.

Picture a common setup. A team chat app shows a preview card under every link, and a worker on its cloud servers fetches each page to build the card. The setup is made up, but every check in this guide ran in our own tests.

Link preview system design is the work of making that worker safe and cheap. Four things go wrong without it. A link points at an internal admin page or the cloud metadata service, where a server reads facts and keys about itself, and the worker happily reads it. A link points at a 5 GB file or a page that sends one byte a second, and workers pile up.

A popular link is pasted in 500 channels at once, so you fetch the same page 500 times. And the site, or the reader, learns things about your users that they never agreed to share.

This guide builds the worker step by step, from parsing tags to blocking private addresses, pinning DNS answers, handling redirects, caching and privacy. Every code behaviour here was run on Python 3.13.18 with the standard library only, on 29 October 2026. Product and spec facts come from their own pages.

Eight words carry the whole guide. Each one is tied to the chat app's worker.

• URL: a web address, such as a link to a news story. The user picks it, so the worker must treat it as untrusted input.
• DNS: the internet's phone book. It turns a name like news.example.com into one or more IP addresses, the numbers computers use to reach each other.
• Private address: an IP address that only works inside one network, such as 10.0.0.5. Your servers can reach these; outsiders cannot.
• SSRF, server-side request forgery: tricking a server into sending a request to a place the attacker cannot reach directly.
• Redirect: a reply that says the page has moved, with a Location header naming the new address. Browsers and many libraries follow it automatically.
• Open Graph: a set of meta tags in a page's head, such as og:title and og:image, that say how the page should look when shared.
• oEmbed: a format where a site offers a small JSON description of a link, fetched from a separate address the site advertises.
• TTL, time to live: how long a cached copy or a DNS answer may be reused before it must be fetched again.

Link preview system design covers the pipeline from a pasted link to a finished card. The message is saved first, and the preview arrives a moment later. Nobody waits on a slow website before their message appears.

In the chat app, the message service spots a URL in a new message and looks it up in the preview cache. On a hit, the card is attached at once. On a miss, it puts a job on a queue, which is a waiting list that workers take tasks from.

A worker takes the job, checks the URL, fetches a capped slice of the page and parses the tags. It stores the result in the cache and pushes the card to everyone in the channel.

Real products publish parts of this. Slack's robots page, as of 29 October 2026, says its unfurler fetches as little of a page as it can with Range headers. A Range header asks for only some bytes. Slack also caches results globally for around 30 minutes.

Mostly from Open Graph meta tags, with plain HTML and oEmbed as fallbacks. The worker reads them from the top of the page, so it rarely needs the whole document.

Open Graph tags

The Open Graph protocol page lists four required properties: og:title, og:type, og:image and og:url. It recommends optional ones such as og:description and og:site_name. Each is a meta tag in the page's head, with the name in a property attribute and the value in a content attribute.

Two rules from that page matter for parsers. When a property repeats, the first tag from the top wins. When og:image is set, og:image:alt should be set too, and it describes the image rather than captioning it.

If a page has no Open Graph tags, a sensible fallback is the title element and the meta tag named description. If even the title is missing, show the bare URL.

oEmbed

oEmbed is a format for showing an embedded representation of a URL on another site. A page can advertise its oEmbed address with a link tag whose type is application/json+oembed. The worker then asks that address for JSON.

The spec defines four response types. A photo carries an image url with width and height. Video and rich carry ready-made HTML, and link carries only generic fields. Every response has a type and a version, which must be 1.0.

The oEmbed address is just another URL chosen by the page, so it goes through the same guard as the first link. The spec also warns that embedded HTML is a route for XSS, cross-site scripting. It recommends showing that HTML in an iframe, a boxed-in frame for another page, served from a different domain.

How many bytes to read

Meta's crawler page for link sharing, as of 29 October 2026, says Open Graph properties must sit within the first 1 MB of a page or they are cut off. A cap near 1 MB is therefore a reasonable starting point for your own worker.

We tested a small parser built on Python's html.parser module on a sample page. It kept the first og:title of two and turned a relative og:image path into a full address. With the page cut to 200 bytes, it still returned the title it had seen.

Pause and predict: a page's og:title contains an encoded bold tag, written with ampersand entities. What does the parser hand you?

It handed back a real bold tag. The parser decodes entities in attribute values, so encoded markup arrives as markup. Escape every field when you display it, and drop image values whose scheme is not http or https. Our parser dropped a data: image value for that reason.

Because the request leaves from inside your network, with your network's access. A firewall that blocks outsiders does not stop a request your own server sends. That gap is SSRF.

An everyday picture helps. Think of an office receptionist who will phone any number a visitor writes down. Outside numbers are fine. But the receptionist can also dial internal extensions, such as the finance office, that no outsider can call.

Map it part by part. The visitor is the user pasting a link. The receptionist is your worker. Internal extensions are private addresses, and the phone book is DNS. Blocking SSRF means the receptionist checks the number before dialling.

Where the picture breaks: a real phone book does not change between looking up a number and dialling it. DNS answers can change from one second to the next, which a later section shows.

Why the cloud metadata address matters most

On AWS, Google Cloud and Azure, a server can ask a metadata service about itself at 169.254.169.254. That is a link-local address, one that only works on the local network link. AWS documents that address, and an IPv6 one, fd00:ec2::254, for some instances. AWS's 2019 security blog post says the service makes credentials available for the role attached to the server.

So a worker tricked into fetching that address may return cloud credentials inside a preview card. The OWASP SSRF Prevention Cheat Sheet lists 169.254.169.254 for AWS, Google Cloud and Azure in its deny list.

AWS added IMDSv2, version 2 of its Instance Metadata Service, as a second layer. It needs a session token obtained with a PUT request, an HTTP method mostly used for uploads, and the EC2 docs say PUT requests carrying an X-Forwarded-For header are refused. The token reply also has a hop limit of 1 by default. AWS's blog says this stops the token leaving the instance if the instance is misconfigured as an open router, firewall or NAT device.

The EC2 docs, as of 29 October 2026, say both versions work by default and that you can require IMDSv2. AWS's November 2023 news post says Amazon Linux 2023, console Quick Start launches and instance types released from mid-2024 use IMDSv2 only. Require it on unfurl workers. But treat it as a backup, not the fix, since other internal services have no such token.

The naive fix and why it fails

The naive fix is to scan the URL text for words like localhost or numbers starting with 10. This is the wrong approach. An address can be written in several spellings, and a harmless-looking name can resolve to a private address.

In our test, the system resolver turned a short form and a plain integer form of the loopback address into 127.0.0.1. A text check would have missed both. The fix is to check the IP addresses the name resolves to, not the text.

Everything that is not a normal public unicast address. That means private, loopback, link-local, shared, documentation, reserved, broadcast and multicast ranges, in both IPv4 and IPv6. Unicast means one sender talking to one receiver.

You do not need to invent this list. IANA, the body that manages internet number registries, keeps IPv4 and IPv6 special-purpose address registries. Each row has a Globally Reachable column. RFC 6890 set out how those registries are structured. Both registries show a last update of 9 October 2025, as of 29 October 2026.

Python's ipaddress module gives each address an is_global flag based on the IANA registries. Use is_global, not the absence of is_private. The docs say both are False for 100.64.0.0/10, and our run confirmed it.

Two gaps showed up in our tests, and one worry did not. Multicast addresses such as 224.0.0.1 reported is_global True, so add an is_multicast check. IPv4-mapped IPv6 addresses followed their IPv4 address, as the docs promise.

The second gap is NAT64, a gateway that lets IPv6-only machines reach IPv4 by wrapping the IPv4 address inside 64:ff9b::/96. An address there that wraps a link-local IPv4 address reported is_global True. On networks with NAT64, unwrap it and check the inner address.

Versions matter here. The docs mark is_private and is_global as changed in Python 3.13, with fixes for parts of 192.0.0.0/24, 2002::/16 and 2001::/23. Run your own address tests on the exact Python version you deploy.

This is the check we ran. The test file fed it 24 blocked addresses and 4 public ones, and every one landed on the expected side.

import ipaddress

NAT64_WKP = ipaddress.ip_network("64:ff9b::/96")  # RFC 6052 well-known prefix


def is_public_ip(ip):
    """True only for globally reachable unicast addresses."""
    ip = ipaddress.ip_address(ip)
    if ip.version == 6:
        if ip.ipv4_mapped is not None:  # ::ffff:a.b.c.d
            ip = ip.ipv4_mapped
        elif ip in NAT64_WKP:  # NAT64 gateways turn these into the embedded IPv4 address
            ip = ipaddress.IPv4Address(int(ip) & 0xFFFFFFFF)
    if ip.is_multicast:  # is_global is True for many multicast addresses
        return False
    return ip.is_global  # NOT "not is_private": 100.64.0.0/10 is neither

One surprise for testers: documentation addresses such as 192.0.2.10 are not global, so this check blocks them. To test the allowed path, give your code a fake resolver instead of a real public address.

Resolve the hostname once, refuse if any returned address is not public, and connect to one of those checked addresses. Pinning means the connection uses that exact checked IP, with no second DNS lookup.

1. 1

   Check the URL's shape

   Allow only http and https. Refuse user names or passwords inside the URL. Allow only ports 80 and 443 unless you have a reason.

2. 2

   Resolve once

   Ask DNS for every A and AAAA record, the IPv4 and IPv6 answers. Collect all of them.

3. 3

   Check every answer

   If any answer is not public, refuse the whole URL. A name with one public and one private answer is a trap, not a coin toss.

4. 4

   Pin and connect

   Open the socket, the network connection, to a checked address. Send the hostname in the Host header, which names the site you want. Use it for TLS, the encryption behind HTTPS, so certificates are still verified.

5. 5

   Never let the library follow redirects

   Read the Location header yourself and send the new URL back to step 1.

The OWASP cheat sheet gives the same order for apps that must reach unknown sites. It says to check that each resolved address is public, connect only to a validated address, and turn off redirect following in the HTTP client.

Why the second lookup is the bug

DNS rebinding means a name gives a public answer when you check it and a private one when you connect. Whoever controls a domain's DNS can set very short TTLs, so the first answer is not reused.

We simulated it on loopback with a resolver whose answer changed after the first call. Our fake public site sat on 127.0.0.1, and a fake internal service on 127.0.0.2.

The naive code checked the name, passed, then let Python's urllib fetch the page. urllib looked the name up a second time and got the internal answer. It returned the internal page's body. The pinned fetch made one lookup and never touched the internal service.

Pinning must not break HTTPS. In a separate test with a self-signed certificate for a made-up name, our pinned client sent that name as SNI, the field that tells the server which site you want. A different name failed with Hostname mismatch.

The OWASP cheat sheet makes the same point. Bind the connection to the validated address, keep the original hostname for the Host header, SNI and certificate checks, and apply the rule to retries and fallback connections too.

Follow redirects yourself, with a small limit, and run each hop through the full check. Cap the bytes you read and cap the total time, not just each socket read.

Redirects

Pages move, so a worker that ignores every redirect will show many blank cards. The fix is to follow them manually. Each new address goes back through the scheme check, a fresh resolve, the address check and pinning.

We tested both ways on loopback. A fake public page answered with a 302 redirect to a name that resolved to the internal address. Python's urllib, given the first URL after a passing check, followed it and returned the internal page.

Our guarded fetch refused the second hop and never contacted the internal server. With a page that redirected to itself and a limit of three, the guard made four requests and stopped.

Byte and time limits

Read in chunks and stop at your cap. Against a 5 MiB test page, our fetch read exactly 1,000,000 bytes and closed the connection. MiB means mebibytes, units of 1,048,576 bytes.

Time needs a total deadline. A socket timeout counts the gap between reads, so a server that sends one byte every 0.3 seconds never trips a 2-second timeout. Our test server did exactly that, and only the total deadline stopped the fetch, after 2.1 seconds.

Check the Content-Type header too, and skip anything that is not HTML unless you build image or video cards. We asked for uncompressed bodies. If you accept compression, apply your byte cap to the bytes after decompression.

Because code has bugs, and a second layer catches what the first one misses. The OWASP cheat sheet recommends a firewall that defines legitimate flows and network segregation that blocks bad calls at the network level.

• Run unfurl workers in their own subnet, a separate slice of the network, with outbound rules that deny private ranges and the metadata address.
• Or send all worker traffic through an egress proxy, a single outbound gateway, that applies the same address checks.
• Give the workers no cloud role and no secrets. If the metadata service is reached anyway, it has nothing valuable to hand out.
• Require IMDSv2 on the worker instances, so a plain GET to the metadata address fails.
• Log every blocked URL with the resolved addresses. A spike usually means someone is probing you.

The same thinking applies to webhooks, where your server calls URLs that customers register. SevyDevy's webhook design guide covers delivering webhooks, and those outbound calls need this same guard.

Cache per URL, not per message, with a time limit, and cache failures for a short while too. The same news link pasted in 500 channels should cost one fetch.

Normalise the key first: lower-case the scheme and host, drop the fragment after the hash sign and remove default ports. Be careful with query strings, since they often change the page. Keep them unless you know a parameter is only for tracking.

When many requests miss the cache for the same URL at once, let one worker fetch while the others wait for its result. This is often called request coalescing. Without it, a viral link sends a burst of identical fetches, the stampede problem covered in SevyDevy's caching guide.

Let the cache serve the card, not the original page. You do not need the HTML after parsing, and keeping less data means less to protect.

It depends on your policy, and big products differ. RFC 9309, the robots exclusion standard from September 2022, defines crawlers as automated clients. A preview fetch triggered by one person sits in a grey area.

Whatever you choose, identify yourself honestly. Use a clear user agent, the name a client sends to say what it is, with a page explaining the bot and how site owners can opt out. RFC 9309 also says its rules are not a form of access authorisation, so never treat robots.txt as security.

If you do honour robots.txt, fetch it through the same guard. Python's urllib.robotparser read method calls urllib's urlopen directly in 3.13.18, so it bypasses your checks. Fetch the file yourself and pass its lines to the parse method instead.

Know the parser's choices too. In the 3.13.18 source, read treats 401 and 403 replies as disallow everything. RFC 9309 treats replies from 400 to 499 as unavailable, and then a crawler may access any resource. For replies from 500 to 599, both the source and the RFC assume complete disallow.

Someone always learns that a link was shared. The design question is who: your server, the target site, or the people in the chat.

If each reader's app fetched the preview, the site would see every reader's IP address. Any automatic fetch, by readers or by your server, can also use up a link that works only once. Server-side fetching hides readers from the site, but your server now sees every link.

Signal's 2019 launch post describes a third path. The sender's app fetches the preview through a proxy that hides its IP address. TLS runs directly between the app and the site, so the Signal service never sees the URL. Previews are optional, need HTTPS and, at launch, worked only for Imgur, Instagram, Reddit and YouTube links.

For the chat app, three habits help. Serve images through your own image proxy, so readers never contact the original site. Slack's robots page says its image proxy also hides referrer information. Let users turn previews off. Skip previews for links inside private or end-to-end encrypted chats unless the sender's app builds them.

Interviewers use this question to see whether you spot the security problem hiding inside a simple feature. Lead with scope, then spend your time on the fetch path.

1. 1

   Clarify the scope

   Chat or social feed, rough messages per second, and whether previews must appear instantly or a second later.

2. 2

   Draw the async flow

   Save the message first, check the preview cache, queue a job on a miss, push the card when ready.

3. 3

   Name the SSRF risk

   The worker fetches attacker-chosen URLs from inside your network, including the cloud metadata address.

4. 4

   Describe the guard

   Allow http and https only, resolve once, check every address with the registry-based rules, pin the IP, keep the hostname for TLS.

5. 5

   Handle redirects and limits

   Follow redirects manually through the guard, cap hops, bytes and total time.

6. 6

   Add defence in depth

   Isolated subnet or egress proxy, no credentials on workers, IMDSv2 required, alerts on blocked fetches.

7. 7

   Cache and coalesce

   Per-URL cache with a TTL, short negative caching, one fetch per URL at a time, images through a proxy.

8. 8

   Close with policy

   A clear user agent, an opt-out, a robots.txt decision and a privacy choice about who fetches.