Sockets, Buffers, and the Stream

Be conservative in what you do, be liberal in what you accept from others.

— Jon Postel, RFC 793

Every program that talks to a network eventually calls a function named something like recv(). It’s entirely possible to write networked software for years without asking what that function does, what it’s reading from, or why every serious library wraps it in a “buffered” something-or-other. I know, because I did.

This is the article I wish I’d read before that. It starts below the socket, with what the network actually delivers, and works its way up through the socket, the buffers on both sides of it, the protocols that give the bytes meaning, and finally the async machinery that most of us meet the socket through today. Each section leans on the one before it. If a later section seems to skip a step, the step is in an earlier one.

Skip the margin notes and the article still works. Read them and you’ll find out who got paged at 3 a.m. so that you don’t have to be.The margin is where the history lives. Every concept here was invented by a specific person who was annoyed by a specific problem, and I find that makes them a lot easier to remember. Some notes are that; the rest are commentary.

Part I — What the network hands you

1. A connection is a stream of bytes

Here’s the first thing to get straight, because it quietly underlies half the confusion in this subject: the network is made of packets, and your program will never see one.Vint Cerf and Bob Kahn published the design that became TCP in 1974. The first version was a single protocol that did everything: routing packets across networks and making them reliable. In 1977 Jon Postel argued, in a memo that is still a bit of a mic drop, that this violated layering: the reliability machinery belonged in the endpoints, not in the network. So TCP was split. IP delivers packets and promises nothing; TCP, sitting on top, turns them into a stream.

The wire carries packets, individual chunks of at most a few hundred to a couple of thousand bytes. On Ethernet the ceiling is about 1,500 bytes per frame, which after the IP and TCP headers leaves roughly 1,460 bytes for your data. Packets get lost. They arrive out of order. Occasionally they arrive twice. This is not a defect; it’s the deal you make for being able to route data across tens of thousands of independently owned networks with no central coordinator.

TCP’s job is to hide all of that. It numbers every byte, has the receiver acknowledge what arrived, retransmits what didn’t, and holds early arrivals until the gaps are filled. What it presents to your program is a stream: an ordered, reliable, unbroken sequence of bytes, exactly as sent, with no indication of how they were chopped up in transit.

So if you send a 10,000-byte message, it leaves as seven-ish packets and arrives as… whatever the receiving program asks for. One read might return 4,096 bytes, the next 2,000, the next the remaining 3,904. The boundaries between packets are gone by the time you look; they were reassembled and discarded inside the kernel. Nothing in your program, not your buffer sizes, not your message sizes, needs to know or care about the packet size. If you ever catch yourself reasoning about “the TCP limit” while sizing an array, stop: that limit lives below the floor you’re standing on.UDP is the other transport, and it’s the opposite deal: it hands you packets as packets, called datagrams, with no ordering and no retransmission. You use it when a late answer is worse than no answer, like a video call, or when you’d rather do reliability yourself, which is what HTTP/3 does.

2. Addresses and ports name a machine and a program

An IP address tells the network which machine. It does not tell the machine which program. That’s what a port is: a 16-bit number, 0 through 65535, that the kernel uses to route incoming data to the right process. A web server sits on port 443, listening. Your browser connects to port 443 and gets the web server, not the mail server.For about thirty years, the person who decided which number meant which service was Jon Postel, more or less by himself. He kept the “Assigned Numbers” list, which grew into IANA. Ports 0 through 1023 were reserved as “well-known,” and on Unix only root can bind to them. Which is why so many services historically ran as root, which is why so many services historically got broken into.

A single connection is therefore identified by four things together: the client’s address and port, and the server’s address and port. Networking people call this the four-tuple, and it matters more than it looks. The server’s port is the same for every visitor, but each visitor comes from a different address, or at least a different port, so the kernel can tell ten thousand simultaneous connections to port 443 apart.Whenever a limit in this subject looks like it was pulled from a hat, check whether the hat is a power of two. 65,535 is 2¹⁶ − 1: the port has to fit in the sixteen bits the TCP header gives it, and that’s simply the biggest number sixteen bits can hold. The same shape explains the 64 KB ceiling on the receive window in §6, the 255 in every IPv4 address, and the 4,096 that keeps turning up in §9. Computers don’t have a favourite number. They have a favourite shape, and it’s 2ⁿ − 1.

(203.0.113.7 : 51384)  →  (198.51.100.20 : 443)     // client addr:port, server addr:port

Change any one of the four and it’s a different connection. A second tab in the same browser, to the same server, gets a different client port and therefore a different tuple.

Where does the client’s port come from? Your browser didn’t pick 443 for its own end. The kernel assigns an ephemeral port from a reserved range whenever a program connects outward. The number means nothing except “this connection, on this machine, right now.”The ephemeral range varies by system; IANA suggests 49152–65535. That’s roughly 16,000 ports, which sounds like plenty until you read §16 and learn how a closed connection keeps its port for a while.

File one rule away for later: only one program can listen on a given port on a given address. Try to start a second and you get “address already in use.” When you hear that a machine serves its website and its API and its admin panel “all on 443,” something else is going on, and §13 explains what.

3. The kernel boundary, and why system calls cost

Your program can’t touch the network card. It can’t touch the disk, the screen, or most of physical memory either. Modern operating systems draw a hard line: your code runs in user space, with access to its own memory and nothing else, and anything involving hardware happens in kernel space, on the other side of the line.Ken Thompson and Dennis Ritchie described Unix to the world in a 1974 paper, and one of its ideas was that a disk file, a terminal, and a tape drive should all be read and written with the same handful of calls. “Everything is a file” wasn’t a slogan yet; it was a shortcut that saved them from writing a separate interface per device. Nine years later, sockets would hitch a ride on the same shortcut.

To get across it, you make a system call. The CPU switches modes, saves your registers, jumps into kernel code, does the work, copies any results back into your memory, and switches back. Every recv() is one of these round trips.

It’s not slow, exactly. On a modern machine a system call costs on the order of a few hundred nanoseconds to a microsecond, plus some collateral damage to the CPU caches. But the thing it competes with, reading a value that’s already in your program’s memory, costs a few nanoseconds. Call it a hundredfold difference. If a program does one system call for every byte it wants to look at, that hundredfold shows up directly in the throughput.The line got pricier in 2018. The fix for the Meltdown CPU bug, called KPTI on Linux, made the kernel swap page tables on every system call, adding a noticeable tax to each one. Software that already buffered its I/O barely noticed. Software that made a system call per byte got a surprise.

Hold onto that ratio. Nearly everything in Part III exists to avoid crossing that line more often than necessary.

Part II — The socket itself

4. The socket: a file descriptor for a conversation

A socket is an endpoint for communication that the kernel creates and manages on your behalf. That sentence is accurate and useless, so let’s make it concrete.In the early 1980s DARPA wanted TCP/IP in a Unix that universities could actually get, and paid Berkeley’s Computer Systems Research Group to put it there. BBN (Bolt Beranek and Newman, the Cambridge, Massachusetts firm that had built the ARPANET’s original packet switches back in 1969) had already written a TCP/IP implementation under a separate contract, and handed it over. Berkeley, being Berkeley, rewrote it, and it ran faster, which BBN was not thrilled about. The result shipped in 4.2BSD in August 1983 with the sockets interface bolted on top.Kirk McKusick’s essay “Twenty Years of Berkeley Unix” tells the story from the inside.

When your program calls socket(), the kernel allocates an object internally and hands you back a small integer, a file descriptor, that refers to it. You never see the object itself. The kernel keeps the connection state, the peer’s address, and, importantly, its own queues of bytes waiting to go out and bytes that have come in. What you hold is a handle.

The reason it’s a file descriptor and not some new kind of handle is the single best decision in the API’s history. Unix already had file descriptors for files, terminals, and pipes, and a small set of calls, read(), write(), close(), that worked on all of them. The socket designers made network connections fit the same mould. Once a socket is connected, you can read() from it and write() to it with code that doesn’t know whether it’s talking to a disk or to a machine on another continent.

Getting to “connected” takes a small dance, and it’s asymmetric. The server side:The API outlived every competitor because it copied the file model and then got copied itself: Windows got a near-verbatim version as Windows Sockets, and POSIX eventually standardized it. A networking tutorial from 1985 mostly still compiles, which is not a sentence you can write about much else in computing.

int s = socket(AF_INET, SOCK_STREAM, 0);
bind(s, &addr, sizeof addr);        // claim port 8080 on this machine
listen(s, 128);                     // start queueing connection requests
int c = accept(s, NULL, NULL);      // block until a client arrives; c is the conversation
recv(c, buf, sizeof buf, 0);
send(c, reply, len, 0);
close(c);

And the client:“Handshake” is borrowed from electrical engineering, where it meant the exchange of control signals two devices go through before either sends data. A modem’s screech was literally its handshake, and serial cables carried dedicated wires named “request to send” and “clear to send.” TCP’s version is a three-step nod, SYN, SYN-ACK, ACK: I’d like to talk, and here’s my starting number. Fine, here’s mine. Got it. Three, because two isn’t enough: after two messages the client knows the server heard it, but the server has no idea whether its reply arrived. Ray Tomlinson, who also put the @ in email addresses, gets the credit for working out that it had to be three.

int s = socket(AF_INET, SOCK_STREAM, 0);
connect(s, &server_addr, sizeof server_addr);   // handshake with the server
send(s, request, len, 0);
recv(s, buf, sizeof buf, 0);
close(s);

send() and recv() are the socket-flavoured names for write and read; they take an extra flags argument and are otherwise the same thing. Two details in the server code repay attention. First, bind() is where the port from §2 gets claimed, and where “address already in use” comes from. Second, accept() returns a new socket for each client. The original keeps listening; the new one is the actual conversation. The word “socket” covers both, which is a minor naming crime we’re stuck with.

5. Domains and types: what “protocol agnostic” really means

socket() takes three arguments, and they answer a question that trips people up: if sockets are “protocol agnostic,” how come you have to pick a protocol?

The first argument is the domain, or address family: AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX for the local machine. The second is the type: SOCK_STREAM for a reliable byte stream, SOCK_DGRAM for individual messages, SOCK_RAW for “let me build the packets myself.” The third is the specific protocol, which you can almost always leave as 0 to mean “the obvious one”: TCP for a stream in the internet domain, UDP for datagrams.

So here’s the precise version of “agnostic.” A socket knows nothing about application protocols. HTTP, SMTP, your game’s message format, the JSON your microservice emits: the socket has no idea any of that exists. It ships bytes. But you do choose the transport mechanism when you create it, and the kernel implements that mechanism for you. Agnostic about what the bytes mean; opinionated about how they’re delivered.

That also answers where a socket sits in the layer diagrams. It is not the transport layer. TCP and UDP are the transport layer, and they live inside the kernel. The socket is the doorway your program uses to reach them.In OSI terms: layers 1 through 4, physical, data link, network, and transport, live in the kernel. Layers 5 through 7, session, presentation, and application, which the internet protocols collapse into a single “application” layer, live in your program. The socket is the line between 4 and 5, not a layer of its own. Picture the counter at a post office. Everything behind it, the sorting, the trucks, the planes, the carrier’s route, is the postal system’s business: layers 1 to 4. Everything in front of it, what you wrote, what language it’s in, whether it’s a bill or a love letter, is yours: 5 to 7. The counter is where envelopes change hands, and nobody would call the counter a stage of delivery.

Two kinds of socket prove that the doorway isn’t nailed to one layer.

A raw socket opens onto the network layer instead. You get IP packets and build the headers above them yourself. ping and traceroute work this way, as do packet sniffers, which is why raw sockets usually need root.

A Unix domain socket has no network layers at all. Instead of an address and port, it’s named by a path on the filesystem, like /var/run/docker.sock, and the kernel simply copies bytes from one process to another. It’s the same API, the same connect() and accept() and recv(), with no TCP, no IP, no checksums, and no possibility of the other end being on another machine. That makes it faster than talking to localhost over TCP, and it can be permission-controlled like a file. It can also do a trick no network socket can: hand an open file descriptor from one process to another. Docker, PostgreSQL, systemd, D-Bus, and the X Window System all use them.Unix domain sockets are not a later add-on. The “local” domain shipped in the same 4.2BSD release as the internet one, in 1983, because the designers wanted one API for all inter-process communication, not just the networked kind. Windows covered the same ground with named pipes for decades and added AF_UNIX to Windows 10 only recently, largely so Linux software would port more easily.

And there are odder relatives: netlink sockets on Linux, for talking to the kernel itself (the ip command uses them to configure routes), and datagram sockets for UDP, where each recv() returns exactly one message as sent, boundaries intact, no ordering, no retransmission.“But I connect to localhost:5432 and it works fine.” It does. TCP to localhost never leaves the machine, and the kernel takes a shortcut, but it still runs the whole TCP state machine, checksums included. The reason to do it anyway is that the same code works when the database moves to another box. The reason not to is that a Unix domain socket is faster and can’t be reached from the network by accident.

So the honest one-line definition is: a socket is a kernel-provided endpoint that plugs your program into whichever communication mechanism you asked for. For most of us, most of the time, that mechanism is TCP.

6. The kernel’s own buffers and flow control

There’s a buffer you never see, and understanding it clears up two questions at once: where data waits when your program isn’t reading, and how the other side knows to stop sending.

Every socket has, inside the kernel, a receive queue and a send queue. The full path of an incoming byte is: network card, driver, IP, TCP reassembly (§1), then the socket’s receive queue. It sits there until your program calls recv(), at which point the kernel copies as many bytes as you asked for into your memory. The kernel sizes that queue automatically, growing it for a fast connection with a slow reader.

By the time data is in that queue, it’s bytes, not packets. TCP has already stripped the headers and put everything in order. A recv() of 4,096 bytes gets the next 4,096 bytes of the stream, whether they came from one packet or four.

Now the other side. Suppose your program stops calling recv() for a while. The queue fills. What happens to the bytes still arriving?Flow control was in TCP from the 1974 paper. Congestion control was not, and in October 1986 the internet found out the difference: a link between Lawrence Berkeley Laboratory and the Berkeley campus, rated at 32 kilobits per second, slumped to about 40 bits per second as every host retransmitted into a network that was already full. Van Jacobson and Mike Karels diagnosed it and added slow start and congestion avoidance, shipped in 4.3BSD-Tahoe and written up at SIGCOMM in 1988. The distinction: flow control protects the receiver from being overrun. Congestion control protects the network.

Mostly, they don’t arrive, and that’s on purpose. TCP has a mechanism called flow control. Every acknowledgment your kernel sends back carries a number, the receive window: “I currently have this much free space.” The sender is not allowed to have more than that many unacknowledged bytes in flight. As your program reads, space frees up, the next acknowledgment advertises a bigger window, and the sender resumes.

If your program stops reading entirely, the window shrinks to zero and the sender stops. Completely. It sends a tiny probe every so often to ask whether space has opened up, but the data waits in the sender’s kernel queue, and if that fills too, the sending program blocks in send(). Back-pressure travels the whole way to the other application. Nothing is lost; it just isn’t sent yet.The window field in the TCP header is 16 bits, so the original maximum was 64 KB, which on a fast long-distance link is nowhere near enough to keep the pipe full. A 1992 extension added a scale factor, negotiated at connection time, that multiplies it by up to 2¹⁴. Every modern connection uses it, invisibly.

The trigger for all of this is your program not reading. Not packet loss, not a slow network. If a packet does show up when there’s no room, say because the window hit zero at the same moment the sender was already transmitting, the kernel drops it, never acknowledges it, and the sender retransmits it later. That’s the safety net, not the mechanism.

Part III — Making sense of a stream

7. Framing: drawing lines on a stream that has none

A stream has no boundaries. That’s the whole point of it, and it’s also the source of the most common bug in hand-written socket code.

If you send() three 100-byte messages, the receiver might get them as one 300-byte recv(). Or as 250 bytes followed by 50. Or 1 followed by 299. TCP promises the bytes will arrive in order and none will be missing; it promises nothing about how they’ll be grouped. One send() is not one recv().

Worse, localhost will lie to you about this. On a single machine, with no real network in the way, sends usually do arrive intact and separate. So the code works in development, passes the tests, and then corrupts data in production the first time a packet boundary falls in the middle of a message. Every socket programmer does this once.It gets worse the other direction too. The kernel is free to merge your three small sends into one packet, and in §16 you’ll meet an algorithm that does so on purpose. “Message” is a concept your protocol has and TCP does not.

Since the stream won’t draw lines, the protocol has to. There are only a handful of ways:

• Delimiters. “A line ends at \r\n.” “A header section ends at an empty line.” You scan for the marker; everything before it is one unit.
• Length prefixes. “The next 4 bytes say how long the message is; then that many bytes follow.” You read the length, then count. No scanning.
• Fixed sizes. “Every record is exactly 64 bytes.” Rare on the wire, common in binary file formats.
• Self-describing chunks. “Size, then that many bytes, then size, then that many bytes, then a size of zero.” The data announces its own end. HTTP’s chunked encoding works this way.

Whatever the rule, the reader has to cope with reads that cut across it. A single recv() might return the tail of one message and the head of the next, or half a line. The parser keeps what it has, waits for more, and only acts when a complete unit is present. Where does “what it has” live while it waits? In a buffer. We’re nearly there.Why \r\n and not just \n? Because the early protocols were designed to be typed at, and read from, a Teletype, where “carriage return” and “line feed” were two separate mechanical actions: slam the print head back to the left, then advance the paper. Telnet standardized the pair as the network newline, SMTP and HTTP inherited it, and here we are, still sending a byte whose job was to move a metal carriage.My favourite framing scheme for its sheer minimalism is Dan Bernstein’s netstring: the length in decimal, a colon, the bytes, a comma. 12:hello world!, The comma is redundant, which is the point: if it isn’t there, something upstream is broken and you’d like to know now.

8. Application protocols are byte conventions: HTTP

HTTP is not built into sockets, and neither is anything else. HTTP is an agreement about what the bytes on a socket mean. It lives entirely in your program, or in the library your program uses.

When a browser asks for a page, the bytes it writes into the socket look like this:The first HTTP, retroactively named 0.9, was one line. The client sent GET /path, the server sent the document and closed the connection. No headers, no status codes, no content types. Tim Berners-Lee wrote it in 1991, and everything since is elaboration on that line. It was text because text is debuggable: you can type it at a server yourself and read what comes back. HTTP/2, in 2015, finally went binary for efficiency, and you can no longer telnet to a web server and have a conversation. That was the trade.

GET /index.html HTTP/1.1\r\n
Host: example.com\r\n
User-Agent: Mozilla/5.0\r\n
\r\n

Plain text. Lines end in \r\n, a blank line marks the end of the headers, and that’s the request. The socket sees ninety-odd bytes going out. It has no opinion about them.

The server reads those bytes and parses them by the HTTP rules: first line is method, path, version; then header lines until the blank one. It writes back:

HTTP/1.1 200 OK\r\n
Content-Type: text/html\r\n
Content-Length: 1256\r\n
\r\n
<html>... exactly 1256 bytes ...

Now notice how the parser knows where it is in the stream. It’s a small state machine, and it keeps its place by remembering what it’s done so far. I’m reading the status line. Then: I’m reading headers; keep going until an empty line. Then: That was the empty line, and somewhere in the headers I saw Content-Length: 1256, so now I’m reading a body with 1,256 bytes to go. Then: Done. The very next byte is the start of a new response. The socket never tells it any of this. It knows because it has followed the rules from the first byte.Try it. On most machines, nc example.com 80 opens a raw socket to a web server, and if you type GET / HTTP/1.1, then Host: example.com, then an empty line, the response comes back as text in your terminal. There is no better cure for the feeling that HTTP is something magical the browser does.

Two things people get backwards about that parser. It’s usually one component doing all the reading in sequence, not different parts of the program each grabbing their piece; only after the whole response is assembled does it hand a structured object, with a status code and a headers dictionary and a body stream, to the rest of the application. And it doesn’t know it’s about to read Content-Type. Headers come in any order. It reads a line, then looks at what it got.

You could write a working HTTP client in a dozen lines: open a socket to port 80, write that request text, print whatever comes back. Nothing is hidden. HTTP is a byte-format convention on top of a stream.

With HTTPS there’s one more layer, TLS, squeezed between HTTP and the socket. The request text is encrypted before it’s written and decrypted after it’s read. The socket still sees only bytes; they’re just scrambled ones.

9. The user-space read buffer

Put §3, §7, and §8 together and the problem writes itself. The parser wants to read small, specific pieces: a line, then a line, then exactly 1,256 bytes. Every recv() is a system call, a hundred times dearer than a memory read. If the parser called recv() once per line, it would spend most of its time crossing into the kernel and back.

So it doesn’t. It puts a buffer in the way.

A buffer, here, is nothing more exotic than an array of bytes your program owns, say 4,096 of them, plus a position marker. When the parser wants a line, it first looks in the array. If the array is empty, the program makes one system call: “read up to 4,096 bytes from the socket into this array.” That call comes back with whatever was available, often the whole header section and the start of the body. From then on, “read a line” means “scan forward in the array to the next \r\n and advance the marker.” No kernel. No system call. Just memory.This exact trick, read a block, dole out the pieces, refill when empty, is older than sockets. It’s what the C standard library’s getc() has done since the “standard I/O” package appeared in Seventh Edition Unix in 1979. Every buffered reader in every language since is a descendant.

It works until the marker reaches the end of what was fetched. That’s the buffer running empty, and it’s not mysterious: the array only ever held a finite slice of the stream, you’ve consumed the slice, and the next read has to go back to the socket for another one. Refilling may bring back fewer than 4,096 bytes if that’s all the kernel had at that moment; that’s fine, you just have a smaller slice this time.

Why 4,096? No deep reason. It’s a convention. It matches the memory page size on most machines, which makes allocation tidy, and it’s larger than nearly any HTTP header section, which means one system call typically covers all the small reads that follow. But it’s a knob, not a law. Go’s bufio defaults to 4 KB, Java’s BufferedReader to 8 KB, Node’s streams to 16 KB. Bigger means fewer system calls and more memory per connection.Why not make the buffer enormous and be done with it? Because a server with fifty thousand idle connections would then hold fifty thousand enormous buffers, mostly empty. Some libraries dodge the trade-off by renting buffers from a shared pool only while a read is in flight, which is what .NET’s ArrayPool is for.

What the size is not related to is packet size. Remember §1: packets were dissolved into a stream inside the kernel before your program ever saw a byte. Your 4 KB read pulls from the kernel’s receive queue, which is holding an ordered run of bytes that came from however many packets it took.

So there are three buffers stacked along the path, and it helps to see them together:

1. The sender’s kernel send queue, which holds what it wasn’t allowed to transmit yet (§6).
2. Your kernel’s receive queue, which holds what has arrived and hasn’t been read (§6).
3. Your program’s buffer, which holds what has been read and hasn’t been parsed (this section).

The framing problem from §7 lands in the third one. When a read ends with half a line, the half-line stays in your buffer, and the next refill appends the rest after it. Libraries usually wrap this whole arrangement in an object called a buffered stream or buffered reader that presents the same read interface as the socket underneath, while quietly doing far fewer real reads. Keep that object in mind. It’s about to become the answer to a puzzle.

Part IV — Many connections at once

10. Blocking, non-blocking, and readiness

By default, a socket blocks. Call recv() with nothing in the receive queue and your thread stops, mid-instruction, until something arrives. Same for accept() with no client, connect() during the handshake, and send() when the send queue is full. It’s simple, and it’s exactly how reading a file behaves, which was the point.

It’s also why a server that handles connections the obvious way needs a thread per connection. Each thread spends nearly all its life parked in recv(), waiting. Ten of those is nothing. Ten thousand is a problem: each thread reserves a stack, typically a megabyte or more of address space, and the kernel’s scheduler has to keep them all straight. This is the model everyone used from 1983 until it fell over.

The alternative starts with a flag. Set a socket to non-blocking and the same calls return immediately. If data is there, you get it; if not, recv() returns an error whose name is literally “would block”: try again later. Now one thread can service many sockets. But it can’t just loop over them calling recv(), because a loop that gets “would block” ten thousand times in a row is a CPU pegged at 100% doing nothing.It fell over in public in 1999, when Dan Kegel put up a web page titled “The C10K problem” pointing out that a cheap PC could plausibly serve ten thousand simultaneous connections, and the operating system interfaces were what stopped it. select(), the original readiness call from 4.2BSD, took a bitmask of descriptors and rescanned all of them on every call, so its cost grew with the number of connections rather than the number that had anything to say. Jonathan Lemon’s kqueue landed in FreeBSD in 2000 and Davide Libenzi’s epoll in Linux in 2002–03; both let you register a socket once and pay only for the ones that are ready.

What it needs is a way to sleep until something is ready. That’s a family of system calls: select(), poll(), and their modern descendants epoll on Linux and kqueue on BSD and macOS. You hand the kernel a set of sockets and say “wake me when any of these can be read or written.” The thread sleeps. It wakes with a list of the ready ones, does the reads and writes that it now knows won’t block, and goes back to sleep. That loop is the heart of nginx, Node.js, Redis, Go’s scheduler, and Python’s asyncio.

One thing to get straight before moving on: blocking, non-blocking, and “async” are not three kinds of socket. The first two are a flag on the same socket. The third is a programming style built on the second plus a readiness call. Same socket, same kernel queues, same four-tuple. The difference is entirely in who waits, and how.Windows has select() for compatibility, but it was never the fast path there. Windows servers went a different way entirely, which is the next section.

11. Reactor and proactor: readiness versus completion

There are two shapes for that loop, and the difference between them comes down to who does the I/O and what the wake-up means.

In the reactor shape, you tell the kernel what you’re interested in: “let me know when socket A is readable.” When data arrives, the loop wakes with “A is readable.” Your code then calls recv() itself, and it succeeds without blocking because you were promised it would. The notification is about readiness. The actual read is still yours to do, after the fact.The names come from Douglas Schmidt and colleagues, who catalogued both as design patterns in the second volume of Pattern-Oriented Software Architecture in 2000. “Reactor” because the loop reacts to events; “proactor” because it proactively starts the operation and gets told when it’s done. The proactor shape had already been the way of Windows NT for years by then, where it’s called an I/O completion port: NT was built to be a server, and completion ports let a small pool of threads handle a large number of in-flight operations.

In the proactor shape, you start the operation and walk away: “read from socket A into this buffer, and tell me when it’s done.” The kernel, or a helper thread, performs the read. Later the loop wakes with “the read on A finished; 512 bytes are in your buffer.” The notification is about completion. By the time you hear about it, the I/O has already happened and the data is sitting where you asked.

A restaurant pager versus room service. The pager tells you your table is ready; you still have to walk over and sit down. Room service knocks when the food is already at the door.

Practically: the reactor is simpler and lets you decide how to read once you know you can, but every wake-up still costs a system call for the read itself. The proactor folds the read into the notification, so there are fewer boundary crossings and a natural fit for batching, but the buffer you handed over has to stay put, pinned in memory, until the kernel is finished with it, which complicates life for garbage-collected runtimes that would like to move things around.Linux resisted the proactor shape for a long time; POSIX asynchronous I/O existed but was widely regarded as a disappointment for sockets. Then in 2019 Jens Axboe added io_uring: two ring buffers shared between your program and the kernel, one for submitting operations and one for collecting completions. Batches of reads can be queued and harvested with no system call at all. It’s the most significant change to how Linux programs do I/O in two decades, and it’s a proactor.

Most cross-platform libraries pick one shape for their public API and emulate the other where the OS doesn’t provide it. libuv, under Node.js, exposes a reactor-ish interface and uses completion ports underneath on Windows. .NET exposes a proactor-style interface, BeginReceive and friends, and on Linux implements it on top of epoll, doing the recv() itself the instant the socket reports ready. Which brings us to the puzzle from §9.

12. Language async and the buffered stream: why a callback can fire synchronously

When a language gives you async/await, or promises, or a BeginRead/EndRead pair, it’s not offering a new kind of socket. It’s wrapping §10 and §11: underneath is a non-blocking socket plus a readiness or completion mechanism, and the runtime turns “wake me when ready” into “resume this function when ready.” Your code reads like blocking code. No thread is actually parked on the socket.

Now stack the buffered reader from §9 on top of that async socket and follow one HTTP response through it. The parser asks the buffered stream for a line. The buffer is empty, so the stream issues a read to the socket. Nothing has arrived yet. The operation can’t complete, so the call returns immediately with “pending,” and some time later, when bytes land in the kernel queue, the callback fires or the awaiting function resumes. That first read was truly asynchronous..NET has had three generations of this. The original Asynchronous Programming Model from 2002, with paired BeginX/EndX methods and a callback. The Task-based model in 2010, which turned the callback into an object you could compose. And async/await in C# 5 in 2012, which made the compiler write the callback for you. All three sit on the same completion ports and, on Linux, the same epoll loop.

But that read pulled in 4 KB, and 4 KB is the whole header section plus the start of the body. The parser asks for the next line. The buffered stream looks in its array, finds the line already there, and hands it over. No socket. No kernel. The operation finished before the call that started it returned.

In a completion-style API, this is called completing synchronously, and it’s common enough that the API has a flag for it. .NET’s IAsyncResult has a property named CompletedSynchronously; the Begin method sets it when it found it could satisfy the request on the spot, and it invokes the callback right then, on the calling thread, instead of handing it to the event loop. Every header line of that response, Content-Type, Content-Length, the blank line, completes this way. Only when the parser gets into a body larger than what’s left in the buffer does the stream drain, go back to the socket, and flip to being asynchronous again.

// Inside a buffered stream, roughly.
IAsyncResult BeginRead(byte[] dest, int count, AsyncCallback cb, object state)
{
    if (buffered > 0) {
        // Bytes are already here. Copy, mark it done, call back immediately.
        int n = Math.Min(count, buffered);
        Array.Copy(buffer, pos, dest, 0, n);
        pos += n; buffered -= n;
        var done = new SyncResult(n, state);   // CompletedSynchronously == true
        cb(done);
        return done;
    }
    // Nothing buffered: this one has to go to the socket.
    return socket.BeginReceive(buffer, 0, buffer.Length, cb, state);
}

There’s a hazard hiding in this, and it’s why the flag exists rather than being an implementation detail. If your callback’s job is “process the line, then BeginRead the next one,” and the next one also completes synchronously, then the callback calls BeginRead, which calls the callback, which calls BeginRead… Every synchronously completed line adds a frame to the stack. Enough of them and the stack overflows. Well-behaved code checks CompletedSynchronously and loops on the synchronous path, reserving the callback for the case where it actually had to wait. Task-based APIs hide this by design, but the same distinction is why modern .NET has ValueTask: a cheap way to represent “this already finished” without allocating a Task to say so.Go and Node never show you this distinction. Go parks the goroutine on its network poller and resumes it when the socket is ready, so a buffered read that doesn’t need the socket simply doesn’t park; from the outside it’s all the same code. Node’s streams do a version of the same thing with their internal high-water-mark buffer. The concept is there in every runtime. Only some of them make you look at it.

So the full answer to “why did my asynchronous read complete synchronously?” is: because a buffer above the socket already had the bytes, and going to the kernel for them would have wasted the exact thing buffers exist to save.

Part V — Sockets in the wild

13. One port, many programs

Back to the rule from §2: one listener per port per address. Here’s how a machine appears to break it.

First, the non-answer that resolves most of the confusion. One listener handles thousands of connections at once, because connections are identified by the four-tuple. Two browsers connecting to port 443 differ in their own address or port, so the kernel keeps them apart and hands each one to the listener as a separate accepted socket. “One program per port” and “ten thousand users on port 443” were never in conflict.

When a machine really does host several services behind one port, the usual answer is that only one program listens on 443, and it’s a traffic director. nginx, HAProxy, Caddy, or a cloud load balancer accepts the connection and routes each request based on what’s inside it: the Host header (api.example.com versus www.example.com), the path (/api/ to one backend, /static/ to another), or for HTTPS the server name the client announces during the TLS handshake, which lets the proxy pick the right certificate before decrypting anything. The actual services listen on other ports, 3000, 8080, or a Unix domain socket, that the outside world never reaches.HTTP/1.0 had no Host header, so a server had no way to tell which website a request was for. The only fix was one IP address per site, and in the mid-1990s hosting companies were burning through IPv4 addresses to run virtual hosts. HTTP/1.1, in 1997, made Host mandatory and the problem went away. Then TLS brought it back: the certificate has to be chosen before any HTTP is readable. The Server Name Indication extension, first specified in 2003, has the client announce the hostname in the clear during the handshake, and it took about a decade of waiting for old browsers to die before everyone could rely on it.

Second, the rule is per address. A machine with several IP addresses can have a different listener on 443 for each. Containers exploit this: every Docker container gets its own internal address, so each can run its own web server on 443 without conflict, and port mapping decides which ones are exposed.

Third, there’s a real exception. Linux and the BSDs support an option called SO_REUSEPORT that lets several processes of the same program listen on the same port, with the kernel spreading new connections across them. It’s for servers that want multiple worker processes without a proxy in front, and for zero-downtime restarts where the new version starts listening before the old one stops. It requires cooperation; it’s not a way for two unrelated programs to fight over a port.The BSDs had SO_REUSEPORT first, with looser semantics; Linux added its load-balancing version later. On Linux, every process sharing the port has to run as the same user, precisely so a stranger can’t quietly attach to your listener and start receiving a share of your traffic.

And on your own machine, the browser, the mail client, the chat app, and the package manager are all talking to remote port 443 at the same time, each from a different local ephemeral port. The four-tuple stays unique, and the kernel never mixes them up.

14. Firewalls, default-deny, and the funnel to 443

Corporate firewalls, home routers, and cloud security groups all tend to arrive at the same policy: block every inbound port except the ones explicitly opened, which for most machines means “none” and for a web server means “80 and 443.” It’s natural to ask whether that makes sense. Can’t 80 and 443 be attacked too?

Yes. The logic isn’t that those ports are safe. It’s that a port is only dangerous if something is listening on it, so fewer open ports means fewer programs an attacker can reach.

A port number is just a label saying which program gets the traffic (§2). Port 22 isn’t risky; an SSH daemon with password logins enabled might be. Port 3306 isn’t risky; a MySQL server facing the internet with a default password is. When a firewall drops everything except 443, what it’s really saying is: only the web server is reachable from outside. The database, the admin panel, the file share, the monitoring agent, and whatever else someone installed and forgot about are not. That’s called reducing the attack surface, and historically most break-ins came through exactly those forgotten services.The perimeter firewall is, more or less, a reaction to November 1988, when Robert Morris’s worm spread through a few thousand machines, a large fraction of the internet at the time, by walking in through services that were listening on ports nobody was thinking about: a debug mode left on in sendmail, a buffer overflow in finger. The lesson people drew was that you can’t audit every listening program, so you’d better make most of them unreachable. Bill Cheswick and Steve Bellovin’s 1994 book Firewalls and Internet Security turned that into doctrine.

Port 443 is open because the business needs it, and every security team knows it’s a risk they’re accepting deliberately. SQL injection, authentication bugs, path traversal, a vulnerable framework: all of them travel over perfectly ordinary HTTPS requests, and the firewall’s view of them is “a GET to port 443,” which is exactly what it was told to allow. That’s why other tools exist for that layer: web application firewalls that inspect the HTTP inside, intrusion detection, patching, and, unglamorously, writing the web server correctly.

There’s also a direction to it. On a corporate or home network the firewall blocks nearly all inbound connections, since nobody outside should be initiating a connection to your laptop, but permits outbound connections on 80 and 443 so people can browse. That asymmetry is the thing the next section exploits.

Because 443 is nearly always open outbound, everything now tries to travel over it: VPNs, remote-desktop tools, chat, software updates, and malware phoning home. Port filtering has therefore become much less meaningful, and defenders responded with deep packet inspection and, in strict environments, TLS interception, where the firewall decrypts and inspects traffic before re-encrypting it. That, in turn, pushed protocol designers to make their traffic look even more like ordinary browsing. The general name for this is ossification: the network gets so tuned to the protocols it already knows that anything new has to disguise itself as one of them to be deployable.Two recent examples of protocols dressing up. TLS 1.3 announces itself on the wire as TLS 1.2 and tucks its real version into an extension, because too many middleboxes choked on an honest version number. And HTTP/3 runs on UDP partly because TCP’s own innards had become impossible to change without something in the path breaking.

15. WebSocket: a protocol that starts as HTTP

Despite the name, a WebSocket is not a socket. It’s an application-level protocol that runs on top of an ordinary TCP socket, the same place HTTP lives, and it was named to sound like a raw socket because that’s roughly what it gives browser JavaScript, which is otherwise forbidden from opening one.

It starts life as HTTP. The client sends a normal-looking request with a few extra headers:Ian Hickson drafted the protocol as part of the HTML5 effort starting around 2008; it then moved to an IETF working group and became RFC 6455 in December 2011. In between, the disguise came back to bite: the proxy attack described below was demonstrated against the early drafts, and the protocol shipped only after masking was added.

GET /chat HTTP/1.1\r\n
Host: example.com\r\n
Upgrade: websocket\r\n
Connection: Upgrade\r\n
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n
\r\n

If the server agrees, it answers 101 Switching Protocols, and from that byte on, both sides stop speaking HTTP. The TCP connection stays open; the bytes on it now follow WebSocket’s rules instead.

Those rules are what a raw socket doesn’t give you: framing. Each message is wrapped in a small header, two to fourteen bytes, that says how long it is and whether it’s text or binary. A WebSocket library hands you whole messages, not arbitrary slices of stream. The protocol also adds ping and pong frames for keepalive, a proper close handshake, and one strange requirement: every frame from a browser must be masked, XOR-ed with a random key, before it goes on the wire.

Compared to HTTP, the difference is direction. HTTP is request and response; the server can’t speak until asked. A WebSocket, once open, is full-duplex and persistent: either side sends whenever it likes, with a few bytes of overhead per message instead of a fresh set of headers. That’s what makes it fit for chat, live dashboards, multiplayer games, and collaborative editors, where the server needs to push.

Now, why start as HTTP at all? Section 14. A brand-new protocol on its own port would be blocked by a large fraction of the firewalls in the world. But a GET to port 443 with normal headers and a normal-looking response sails through: the middlebox checks everything it knows how to check, decides the connection is legitimate, and from then on mostly just forwards bytes. The handshake is a compatibility disguise. It borrows HTTP’s ports and HTTP’s opening so that infrastructure built for HTTP will let it in, then uses the connection for something HTTP can’t do.

The masking exists because the disguise is imperfect. Some caching proxies didn’t understand the upgrade and kept interpreting the bytes as HTTP. Researchers showed that a malicious page could send WebSocket data crafted to look like an HTTP request to such a proxy and poison its cache for other users. Masking scrambles the client’s bytes with a random key so they can never accidentally resemble a valid HTTP request, whatever the page tries to send.

In the stack you’ve been building: application, WebSocket framing, TLS if it’s wss://, TCP socket, IP. Compare HTTP: application, HTTP formatting, TLS, TCP socket, IP. Same socket, same buffers, same kernel queues. Only the convention for what the bytes mean has changed. And it still has the §7 problem: a 4 KB read may cut a frame in half, so the library keeps its own buffer and reassembles, just as the HTTP parser does.

16. Misconceptions that survive first contact

A short list of things that are true, surprising, and responsible for a lot of 3 a.m. debugging.

send() returning means the kernel accepted the bytes, not that they were delivered. It copied them into the send queue (§6). The other side may not have received them and may never. The only proof of delivery is an acknowledgment from the application on the other end. Also, send() may accept fewer bytes than you offered; you have to loop.

A dead connection is silent. A crashed peer, an unplugged cable, and a NAT router that forgot about the connection all look identical to “nothing to say.” An idle connection can sit for hours apparently healthy. Detecting death needs TCP keepalive, application-level pings (WebSocket’s ping frame exists for this), or a timeout on your reads.TCP keepalive is off by default, and even when on, Linux waits two hours of silence before the first probe. The number dates from an era when a probe cost real money on a leased line. Nearly everyone who turns it on also turns that down.

Closing isn’t instant. After close(), the side that closed first parks the connection in a state called TIME_WAIT for a minute or so, so that late packets from the old connection can’t be mistaken for a new one. A client that opens and closes tens of thousands of short connections can run out of ephemeral ports this way. This is one reason connection pools and HTTP keep-alive exist.

Half-closed is a real state. shutdown() can say “I’m done sending” while continuing to receive. On the other side, recv() returning zero bytes means “they’re done,” not “error.” Code that treats zero as an error, or never handles it at all, spins or hangs.

Small writes can be mysteriously slow. By default TCP may hold a small write briefly to see whether more is coming, so it can send one larger packet instead of several tiny ones. Combined with the receiver’s habit of delaying acknowledgments, this can add tens or hundreds of milliseconds to a chatty request-and-response protocol. The option TCP_NODELAY turns the holding off, and most latency-sensitive libraries set it.The small-write delay is Nagle’s algorithm, from a 1984 RFC by John Nagle, who noticed that a Telnet session sending one keystroke per packet was 40 bytes of headers per byte of payload. His fix, don’t send a new small packet while an earlier one is still unacknowledged, was right for 1984 and is still the default. Its bad interaction with delayed acknowledgments, where each side waits politely for the other, wasn’t his idea, and he has said so, repeatedly, on the internet.

localhost is not the network. No latency, no loss, no reordering, and sends that arrive intact. Code that only ever ran on localhost has not been tested for any of §7.

The socket isn’t the slow part. The system call is (§3), and the thread is (§10). A thread-per-connection server doesn’t collapse because sockets are expensive; it collapses because threads are. That’s the whole reason Part IV exists.

17. References and further reading

• V. Cerf and R. Kahn, “A Protocol for Packet Network Intercommunication,” IEEE Transactions on Communications, 1974.
• D. Ritchie and K. Thompson, “The UNIX Time-Sharing System,” Communications of the ACM, 1974.
• RFC 793, “Transmission Control Protocol,” 1981; superseded by RFC 9293, 2022.
• M. K. McKusick, “Twenty Years of Berkeley Unix,” in Open Sources: Voices from the Open Source Revolution, O’Reilly, 1999.
• M. K. McKusick, K. Bostic, M. Karels, J. Quarterman, The Design and Implementation of the 4.4BSD Operating System, Addison-Wesley, 1996.
• S. Leffler, R. Fabry, W. Joy, P. Lapsley, “An Advanced 4.4BSD Interprocess Communication Tutorial,” in the 4.4BSD Programmer’s Supplementary Documents.
• RFC 3549, “Linux Netlink as an IP Services Protocol,” 2003.
• V. Jacobson and M. Karels, “Congestion Avoidance and Control,” SIGCOMM, 1988.
• RFC 1323, “TCP Extensions for High Performance,” 1992; superseded by RFC 7323, 2014.
• D. J. Bernstein, “Netstrings,” 1997.
• RFC 2068, “Hypertext Transfer Protocol — HTTP/1.1,” 1997; current: RFC 9110 and RFC 9112, 2022.
• D. Kegel, “The C10K problem,” 1999, kegel.com/c10k.html.
• J. Lemon, “Kqueue: A generic and scalable event notification facility,” USENIX Annual Technical Conference, 2001.
• D. Schmidt, M. Stal, H. Rohnert, F. Buschmann, Pattern-Oriented Software Architecture, Volume 2, Wiley, 2000.
• J. Axboe, “Efficient IO with io_uring,” 2019.
• RFC 3546, “Transport Layer Security (TLS) Extensions,” 2003; current: RFC 6066, 2011.
• W. Cheswick and S. Bellovin, Firewalls and Internet Security, Addison-Wesley, 1994.
• RFC 8446, “The Transport Layer Security (TLS) Protocol Version 1.3,” 2018.
• RFC 6455, “The WebSocket Protocol,” 2011.
• RFC 896, J. Nagle, “Congestion Control in IP/TCP Internetworks,” 1984.
• D. Ritchie, “A Stream Input-Output System,” AT&T Bell Laboratories Technical Journal, 1984.
• Microsoft .NET documentation for IAsyncResult.CompletedSynchronously.

Further reading, in the order I’d read them.

1. Brian “Beej” Hall, Beej’s Guide to Network Programming. The shortest path from §4 to a program that actually runs; it walks the same calls with working C.
2. W. R. Stevens, B. Fenner, A. Rudoff, UNIX Network Programming, Volume 1: The Sockets Networking API, 3rd ed., 2003. Where to go when Beej stops answering; every option and state in §16 has a chapter.
3. Dan Kegel, “The C10K problem.” The hinge between Part III and Part IV, written while it was happening; read it to feel why epoll had to exist.
4. Michael Kerrisk, The Linux Programming Interface, 2010, the chapters on sockets and alternative I/O models. The Linux specifics behind §10 and §11, with the exact semantics of epoll that libraries rely on.
5. Jens Axboe, “Efficient IO with io_uring.” Where the proactor shape from §11 is going on Linux, from the person taking it there.