Data Centre Landscape: Part 1 - Understanding Applications and Services

Much is being written about data centres these days. Yet with the shrouding of AI over everything has come a fog of information about the design, use and future of data centres. In this series, I briefly introduce data centres and provide the background to understand their future.

This first part explores the diverse applications and services that drive data centre design.

Skipping over history

While the history of computing can help to explain how we arrived at the current design of data centres, it’s a long journey to go from their first origins in 1946 to today’s behemoths. If you’re interested in the history, there’s a brief introduction written by Digital Reality, a Wikipedia article and this article by data centre operator TRG.

Applications & services

To understand data centres, I think you first need to know what problem a data centre is solving. Unfortunately that’s a little bit easier said than done because modern data centres solve numerous different problems. We’ll start with the most basic example and build up.

1. Files - the traditional storage unit of data

Long before we had web apps and REST APIs, we had files. Lots of them. We quickly found we needed better ways to share the data contained in files than printing them on paper or writing them to floppy disks to be passed around. (No, I wasn’t alive to experience the heyday of the floppy disk. Yes, I’ve used floppy disks for actual work. No, it wasn’t full of nostalgic fun.)

To that end, networks of computers were invented. This required solving several challenges which are still being improved upon to this day.

1. Physical connections: Wires and optical fibres running between computers
   • It is profound to realise that, except for WiFi connections from your device, there are physically continuous lines from your home to servers all around the world (albeit, with computer boxes in between, that connect different parts of wire or optical fibre together).
2. Signalling protocols: The patterns to vary voltages on the wires, or pulses of lights sent down the optical fibres, needed to communicate ‘bits’ (1s and 0s)
3. Transmission protocols: Given some data to send, how to transform that into a pattern of 1s and 0s to send over the physical connection.
4. Communication protocols: Layered on top of transmission protocols to add more complex communication, things like:
   • Ensuring data was correctly received (and re-attempting if it wasn’t) e.g. TCP
   • Structured request/response protocols, e.g. HTTP
   • Some of these layer on top of each other
5. Network protocols/layers: The communications that coordinate the creation, maintenance and closing down of communication channels between pieces of software on different machines. Also the structure of the network - routing.
   • Routing: Directing information from one computer to another and on to its destination.
6. Drivers: The software which drives these various protocols in response to a request by application software.

Having solved the fundamental problem of how to communicate some data from software on one machine to software on another, we were able to solve the problem of sharing files.

Transmitting files from one machine to another is all well and good but it opens up new issues. In particular, how do we share two versions of the same file? How can one user access a file from a machine which is currently switched off? What happens if the hard disk drive in a machine fails? How can all files in a network be backed up? And so on…

Today’s solutions to these problems are known generally as Object Storage (and, in a related use-case, databases). Amazon Web Services’ S3 (“Simple Storage Service”) has become a standard design for this kind of file storage, accessed through APIs rather than traditional file systems. Users don’t see this behind-the-scenes design - most will use some kind of interface, whether that’s a web app like Google Drive, or a file system integration like One Drive on Windows.

S3 is one of the largest services provided by modern data centres. It stores files (“objects” or “blobs”) with extremely high resiliency.

2. Websites

To understand websites in the context of data centres, I’ll draw a useful (if somewhat artificial) distinction: websites serve static content that users download and consume, while apps are interactive systems where users both input and retrieve data. In this model, websites are essentially specialized collections of text files, images, videos, and other media formatted for web browsers.

At first glance, websites might seem like they should simply use the file storage systems described earlier. Indeed, many websites do exactly this - static site hosting on AWS S3 or similar services is a common pattern for simple websites. However, modern websites have evolved far beyond static file delivery.

The key difference lies in dynamic content generation. Even websites that don’t accept user input often change their content over time. News sites pull stories from databases, advertising sites select a new advert to serve on every request, and content delivery networks serve different versions based on geographic location. Two users requesting the same URL might receive entirely different content depending on their location, the time of day, or browsing history - advertising networks like Google’s DoubleClick exemplify this personalization.

This shift from static to dynamic content represents a crucial evolution in data centre architecture. Rather than simply retrieving files from storage, servers must now compute responses in real-time. When a page is requested, the request gets routed by a load balancer to a server capable of executing the necessary logic - whether that’s querying databases, processing templates, or integrating multiple data sources.

It’s worth noting that “server” is simply our term for any computer fulfilling this serving role, regardless of its physical form. However, when we use “server” without qualification, we typically mean machines with substantial processing power and memory capacity designed for this purpose.

This computational requirement introduces workload diversity within data centres. Some servers respond directly to user requests, while others run databases and respond to internal queries from the web servers. This specialization marks the beginning of the complex, heterogeneous computing environments that define modern data centres.

3. Apps

Apps introduce the next level of complexity beyond websites by enabling bidirectional communication between users and servers. While websites primarily deliver content to users, apps allow users to send data back - ranging from simple form submissions containing usernames and passwords to massive file uploads like video content that can be gigabytes in size.

This fundamental shift transforms server requirements dramatically. Servers must now accept incoming data, validate and sanitize user input, and compute results using potentially untrusted data. This introduces numerous new architectural challenges: expanded memory capacity to buffer incoming data, robust security boundaries since user input cannot be trusted, and sophisticated error handling for malformed or malicious requests.

The demand for bidirectional communication also creates entirely new scaling problems. Traditional websites could handle traffic spikes by replicating static content across multiple servers distributed globally - a straightforward horizontal scaling approach. Apps complicate this model significantly. When a user uploads a video to YouTube or TikTok, a specific server must be available to accept, process, and initially store that content. The video then needs sophisticated replication across content delivery networks (CDNs) to make it available to millions of users worldwide with acceptable latency. For TikTok, which prioritizes ultra-low upload latency to keep creators engaged, this requires edge computing infrastructure positioned close to users worldwide.

Geographic distribution becomes critical for user experience, but physical infrastructure ownership everywhere is prohibitively expensive. Request latency is fundamentally constrained by the speed of light - for applications requiring real-time interaction, such as VR experiences that demand maximum latency of 20ms, this creates a maximum practical distance of approximately 350km between user and server when accounting for processing time and transmission delays (the limitation here is the speed of light in a fibre optic cable)!

These geographic constraints helped to drive the emergence of cloud computing and virtualization. Companies need the ability to rent compute capacity on physical machines shared with other organizations, while maintaining the flexibility to scale operations up and down rapidly without physical hardware reconfiguration. Virtual machines (VMs) provide the solution - appearing to software as dedicated physical machines while enabling multiple isolated environments to coexist on shared hardware. VMs are managed by a hypervisor that abstracts the underlying hardware and can be provisioned and destroyed in minutes rather than the days or months required for physical server deployment.

However, virtualization introduces its own challenges, particularly the “noisy neighbour” problem where one virtual machine’s resource consumption can impact the performance of others sharing the same physical hardware. While various mitigation strategies exist, some aspects of this challenge remain unsolved, particularly for workloads requiring consistent, predictable performance.

4. Trading

Our next example takes us in a fundamentally different direction from traditional web services. Trading, particularly high-frequency trading (HFT), requires making rapid decisions using imperfect and incomplete data streams. Success depends not just on algorithmic sophistication, but on maintaining ultra-low latency connections between all servers involved in a trade’s execution.

HFT requires processing massive streams of real-time market data using complex mathematical operations - work that differs entirely from reading files from disk and transmitting them over networks. Servers optimized for web requests, file serving, and database queries are fundamentally unsuited for trading workloads.

Understanding this distinction illuminates a key trend in modern data centre design: workload specialization driving hardware heterogeneity.

Consider the computational flow of rendering a web page. A server receives a small request, parses it character-by-character to identify structure and extract information like the requested URL, then dispatches internal requests to other servers for files or database responses. The server doesn’t sit idle waiting for these responses - it context-switches to handle additional incoming requests. When internal responses return, it resumes the original request, generates the web page, and transmits the final response to the user. This request-response pattern demands three key capabilities: substantial RAM memory to buffer requests and responses, high-bandwidth networks for data movement, and CPUs optimized for memory-intensive operations, highly-branching code execution, and maintaining numerous parallel threads simultaneously.

Trading servers require entirely different capabilities. Rather than discrete request-response cycles, trading operates as continuous data streams. Incoming market data flows through algorithmic processing pipelines that integrate information into specialized time-series databases. Quantitative trading algorithms then scan across this temporal data, often processing ranges in parallel blocks before aggregating results for decision-making algorithms.

This streaming architecture requires CPUs specialized for: consuming single-use data efficiently, streaming from external time-series sources, performing massively parallel mathematical operations across data points simultaneously, and aggregating distributed computations into unified results. In practice, this necessitates processors with vector operations, shallow cache hierarchies, and extremely low-latency network connections.

While general-purpose CPUs have dominated computing for decades, trading firms have increasingly adopted FPGAs (Field-Programmable Gate Arrays) and custom ASICs (Application-Specific Integrated Circuits) to achieve dramatically superior performance for their specialized workloads. Companies like Jane Street and Citadel Securities have invested heavily in custom hardware and co-location facilities to minimize latency. This trend represents a broader movement away from one-size-fits-all computing toward purpose-built hardware architectures.

5. Gaming

Real-time multiplayer gaming represents another fascinating departure from traditional web services, with titles like Fortnite, Counter-Strike, Dota 2, and Rocket League creating entirely unique demands on data centre infrastructure. While the computational complexity of game logic itself remains relatively modest - tracking player positions, processing simple physics calculations, and validating moves - the challenge lies in managing thousands of tiny data packets flowing between server and clients every second.

This creates a fundamentally different scaling challenge than the applications we’ve examined so far. Gaming servers must maintain extremely low latency while simultaneously handling dozens or hundreds of concurrent players. The data volume per player is remarkably small, often just a few bytes containing position updates or action commands, but the timing requirements are unforgiving. A delay of even 50 milliseconds can render competitive games unplayable, creating what players experience as “lag”, “rubber banding”, missed hits and score incoherence. Since many players use some kind of real-time communication with friends (e.g. Discord voice chats), these irregularities are very noticeable.

The network packet handling requirements for gaming directly conflict with standard data centre optimization strategies. Traditional web services benefit from packet coalescing - bundling multiple small data transmissions into larger, more efficient bursts that maximize network throughput. Video streaming services exemplify this approach, as user devices can buffer content and slight delays remain invisible to viewers.

Gaming demands the exact opposite: immediate transmission of every individual packet, regardless of efficiency concerns. Delaying a player’s movement command while waiting to bundle it with other data creates the stuttering, unresponsive gameplay that destroys user experience. This requirement puts gaming servers at odds with standard network security measures, since flooding networks with millions of small packets is a common Denial of Service (DoS) attack pattern.

Data centre operators must therefore implement specialized packet prioritization policies and, in some cases, dedicate specific physical network infrastructure to gaming workloads. Some gaming companies require servers connected to dedicated routers and switches, isolated from general web traffic, to ensure their packet handling requirements can be met without triggering security systems designed to block malicious traffic patterns.

The challenge becomes even more complex when gaming companies attempt to leverage public cloud infrastructure rather than operating dedicated gaming-specific data centres. Major cloud providers optimize their networks for the high-throughput, bursty traffic patterns of web services and data processing workloads. Adapting these networks for gaming’s continuous, low-latency requirements has proven a significant stumbling block for many game hosting operations, though specialized gaming-focused cloud services like AWS GameLift and Google Cloud Game Servers are beginning to address these constraints. Companies like Riot Games have built their own global network infrastructure to ensure consistent performance for League of Legends players worldwide.

6. Number crunching

Raw computational workloads represent another distinct category of data centre applications, encompassing everything from weather prediction models to genomics research to sports analytics. While the umbrella term “number crunching” might suggest similarity, these applications actually impose dramatically different requirements on server infrastructure.

Consider the computational patterns involved. Weather forecasting requires processing massive datasets where each geographic grid point can be computed independently before results are aggregated into global models. The European Centre for Medium-Range Weather Forecasts (ECMWF) operates some of the world’s most powerful supercomputers specifically for this highly parallel workload - computational problems that can be divided into many independent tasks. Similarly, Netflix uses massive parallel computing to process video encoding, creating multiple quality versions of each title for different device capabilities and network conditions.

Genomics research presents an entirely different challenge. DNA sequencing analysis often requires processing long chains of calculations where each step depends on the results of previous computations. Rather than independent parallel processing, these workloads demand sequential computational pipelines with substantial intermediate data storage and high-speed data movement between processing stages.

Sports analytics illustrates yet another pattern: processing continuous streams of real-time data to generate insights during live events. This requires servers optimized for streaming computation with minimal buffering delays, as companies like Second Spectrum demonstrate in their real-time basketball analytics platforms. Amazon’s statistical analysis for NFL games and ESPN’s real-time win probability calculations require similar streaming computational architectures.

These divergent computational patterns drive fundamentally different server requirements. Weather forecasting benefits from processors with massive numbers of cores operating in parallel, substantial high-bandwidth memory to hold large datasets, and relatively modest storage requirements. Genomics research may demand processors optimized for sequential throughput, enormous amounts of both memory and storage, and specialized interconnects for coordinating complex computational pipelines. Sports analytics requires processors optimized for streaming workloads with minimal latency, moderate memory capacity, but sophisticated network interfaces for consuming continuous data feeds.

The heterogeneity of computational workloads has driven data centres toward increasingly specialized hardware architectures, with some operators dedicating entire facilities to specific types of number crunching applications.

7. AI

My final example in this section explores AI - specifically Large Language Models (LLMs) that now dominate through Generative AI applications. For hardware vendors, AI has been nothing short of transformational. The reason becomes clear when you examine what these systems demand: AI workloads push virtually every aspect of computing to its limits, simultaneously.

Consider the computational requirements of a single LLM inference request. When you submit a prompt to ChatGPT or Claude, the server must process your input through massive neural networks containing billions of parameters. This requires enormous amounts of high-speed memory to hold the model weights, sophisticated parallel processing capabilities to perform matrix multiplications across thousands of compute units simultaneously, and substantial throughput to handle the cascading calculations that generate each token of the response.

Unlike our previous examples, AI workloads combine the most demanding characteristics from every category we’ve discussed. They require the massive parallel computation of number crunching applications, the low-latency responsiveness expected from interactive apps, the high-throughput data movement of file storage systems, and the complex user session management of traditional web services. A single AI inference request might involve terabytes of parameter data flowing through specialized processors, followed by careful orchestration to serialize millions of parallel computations into a coherent text response.

The storage requirements alone are staggering. GPT-4’s parameter count is estimated in the hundreds of billions, requiring hundreds of gigabytes just to store the model weights. Training these models demands even more extreme resources - Facebook’s LLaMA 2 training reportedly required millions of GPU hours, while Google’s PaLM training consumed computational resources equivalent to running thousands of high-end servers continuously for months.

Network demands compound these challenges further. Users upload everything from simple text prompts to images, documents, and increasingly large context windows. The responses flow back as continuous streams of generated tokens, creating bidirectional traffic patterns that stress both upload and download infrastructure simultaneously. Meanwhile, the underlying systems must handle traditional application concerns: parsing user requests, managing authentication and rate limiting, coordinating between multiple specialized servers, and maintaining session state across potentially lengthy conversations.

This convergence of extreme requirements helps explain why Generative LLMs only became viable in the last decade. The cloud data centre architecture - with its ability to disaggregate massive computations across heterogeneous server types connected by high-speed networks - provided the essential foundation. Modern data centres can orchestrate specialized GPU clusters for model inference, high-memory servers for parameter storage, traditional CPUs for request handling, and sophisticated networking infrastructure to coordinate these disparate components into unified AI services using multi-tenant architectures that efficiently share resources among multiple customers.

The hardware implications extend beyond individual servers to entire facility design. AI workloads generate exceptional heat loads, requiring enhanced cooling systems. They demand reliable power delivery at unprecedented scales - NVIDIA’s H100 GPU consumes up to 700 watts per chip, and AI servers often contain eight or more such processors. The result is that AI-optimized data centres represent a fundamental shift in infrastructure design, optimized for power density and cooling capacity that would have been unthinkable just a few years ago. Companies like Microsoft and Google have built specialized AI infrastructure, while OpenAI partnered with Microsoft to access the massive computational resources required for training and serving models like GPT-4.

Coming Next

In Part 2 of this series, we’ll dive into the physical structure of data centres - exploring how these diverse application requirements translate into actual hardware configurations, facility design, and the complex systems that keep everything running.

Series Navigation

• Part 1: Understanding Applications and Services (this post)
• Part 2: Physical Architecture of a Single Data Centre
• Part 3: Virtual Architecture of a Single Data Centre
• Part 4: Spanning Multiple Data Centres
• Part 5: Business Models and Economics
• Part 6: Evaluation and Future Trends