Clocking in data centers

Figure 3 and Figure 4 illustrate the PCIe internal clock and external clock architectures on server motherboards, respectively. On internal clock servers, the CPUs generate PCIe clocks. These PCIe clocks are then fanned out by PCIe clock buffers. The buffered outputs clock various endpoints or pass to daughtercards through PCIe connectors.

On external clock servers there could be various PCIe clocking sources: local PCIe clocks from the external clock generator or PCIe clocks generated by the CPUs. Every endpoint or connector can select from one of these sources, depending on the clock domains they belong to.

Devices and interfaces usually require low-frequency single-ended clocks provided by oscillators or a clock generator.

A data center security control module (DC-SCM) is an add-on card defined by the Open Compute Project. In the example shown in Figure 5, a PCIe reference clock is provided to the DC-SCM card from the server motherboard. However, both the baseboard management controller and USB host controller need PCIe clocks. You cannot simply divide the trace and route one clock to both devices, because that would halve the amplitude and degrade signal integrity. As a result, the clock signal will no longer meet PCIe compliance, which is why a PCIe clock buffer is necessary. The clock buffer receives one clock input and generates multiple copies of the input without degrading signal integrity.

Similar to a DC-SCM, other expansion cards or PCIe add-in cards may also need a clock buffer for PCIe clock distribution, as shown in Figure 6.

A network interface card (NIC) connects a server to the network. A SmartNIC provides additional computing resources to offload the server CPUs. Both NICs and SmartNICs need PCIe and Ethernet clocks. In the example shown in Figure 7, there are two PCIe clock sources: a common PCIe clock from the motherboard for the CC architecture and a local PCIe clock for the IR architecture. In normal operation mode, the NIC operates on the CC. But if the CC is lost or unavailable, the NIC can switch to IR and use the local PCIe clock instead. Additionally, because a NIC connects to a switch through Ethernet ports, the Ethernet SerDes within the application-specific integrated circuit usually requires a high-performance 156.25-MHz clock.

The PCIe clocking requirements for a hardware accelerator, used for specific computing tasks such as AI training, are similar to a SmartNIC. Figure 8 shows an example of a PCIe clock architecture. Instead of having both CC and IR architectures and switching between the two, the PCIe clocks are only generated locally. In this example, the CPUs, graphics processing units and other endpoints require many clocks. Therefore, a two-channel clock generator along with two 20-channel clock buffers can generate as many as 40 PCIe clocks. A hardware accelerator does not need an Ethernet clock because it is not connected to a ToR switch like a SmartNIC. There may be proprietary links other than PCIe that could require additional high-performance clocks, however, similar to Ethernet clocks.

Figure 9 is another example of using only an IR PCIe architecture. A two-channel PCIe clock generator is used to clock the solid-state drive (SSD) controller.

Ethernet lane speeds 56 Gbps or higher significantly affect the insertion loss of any passive cable. Therefore, “active” interconnection is necessary to reduce losses and improve data quality. Depending on the distance, there are different types of active interconnection. Active cables, including active electrical cables based on copper and active optical cables based on fiber, can connect over short distances, such as the distance from a NIC to a ToR switch.

Optical modules are used to connect over longer distances. There are also different types of optical modules. Some are used between the ToR switch and the spine or fabric switch within a data center, while others can be used between data centers.

Because of the high Ethernet lane speeds, the optical module digital signal processor requires a very low noise Ethernet clock, as shown in Figure 10. On the other hand, the Ethernet retimers in an active cable only need a regular clock, as shown in Figure 11.

A 1-PPS signal carries clock synchronization information and is passed down from the spine or fabric switch to the ToR switch, and then to the NIC or SmartNIC. You may need a 1-PPS buffer or level translator in the active cable paddle card for level translation and to generate additional copies.

Generating the reference clocks for high-speed Ethernet SerDes requires an extremely low-jitter clock generator, as shown in Figure 12. A spine or fabric switch also requires similar or better Ethernet clock performance compared to a ToR switch. In addition, you’ll need a timing digital phase-locked loop (DPLL) for network synchronization, as shown in Figure 13.