Differential architectures sometimes tempt you to ignore return-current issues, assuming that the signal current returns on the other trace. Although in some cases this assumption may provide a useful mental image, it is not true. Even in a differential configuration, current flows separately on the planes under each trace, almost as if they were two independently routed, single-ended signals. Figure 1 dramatically illustrates this point.
Imagine that you inject a differential signal at Point A onto Layer 3 using edge-coupled striplines. Layer 2 (pink) carries the return current associated with the two elements of the differential pair. Layer 2 happens to be a power plane, which works OK in many cases, except that all the noise on the power plane superimposes onto the signal traces as a common-mode signal. If the common-mode rejection of the receiver is sufficiently good, the circuit works.
At Point B, the signal traces diverge to accommodate SMA connectors residing on the surface. The divergence creates two completely independent, single-ended signals. Clearly, the return current associated with each signal flows separately on the underside of the power plane (pink), hugging tightly to the trace, following wherever it goes.
At the SMA locations C and D, the SMA connectors carry their returning signal current on ground pins, connected in a pattern of four vias to the ground plane. Apparently, somewhere near the SMA connectors, your the returning signal current must change reference planes. It has to get from the pink plane (near the traces) to the blue plane (near the connector). How? Your guess is as good as mine, because no intentional connection exists between the planes near the connector locations. The returning signal current probably diverts far away from the SMA connector in its search for a suitable connection between the planes. It will eventually flow through a bypass capacitor or, once it spreads far enough, through the parasitic capacitance between the planes themselves. Such a diversion in the path of returning signal current degrades your signal just as if you had passed the signal itself through the same circuitous pathway. In a 2.5-Gbps XAUI configuration, this layout produces a horrible-looking eye pattern at the receiver.
To fix the problem, if Layer 2 must for other reasons remain a power layer, carve a region out of the power layer near the SMA connectors. Make it big enough to fill the area under both SMAs, incorporating all their ground pins and also including Point B. Fill this region with ground. Leave only a small gap (6 mils, or whatever is your minimum air gap) between the power and ground regions on that layer. As your traces pass over the boundary between the power region and the ground region, keep the traces close together. (A space of 12 mils is probably close enough for XAUI.)
As the differential traces cross the power-to-ground gap on Layer 2, the returning signal currents will execute a tiny U-turn (Johnson, Howard, "Differential U-Turn," EDN, Sept 1, 2000, page 32). As long as the gap between the power and ground regions remains small and the traces lie close together at that point, the configuration works.
Once you cross into the ground-referenced region, tie the Layer 2 ground region to the SMA ground pins in the normal way. This approach has the advantage of never requiring the returning signal current to dive down to the inner ground layer.