Articles

Datacentres and the Earthing Design Challenges they Present

The rest of the world including Australia has experienced remarkable growth in recent years in datacentre developments, and this trend is expected to continue. Datacentres are infamous heavy consumers of electricity with the servers, data storage and cooling being the predominant load of electrical power. Depending on the expected datacentre energy consumption, electrical connection may be at distribution voltage levels where one or more kiosk type substations are established at 11kV or 22kV.

From this article’s perspective datacentres will be referred to according to their electrical consumption, or load, as they have grown in size from 5MW only a few years ago, and will reach the 100+MW size in years to come. One IT magnate is talking about a datacentre campus that may exceed a GW in a few years. And these datacentre developers don’t stop at one; developers build these things in groups called Availability Zones. According to Microsoft Availability Zones are ‘separated groups of datacentres within a region’.

“Availability Zones are close enough to have low-latency connections to other Availability Zones. They’re connected by a high-performance network with a round-trip latency of less than 2ms. However, availability zones are far enough apart to reduce the likelihood that more than one will be affected by local outages or weather. Availability zones have independent power, cooling, and networking infrastructure. They’re designed so that if one zone experiences an outage, then regional services, capacity, and high availability are supported by the remaining zones. They help user’s data stay synchronized and accessible when things go wrong.”

Source: (https://learn.microsoft.com/en-us/azure/reliability/availability-zones-overview?tabs=azure-cli)

Larger datacentres and campuses need to be connected at sub-transmission to transmission voltages such as 66kV or 132kV and likely higher as consumption needs grow! As such, High Voltage zone style substations are required to be designed and built that supply power to the datacentre, who in turn place their own transformers as required to supply their equipment at medium voltages.

So, what are the electrical challenges faced by these incredible infrastructure projects? Reliability must feature front and centre so key aspects like power quality, harmonics, defence against transients, even electromagnetic cyber security are all important factors that must be considered. Amongst all these issues, the datacentre earthing system and lightning protection design is fundamentally critical to safe and reliable ongoing operation of the datacentre.

Datacentre earthing and lightning design can present some interesting challenges, with those challenges depending on the jurisdiction the datacentre is proposed to be established. What has become apparent is that it depends heavily on who the electrical utility company the datacentre takes their electrical connection from. The reason for this is that the electrical networks in the various jurisdictions across Australia differ considerably, as do the earthing connection philosophies of the different electrical networks and service providers (DNSPs). Some DNSPs are also fully cognisant of substation, structure, and transmission line lightning protection requirements, whereas others have little to no lightning protection systems or design experience within their networks. In many cases the datacentre developer has competing earthing and lightning protection requirements with the DNSP, with some of these competing requirements being founded with reasonable basis, while some are still based on older and no longer current best practice.

Some recent examples presented as two case studies highlight where these incompatibilities can cause issues or worse, hazards and increased risk.

Case 1 – 50MVA Datacentre situated in a light industrial setting, in the outer suburbs of a capital city. DNSP will own the HV substation up to the demarcation line of the secondary voltage of the power transformer terminals. The DC developer will own distribution voltage cabling and switchgear etc.

Here we have the all-too-common scenario where a medium sized datacentre is proposed for installation in an DNSP network area where the subtransmission supply high voltage overhead power lines were never installed with overhead aerial earth wires (OHEWs or OPGWs). The reason for this is historical and that lightning shielding of overhead lines never really seemed to be needed and was never a priority. Thankfully for this region the incidence of lightning is very low, however this means the earthing systems of the network are disparate and unable to be used as part of the earthing solution.

It follows that the DNSP, being the owner/operator of the HV substation including the transformer, has a policy that requires the datacentre owner to demonstrate earthing compliance in their own right – sometimes referred to as standalone compliance. In some cases, this may indeed be a viable option but is heavily dependent on so many factors that must be analysed early in the project, at the concept stage. Disappointingly these important factors are often neglected until it is too late and many commitments to the system have already been made. Key factors include size of the installation, distance separating the HV substation and the datacentre, soil electrical properties and fault levels. And all the while, as attempts to isolate the systems are being tossed around, they are forgetting that in Australia, now in all states, we are required to eliminate risks, or reduce risks to as low as reasonably practicable. Meaning that interconnection of the substation and the data centre is almost always the most responsible action to take.

Case 2 – 100MW Datacentre situated in a light industrial setting, in the outer suburbs of a capital city (there seems to be a pattern here!). The difference this time, however, is that the datacentre owner will own the high voltage system entirely but connect to the utility at transmission voltage level. It is a bigger datacentre footprint and soil electrical resistivity is favourable to achieving an overall very low earth grid resistance.

With the transmission connection come an OPGW installed on the line offering fast protection options but also reliability improvements due to the lightning shielding afforded by the aerial earth wire. Importantly though, a good quality OPGW conductor and transmission earthing system will return 50% or more of the earth fault current to its source, reducing local hazards markedly. With so many options the datacentre owner can arrange the site optimally and even bond medium voltage cables at both ends, creating a very secure and reliable earthing system which also happens to exhibit low risk to Australian Standard AS2067. The entities of the datacentre may achieve low risk standalone, however a key and often forgotten requirement of Australia’s risk-based earthing framework is that if there is common or cost-effective solution or mitigation, then it should be implemented irrespective of the final risk level of the system. Therefore, the HV substation and datacentre and transmission earthing systems invariably are all connected.

Conclusion

The DNSPs across Australia have generally embraced the evolution to adopt risk-based earthing in line with work health and safety legislation and Australian standards such as AS2067. However, most of the datacentre companies are American based and their expectation is to design for so-called ‘safe’ voltage hazards according to IEEE Standard 80. Unfortunately, it is very difficult to make comparison between AS2067 and Std 80. There are vast differences that is true, but the key dissimilarity is that one is based on a broader range of probabilistic analysis of data whereas the other is based on many simplifications which makes it difficult to incorporate into a proper risk evaluation. Regrettably Std 80 has fallen out of sync here in Australia which makes its application in the context of a risk-based framework, dare I say it – risky.

Datacentre developments here in our major cities are subject to difficulties and complications that are forced upon them by location. Good soil conditions versus bad. A DNSP with an interconnected earthing network versus one that does not. A philosophical legacy to isolate systems versus preparedness to bond systems. With all these challenges facing the datacentre developer the most important thing the datacentre developer can do is initiate the concept earthing and lightning protection strategy as early on the project as possible.

Disclaimer – Safearth has intentionally not been specific about any one datacentre developer or electrical utility in this article. As such some of the figures and details are fabricated but based on recent experiences.

By Bill Tocher – Engineering Director | Manager Strategic Projects, Safearth

For more information: enquiries@safearth.com or 1800 327 844