Gadi appears in 24th on the Top500 ranking - 23 June 2020
A little over six months since users first logged in to Gadi, the new supercomputer has now appeared high up on the Top500 list released overnight at the ISC High Performance Digital Conference. Placing 24th with a peak performance of 9.26 Petaflops, Gadi is 8 times as powerful as Raijin, the supercomputer it replaces. We are thrilled at the great reception from our users, as well as the broader Australian science community. We are so proud to feature an artwork from Ngunnawal artist Lynnice Letty Church on our supercomputer – her art and the Gadi name perfectly encompass NCI’s mission of scientific research and high-performance computing: to search for knowledge that can make the world a better place through enabling innovative world-class research.
Gadi is open for business - 18 November 2019 - Day 110
The Gadi supercomputer has been opened to users for the very first time today as 'Phase 1' is completed. Eighteen racks of Gadi are now online and currently being battle-tested by Australian researchers, with the rest of the supercomputer to be made available in Q1 2020.
Today marks a milestone in Australian HPC. Staff have been working round the clock to make sure that Gadi is the best possible environment for today's researchers to land that important scientific breakthrough, and make our world a better place.
A lot has changed in 110 days at NCI!
A very cool delivery - 1 November 2019 - Day 92
All the pieces are coming together now. Some of the last critical components, the Cooling Distribution Units (CDUs), arrived on site last week and were rapidly connected up to the network of water pipes, tanks and cooling towers that maintain the temperature of our supercomputer. Manufactured to our specifications and specific cooling needs, these CDUs are a vital piece of the Gadi puzzle. The only way that a supercomputer like Gadi can operate is with a robust cooling infrastructure behind it to take away all of the heat its powerful processors produce.
Building a high-koalaty network - 30 October 2019 - Day 90
Nearing the final stages of the Gadi build, over the past few weeks a key component has been installed. Our Infiniband network is now in place, connecting the new Gadi compute nodes to their new /scratch filesystem, the /g/data filesystems, and MDSS. Hundreds of fibre-optic cables are now ready to deliver a 200Gb/s HDR Infiniband network to all our Gadi users. And all with a little help from our little NCI koala friend!
Australia's most powerful supercomputer takes shape at NCI – 16 October 2019 – Day 76
It’s the home stretch. The core components of Gadi are now located in NCI’s data centre, and work is underway to ‘plumb it all in’ – literally and metaphorically.
The plumbing that is used to keep Gadi cool has been connected to the in-row coolers. These coolers will – as the name suggests – keep Gadi running cool, and are placed at regular intervals beside compute hardware.
These pipes will pump special water to prevent the build-up of minerals inside the plumbing, ensuring that Gadi will stay chill for years to come.
At the same time, connection of power and networking for each of the Gadi nodes is currently underway. Extra power cabinets have been installed along the western side of the data centre, and network cabling is being laid overhead.
Meanwhile, Gadi’s new file system has been configured and is ready for a supercomputer. The blinking lights are a dead giveaway.
Full to the brim – 27 September 2019 – Day 57
With all our compute racks, cooling units, power distribution racks and more all filling up our data hall, it's getting pretty crowded in there! In the next few weeks, all the racks will be arranged into neat rows and connected up to power and water.
It's all happening now! An update in THREE parts – 18 September 2019 – Day 48
Part 1: Our new data arrays, installed during August, have now been joined by their management servers. These specialised nodes are the final major piece of the Gadi storage solution, so getting them in place is a big deal.
Part 2: The electrical distribution boards that will send power through the Gadi system are currently being installed. Following the underfloor piping and electrical work, all those orange cables now need to be connected up to the big orange boxes that split the power up for use across the entire system.
Part 3: Our all important compute racks have started arriving! Shipments of compute racks with pre-installed compute nodes are arriving this week, getting us ever closer to operations. Soon they will be mounted in place on the floor with their dedicated in-row coolers in position too. There are still some key components to come before we can switch them on, but the excitement is building!
Fabulous flooring, fit for a supercomputer – 13th September 2019 – Day 43
The super-extra-strong floor tiles are now in place. This void won't stay empty for long, as soon we'll have the first components of the Gadi supercomputing cluster being rolled into place.
The floor tiles conceal all of the complex plumbing and electrical infrastructure that was installed prior. When taking visitors on tours of NCI, we often say that the maze of pipes and cables is just as impressive as the computing cluster itself, perhaps even more so.
Meanwhile, new switches have been installed above the NetApp storage arrays, the last few pieces of a very complicated data storage puzzle. These file systems will be the fastest of their class in the Southern Hemisphere.
Even supercomputers need to keep cool! – 6th September 2019 – Day 36
We are almost finished installing the cooling infrastructure that will sit under the floor and provide Gadi with the circulating water it needs to keep cool. The water pipes are specifically designed for our needs, in particular the specific range of water temperatures that we will be working with to draw excess heat away from the supercomputer. Constructed off site, the pipes are trucked to our building and welded into place one by one.
We're almost ready for the floor to go back on!
Premium plumbing and a pint-sized supercomputer – 30th August 2019 – Day 30
You can't have a world-class supercomputer without a way to keep everything cool. Gadi will use evaporative cooling to beat the heat, just like Raijin. Instead of moving and removing heat using air (fans), heat from the supercomputer is transferred using water. The heated water is pumped away from Gadi, then evaporated via large cooling towers on the data centre roof, while cool water is then pumped back into the data centre to start the cycle all over again.
These water 'loops' require purpose-built plumbing to stand up to the rigours of operation. The new plumbing for Gadi is especially treated to stop the build up of micro-organisms in the water supply, while also preventing the metal components themselves from degrading and contaminating the system.
While new plumbing is laid down for Gadi, a rack of mysterious looking equipment has since been plumbed in and switched on. These compute nodes are a miniature emulation of Gadi, and will be used to road-test the technology behind what is soon-to-be Australia's most powerful supercomputer. Our system administrators will be able to use this now, and in the future, to experiment with improvements to Gadi without interrupting the whole system.
With the last of the storage arrays being installed this week, we're really excited to see what's just around the corner...
Data for days – 26th August 2019 – Day 26
The finishing touches are being readied for Gadi's new file systems. Nearly all of the storage arrays are now in place - now all that's left is for them to be plugged in and powered up (that might be a massive oversimplification).
They won't become accessible until Gadi is brought online later in the year, but they'll be ready all the same.
Data by the truckload – 21st August 2019 – Day 21
As soon as there is free space in the data centre, it's filled by fresh new hardware. Storage arrays and hard disk drives for Gadi's new file systems continues to be delivered by the truckload, and it won't be long before this new storage infrastructure is installed and ready to be fired up.
Whoa we’re half way there… Whoa-oh! Look at all our hardware! – 14th August 2019 – Day 14
Shiny, right?
We’re now past the half-way point for installing the new Gadi file systems. Drawers are being filled with fresh hard disk drives and cabling is being readied.
Every hard disk drive will be tested for performance and reliability before becoming a part of Gadi. These are all important steps before the compute hardware is delivered in the coming weeks and months. Work in the western side of the data centre is continuing in earnest.
Gadi grows in bits and bytes – 8th August 2019 – Day 8
The data centre has been a hive of activity today. While flooring continues to come up at one end the building, the other end is being populated with new storage for Gadi. Thousands of hard disk drives have been delivered over the past two days, and will make up the /home (‘slash home’) and scratch file systems for Gadi.
Floor tiling at the other end of the data centre has almost all been removed, and cable trays are currently being installed along the ceiling.
Floor the record, we’re making progress – 7th August 2019 – Day 7
It’s demo day! We have started pulling up floor tiles in the data centre to begin installation of essential plumbing and electrical services to support Gadi. Obviously, all of this work has to be done before anything else can happen.
A new set of in-row coolers also arrived today, giving us another behind-the-scenes look at our infrastructure. They look a lot like concert speakers, but these coolers are integral to NCI’s cooling requirements. Without them, everything would overheat very quickly.
Heat from the supercomputer is trapped in ‘hot aisles’, contained spaces between the rows and rows of computer equipment. This heat is then drawn over the cool-water radiators and transferred away from the equipment.
Meanwhile, cooled air is then drawn back into the data centre to begin the process all over again. The hot air makes its way up to the data centre roof, where it evaporates in Canberra’s dry air. This is commonly known as ‘free cooling’.
We have also taken delivery of hard disk drives and other storage hardware that will make up the /home (‘slash home’) file system of Gadi. These are being set aside for the moment while supporting infrastructure is assembled and put in place.
It’s just a ‘phase’ we’re going through – 6th August 2019 – Day 6
I’ve got the power!
Each storage rack contains six of these super-sized power boards.
New Storage Infrastructure Arrives – 2nd August 2019 – Day 2
Our very first deliveries are already here. These racks will be used to house new storage infrastructure that will expand NCI’s total storage capacity and support Gadi. Currently, NCI’s total data storage capacity stands around 50 Petabytes, or 50 million Gigabytes.
They will be installed at the opposite end of the data centre, as Gadi itself will require all the space currently available on the western side of the building. Colocation of storage and compute at NCI has been an important part of our organisation for decades, and will continue to be once Gadi is up and running. Our HPC and data teams are expecting transfer speeds of up to 200Gb/s under the new system, more than twice as fast as what is available now with Raijin.
Small steps again today, but the best is yet to come.
Gadi is a GO – 1st August 2019 – Day 1
Following last week’s announcement, work has already commenced in NCI’s data centre to make way for the new hardware. From the very beginning, NCI’s data centre has kept an empty space available for Raijin’s eventual successor, and we are going to need every bit of it.
This forward thinking means that Raijin will stay online and available for our thousands of researchers while Gadi is assembled.
The overhead lights that spanned the empty space are starting to come down to make way for new lights and cable trays, while new lights will be installed to keep everything illuminated. This work is just the start of an incredibly busy period for NCI. The first truckloads of equipment are expected to arrive in the coming weeks.
