Original Coverage & Source Attribution: maketecheasier.com
In late 2012, physicist Gaurav Khanna was offered a supercomputer’s worth of used PlayStation 3 consoles for free, and he had nowhere to put them. The consoles came from Condor, the Air Force Research Laboratory (AFRL) cluster in Rome, New York, that wired more than 1,700 PS3s to Intel servers and Nvidia graphics cards. How that machine worked, and why the Air Force chose consoles at all, is covered in Make Tech Easier’s earlier story on the Air Force’s PlayStation supercomputer. This is what happened to some of its hardware next.
The consoles were free. Running them was not. “One challenge we had is that they had substantial power and cooling requirements,” Khanna told DCD in 2018. “The bill we had to renovate a lab to house 400 units was more than $500,000.”
The answer, announced by the University of Massachusetts Dartmouth in June 2014, was a refrigerated shipping container parked on campus, drawing power and network connections from a nearby building and holding 176 PS3s.
How a black hole physicist ended up with Air Force consoles
Khanna got to the consoles before the Air Force did. He began clustering PS3s for black hole simulations in 2007, an approach UMass Dartmouth says he pioneered, starting with a handful of consoles in his department. AFRL, which was looking for chips that combined high performance with low power consumption, took notice. “They were quite impressed with the performance of a PlayStation, especially its power efficiency,” Khanna told DCD.
According to DCD’s 2018 account, he was brought in to explain what he had learned and helped the lab with its system in exchange for time on it, eventually becoming Condor’s most active user. When AFRL decided to repurpose the cluster’s floor space, it offered him all of the consoles. He worked out how many the campus could support and took 400.
The first batch was smaller. A 2014 account by Kai Staats, published by Space.com, says AFRL granted Khanna’s group “four racks of PS3s,” or 176 units, along with networking equipment, cables and software, under a Department of Defense research agreement. Those 176 consoles are the system UMass Dartmouth announced in June 2014. Hackaday reported that December that a further 220 were on their way, and Khanna’s group page now describes a “400+ PS3 cluster.” This story is about the 176.
Why a refrigerated container made sense
A university physics department had no room ready for hundreds of consoles. A computer room needs three things: electricity, a network and a way to remove heat. Khanna’s $500,000 figure was the quote for renovating a lab to house the 400 consoles he planned to take, and he tied it to their power and cooling requirements.
A refrigerated shipping container, or “reefer,” is built to keep perishable cargo cold on long journeys. That makes it an insulated box with its own cooling unit already attached, and shipping lines use them in large numbers. “For $30,000 we had this relatively huge freezer unit that basically gives us cooling capacity of 80,000 British Thermal Units per hour, which is exactly what we needed,” Khanna told DCD. That is the price of the container, not the full cost of running the cluster over its life.
Some rough arithmetic gives a sense of scale. Eighty thousand BTU per hour is about 23 kilowatts of heat removal. Khanna has said a PS3 drew about 100 watts at full load. Under that assumption, the 176 consoles alone would produce about 17.6 kilowatts of heat, below the quoted cooling capacity. That comparison leaves out other equipment and operating conditions, so it is an illustration rather than a measured cooling margin.
The container idea has a second author. Staats, who co-founded Terra Soft Solutions, the developer of Yellow Dog Linux and of high-performance systems built on IBM’s POWER architecture, wrote that the container was used “at the suggestion of this author,” drawing on his “experience in working with unique clusters.” Khanna then bought it. Staats says the project took months, and that the cluster had been “in full production operation since Winter 2014.”
The direct evidence on how well it worked comes from Khanna. “At a tenth of the cost, we had a mini data center,” he told DCD four years later. “We’ve had no trouble, it’s been able to maintain the temperature in a very stable way.”
From radar images to black holes
At AFRL, Condor’s main applied job was turning radar echoes into images. In the container, the same kind of hardware went to work on gravitational waves.
Khanna’s broader research concerns gravitational-wave astronomy, work relevant to the LIGO detectors, which at the time had not yet made a direct detection. The specific model described below, in which a small object such as a star-sized body spirals into a much larger spinning black hole, concerns signals relevant to space-based detectors such as the planned LISA mission. Graduate students used the container for their own simulations. “On a typical machine, those simulations would take months to run, whereas on the PlayStation, they only take a few days,” physics graduate student Tyler Spilhaus said in a December 2014 university feature.
The university’s announcement also mentioned cybersecurity, with few details. The clearest documented example is a cryptographic key-search challenge, not an attack on real systems. In September 2014, UMass Dartmouth said the cluster had taken it to the top of the list for the RC5 challenge, a contest started by RSA Laboratories in 1997 in which participants search for the secret key to an encrypted message by trying possible keys. The cluster tested about 50 billion keys per second. That is close to an ideal cluster job, because each console can test its own range of keys without waiting for any other. Computer and information science undergraduate Violet Pfeiffer was set to use the system to study cryptography and cybersecurity.
Making black hole code fit a PS3
The PS3’s Cell processor, as the earlier Condor story explains, paired one general-purpose core with six usable arithmetic engines called SPEs, each with only 256 KB of its own memory. What mattered for Khanna was finding the part of his own code that suited them.
His code represents the space around the black hole as a grid of points and steps the whole grid forward in time. When Khanna and Justin McKennon profiled it for a 2009 study, one piece, the “source term” describing the small object’s influence, took 99% of the running time. It involves heavy arithmetic at each grid point, and the calculation at one point does not need to wait for the result at another, which is exactly the kind of work the SPEs were built for. According to the group’s PS3 project page, handing that piece to a PS3’s six SPEs made it more than six times faster than the general-purpose core alone, in double precision.
Across consoles, the group split the grid into slices, one per PS3. During the simulation, neighboring consoles exchanged boundary data as needed. Each console did a great deal of arithmetic and sent relatively little over the network, which is what let cheap hardware do useful work as a cluster.
The limits that applied at AFRL applied in the container too. UMass Dartmouth compared the installation’s performance with nearly 3,000 conventional processor cores, but its announcement did not specify the benchmark or comparison hardware, so the figure should not be treated as a universal speed rating. On HPL, the standard benchmark used to rank supercomputers, the group’s earlier 16-console cluster managed only about 2.5 billion floating-point operations per second per console, in a test its page calls “far from optimal” because of the consoles’ memory. And because Sony had removed Linux support from new firmware in 2010, every console in the container had to be an original model running firmware older than version 3.21, which made replacements harder to find.
Moving beyond the PS3s
Khanna’s group was already moving to graphics cards as the PS3 cluster expanded. In June 2015, UMass Dartmouth announced donations of GPU-accelerated systems originally built for bitcoin mining, and a 2017 paper from the group credits GPU computing for its long simulations. The consoles kept running anyway: in 2019, The Verge described PS3s still working in the container on campus. Khanna is now a professor of physics and assistant vice president for research computing at the University of Rhode Island.
The Air Force’s consoles were free, but making them useful required a place to house and cool them, suitable power connections, and software written for their unusual processors. The container supplied an affordable home. The programming turned retired military hardware into a research tool.
