Daisuke Wakabayashi and Cade Metz report that Google “has fired a researcher who questioned a paper it published on the abilities of a specialized type of artificial intelligence used in making computer chips”:
The researcher, Satrajit Chatterjee, led a team of scientists in challenging the celebrated research paper, which appeared last year in the scientific journal Nature and said computers were able to design certain parts of a computer chip faster and better than human beings.
Dr. Chatterjee, 43, was fired in March, shortly after Google told his team that it would not publish a paper that rebutted some of the claims made in Nature, said four people familiar with the situation who were not permitted to speak openly on the matter. . . . Google declined to elaborate about Dr. Chatterjee’s dismissal, but it offered a full-throated defense of the research he criticized and of its unwillingness to publish his assessment.
“We thoroughly vetted the original Nature paper and stand by the peer-reviewed results,” Zoubin Ghahramani, a vice president at Google Research, said in a written statement. “We also rigorously investigated the technical claims of a subsequent submission, and it did not meet our standards for publication.” . . .
The paper in Nature, published last June, promoted a technology called reinforcement learning, which the paper said could improve the design of computer chips. . . . Google had been working on applying the machine learning technique to chip design for years, and it published a similar paper a year earlier. Around that time, Google asked Dr. Chatterjee, who has a doctorate in computer science from the University of California, Berkeley, and had worked as a research scientist at Intel, to see if the approach could be sold or licensed to a chip design company . . .
But Dr. Chatterjee expressed reservations in an internal email about some of the paper’s claims and questioned whether the technology had been rigorously tested . . . While the debate about that research continued, Google pitched another paper to Nature. For the submission, Google made some adjustments to the earlier paper and removed the names of two authors, who had worked closely with Dr. Chatterjee and had also expressed concerns about the paper’s main claims . . .
Google allowed Dr. Chatterjee and a handful of internal and external researchers to work on a paper that challenged some of its claims.
The team submitted the rebuttal paper to a so-called resolution committee for publication approval. Months later, the paper was rejected.
There is another side to the story, though:
Ms. Goldie [one of the authors of the recently published article on chip design] said that Dr. Chatterjee had asked to manage their project in 2019 and that they had declined. When he later criticized it, she said, he could not substantiate his complaints and ignored the evidence they presented in response.
“Sat Chatterjee has waged a campaign of misinformation against me and [coauthor] Azalia for over two years now,” Ms. Goldie said in a written statement.
She said the work had been peer-reviewed by Nature, one of the most prestigious scientific publications. And she added that Google had used their methods to build new chips and that these chips were currently used in Google’s computer data centers.
And an outsider perspective:
After the rebuttal paper was shared with academics and other experts outside Google, the controversy spread throughout the global community of researchers who specialize in chip design.
The chip maker Nvidia says it has used methods for chip design that are similar to Google’s, but some experts are unsure what Google’s research means for the larger tech industry.
“If this is really working well, it would be a really great thing,” said Jens Lienig, a professor at the Dresden University of Technology in Germany, referring to the A.I. technology described in Google’s paper. “But it is not clear if it is working.”
The above-linked news article has links to the recent paper in Nature (“A graph placement methodology for fast chip design,” by Azalia Mirhoseini) and the earlier preprint (“Chip placement with deep reinforcement learning”), but I didn’t see any link to the Chatterjee et al. response. The news article says that “the rebuttal paper was shared with academics and other experts outside Google,” so it must be out there somewhere, but I couldn’t find it in a quick, ummmmm, Google search. The closest I came was this news article by Subham Mitra that reports:
The new episode emerged after the scientific journal Nature in June published “A graph placement methodology for fast chip design,” led by Google scientists Azalia Mirhoseini and Anna Goldie. They discovered that AI could complete a key step in the design process for chips, known as floorplanning, faster and better than an unspecified human expert, a subjective reference point.
But other Google colleagues in a paper that was anonymously posted online in March – “Stronger Baselines for Evaluating Deep Reinforcement Learning in Chip Placement” – found that two alternative approaches based on basic software outperform the AI. One beat it on a well-known test, and the other on a proprietary Google rubric.
Google declined to comment on the leaked draft, but two workers confirmed its authenticity.
I searched on “Stronger Baselines for Evaluating Deep Learning in Chip Placement” but couldn’t find anything. So no opportunity to read the two papers side by side.
Comparison to humans or comparison to default software?
I can’t judge the technical controversy given the available information. From the abstract to “A graph placement methodology for fast chip design”:
Despite five decades of research, chip floorplanning has defied automation, requiring months of intense effort by physical design engineers to produce manufacturable layouts. Here we present a deep reinforcement learning approach to chip floorplanning. In under six hours, our method automatically generates chip floorplans that are superior or comparable to those produced by humans in all key metrics, including power consumption, performance and chip area. . . . Our method was used to design the next generation of Google’s artificial intelligence (AI) accelerators . . .
This abstract is all about comparisons with humans, but it seems that this is not the key issue, if the Stronger Baselines article was claiming that “two alternative approaches based on basic software outperform the AI.” I did find this bit near the end of the “A graph placement methodology” article:
Comparing with baseline methods. In this section, we compare our method with the state-of-the-art RePlAce and with the production design of the previous generation of TPU, which was generated by a team of human physical designers. . . . To perform a fair comparison, we ensured that all methods had the same experimental setup, including the same inputs and the same EDA tool settings. . . . For our method, we use a policy pre-trained on the largest dataset (20 TPU blocks) and then fine-tune it on five target unseen blocks (denoted as blocks 1–5) for no more than 6 h. For confidentiality reasons, we cannot disclose the details of these blocks, but each contains up to a few hundred macros and millions of standard cells. . . . As shown in Table 1, our method outperforms RePlAce in generating placements that meet design criteria. . . .
There’s a potential hole here in “For confidentiality reasons, we cannot disclose the details of these blocks,” but I don’t really know. The article is making some specific claims so I’d like to see the specifics in the rebuttal.
It doesn’t sound like there’s much dispute about the claim that automated methods can outperform human design. That is not a huge surprise, given that this is a well-defined optimization problem. Indeed, I’d like to see some discussion of what aspects of the problem make it so difficult that it wasn’t already machine-optimized. From the abstract: “Despite five decades of research, chip floorplanning has defied automation, requiring months of intense effort by physical design engineers to produce manufacturable layouts,” but the article also refers to “the state-of-the-art RePlAce,” so does that mean that RePLAce is only partly automatic?
The whole thing is a bit mysterious to me. I’m not saying the authors of this paper did anything wrong; I just don’t quite understand what’s being claimed here: in one place the big deal seems to be that this procedure is being automated; elsewhere the dispute seems to be a comparison to basic software.
Google’s problems with reproducibility
Google produces some great software. They also seem to follow the tech industry strategy of promoting vaporware, or, as we’d say in science, non-reproducible research.
We’ve seen two recent examples:
1. The LaMDA chatbot, which was extravagantly promoted by Google engineer Blaise Agüera y Arcas but with a bunch of non-reproducible examples. I posted on this multiple times and also contacted people within Google, but neither Agüera y Arcas nor anyone else has come forth with any evidence that the impressive conversational behavior claimed from LaMDA is reproducible. It might have happened or it might all be a product of careful editing, selection, and initialization—I have no idea!
2. University of California professor and Google employee Matthew Walker, who misrepresents data and promotes junk science regarding sleep.
That doesn’t mean that Chatterjee is correct in the above dispute. I’m just saying it’s a complicated world out there, and you can’t necessarily believe a scientific or engineering claim coming out of Google (or anywhere else).
P.S. From comments, it seems that Google no longer employs Matthew Walker. That makes sense. It always seemed to a misfit for a data-focused company to be working with someone who’s famous for misrepresenting data.
P.P.S. An anonymous tipster sent me the mysterious Stronger Baselines paper. Here it is, and here’s the abstract:
This all leaves me even more confused. If Chatterjee et al. can “produce competitive layouts with computational resources smaller by orders of magnitude,” they could show that, no? Or, I guess not, because it’s all trade secrets.
This represents a real challenge for scholarly journals. On one hand, you don’t want them publishing research that can’t be reproduced; on the other hand, if the best work is being done in proprietary settings, what can you do? I don’t know the answer here. This is all setting aside personnel disputes at Google.