700 Tokens Per Second: New Google Chip Speed Challenges Nvidia Dominance

Google’s custom AI chips are now powering large-scale AI workloads at Meta, marking the first time these processors have been sold to an external major customer in this capacity.

The announcement, made on Google’s developer blog, details a new version of the SGLang inference engine that is native to PyTorch. This software update allows Google’s tensor processing units, or TPUs, to run directly on the PyTorch framework without requiring extensive code rewrites. Previously, developers had to use Google’s proprietary JAX framework to operate on this hardware. A corresponding open-source version of vLLM, another popular inference engine, was also released to support this new workflow.

This development represents a strategic pivot for both companies. Meta, which created the widely used PyTorch framework, had previously relied on a bridge called PyTorch/XLA to connect its software to Google’s hardware. That setup was considered a predecessor to the new system. According to reports from SemiAnalysis, Meta used TPUs through that older bridge between 2020 and 2023 before canceling those contracts. The new TorchTPU stack was co-developed by Google and Meta’s PyTorch team to address the need for a more mature and native serving environment.

Zongwei Zhou, who leads the PyTorch-TPU effort at Meta, confirmed that the new SGLang-TorchTPU system is already running in production. He described the achievement as the result of a close three-way collaboration involving Meta, Google, and the SGLang team. Zhou works at RadixArk, a startup founded by the creators of SGLang, which built the engine in partnership with the tech giants. He noted that the new stack provides a remarkably mature environment for running PyTorch-native services.

Jana van Greunen, director of PyTorch engineering at Meta, emphasized the importance of hardware flexibility for their infrastructure. She stated that Meta cannot be constrained by a single hardware architecture. By adopting this new technology, the company can deploy its existing PyTorch workloads directly onto Google’s hardware. This move is part of a broader industry trend to reduce dependence on a single chip supplier.

The timing of this announcement is significant. It came just one day after Amazon revealed it was open-sourcing TorchNeuron, a similar native backend for its own Trainium2 and Trainium3 chips. Both companies appear to be making concerted efforts to loosen the software lock-in associated with Nvidia’s market leadership. For businesses in Texas and across the South Central region that rely on cloud computing, these shifts could mean more options and potentially lower costs for running advanced AI models.

In terms of raw performance, the new setup shows impressive results. A tuned vLLM configuration serving a large open-weight model called Kimi K3 reportedly achieves speeds exceeding 700 tokens per second for a single user. This test was conducted on the seventh-generation Ironwood TPU. The figure comes from Inferact, a startup founded by the creators of vLLM that helped build the new software. The test utilized 16 Ironwood chips with speculative decoding, a technique that uses a smaller model to speed up the process. In comparison, a published setup using 16 Nvidia GB200 chips reached 452 tokens per second. The new kernels also show throughput that is 1.4 to 2 times higher than the Nvidia baseline at small batch sizes when speculative decoding is not used.

Looking ahead, the SGLang-TorchTPU project plans to support additional models, starting with DeepSeek V4.1 Flash and Reflection Beam. The team is also working to provide support for new open models on the day they are released. This goal of day-zero support is targeted for the first quarter of 2027. While it is not yet clear where Meta’s production TPUs are physically located, previous talks included Google Cloud rentals in 2026 and purchases in 2027, followed by a multi-year rental agreement.

Google, Meta, and RadixArk plan to share more details about this collaboration at the SGLang Summit 2026. The event is scheduled for November 12 and 13 in San Francisco. As these companies continue to integrate their software and hardware ecosystems, the competitive landscape for AI infrastructure is likely to evolve rapidly over the coming months.