00:00OpenAI is out with performance results for its first custom inference chip called Jalapeno.
00:05The ChatGPT maker says in tests its chip was faster and more efficient than NVIDIA's GB300 system.
00:12Richard Ho, OpenAI's vice president of hardware, is with us here in San Francisco.
00:16Let's start with the basics of the testing and the performance benchmarks that you've published.
00:21What does it tell us about Jalapeno so far?
00:24First of all, thanks, Ed, for having me here.
00:26The benchmarks that we did were with an open source kind of benchmark called InferenceX.
00:31And we did that because it was a fair kind of neutral benchmarking method.
00:35What we're seeing with Jalapeno is that we're able to get both high throughput and low latency with the same
00:42device, which is kind of a first in the industry.
00:45And why does this matter is because high throughput means that you can serve a lot of customers with a
00:50lot of tokens much cheaper than you could if you otherwise.
00:54Low latency gives you very fast response times.
00:56And so, you know, the agent will come back faster.
00:58You can do, you know, more coding for her.
01:01And that really, I think, is a real benefit for our products.
01:03One of the questions that we got immediately was why didn't you benchmark against Vera Rubin, NVIDIA's latest system that's
01:10currently in full production?
01:11I guess the answer is quite simple, that it's just come out.
01:13It's just come out.
01:14We wanted to use numbers that have been published, that have been optimized.
01:18Those were available, the GB numbers.
01:20The other chip makers also have the numbers in there.
01:23It just happens that GB is the best, indeed, product in the current benchmarks that are there.
01:28This is it.
01:29This is Jalapeno.
01:31You very graciously allowed me to get my hands on it.
01:35And, again, the whole emphasis is on throughput and ultra-low latency.
01:40And, actually, in the ultra-low latency piece, you'd associate that with SRAM-based systems.
01:45This is both HBM and SRAM.
01:49Talk a little bit about its design and architecture, but why that ultra-low latency matters.
01:53Yeah.
01:54So, it is an HBM-based design.
01:56It's HBM4.
01:57It's one of the first, you know, products that's going to go to volume with HBM4, right behind NVIDIA's products.
02:04And it is very low latency.
02:07The reason how we got it was really taking a blank slate approach to this design.
02:11So, my team arrived at OpenAI, and we basically were able to work with the research team to understand where
02:18the bottlenecks were in large language models.
02:21And with a blank slate design, we figured out that you had to reduce your data movement a lot, and
02:25we figured out a way to do it.
02:26And so, this architecture is different from GPUs.
02:29It's different from TPUs and other chips in that nature.
02:32It really reduces the data movement.
02:34It optimizes the algorithm, and then it basically is able to get this very low latency, which matters for the
02:41agentic workloads, for example, right?
02:43We're getting to why a Frontier Lab would look to a custom silicon program, the why.
02:48Yeah.
02:48What is it that you can optimize for with Jalapeno or any custom ASIC or accelerator that you just can't
02:56with an NVIDIA-based system, for example?
02:58It's the full stack control that we're really looking for, all the way from the models, through the software, all
03:04the way down to the silicon.
03:05By having that control, we can make tradeoffs.
03:08We can make tradeoffs and optimize it really well.
03:11And that's something that's unique from being inside an AI Frontier Lab, that this tradeoff can actually be made all
03:16the way down into the silicon.
03:18And with that, right, so I think that the key thing here is that this is quite a novel architecture.
03:25I don't think other people have done it this way before, where there's affinity between the memory and the cores
03:31in a very integrated way.
03:33One of the worries has always been, is this programmable, right?
03:37And new architecture is sometimes it's hard to get the software up, but we're able to show, because of how
03:41fast we were able to bring these three very different models
03:45that were in the benchmark up and running on this new hardware, which we only got a couple of months
03:49ago,
03:50is indication of, one, how good the programming model is, and two, how good our models, our AI models,
03:58have gotten in optimizing the firmware, the kernels, in order to make this performance possible here.
04:04This is an inference chip.
04:07It is.
04:07So very simply, why focus on inference, not on the training side with your custom silicon program?
04:12Well, I think that if you look at it, right, OpenAI has this enormous need of compute.
04:17And where that growth of compute is, is really on the inference side.
04:20We need a lot of training, and we have a really strong partner in NVIDIA who has, you know, really
04:24got a great training chip.
04:25So we focused on where we thought we needed to increase the amount of compute that OpenAI has available.
04:30With the growth in our users, with our weak active users growing very, very fast,
04:35we saw the need to continue to have a wide range of inference products.
04:39And this is going to be part of our fleet, along with our existing partners who are providing us other
04:45chips.
04:46How do you decide which workload goes to Jalapeno versus NVIDIA or Cerebris, for example?
04:54You have options, but how do you make that decision?
04:55It's a great question, right?
04:57I think with this optionality that we're now creating for the company,
05:00I think that we will be able to provide users with a range of models that have different cost benefits
05:06and different latency, you know, capabilities.
05:12We will map that ourselves.
05:14We will map out which product, which SKU, which chip is best for which one of these models.
05:19But the whole goal of this is to deliver more intelligence to our users at a lower inference cost,
05:24at a lower infrastructure cost.
05:25And that's what this is part of, that entire strategy that, you know, we've been working on for a while
05:30now, right?
05:31You know, lots of people struggle with the benchmarks.
05:33You know, to the layperson, it's difficult to decipher.
05:36So in very simple terms, on a dollar per token basis,
05:40how does Jalapeno stack up against the other options that are out there?
05:45How has this changed the economics for you?
05:47It should drop.
05:49It's a really good chip.
05:51It should drop it by a lot.
05:53Like, you know, the performance per watt that we're seeing,
05:57which is a good proxy for how much a token will cost,
06:00is somewhere in the region between, you know, 1.8 to 4x better than other existing chips right now.
06:05And so we should see that.
06:07Ultimately, when this rolls out into production, customer token prices should come down.
06:11That's a big, you know, drive that we've had at OpenAI for a little while now.
06:14And this really is part of that strategy.
06:17And again, it goes back to attacking not only in the software,
06:20but attacking all the way down into the hardware and even into the data centers as well.
06:24You are the VP of hardware, not just a single chip.
06:28Talk about the challenge of scaling Jalapeno.
06:30You do this in partnership with Broadcom, right?
06:33You know, there are still bottlenecks in high bandwidth memory.
06:37Ramping output takes time on the fab side.
06:40You know, what's that like?
06:41Yeah, totally.
06:41I mean, yes, Broadcom was our big partner in the chip design.
06:45Seleska is our system partner who basically did the racks, you know, doing the racks with us.
06:50This is the big challenge today, right?
06:52Memory is highly constrained.
06:54Wafers are highly constrained.
06:56We rely very heavily on our partners.
06:58We also take control of that and have direct relationships with many of the vendors that we operate with.
07:03And I think it's one of those things where, you know,
07:05I think we have to work together to basically be able to ramp up the volume
07:08and get us into, you know, high volume production.
07:10Richard, two rapid-fire questions.
07:11First, percentage, how much content does OpenAI own of the server design?
07:17We did the entire design.
07:19You know, Broadcom has done the physical design,
07:21which is like after you take the logic design,
07:23you turn it into the masks that TSMC would manufacture, for example.
07:28That was, you know, so that's where the separation is.
07:31On the system side, we did the design, and Seleska co-designed it with us.
07:35I ask because these days, you know, NVIDIA is pushing 55% of the ownership of the content of the
07:39server design.
07:41In the future, how much proportionally of your inference workloads do you see being run on custom silicon versus third
07:47parties?
07:48I think, you know, if we can ramp it and it's a cost, you know, profile looks good.
07:53If it's good as I say, I think we'll see a lot more of it coming up.
07:56And I think the other thing I want to highlight is this is where it's a multi-generational roadmap.
08:00So we're already deep into development of our second generation of this device,
08:04and we're already starting concept work on our third.
08:06So we see this as a long-term bet to lower the overall infrastructure cost for OpenAI.
Comments