00:00Chip startup SambaNova has completed the first close of its Series F, raising a billion dollars at an $11 billion
00:06valuation.
00:07The company's among a growing group of startups aiming to challenge NVIDIA and supply AI infrastructure specifically for inference.
00:14SambaNova CEO Rodrigo Liang is with us with more.
00:18Let's start with the basics of what that capital allows you to do in scaling the platform.
00:23Well, look, you know, it's the inference market has broken everything open and that's where we focus.
00:30And so we're seeing this incredible demand in the market for inferencing and inferencing at really high token speeds.
00:37And so the capital allows us to really accelerate the supply chain, accelerate the build out of these racks and
00:42deliver the racks to a broad range of customers.
00:46Inference is split in two phases, pre-fill, decode.
00:49And so, you know, to the audience that might not be familiar with that, take the opportunity, Rodrigo, to explain
00:54why a specific platform for the decode phase is important, why it works better for inference.
01:01Yeah, I mean, the analogy I make is like, you know, if you're at the Disneyland ride and you have
01:05the rides that are kind of really busy and you want to organize the queues ahead of time.
01:09And so in pre-fill, what you're really able to do is organize this traffic to maximize what you need
01:14to do when the decode phase of inference shows up.
01:17And so what you're seeing is that if we use NVIDIA for the pre-fill portion and then replace that
01:22NVIDIA rack using some Adobe for decode, you're getting two to three X throughput advantage on the same infrastructure and
01:29the same cost.
01:30So driving your output to a much higher level without increasing your investment.
01:35So this is the bit that I find fascinating.
01:37SM40, SM50, your proprietary tech, the way that you put it out there, five to ten times faster on inference
01:45on the decode phase when compared to NVIDIA GPU.
01:49But the reality is it is working alongside other accelerators on the platform, right?
01:54Explain that.
01:56Yeah, that's right.
01:57So if you look at someone of SM40, it was originally designed for the enterprise for on-prem use cases.
02:03So you could do the pre-fill, you could do the decode altogether, and you can actually run these models
02:07at a really, really high performance, very low power, 10 kilowatts per rack versus 100 kilowatts on a typical GP
02:14rack.
02:15Now, if you actually go at scale and you look at these cloud players, these very large frontier labs, what
02:21they're doing is actually running lots and lots of traffic behind it.
02:24And so now what you want to do is actually take NVIDIA chips, who are very good for certain things,
02:30marry them together with some of the racks, which are made for very fast decode of inference, and then getting
02:36the maximum throughput and maximizing the output of whatever power you're able to secure for the data center.
02:44Let's give some size and scope, please, Rodrigo.
02:47How many customers are actually running production workloads on SM40 or SM50 today?
02:53And I guess you'd measure that as tokens processed per day, or what would the unit of measurement be?
02:59Well, today, look, you know, some of it, we're in the business of moving racks.
03:03Our customers are service providers, cloud players, model builders.
03:07Those are our customers.
03:08We don't sell tokens, we really sell racks.
03:10And so if I look at kind of what we did this summit over, we're already in the dozens of
03:15customers and we'll touch triple digit by the end of the year.
03:18And so you were in the model of deploying racks, which then those customers of ours would turn into token
03:24services that allow them to offer these premium tokens, very fast tokens on the largest models, out to their developers,
03:32out to their users.
03:33J.P. Morgan's the big one, right?
03:35It's a really interesting case study.
03:37Explain what it is that J.P. Morgan's able to achieve with your technology.
03:42Yeah, really excited about this announcement.
03:44J.P. Morgan has selected someone over to be the inference provider for the bank.
03:48And what you're seeing now is where the attention continues to be on the frontier labs, on the hyperscale clouds.
03:56One big segment of inference or AI is waking up, and that's enterprise.
04:01The enterprise, frankly, J.P. Morgan has always been the leader in actually using technology to drive the business.
04:09And so they're using Salmonova because they can bring this technology onto their premise, private data put into these racks,
04:17secure data completely within their own firewalls.
04:20And you can do all of this highly regulated business in production within the confines of a nicely protected environment.
04:29J.P. Morgan, I think you'd admit, right, there are many inference-specific platforms out there in the world at
04:35different stages in their life cycles.
04:36But, you know, I want to go back to this idea that the SM40 and SM50 work in conjunction with
04:42other accelerators, but in particular the GPU.
04:46The GPU architecture relies on high bandwidth memory.
04:49Cerebris, for example, has talked a lot about its use of SRAM.
04:53So it's not affected by the bottleneck that is HBM.
04:57How does that play out for Salmonova?
05:00Salmonova, you know, it's a data flow architecture which is very SRAM-based.
05:04And so what it does, it actually is able to actually take all the benefits of the SRAM, the very,
05:11very fast SRAM, and generate the speeds.
05:13One of the benefits of having HBM is now you're able to run the big models.
05:18Where the SRAM, traditional SRAM-based architectures that don't have HBM, they can't run the trillion parameter models.
05:27They have to quantize those models.
05:29And so we want to tackle the hard models, and so we use HBM.
05:32But here's what we did on Salmonova.
05:34We used HBM that was N-1 technology, HBM that was already in mature production, which allows us to actually
05:42generate significantly more supply versus competing with NVIDIA on, say, the latest and greatest, newest HBM.
05:49And so we're delivering high performance using mature HBM technology where the supply is significantly better.
05:55Let's end on by going back to the round.
05:58Very interesting who's now on the cap table or increased position.
06:01What happens next for you guys?
06:03Like capital intensive on the R&D side, on the scaling side?
06:07How are you going to manage your finances going forward?
06:10Look, I mean, this is something that, you know, we're heading to scale.
06:13And you see the investors coming in, you know, from General Atlantic to Capital Group, Tiro Price, Columbia Seligman.
06:19And, you know, just incredible investors that have great track record going to the public markets.
06:24And so we're actively driving the scale.
06:27We see incredible ramp up on our demand.
06:30And so we're just focused on delivery and driving the revenue to the point that allows us the option of
06:35actually going into the public markets.
06:37We'll see you next time.
Comments