Rendered at 12:14:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
TechTechTech 2 hours ago [-]
I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device.
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
mixermachine 2 hours ago [-]
Scaling a model on a chip is quite hard.
ChatJimmy is based on Llama 3.1 8 billion.
Kimi K3 has 2.8 trillion parameters.
That are 350x more parameters.
I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
formerly_proven 1 hours ago [-]
It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about half the area on ROM. There's a reason AI accelerators just use a ton of silicon area (each HBM3 die is >1000mm² of silicon). I imagine this is not terribly viable unless they make it a lot more space efficient e.g. using MLC ROM if they don't already, or use stacked dies with a ROM-optimized process. And then we're back to not cheap, though reticle chips were never in the cheap area to begin with.
Tuna-Fish 1 hours ago [-]
Taalas exploits the low cardinality to store one 4-bit weight with one transistor. (They are using metal layer traces for the ROM, and connecting an access transistor to light up one of 16 options.)
Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent advancements that made attention more efficient are probably going to be very useful for them.
Frannky 1 hours ago [-]
Yes, it also opens up a faster recurring revenue model for hardware companies, faster model obsolescence than how often you change a computer, a server, or a GPU card. I hope they can figure out trillion-parameter models rapidly.
Nvidia happened to be the best option for AI after building machines for graphics, so it makes sense they weren't the best idea from scratch for this specific use case. Especially given the scale of the demand and the possibility of recurring revenue, I hope a lot of smart people will try to solve it, compete with each other, and deliver us extremely fast and cheap intelligence.
And one can say LLMs are not as smart as a human, but a lot of the reasoning humans do for product and service generation isn't smart at all—it's just a bit of fuzzy input/output plus some reasoning rules. And then if you hook a robot up to the LLM, you can get results in atoms instead of bits.
I'm very excited about the future. I also hope it will stop money from flowing to bureaucrats who are incentived to keep the problems open to keep the money flowing, and instead facilitate sharing directly with the people (for example, no money to the state to solve homelessness—instead, spay instead with intelligence output to build a house and provide food as part of taxes.)
joshspankit 51 minutes ago [-]
I think you’re making another good point as well:
Specific traces for specific inferencing will mean that some generations get deprecated. Look at H265.
LarsDu88 15 hours ago [-]
I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition.
Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
anthonypasq 14 hours ago [-]
Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.
Melatonic 14 hours ago [-]
The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )
chorizo 12 hours ago [-]
Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
coderbants 4 hours ago [-]
I think the lifecycle for these chips could stretch far longer. If you're offering these models on a two year lifecycle, then you'd be able to stand up your top tier (wouldn't need to be frontier) at high speed. Run (for example) Kimi K3 on it and give it a brand name:
AcmeAI Carbon
Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely new hardware, your lineup becomes:
If you just kept pushing the same model down your pricing tier over time you could still extract a lot of value from an old model, even years after it's been set in stone. Working on brand new code/frameworks? Pay to use the newest model. Working on legacy code? Use the lower tier models that will already know your legacy frameworks, pay far less and still get massive throughput. I've worked on a lot of government projects that this would be absolutely brilliant for.
The other side of this is that agent harnesses are NOT set in stone, so even a legacy model with a knowledge cut-off that's years out of date can likely still be helped quite a bit by harness and fetch behaviours that are still developing rapidly. Especially at this kind of throughput.
hypfer 4 hours ago [-]
I'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon.
This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints.
___
Specification HC1
Model Llama 3.1 8B (hardwired)
Process TSMC 6nm
Die size 815mm²
___
So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.
A_D_E_P_T 3 hours ago [-]
Kimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
petra 5 hours ago [-]
AMD can let the SRAM be on a different chip. Maybe even something similar to their 3D cache. that could increase density to 20B[1]. They could also move from 6nm to 2nm. that would probably increase density by another 3x to 60B.
Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others.
Baking the base models on to ROM makes a lot of economic sense.
Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
ronsor 7 hours ago [-]
It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
ben_w 2 hours ago [-]
Right now the models are doubling in performance (by the METR time horizon metric at least) every 4 months, so 3 doublings in a year; conversely, I hear (not my field) it takes around a year to make a prototype IC and another year to turn that into mass production, i.e. if the next (late-2026 model) iPhone has a chip like this, it will likely be with, at best, a late-2024 set of weights. I think you can get open-weights models today that have performance equivalent to the SOTA-late-2024 while fitting in the RAM of a (high end) 2025-26 phone.
At some point the music will stop on training bigger models, and when that happens it will make sense to have ROM weights (or 100% analog circuits given how noise-resistant LLMs are), but we'll know when that is because the investment bubble funding the training of new models will have burst.
close04 26 minutes ago [-]
At some point it’s got to be good enough for the normal “phone stuff” that appeal to most users. So they wouldn’t suffer from FOMO because they didn’t wait for the next model. Every phone gimmick went through the same evolution curve until it passed the “good enough” point and eventually plateaued.
ben_w 20 minutes ago [-]
Yes, but irrelevant. While these models are improving at the present rate, the manufacturer can save money at no loss of feature-bullet-point-on-website by letting you download a model after you bought the thing and running it on normal hardware.
The rate of change to the models has to be slower than the hardware roll-out to be worth a hardware solution. If "good enough" happens before then, that just means the user gets a software solution.
miohtama 5 hours ago [-]
This stopped working some years ago.
Now you sell the same phone with higher price tag.
fsloth 6 hours ago [-]
I don’t think average user _needs_ to solve frontier challenges. ”Call to Jane”, ”turn on the lights” and ”what’s the weather this afternoon” is more like it I would guess.
Ofc if the model has some critical bugs that’s another matter.
dzhiurgis 5 hours ago [-]
Your examples worked on phones for over a decade.
Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.
gf000 4 hours ago [-]
It barely works even today, like Siri is laughably bad.
dzhiurgis 4 hours ago [-]
I mean the examples he gave definitely work. Mostly well I'd say as they are pretty primitive.
What Siri is missing is more logical solutions and answers for recipes, etc (still suck even with chatgpt integration).
darkwater 3 hours ago [-]
No, they don't work. Just asked Siri the other day "what's the weather tomorrow in $LOCATION" (where $LOCATION is a broader zone and not strictly a city) and the answer was the weather in a street called "$LOCATION Avenue" in a city 150km away.
gf000 3 hours ago [-]
Works, sometimes. But they can fail spectacularly and unexpectedly even on very basic questions/instructions, like so simple that a hand-coded word-matching style logic could get them right 20 years ago.
logicallee 5 hours ago [-]
>Your examples worked on phones for over a decade.
Nope. And not only not a decade ago, right now.
If you have an Android or iPhone, you can give it clear and easy to understand instructions that Gemma 4 could complete[1] if it had tool calls on it, and that 100.00% of Claude, ChatGPT, Grok, Kimi, you name it, could understand and all complete if they had the access.
The phones will fail to complete it. I just tried Siri. I said "hey Siri", waited for Siri to come up, and then I asked one of the exact sentences you replied to: "what's the weather this afternoon?" It thought for around 20 seconds, and said "Something went wrong. Please try again."[2]
I have Wifi, I have mobile Internet, I have free storage space, I have up to date software. What went wrong is that phones have never properly connected agents, not ten years ago, not last year, not this year, and probably not next year.
But don't settle for what Google could do in 1999 by hotlinking the keyword "weather" in any query to the weather being shown in the results.
Tell your phone (any phone): "Please call back the last number that called me that is not an unlisted number, regardless of who it came from."
0 out of any phone will complete that today, tomorrow, a year from now, five years from now, ever, because phone makers are not going to let them do that.
Meanwhile, 100% of all frontier agents could complete it if they had tool calls on the phone. Which they don't, and won't ever, thanks to the duopoly.
Okay, that's a bit dismissive, I would love to be wrong!
[1] after any voice recognition to text - which does work really well on both Android and iPhone!
[2] screenshot: https://ibb.co/21rtDnfV
narism 44 minutes ago [-]
I’m on the IOS 27 beta and Siri did those two tasks (weather/phone) flawlessly. It’s a lot better than it used to be.
logicallee 16 minutes ago [-]
Thanks for trying that! Very interesting.
Can you say this to it: "Hey Siri [wait for it to come up] - please send me an email with the temperature right now so I have it for my records." and see if it can complete the task without any backtalk or misunderstanding, and if you get exactly what you asked for. (It's a really clear request.) Should be 1 statement, no clarification, conversation, random search results, ("Here's what I found!"), etc.
A normal frontier model can do that - or Siri can do it if it is properly connected to Claude, ChatGPT, Gemini, Grok, or any other frontier AI - but previously it was never properly connected.
If it can do this task, I might have to look into this again. It counts as a success if it sends yourself any email with the current temperature and you actually get it (it can include whatever other text in the email), and a failure if it talks back, says "here's what I found", says it can't, asks you any question, sends you an email that doesn't actually contain the current temperature, just reads you the temperature and then asks if you want it to send an email, etc. Should be 1 shot.
let me know if it works!
chorizo 7 hours ago [-]
And the customers can wait for the new phone released next year. These are edge models - the average customer doesn’t need the latest frontier model. Just needs to be good enough for the features you promised.
otabdeveloper4 7 hours ago [-]
That's a software engineering problem. They just need to figure out how to fine-tune for alignment and tool usage.
That's the only thing the normie consumer cares for really.
tesnorindian 6 hours ago [-]
We also have ReRAM (Analog Computing), which also holds a promising future given its efficiency and low power. Though ReRAM of larger size is still a research area.
adrianN 9 hours ago [-]
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
lelanthran 6 hours ago [-]
Did you try using the the talaas chat? Something stupid like 18k tokens/second.
Think it's called Askjimmy or similar.
TeMPOraL 4 hours ago [-]
What is the model they're using there though? Interrogated, it claims it's a BERT variant and has capabilities around GPT-3 and below GPT-4.
(Not that I believe it, it writes too well for GPT-3.)
Hosted frontier models from two years ago would be much faster today, too.
egorfine 4 hours ago [-]
They run Llama 3.1 8B.
jgilias 6 hours ago [-]
Oh boy, thanks for sharing this, truly mind blowing. It was chatjimmy.ai
chorizo 9 hours ago [-]
The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
klodolph 9 hours ago [-]
I thought DRAM was pretty dense already. Is mask ROM that much denser?
chorizo 8 hours ago [-]
Yes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc.
Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
ReactiveJelly 8 hours ago [-]
Hm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope
selcuka 11 hours ago [-]
Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.
aenis 7 hours ago [-]
Its a cool demo, but its gpt-3.5 level stupid, or worse.
edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.
selcuka 6 hours ago [-]
It's a quantised 8B model (Llama 3.1 8B to be exact).
I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster
trebligdivad 11 hours ago [-]
What, even if it means you can run models without relying on the currently backlogged DRAM production?
adgjlsfhk1 10 hours ago [-]
The size of model we're talking about running doesn't need much if any dram.
trollbridge 5 hours ago [-]
The chatjimmy demo is using a model that needs 6-18GB of VRAM. That's not exactly trivial.
I could see it being feasible to get a Qwen-3.6-27b type of model done on something like this. Qwen-3.6-27b at 18tok/s would be a game changer.
teaearlgraycold 14 hours ago [-]
My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.
QuiDortDine 12 hours ago [-]
Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model.
Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).
But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).
But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!
And then the economy implodes.
I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!
manmal 7 hours ago [-]
Errors compound, and making 1000 wrong decisions per hour, will not result in something useful. Maybe you‘ve tried setting up guardrails for good design or architecture at some point? I think it’s simply not possible to do that.
It would certainly be an accelerator for people who know exactly what they want. And it would remove multi tasking, which I‘d appreciate.
Ericson2314 4 hours ago [-]
If your task has incremental rewards/feedback, you can push the "intelligence rate" simply by sampling the reward function faster. That's not fake, even if it not a substitute either.
This is the "dumber but honest person that works harder" phenomenon, vs "lazy genius".
gf000 4 hours ago [-]
That's a good way to put it, but still my experience is that worse code bases are non-linearly harder to maintain and improve in the future, software tends to break down without a good enough base.
Sure, in the future full rewrites and stuff like that will be just another "throw money at it" problem, but fundamentally software can get arbitrary complex and we barely know how to write large, maintainable code bases.
Nonetheless, I think testing (and maybe proofs) will have its long-awaited time to shine, as being the "reward function".
visarga 7 hours ago [-]
AI helps you but also your competition, and gets factored in by investors while customers can use it to find better deals. The whole market is different even if a company did nothing.
Whatever you can cheaply do with AI is not a moat, if there is profit in there there will be quick imitation and competition will eat away those profits.
Models can be replaced easily, harnesses & AI tools too. And if cloud inference gets too expensive there are local models keeping the cloud prices hard capped.
Probably AI won't make anyone very rich.
pastel8739 8 hours ago [-]
This is the same pitch that people make about AI today. Speed isn’t the differentiator, quality is
tiborsaas 2 hours ago [-]
Speed will be one of the killer features once you get closer to instant speeds of 300ms. Just remember what changes were made possible simply by upgrading from ADSL to broadband.
If inference speed goes up, I can launch the same query 5 times, evaluate the best result and proceed from there. Of course, evaluation is also instant, so in seconds I can get a near perfect solution. Or maybe 10 and I can pick what I like the best.
sroerick 5 hours ago [-]
They are both the differentiator.
AI previously provided speed but not quality. As soon as quality reached an acceptable threshold, the speed became the reigning factor.
In my opinion the quality is still much lower, but speed means the cost is significantly lower also.
TeMPOraL 4 hours ago [-]
AI is already fast enough that human is a bottleneck. Hell, typing speed became a bottleneck like it was never before.
I mean, if an agent can do half-decent work in less time than it takes the user to prompt them (and "user" in this context is a fast touch-typist like most programmers are), it's obvious it's not the agent that's the bottleneck anymore.
tripzilch 1 hours ago [-]
Because finishing someone else's (or something else's) "half decent work" to the point of "actually decent" becomes the bottleneck.
This has always been the case for human project management, and LLMs just aren't at that level yet.
It's more like everyone is speed running to how fast they can convince others that "half decent" is good enough. And for sure, newer models of LLM seem to be getting better at that.
Certhas 7 hours ago [-]
So for every work produced by AI have ten separate agents review it thoroughly.
xur17 11 hours ago [-]
I'm not sure inference speed is always the slowest thing for me right now. The agent is running tests, loading webpages, etc, which all take time. I don't know if a fast agent would speed things up in all cases.
That said, it obviously depends on the project.
jodrellblank 11 hours ago [-]
> "The agent is running tests, loading webpages, etc, which all take time"
A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.
evilduck 10 hours ago [-]
It's already kind of that way with MCP servers popping up everywhere. The JIRA MCP server is like a couple orders of magnitude faster to work with than the website itself.
manmal 7 hours ago [-]
That’s their API with extra steps, or am I missing something? That was always faster.
andersonpico 9 hours ago [-]
They finally cared about clear requirements and documentation when that meant getting rid of devs.
layla5alive 9 hours ago [-]
That happened at corpo work for each of:
* Build times
* CI latency
* Developer tooling
* Documentation
* Modularity
wsintra2022 11 hours ago [-]
I think this reads like Ray Kurzwheil (sorry not able to spell that off top of my head, that bloke who wrote that book about the future) .. But yeah very dystopian and totally realistic. Not if but when..
QuiDortDine 11 hours ago [-]
I LOVE Kurzwheil! Thank you for the compliment, I'm very far from having his writing skills. But yes sci-fi is looking more and more like, well, sci.
IOT_Apprentice 6 hours ago [-]
I tried it. I asked where Bruce Lee was born. It stated he was born in Hong Kong. I challenged it and it went further naming a hospital there. I stated he was born in San Francisco and it apologized and then said his father was a missionary traveling in America, which was also wrong. Bruce’s father was a famous Cantonese Opera singer and actor.
This model had zero information right, while being fast in responding.
Unacceptable.
TeMPOraL 4 hours ago [-]
That's not what speed is useful for.
I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}.
Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Generated in 0.024s - 14,163 tok/s", the page says). Slightly more powerful model while keeping this under a second, and this could easily become a qualitatively different form of autocomplete/text suggestion. Running in the background every couple keystrokes, or every time user stops typing for more than 500ms.
logicallee 2 hours ago [-]
>That's not what speed is useful for.
>I just pasted your comment and its whole inheritance chain to it,
Good idea. Only problem is it doesn't work. I just did the same thing with exactly this prompt:
>did the user IOT_Apprentice participate in the thread below and if, number and quote all of their comments. Only just number and quote the comments or write "Did not participate", do not add any commentary. Quote any comments by this user verbatim, exactly as input. Thread:
followed by pasting the thread[1]
And received the answer "IOT_Apprentice did not participate in the thread."[2] in 0.001s, even though they have literally the last comment in my quote and it's clearly legible.
It's particularly insidious because the understanding and thinking that is required to follow my requested answer format exactly is substantial - so based on the fact that it gets the format right and clearly understood the assignment, I would be inclined to believe that it would also be correct!
So to use your example, it's not just autocomplete, it's autocomplete that confidently returns "No matching results" in 0.001 seconds, even though there is a search term matching what you put in, right in the prompt itself that was sent to it. That is much worse than useless.
It gave the correct answers to both questions for me:
> Bruce Lee was born in San Francisco, California, USA on November 27, 1940.
> Bruce Lee's father was a Chinese opera singer
That being said, this is not a good test. It is a language model (a very small one), not an encyclopedia.
ChatJimmy interface is just a tech demo. Without tool calling functionality we can't expect it to be factually correct.
logicallee 4 hours ago [-]
if it's baked into silicon how can you two get different answers?
v9v 3 hours ago [-]
It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.
logicallee 1 hours ago [-]
Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.
tliltocatl 3 hours ago [-]
Using LLMs for information retrieval is the most stupid thing one can do. Especially when old methods work much better.
bdangubic 10 hours ago [-]
> I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless).
you need to launch 10-15 more terminals, who is waiting these days? :)
TeMPOraL 4 hours ago [-]
It's pretty clear if you see what's happening on current phones.
Autocorrect that works. Reply suggestions that almost work, just need to be tad more accurate (probably more of a data access issue than model) and a tad faster to look completely seamless. Screenshots with automated text detection and OCR and automatic interpretation (different suggested actions for when something on the picture looks like a web link, phone number, postal address, e-mail, or QR code, or an event poster). That's just a fraction of things I saw showing up on my Samsung phone over the last 6 months.
For over a year now, you could get a much better autocorrect and spell/grammar check, and a translator all in one, if you just pasted your text to a frontier model and asked it to check for errors or translate into target language. Now imagine being able to go through a round of such checks in a 1/100 of a second. You could have this running every keystroke, and suddenly the inline autocorrect/checks would not suck anymore.
Auto-linkifying that can correct for typos and doesn't need careful regex tuning because it understands from context what is meant to be a link or not. That's just one of many obvious things possible once you get local models running fast enough. Tip of an iceberg, and the first step to imagining all the other potential uses is to let go of the two mistaken beliefs people hold on to:
1. That LLMs are about written language. They're not; ever since "multimodal models" became a thing, tokenization extended to visual and audio space, and now textual and visual and aural inputs are all just regular, first-class tokens.
2. That chatting with the models is the only optimal way for end users to interact with AI. That's just artificially limiting yourself to the space of chat-based UI.
Godsend69 47 minutes ago [-]
[dead]
in_a_society 13 hours ago [-]
The best way I can explain it is that it's the same feeling when I upgraded from 56k dialup to cable broadband.
HDBaseT 12 hours ago [-]
In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results.
Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.
manmal 7 hours ago [-]
Problem is, there exists no judge model that will really pick the same winner that you would.
RussianCow 14 hours ago [-]
That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.
spijdar 13 hours ago [-]
I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like.
But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.
That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
manmal 6 hours ago [-]
> At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents"
That also needs server class hardware though. A phone won’t happily service the insane amount of IO, compute, and network that this cascade would require.
intrasight 12 hours ago [-]
Box that plugs into my desktop would be fine. Or perhaps in SSF form factor.
retatop 13 hours ago [-]
But wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?
dabbz 12 hours ago [-]
This is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.
manmal 7 hours ago [-]
I think that only works if you have checkpoints where all that reasoning can be checked against reality. Otherwise you get an army of armchair experts. LLMs are hilariously bad at home improvement advice btw, where reasoning alone won’t get you far.
nvme0n1p1 13 hours ago [-]
That order of magnitude could be the difference between "the users wants me to open the notes app, let's open it" and "I've scanned all your notes before you could blink and found what you're looking for".
p1esk 13 hours ago [-]
If Siri is using a 3T model in high reasoning mode to answer your question you will.
LarsDu88 8 hours ago [-]
Massive economic simulations with thousands if not millions of agents to front run the global economy and stock market.
Fully interactive realtime NPCs in videogames at scale.
Recommender systems that simulate individual consumers.
Crazy shit
manmal 7 hours ago [-]
About your first example, isn’t the butterfly effect preventing this from being useful? One agent in your simulation decides to sell, and starts an avalanche, that won’t happen in reality?
bostik 6 hours ago [-]
When you run tens of thousands of simulations for complex economic models, you actually do want to see the extreme outliers too. I can't recall who said it, but in finance the interconnected incentives make so-called Black Swan events much more likely and frequent than models or theories can comfortably account for.
In a way... when it's finance, they should be maybe called Gray'ish Swans?
dboreham 12 hours ago [-]
Works great from a press release perspective though.
bastawhiz 14 hours ago [-]
The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.
wmf 12 hours ago [-]
That math doesn't really work.
8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2
8B model (Taalas) = 4 GB ROM = ~800 mm2
freekh 6 hours ago [-]
It would be cool if the future was a standard fairphone like module system where you could replace the model chip when you felt like it without having to shell out 1-2k $$$s for a new phone
dzhiurgis 2 hours ago [-]
Its wild but if chip is something like $30 and provides frontier intelligence then just throwing them away every 3 months isn't that big of a deal when a lot of us pay $50 to $150 to $1.5k per month on AI tools.
I don't think it needs to be on phone per-se. It can keep chugging in cloud - plenty of people use cheaper older models.
And I suspect the growth will slow eventually making taalas interations slower.
bsaul 14 hours ago [-]
That's actually a really good point... There's currently zero incentive to buying more hardware, and that's one very good reason do have a new one.
sebular 13 hours ago [-]
But this is already happening with iPhones. Apple is touting on-device AI and only the latest phones offer the full capabilities. Newer phones will be able to run better models, so the incentive is there as soon as someone makes the killer app that only makes sense when the model is running locally on your phone.
amelius 3 hours ago [-]
> as soon as someone makes the killer app that only makes sense when the model is running locally on your phone.
I expect this to be around the time when we're finally ready to travel to Mars.
superb_dev 14 hours ago [-]
From what I remember, these chips are not mobile size yet
bradfa 14 hours ago [-]
A small model would be. I think that’s more the point. It’s definitely not SOTA but it’s fast and energy efficient and local.
mdp2021 14 hours ago [-]
> A small model would be [mobile size]
A ~30mm side for the HC1 tech for an 8b model (still unclear the planned HC2)?
teaearlgraycold 14 hours ago [-]
Is that analogue or are they baking floating points into the silicon?
AlotOfReading 12 hours ago [-]
It's entirely possible they're using something like block floating point, where most of the hardware is simply fixed point. AMD's NPU does this, for example.
wmf 14 hours ago [-]
Nope, a small model would be larger than the whole iPhone SoC.
makeitdouble 13 hours ago [-]
Slightly besides your point, but it's interesting how many here naturally ponder about how the current winner could or "should" keep winning, instead of how another company could become a competitor by doing the more clever thing the incumbent isn't thinking about.
krisoft 12 hours ago [-]
It is not a “should”. At least not in the “we wish it were so” sense.
It is more that there are multiple reasons why this idea (burning an LLM into silicone and deploying it into a device in people’s pockets) requires huge piles of cash and the kind of engineering chops only a few company posesses.
Of course i would like it if a small upstart would do this, but it doesn’t seem likely as a posibility. They won’t have the funds to fab the IC. They won’t have the funds to train and validate the model before burning it into silicone. They can’t absorb the risk of the first tape out going wrong. They can’t absorb the risk of the model being faulty in some subtle way. They don’t have a device to integrate the IC into. They won’t have the funds to develop one. If they somehow would make a device they don’t have the marketing and sales channels built out to get the device into people’s hands in sufficient numbers to justify the development cost.
Basically this idea feels ruinously expensive. Apple has deep pockets, they already have working well-regarded phones, and an ethos of privacy preserving innovation. This is why this idea feels well suited for them and not many others.
Do i want the winners to keep winning? No. But not many others can pay for a moonshot crossed with a manhattan project. They just can’t.
whatsThisBtn4 14 hours ago [-]
[flagged]
9 hours ago [-]
bjackman 4 hours ago [-]
Dwarkesh recently pointed out [0] that these guys are almost forced to spend most of their compute on training instead of inference. This is because they need to maintain the appearance (which may also be the truth) that future models will make current models obsolete and be much more valuable.
Completely fixed-function HW can't be used for training, it's inherently a statement that "this model is Good Enough and we are now gonna start just extracting its value instead of extending it". So yeah it's an inference moat but it's not a growth moat.
Makes perfect sense for a company trying to get into the compute business, not companies who wanna be in the creating-ASI business.
Still, I guess/hope they have teams doing it in-house anyway. Just not something they'd wanna make a huge amount of noise about, it doesn't look good for To The Moon valuations.
Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.
breuleux 14 hours ago [-]
If you’re only running models for frontier capabilities, yeah. For tasks where current models are smart enough, running them 100x faster is the most impactful improvement you can make. Consider all the things you could use a model for, but don’t, because the latency is just a bit too high.
zxspectrum1982 13 hours ago [-]
I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.
subroutine 12 hours ago [-]
But Claude Opus 4.6 is not really practical. Taalas' process seems targeted for edge models. Their proof of concept model, for example, is a heavily quantized version of Llama 3.1 8B and even then they acknowledge their custom 3-bit/6-bit representation causes model quality degradation.
Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the maximum size that conventional lithography can expose. They claim they could partition the model across many chips, but I'm not sure if they have tested this process or what it means for compute. The basic storage arithmetic is unforgiving: for a one trillion parameters model at four bits it will take 50–100 chips. To service a sizable customer base will take thousands of 100-chip fabs.
That all said, I'm bullish on this technology, and look forward to seeing it evolve.
Iolaum 7 hours ago [-]
A really fast qwen-3.6-27B type of model could be useful. With a specialized harness and this speed I 'd expect it to find many applications. Implementing a coding plan is the minimum I can think of.
vatsachak 12 hours ago [-]
Yeah. But this kinda feels like a bandaid.
Eventually someone will have to solve compute in memory at scale.
andix 12 hours ago [-]
With thousands of token per second output it would be an enormous waste of resources. Such chips are clearly made to process thousands of conversations simultaneously. Not necessarily in parallel. All LLM workflows are turn based right now, there are often seconds between turns until tool calls finish or users type the next message.
If the LLM response only takes a few milliseconds, the chip can process hundreds of other requests until the first conversation becomes active again.
tiborsaas 1 hours ago [-]
With those speeds I can benchmark a batch of different approaches, compare the results and serve the results all within a second. It's quite amazing.
akie 4 hours ago [-]
> it would be an enormous waste of resources
Sounds a lot like "640Kb ought to be enough for anybody"
Gigachad 13 hours ago [-]
It costs something like $300,000 for the hardware to run a model of that size. You'd pay that for a single model for 1-2 years? Not even the AI companies can justify that kind of spend which is why they keep extending the expected lifespan on their hardware in the accounting.
zxspectrum1982 13 hours ago [-]
I'm expecting the Taalas MSIC version to cost a fraction of that. Then probably have some kind of cheap subscription to Anthropic for updates (yes, Taalas chips can receive a certain kind of updates: they have a small SRAM).
mdp2021 12 hours ago [-]
> It costs something like $300,000 for the hardware to run a model of that size
You did not compute that as the cost for a speculative card from Taalas, right?
Gigachad 11 hours ago [-]
It's the cost of the current nvidia hardware used to run these models. Of course all bets are off if you are accounting for some future chip that doesn't exist yet which could cost less.
NiloCK 9 hours ago [-]
Not so long ago, I was good enough for many coding tasks. But I found that things can change in a hurry.
Yes, a cheap and fast Opus4.6 can drive a lot of value in current context. But if we continue to craft bigger-and-bigger balls of mud, Opus 4.6 may end up hitting its conceptual ceiling and unable to contribute.
Winding the clock back on your statement gives:
> I'd gladly pay for a Claude Sonnet 3.5 in silicon and use it for 1-2 years.
Man, I dunno.
kennywinker 8 hours ago [-]
Assuming moore's law like progress, which I'm 100% sure isn't going to happen - I think we're at the top of the S curve already. But assuming dramatically increased intelligence every year this is still the exact same position as anyone who bought a computer in the last 5 decades. Yet, people did very much buy computers.
nowittyusername 10 hours ago [-]
Depends on how much it costs the consumer. If I could buy a "cartridge" of Kimi K3 for 300 bucks I 100% would buy that shit asap. Even if it's "no good" after lets say 4 months still would be worth it IMO.
desmaraisp 10 hours ago [-]
That's definitely super-enthousiast territory. Paying 80 bucks a month for AI is more than 99.99% of people would be willing to do
Iolaum 5 hours ago [-]
Do think about b2b. Companies are already paying much more for AI. a new K3 (or similar model) every 6 months for a monthly rate of ~100$ per month is something MANY businesses would pay for. Then they could even sell them at half the price to consumers.
nowittyusername 10 hours ago [-]
This will be considered very cheap within the year IMO. The value you get from AI is exponentially increasing and like all tech just takes some time to ramp up. Cell phones, internet and many other amenities when they came out many people were not willing to pay for but that all changed and considering how important AI tech is this will also be the case especially considering if its 100% private such as for that cartridge.
brailsafe 7 hours ago [-]
> The value you get from AI is exponentially increasing.
Perhaps in some cases, but the value I personally and professionally got out of LLMs reached a limit a while ago and has since kind of fluctuated between that limit and a bit less.
If the best model was instant, like the demo here, it could certainly provide more value, I guess, but I think the limit I'd quickly hit is the same one as now, which is how much of it do I want to produce, for what reasons?
kennywinker 8 hours ago [-]
That's because the super-enthusiast will upgrade in 4 months when a better model is released. The casual user would keep it for years. A year of claude at the lowest plan is almost $300
topspin 14 hours ago [-]
"seems like baking models into silicon is speed-running obsolescence"
Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.
mdp2021 14 hours ago [-]
Well, 50TB ROM Taalas HC1 style would be apparently a 400000b transistor system through a chip sized 2.5 meters on the side... :)
preg_match 11 hours ago [-]
Yes but have we considered employing, like, a really big block of ice? Like old-timey surgeries? What if we put a big block of ice on the 2.5 cubic meter CPU what happens then?
thfuran 13 hours ago [-]
Phones were getting too thin anyways.
heywoods 12 hours ago [-]
Or autonomous weapon systems, missiles, and drones.
umeshunni 11 hours ago [-]
Why would they need multi TB frontier models?
adwn 2 hours ago [-]
For pondering trolley problems, maybe?
ray_v 14 hours ago [-]
I could see this making sense when model development start to settle down ... it's going to settle down, right? ...
amelius 14 hours ago [-]
Not sure. You can fix the transistors but leave the connections between them open for flexibility, so you only need to change the manufacturing process for the upper masks for every new model.
tliltocatl 3 hours ago [-]
I think they already do that, except it's not 1980 so you don't fix the upper mask, you fix the lowest metal layer (the upper layer is very coarse and is only useful for power). But even a single mask is still quite expensive.
amelius 1 hours ago [-]
But I suppose the interconnect masks don't have the resolution requirements of the masks for transistors. Therefore it could be a lot cheaper.
(Yes, you could fix a number of masks, e.g. entire logic gates, of course).
tsujamin 14 hours ago [-]
Surely that added flexibility negatively impacts the density/parameter count of the model you could etch?
sroussey 14 hours ago [-]
Or do a hybrid
mdp2021 14 hours ago [-]
Compute the cost of producing n of them devices, imagine a fair price based on that, and see if that local, blazing fast card* can be an asset that could be replaced periodically.
*(It's local: private files managing firm oriented. It's blazing fast: it can be placed into recursive, intensive local workflows.)
LarsDu88 7 hours ago [-]
It depends on how quickly you can bake new architectures.
Text diffusion might be a disruptor here, but let me just say the most cutting edhe form of image diffusion (JiT and DiT) right now is just a big fat stack of alternating attention and MLP matmulls.
Not theoretically hard to bake
alightsoul 14 hours ago [-]
Which is exactly what companies and shareholders want to increase sales.
flyinglizard 14 hours ago [-]
Look at it the other way: compared to the cost of training a model, the cost of making a custom ASIC is trivial.
try-working 13 hours ago [-]
obsolescence is the whole point. apple gets to sell a new phone very 6-12 months because of it.
i have written about this:
"For device makers
Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models.
A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."
They are: https://openai.com/index/cerebras-partnership/
My guess is they only consider Luna "good enough" to justify the immense up-front investment to put it onto silicon, but Luna at 10x the current speed would be killer. If they're really pursuing live voice conversations with a hardware assistant, latency is more important than accuracy (for complex questions the assistant could always say something like "wait a minute, I need to think about this" and hand over to another model).
Philip-J-Fry 3 hours ago [-]
Cerebras doesn't etch the model onto silicon though. They're basically just wafer scale GPUs. They're more flexible than etched silicon though because they can just run the next version of the model almost straight away.
ldng 6 hours ago [-]
How ? Do LLMs actually "know' when they don't "know" ?
planb 5 hours ago [-]
They do this all the time, I'm using ChatGPT in Instant mode and it auto updates to thinking if my question is complex. Most of the time this works.
To answer your question: A large language model itself does not know this (afaik). But chatbots are not "just LLMs" but a whole bunch of systems (and models) around them.
ldng 4 hours ago [-]
Ok, but the article is about etching the model, not a "whole bunch of systems". So far, I still don't if its actually doable or if it is just unsubstantiated speculation.
Certhas 6 hours ago [-]
How do humans?
dgellow 4 hours ago [-]
Always the same trick of not answering the question and deflecting to „what about humans“. Can you folks not evaluate LLMs as the system they are, without vague gestures at how a different system behaves?
Certhas 1 hours ago [-]
Evaluating LLMs is incredibly difficult. They are categorically different from any other system we have intuition about.
That said, I read the question I am replying to as a rhetorical one. If it was meant as a genuine question, curious about the question of meta knowledge, then I misread. Certainly the question is extremely interesting, for both LLMs and humans! But it's also obviously a very difficult one, as we don't even have a clear theory on how "knowing" works in the base case.
throwaway27448 13 hours ago [-]
You need to find customers for several-generations-ago models before this makes any sense. AMD is a lot more incentivized to look than mr vanilla llm is
Haven880 6 hours ago [-]
Chinese already start making DUV which can do the lower end 7nm. They are winning. Once that 7nm and up market cornered by Chinese, AMD Intel and TSMC and Samsung will have to burn thru bleeding edge depreciation faster perhaps from 7yr down to just 18mths. The CPU they generated will be incredibly expensive. Meanwhile Chinese just keep minting the AI cheaply and more efficiently and inching upwards towards 1.4nm.
petra 4 hours ago [-]
They can do 7nm. But they use multiple patterning(printing the same pattern multiple times to get to 7nm), which is expensive. So it's not comparable on cost to western single-patterning 7nm, and of course not to the leading edge on cost/size/power.
giancarlostoro 14 hours ago [-]
ASICs is what took over Bitcoin mining, cheaper in all ways, and lasts longer than Nvidia GPUs for inference.
SR2Z 13 hours ago [-]
> cheaper in all ways,
Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory bandwidth.
giancarlostoro 13 hours ago [-]
My understanding is an ASIC can last 10+ years, where are Nvidia enterprise GPUs are rated for 5...
SR2Z 12 hours ago [-]
Most enterprise GPUs are scrap after 5 years because they're so inefficient compared to newer models. It's entirely possible to make them last longer by undervolting them, people just don't because it doesn't make sense.
Bitcoin OTOH has used the same PoW algorithm for a decade. Barring some really exciting discoveries about the nature of computation, new ASICs are not that much more efficient than old ones.
BTC mining is also not exactly competitive anymore; the nature of the PoW algorithm means that it's dominated by a few large players who've set up shop next to a dam and who pay very little for electricity.
New entrants are highly discouraged because the mining rewards are constantly halving, it's hard to find cheap power, and the price of BTC is now so volatile that a yearslong investment is very likely to lose money.
unsigner 7 hours ago [-]
Their thing is improving the models; it would be extremely counter-company-culture to bet on models plateau-ing. Maybe wise in terms of hedging, but still difficult to pull of as a company decision.
larodi 5 hours ago [-]
We don’t really known (from the outside) how exactly do they move. Besides it may have not been truly viable 1-2 years ago…
mrtksn 14 hours ago [-]
Isn’t that kind of useless for the stock? It sounds complicated, unlike having number of CPUs go up.
It’s like talking about anything else than Megapixels when everyone was convinced that megapixels must go up in certain periods of the smartphone boom.
elAhmo 4 hours ago [-]
They were busy buying open source frameworks and teams behind those.
LPisGood 14 hours ago [-]
I’m surprised Nvidia hasn’t partnered to make a Claude chip yet. It’s a win/win you can license them out, sell them when they become obsolete, etc.
wraptile 9 hours ago [-]
This seems like a very bad and dangerous direction for our society.
Not necessarily: it is relevant to Taalas only if it is a compute-in-memory architecture.
The Jalapeño mentioned («Anthropic is not alone in walking this path») in the article is still a classical Von Neumann architecture.
And Taalas' idea makes sense in a perspective of scale - producing a large number of cards; "for internal use" (a lower order of items) means a high production cost.
UncleOxidant 11 hours ago [-]
I guess I'm not understanding why this makes sense for AMD to buy Taalas unless they plan to get into hosting. It doesn't seem like a great fit.
la6479 13 hours ago [-]
Just to see how fast it is try chatjimmy.ai
tasty_freeze 10 hours ago [-]
It is really fast and ... really hallucinates. I asked "Does the Wang corporation still exist? If not, what happened to it?" and it replied (in part):
"Yes, the Wang Corporation, the company that originally developed and marketed the Wang 2200 computer, still exists as a rebranded company under the name PPL (Precision Pencil and Label), but it has undergone significant changes and challenges over the years.
Here's a brief overview of what happened:
Founding and Growth: The Wang Corporation was founded by An Wang in 1969."
In fact, Wang labs was founded in 1951. PPL seems to be a made up entity. But it did generate those "facts" in 0.033 seconds. If people value speed over accuracy then I can write an LLM that is 100x faster than chatjimmy.ai and make big bucks by responding one of N canned responses to any question.
jjcm 6 hours ago [-]
to be fair, it's running an 8b model from like 2 years ago. Taalas just does the chip design, not the model architecture.
mickaelkerjean 10 hours ago [-]
their tech is a mere demo to open up a new path, the day we can have some asics running a Qwen3.6 27b, this would open up new doors
mr_mph 13 hours ago [-]
Pretty incredible to see. It reminds me of when I first used the Groq chatbot, except in this case it's a full response instantly.
stingraycharles 10 hours ago [-]
Didn’t Anthropic acquire Cerebras? Seems like a move into the same direction.
I also think that etching models into ASICs may be a bit too inflexible for what OpenAI and Anthropic want.
wyrdcurt 8 hours ago [-]
No, that's backwards. OpenAI are the ones investing in Cerebras. Part of the deal is that they can't sell to Anthropic.
alightsoul 14 hours ago [-]
Because Openai and anthropic are not hardware companies. They outsource that to Broadcom and AWS' Annapurna labs.
wmf 14 hours ago [-]
OpenAI and Anthropic are both designing ASICs.
alightsoul 13 hours ago [-]
So they have decided that putting a small LLM on a phone would backfire because people would have a negative perception of their cloud models. Pretty sure AMD will use these taalas chips in data centers, not phones
karmasimida 14 hours ago [-]
A model can't be updated, and a chip that is only relevant for 6 months at max?
anigbrowl 14 hours ago [-]
Depends what you mean by relevant. If you use AI primarily as a search/knowledge engine, it makes no sense. If it's your capable assistant that has a lot of general knowledge, can do tool calls, and has a big context window, very doable.
Indeed, for some kinds of applications involving secure/legal data etc. I can see the consistency of silicon winning out, because it combines performance with immutability and guardrails in hardware. Some chips have write-once PROMs to store password hashes and similar, you could do the same thing with prompt hashing to absolutely force or forbid certain behaviors. A model that can't be updated is also a model that can't be hacked.
askvictor 14 hours ago [-]
People already buy new phones every year, this just creates even more reason to do so
Gigachad 13 hours ago [-]
Outside of this website I've never met a person who buys a new phone every year. It's closer to every 3-4 years for most people.
boelboel 13 hours ago [-]
Closer to every 5-6 years these days and with ram prices going up it will be even longer. Especially with the low/mid range phones, which are most phones outside some developed countries, people will keep their phones as long as they can.
Gigachad 13 hours ago [-]
Would depend on the income levels, but yeah, buying a new phone these days is entirely a non essential luxury. An iphone easily lasts 7 years so the moment money is tight, it's a very easy choice to not buy a new one.
askl 1 hours ago [-]
I mean are there even any reasons to buy a new phone?
If I compare the Pixel 6 Pro I'm using at the moment to current models, they are functionally identical. The only reason to upgrade might be getting a fresh battery and access to firmware updates.
Otherwise I'd be happy to continue using it for the next 10 years.
throwaway240403 13 hours ago [-]
Your location/income bias is showing.
Most people do not buy new phones every year.
winrid 10 hours ago [-]
I live in the bay area and buy a phone maybe every 3 years? Why do people waste so much money :D
askvictor 6 hours ago [-]
I never said most people. But it's not uncommon. I personally find it ridiculous, and hold onto my phone until it's unusable, but plenty of people in middle class Australia seem convinced that they need the new one whenever it comes out.
simpsond 11 hours ago [-]
Base model sure, but the stack will be hybrid. It’s still early days here. Too bad FPGAs have such large feature size.
hamdingers 13 hours ago [-]
One of these chips smart enough to take orders at a drive-thru would be relevant for a decade, minimum.
wolttam 14 hours ago [-]
It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.
nine_k 14 hours ago [-]
Not so if it's embedded in something smart enough for its intended purpose.
Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.
anigbrowl 14 hours ago [-]
This is only true for people who are solely focused on performance. There is absolutely a market for acceptable performance combined with predictability.
teraflop 13 hours ago [-]
True, but predictability cuts both ways.
We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time.
Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food drive-thrus. And then somebody discovers the model has a fondness for goblins[1], and if you role-play convincingly enough, you can get it to accept payment in shiny buttons and rodent skulls instead of cash.
What do you do then? I guess your options are to try and fix the behavior with a better prompt, or put some kind of filter in front of the model to catch attempted exploits. If the filter is cheap and dumb it probably won't work well enough, and if you use another model as a filter, you've negated the cost and speed benefits of putting the first model in hardware.
Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.
I see your argument but your example seems highly contrived. I can't think why you'd want to use something like this for something as dynamic as takeout ordering, where you might have to deal with bad customers, supply chain breakages, public health recalls, or any of many other probabilistic events.
I think it's far more likely to see them used in safety critical applications where you need a capable model that can run on low power and doesn't have multiple layers of operating abstractions between the model and the hardware.
fwip 9 hours ago [-]
What safety critical applications would be a good fit for LLMs?
noisy_boy 10 hours ago [-]
> Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.
Does it though? Isn't that what CPUs are, very fast-not-so-clever computing brain surrounded by layers that protect it?
14 hours ago [-]
askl 1 hours ago [-]
Might be worse because it takes time between designing the silicon and having the first usable chips. So they're outdated the moment they hit the market or even before that.
speed_spread 14 hours ago [-]
If a model is good enough today, it's still gonna be good enough in a year. Except you'll be able to serve it 1/100 of the price. Or 100x the speed.
twobitshifter 13 hours ago [-]
OTOH, people get a new iPhone every year and they are ok with it.
nomel 13 hours ago [-]
How is that in any way related to a consumer device? This method doesn't reduce physical memory requirements, so still results in huge die area. This isn't a for-end-user thing, probably for decades.
twobitshifter 13 hours ago [-]
Ok, how long until nvidia gives us a new GPU?
nomel 13 hours ago [-]
I don't follow. How is that related? GPUs don't have fixed memory. You don't throw them away when you want to load a new model.
NVIDIA will probably give us a new GPU when someone competent in the free market decides they want wheelbarrows full of money. Unfortunately, AMD is entirely, incomprehensibly, incompetent, to the point where I can only assume they're colluding with Nvidia, behind the scenes.
bamboozled 14 hours ago [-]
It googles models suck
dave1010uk 4 hours ago [-]
I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict
For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity.
I'm not good at predicting, but some ideas:
1. All information gets augmented in real time with personalised context.
2. AI interaction seems more like find-as-you-type than a back and forth.
3. AI produces many outputs to pick from. Either the human, AI, or another system can do the deciding.
Even if it's last year's model, speeding up LLM inference could open up all sorts of opportunities.
bearjaws 9 minutes ago [-]
You could quite literally make an OS that generates your software ahead of you, like this VibeOS demo but lightning fast.
Imagine this demo but the apps render in real time generating real code.
Obviously not valuable because we have OS today, but could be your companies "WorkOS"
overfeed 3 hours ago [-]
4. Pervasive , distributed dragnet surveillance under the misrepresentation that it's not a search until a human pulls the data. But a small on-device "E2E preserving" "safety" model that runs on your phone and snitches when illegal communication content is suspected.
Edit: also consider centralized Room-641A-type surveillance when models summarize and/or flag all calls processed by public telephony
dave1010uk 1 hours ago [-]
I hope this doesn't come to BCI.
amelius 3 hours ago [-]
I think the most important direction will be: you spawn hundreds of agents at the same time, and let them work in a breadth-first search style. So you will not necessarily get your answers much faster, but they will be much more thoroughly researched. And if you do want faster responses, you can do that at the expense of quality.
petra 3 hours ago [-]
You could probably do real-time deep research using that method, and than summarize and ask questions about the results.
That is a probably a significant jump in search quality for many queries, that people didn't take the time to research properly.
tybit 4 hours ago [-]
Yes, this is a great point and it’s even one that the Cerebras CEO spruiks for his own chips. Talaas has gone furthest on the spectrum here, so seems to have the most potential to evolve the use cases.
trash_cat 3 hours ago [-]
People are not talking enough how huge this is for robotics and IoT. Current robotics arhitectures are limited by tok/sec. How cares if its not a Fable model?
This move undercuts NVIDIA directly.
Zenst 2 hours ago [-]
With AI models like Mixture of Experts, many of those experts will be the real target here, as polished, refined and little to no change, they become fine candidates for being locked into silicon. Who knows, add some SRAM in there and small changes to those experts could be carried out without needing new silicon.
Maybe AI models may become reduced to a collection of tiles you add to a chips one day, maybe sooner for some areas as you say, motor control for balance, vision systems, speach recognition systems etc, broken down, for robotoics, much is already there and just cost of battery/power holding much back.
Tade0 1 hours ago [-]
An "Expert" is really just an unfortunate name for what amounts to a dense part of a sparse matrix and that's also an oversimplification.
It doesn't actually specialise in anything in particular that one can point to.
For this reason you can really transfer them between models.
runtime_lens 2 hours ago [-]
[dead]
linzhangrun 11 hours ago [-]
Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.
dyzone 8 hours ago [-]
It tells me that they have some kind of insider knowledge that the models have hit their limits and won't be getting much better, and it makes sense economically speaking to just bake the current models and use them for the next 5-10 years. Looks like we're near the top of the S curve.
azernik 2 hours ago [-]
No knowledge (insider or otherwise) is needed — could just be a bet, hedged against all the other bets a company AMD's size makes.
mountainriver 8 hours ago [-]
What would possibly tell you that?
Maxion 6 hours ago [-]
Expensive as fuck to make chips, only makes sense if you believe whatever model you're creating a chip out of will not become completely irrelevant in 5-10 years.
niek_pas 5 hours ago [-]
There’s lots of use cases where the current models do fine though. A chip that can run a current cheap model at 100x would be amazing for things like detecting prohibited content on Facebook. You don’t need the 2030 equivalent of Fable for that, you need something that can cheaply process insane numbers of posts per day.
IshKebab 5 hours ago [-]
No, it makes sense if whatever model you're creating a chip out of will not be irrelevant in 5 years even if it is much much faster & cheaper.
There are probably limited applications but not zero.
7 hours ago [-]
fsloth 6 hours ago [-]
I’m not sure we are at the top. Swarmed lower tier models appear to scale up in competence. So something like a Fable swarm running at 100x the speed might provide surprises.
But ”top of the curve reached” feels like the likelier scenario.
hahahaa 7 hours ago [-]
A model you can run for practically no cost is a new proposition. It is the CPUification of AI. Sure there be supercomputers but you PC will be pretty super too.
threatripper 6 hours ago [-]
It's not just speed, it will consume a lot less energy per token, maybe even more than 100x difference. And cost for a chip that runs that one model will also go down a lot once volume scales up. They will end up way cheaper than flexible GPU chips.
I expect AI models chopped up into building blocks where 99.9% of the compute is fixed but glued together with flexible "fine tuning" layers that will adapt them to specific applications. Those kind of chips will run 99% of consumer AI and at some point be integrated into consumer devices.
akie 2 hours ago [-]
Ohhh that's a very interesting hypothesis. I think you might be right.
matheusmoreira 6 hours ago [-]
It makes me dizzy. I have no idea what is going to happen within even a year from now, can barely even imagine it.
I'm trying to get the most out of it by redlining my AI subscriptions. Hopefully I'll manage to start a business in my niche. I don't even know if my niche will exist in the future.
novaRom 5 hours ago [-]
Not just you. Hassabis in a recent interview looked a bit nervous when talking about AGI is rather sooner than many expected.
ilaksh 11 hours ago [-]
Cerebras already runs large models like Kimi 2.6 or GLM at like 30x speed. 100 times is next year, not six years.
You can actually test it out on their website, just imagine 3 x faster and maybe 15% smarter.
kllrnohj 8 hours ago [-]
Cerebras is literally the entire wafer, so it can't get bigger. So where is the jump from 30x to 100x coming from? Node improvements only yield like 10-20% gains these days...
ilaksh 8 hours ago [-]
They have a next generation, I don't really know if it will be 3 x or what but I heard it was significantly better.
Also there are other people innovating in hardware.
AussieWog93 7 hours ago [-]
Could we not just make bigger wafers, if the technology called for it?
kzrdude 7 hours ago [-]
The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?
rf15 6 hours ago [-]
There's so much you'd need to scale, it's even more expensive than the humble few billions you spend on a new fab. And, beyond R&D, the scale itself also makes it significantly more expensive. This is an energetic dead end, you would need a country to fully commit to making this one thing.
Maxion 6 hours ago [-]
Lithography is AFAIK the most expensive part. If you wanna scale to larger wafers I'd guess you're talking hundreds of billions and at least a decade. And even then, what will your yield be? Surface area grows to the square of the diameter of a circle, and so will errors. The larger your wafer goes the lower your yield will go, especially if you're going to use the whole wafer in one final product.
hhh 6 hours ago [-]
we seem to be in a phase of spending trillions on the computer, so while it isn’t likely, it isn’t impossible
techpression 6 hours ago [-]
You also need to worry about yields, Apple, AMD etc can sell ”bad” chips as lower core versions, if you’re depending on whole wafer you have little room for error.
keepupnow 11 hours ago [-]
This.
__MatrixMan__ 8 hours ago [-]
Last month: agents spend 4 days on a hack, humans spend 3 weeks (so far) digging through the slop to figure out what happened
Next time, one of those number will be smaller, and the other will likely be bigger. How long before the analysis side gets too overwhelming to bother with? Probably less than 6 years.
1saadcodes 9 hours ago [-]
Feels both unreal and dystopian. The speed at which these models are developing is very scary
DiscourseFan 11 hours ago [-]
It will be cool but also violent and terrible.
pizzaiolo 11 hours ago [-]
So, like the present
barbazoo 11 hours ago [-]
With more wealth concentrates at the top, yes.
__MatrixMan__ 9 hours ago [-]
Like some kind of pimple, which we can pop from all sides. And then we'll build something different. Something that works.
bigyabai 10 hours ago [-]
Presumably wealth would concentrate upwards even if AI was never made.
sanex 9 hours ago [-]
Yes it's a function of the monetary system. Absurd amounts of debt only certain people can access.
It failed on my usual test. But it failed really fast:
"A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"
tyre 8 hours ago [-]
This farmer needs a tote.
bogdan 4 hours ago [-]
You probably haven't met a determined goat yet.
AussieWog93 14 hours ago [-]
I read the paste, it got the etymology wrong, no? Schlong comes from shlang (snake), not shlemp (is this even a word? I don't speak Yiddish but couldn't find it on Google).
It did get it wrong but it also got a lot farther than much more recent, but worse models like 6.7GB on disk size ternary bonsai. It at least knows it's from Yiddish. The "schlemp" appears to be a total hallucination or it's confusing it with schlep, which is not related to schlong. One of the reasons why I said it "mostly" passes the test. Something much larger on the size of qwen 3.5 122B, deepseek v4 flash or similar that runs in 120GB to 190GB of RAM in my experience will answer perfectly unless it has been ruined by something like Q2 quantization.
thoughtpeddler 15 hours ago [-]
I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)
walrus01 14 hours ago [-]
There isn't SchlongBench(TM) yet, it's a specific question I've been asking of differently sized models as a randomly chosen gauge of how much less commonly used knowledge is perma-baked into it. In this case a question about a specific yiddish origin slang term. Small/bad models don't know it's from middle high german or Yiddish and get its origin and meaning totally wrong (or it runs into model censorship related to slang related to the male anatomy).
It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not
You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.
thoughtpeddler 14 hours ago [-]
Oh ya, this is like the approach from the Incompressible Knowledge Probes [0] paper - smart!
This is the only demo of 2026 which has blown my mind.
If we can get to this speed with reasoning models, man... I can't even imagine the impact.
tiborsaas 1 hours ago [-]
Looking at the history of technology, it's a question of when do we get there.
wxw 15 hours ago [-]
I freakin' love this demo. It feels magical.
VBprogrammer 15 hours ago [-]
I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?"
I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.
varun_ch 14 hours ago [-]
to be fair, the model used for Chat Jimmy is not very smart, but the world where it is smart is very interesting.
It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…
ricardobeat 12 hours ago [-]
I had the chance to try out MiMo v2.5 Pro Ultraspeed (600-1000tok/s) for a couple weeks and it is amazing.
Developing software becomes 95% about intent and requirements. Can’t wait for the next iteration of that.
Systemerror7A69 7 hours ago [-]
It's not thinking. Not in the way she probably meant. It can "think" that fast the same way a calculator can "think" that fast (kind of).
Because it's not human and not "thinking", it's a mathematical algorithm
VBprogrammer 5 hours ago [-]
I know.
axus 14 hours ago [-]
I asked it some old hardware command line questions I'd recently asked Gemini, it hallucinated parts of the answer.
The characters in the 3-act Shakespearean play had very little depth, many of the names were similar, and they were not very smart, but the simple plot was cohesive.
XCSme 13 hours ago [-]
Wait, is it even thinking? Or is it an instant model?
msdz 9 hours ago [-]
It’s not reasoning, the hardware demo uses a 3.-something generation Llama 8B.
But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.
After all, there’s already very useful models even for productivity at 27 or 35B.
XCSme 8 hours ago [-]
My concern is that reasoning could involve some sequential steps that instant models don't.
Not sure if modern models "think" only by outputting <thinking> blocks, or there is a more complex mechanism at play.
tveita 2 hours ago [-]
> Not sure if modern models "think" only by outputting <thinking> blocks
That's pretty much it - a small refinement to "Chain of Thought" prompting, where you tell the model explicitly in the prompt to "Think step by step" or similar, so it writes out more steps before giving a final answer, potentially catching some errors. The "thinking" models are tuned to do that without being prompted to, and to output the "thinking" markers around it, so they can be hidden from the user.
XCSme 1 hours ago [-]
My question was more about more complex problems, which no seem to be multi-turn somehow, or maybe just the harnesses make it look that way.
I am curious what the drop in thoughput is for multi-turn answers, instead of one-shot. More in line with the current "agentic" use-cases.
4k0hz 7 hours ago [-]
It's not really "instant", i.e. the text is still generated token-by-token, it's just super fast. Reasoning would work with this model without any changes to the chip but it's disabled for speed.
emdash 10 hours ago [-]
I feel like Ray Kroc in the McDonald's movie trying to figure out how his hamburger could possibly be done when he just ordered it
pizzathyme 13 hours ago [-]
For those old enough to remember, this is like dial up internet to broadband. So fast it creates new markets
calgoo 3 hours ago [-]
I was thinking the other day if we could use something like this "old" 8B model, and run 20 or 30 calls at the same time (or in sequence, we wont notice) and use and use the best result. Basically tiny agents that do tiny things but VERY fast.
appplication 12 hours ago [-]
This is the coolest LLM thing I’ve seen since the original ChatGPT announcement a few years ago. IMO much more impressive than marginal gains of frontier models.
senderista 15 hours ago [-]
Wow, feels like Google web search in 1999.
joshvm 14 hours ago [-]
If you still want the experience, go and browse McMaster Carr. Wizards designed that website.
That is insanely fast. I had it generate a basic C FFT library that can handle multi-dimension arrays, and it was instant.
brikym 13 hours ago [-]
The speed is awesome, in the true sense of the word. It's great at knowledge and basic stuff but the output is complete junk for anything concerning new facts or slightly esoteric topics.
itvision 15 hours ago [-]
OMFG this thing is fast.
phoh 14 hours ago [-]
its fast but try to get it to give you pi to 50 decimal places. it didnt go well for me.
walrus01 14 hours ago [-]
I think the same exact model running on CPU-only and RAM, or a small GPU, would do about the same? It's quite an old model now and small, you could throw a GGUF into llama-server or something for a side by side comparison.
As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.
estearum 12 hours ago [-]
That's not how LLMs work
XCSme 13 hours ago [-]
Wow, that's instant, crazy.
mrheosuper 10 hours ago [-]
looklike the training material is stopped at around July 2022, a little too outdated.
hendurhance 15 hours ago [-]
I understand the appeal due to the speed
14 hours ago [-]
hahahaa 7 hours ago [-]
Made me an entire app in 84ms lol
anigbrowl 13 hours ago [-]
15,000 tok/s
....damn. It's very impressive notwithstanding its limitations.
zhoge 11 hours ago [-]
This is the answer I got after asking it twice what's taalas (second time hinting that it's a chip startup):
After a quick search, I found that Ta'ala is actually a Canadian chip startup that produces artisanal, high-end potato chips. They offer a range of unique and creative flavor combinations, often featuring Canadian and international ingredients.
Ta'ala is known for its high-quality, small-batch potato chips made with premium ingredients and care. The company is committed to creating unique and delicious flavor profiles that showcase the best of Canadian ingredients and cuisine.
Is this the Ta'ala you were thinking of?
nsxwolf 15 hours ago [-]
It doesn’t believe it’s running on that chip, it’s arguing with me
shaewest 15 hours ago [-]
It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.
metadat 15 hours ago [-]
What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?
penagwin 14 hours ago [-]
Reasoning models are the same speed. They’re just post trained with RL to do CoT inside tags like <thinking></thinking> before a tag like <response></response>
There’s no difference in the inference implementation, parameter count, or speed.
paytonjjones 11 hours ago [-]
There's a difference in the latency distribution between when you submit a query and you see the response, which is what the comment is (clumsily) asking about.
But yeah, there are a lot of factors, so it's hard to answer, and tokens/s isn't the right question.
dumberquestions 15 hours ago [-]
Which model? Or how many active parameters?
_whiteCaps_ 15 hours ago [-]
Llama 3.1 8B model
dumberquestions 14 hours ago [-]
So this demo is around 90 times faster than typical speeds for the same model at openrouter, and around 30 times faster than the absolute fastest option available (Groq).
im assuming energy expenditure is substantially lower as well
15 hours ago [-]
wmf 14 hours ago [-]
AIs don't intrinsically know anything about themselves so they often give wrong answers to such questions. This can be fixed by putting info in the system prompt but they may consider it a waste of tokens since most usage doesn't benefit from that information.
anigbrowl 13 hours ago [-]
That proves it's conscious!
(/s!)
mintflow 11 hours ago [-]
try let it to get a brief of france history which being reading a while
hit the button and then the brieft jump into my eye
Generated in 0.051s • 14,092 tok/s
Impressive...
Given gpt 5.5 was very good to me and gpt 5.6 series seems not boost too much, i kinda like the way bake the model weight to the chip, and connect multiple chip to serve the large scale model and allow respin some parts(ROM like?) to do model weight update, maybe this seems sustainable, the future is exciting
lelanthran 6 hours ago [-]
> try let it to get a brief of france history which being reading a while hit the button and then the brieft jump into my eye
WTF is this?
mNovak 12 hours ago [-]
What I like about this, is that it significantly increases the probability of a sci-fi scenario where you're picking up a hot chip on the black market; rumor has it, Mythos 9 weights baked in...
arlort 5 hours ago [-]
steadily heading towards the final goal of computers being composed of pluggable crystals as stargate intended
matheusmoreira 6 hours ago [-]
Black market uncensored heretic Mythos weights...
bigyabai 10 hours ago [-]
Plug it in, and it's a old prototype with Gemma 5 weights baked onboard. Dammit, fucked by Craigslist again!
NitpickLawyer 9 hours ago [-]
Back in the kazaa and limewire days, you'd sometimes try to get a movie / episode from a series, wait hours / days for it to download, and when it was done you had a ~50/50 chance to actually watch what you wanted or an old german porn movie :/
rullelito 5 hours ago [-]
Better then vice versa!
Cort3z 3 hours ago [-]
I believe this is big news. Speculations here on out: I imagine this being baked into consumer products, greatly increasing the local token capability for consumers. They will suck the cloud-oriented companies' milkshake. Most users do not need extremely capable models, they just need some automation to do better web-searches, and get simple facts etc. If it can do simple coding tasks too, but at thousands of tokens per second, in stead of tens or hundreds, the development will benefit so much. It will benefit AMD in other ways too. I imagine they can start selling physical chips, usb-drive like devices, that just does llm. If you want a newer, better, model, you simply go to a store and buy one. Need more capability, buy more drives. Similar to physx back in the day, but with usb-c and a smaller footprint.
msteffen 14 hours ago [-]
This is neat but IMO a little crazy.
Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.
I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
Tade0 39 minutes ago [-]
Up to a certain scale a model only needs to effectively do tool calls - that is something which doesn't require too frequent updates.
What I imagine an on-device model should be doing is just translate natural language to search requests and calls to tools manipulating retrieved data - much like no model currently does calculations and instead they open up calculator and use that instead.
jgilias 3 hours ago [-]
What I’m doing is running a kind of a meta harness that uses different models (and underlying harnesses) to work on a problem, or review the solution. Idea is to get to an error rate better than each of the underlying models can provide. Same thing as sensor fusion.
Now, that’s slow and expensive although seems to work quite well (haven’t really evaled this properly, don’t have the time). If inference can be made fast and cheap, multi-model approaches like this would become more viable for more applications.
dumberquestions 13 hours ago [-]
I think you're underestimating both their reliability for standard problems and the usefulness of that level of reliability.
tyre 8 hours ago [-]
This is a good point. Opus does some silly shenanigans sometimes but then catches it later. It’s still an order of magnitude faster at getting to a working system than I am, for ones I don’t know.
It’s really a dream for setting up a homelab
daishi55 13 hours ago [-]
> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I wanted some examples they actually experienced. Because I use these things daily and haven’t seen a hallucination in a long long time.
yoyohello13 11 hours ago [-]
I saw a hallucination just this afternoon about a spurious ca cert error. Definitely happens less often, but I do need to correct it occasionally. Maybe once a week so it still requires vigilance.
tyre 8 hours ago [-]
Search a terminal with Claude Code for things like, “I got it wrong twice. I should look up the documentation instead of guessing.”
Does it about once a day, that I notice.
msteffen 6 hours ago [-]
Well, I wrote my comment after an exchange similar to this, with the linked example chatbot (chatjimmy): https://pastebin.com/uEV7Pm8q
I asked it a slightly tricky math problem (I re-asked it the same problem to create the paste, and it did about as well the second time). It was unable to solve the problem, and it’s a small, old model, so…fair enough…but also its answer was pretty incoherent, with stuff like “Since A is an invariant set, it's always possible to find a cave that the fox cannot be in. Therefore, you can always catch the fox in that cave.” (…catch it…in the cave it can’t be in?).
Then, off the top of my head: Claude somewhat recently generated a Spark Job where the worker timeout was longer than the worker heartbeat, so workers would always inevitably be killed when they didn’t heartbeat within their timeout window. (also…neither option needed to be set?) Before I noticed the problem, I asked Claude why the job was taking so long, and it told me the data set was too large. More recently, there was a blog post by John Scalzi I was having a hard time finding, so I posed the problem to ChatGPT, and it came back with a blog post that didn’t include any version of the text I remembered and wasn’t really topically relevant (and maybe I hallucinated the blog post, but it could’ve said “I can’t find it” instead of “here you go”). On another occasion, I was trying to find a particular episode of Bob the Builder for my kids, so I Googled it, and Gemini kept giving me the wrong season and episode number, even after several rounds of “no, s5e6 is ‘that thing’, I’m looking for ‘this thing’.” Turned out the episode wasn’t on Amazon at all (which I had to tell it), and I had to go find it on YouTube.
That said, as I sit here scrolling through my history to see if I’ve forgotten any particularly good examples, I have to admit they do a better job than I’m giving them credit for. But I still wouldn’t have them write my email for me (the one time I tried that, when I was playing with openclaw, it sent a fairly demanding email to someone I didn’t know that well without asking for confirmation, and I had to go apologize and explain that I hadn’t really written the email, which was embarrassing), nor am I particularly excited to have chatjimmy as a permanent resident of my pocket.
yumraj 14 hours ago [-]
Given the fast churn of the models, how does it work out?
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
craftkiller 12 hours ago [-]
I think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running a flash model could benefit from this since they're already deciding that speed+price is worth using a less capable model.
Maxion 6 hours ago [-]
Or as subagents reading code or other text and summarizing them.
sigmoid10 14 hours ago [-]
I find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty soon see major software companies pop up that are run by a single human.
deadbabe 13 hours ago [-]
Can you give some examples of these tasks that require an hour or more of reasoning?
xyzsparetimexyz 13 hours ago [-]
The recent maths prompts did. The 'you should find a breakthrough' one was several blocks of reasoning, each taking 90 minutes or so
XCSme 13 hours ago [-]
I think this would make sense for consumer hardware, not for AI companies.
AI companies constantly update/change stuff, new models come out, new requirements, etc.
But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.
yumraj 12 hours ago [-]
Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them
throwaway173738 12 hours ago [-]
You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.
XCSme 12 hours ago [-]
It was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)
tyre 8 hours ago [-]
Think of a HomePod. 99% (and likely much more) of what people are asking is super simple.
m463 12 hours ago [-]
subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.
XCSme 12 hours ago [-]
So local personalized ads?
Not sure if that's better or worse than online personalizaed ads...
thunderbird120 4 hours ago [-]
It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip, and bigger chips cost money. You can't have a dishwasher using a chip the size of a B200 die to run a model you could run on a Raspberry Pi.
What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster.
Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors.
815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money.
If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip.
There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports.
Reasoning models are barely usable with contexts that short.
The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM.
Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.
XCSme 1 hours ago [-]
Yeah, makes sense, if it's good for very small models only, then there's no point, as those van already run on cheap consumer hardware.
Yet, maybe it can work well enough, so that as a manufacturer, you don't pay $50 for a PI, but only $0.50 for a tiny "hard-coded" chip.
The advantage can be that, as a LLM, as opposed to other types of chips, the use-cases could be more varied, so same chip could be use in different devices (robo vacuums, security cameras, ball-shooting training robots, etc.)
petra 1 hours ago [-]
Maybe AMD will do a 3D memory layer for this chip for the kv cache. Possibly even something more dense than SRAM, like cgram, when it's ready.
Could this enable a reasonable context size ?
hypfer 3 hours ago [-]
And yield. Good luck getting good yield out of such ginormous chips.
That said, I've been wondering if they could go with multiple smaller ones instead. Like one per layer maybe even?
What are your thoughts on that? You seem to be more qualified than me on that matter.
thunderbird120 2 hours ago [-]
Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic.
I already harped on Cerebras, but their approach of just copy/pasting a whole bunch of identical functional blocks, over-provisioning the chip by ~8%, and then just fusing off blocks with defects allows them to effectively have 100% yield on a wafer-sized monolithic chip. This is very desirable, and just another reason I like their approach better.
heywoods 2 hours ago [-]
[dead]
christina97 13 hours ago [-]
There’s some kind of tradeoff between speed, cost, and quality for every application. I would be perfectly happy with a model 6 months old that was 50x faster for many uses. Right now I use either Opus (for smart stuff) or Flash without thinking (for fast stuff). I would take an even dumber model for more speed (lower latency in particular).
nullbio 11 hours ago [-]
Perfect for consumers. You buy it and then you need to buy a new one in a couple of years. If they can make them affordable they'll sell like hotcakes.
etoxin 10 hours ago [-]
And the second hand market. I'd love to see this integrated into motherboards like RAM. Someone could have a motherboard with 4 sticks of different AI with various models. Swap, change and trade.
chorizo 11 hours ago [-]
That’s not going to be true forever. As models mature, we will hit diminishing returns. Major improvements will come annually rather monthly - matching the roughly annual release of new processors. Model ROM’s will likely get integrated into die packages just like DRAM now.
pennomi 10 hours ago [-]
I’m hoping for SNES style cartridges
brokencode 13 hours ago [-]
Already models have gotten really good at a lot of things.
A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper.
And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.
prinny_ 13 hours ago [-]
They expect a sort of breakpoint at which each subsequent model version will only be marginally better than the previous ones, thus allowing them to retain their value for some time. Their business doesn’t work if each year the new model demolishes the previous one in terms of performance.
mrheosuper 10 hours ago [-]
I'm still using Opus for most daily task because Fable is too expensive.
If they begin etching Fable into silicon now and release it 2-3 years later, i can see the market for it
noosphr 10 hours ago [-]
This is a feature for most local use cases. You don't want all your work flows to start failing because of a model update.
hahahaa 7 hours ago [-]
They still make 6502s right.
laweijfmvo 13 hours ago [-]
pretty much everything is “1 or more versions behind” by the time it comes out. the question is whether or not it’s still useful? at some point, presumably not every application will need the latest cutting edge huge model.
casey2 10 hours ago [-]
There isn't a fast churn in the underlying pretrained model, nor RL. It's mostly orchestration around the model. Said another way you could just pretrain and RL for longer.
Also I believe there is both a market for extremely fast local inference with current model performance and that such fast inference would unlock unforeseen usecases. Especially as TPS approaches early computer clock cycles and data rates.
deadbabe 13 hours ago [-]
You could take your silicon chip and have it re-etched only with model diffs for an upgraded version.
yumraj 12 hours ago [-]
How does that work, as in re-etching of silicon? Any pointers to read?
yassa9 8 hours ago [-]
Can anyone imagine if a video generation model with the speed of ASICs baked into silicon ? real Sci-fi
mikeayles 15 hours ago [-]
AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
Melatonic 14 hours ago [-]
Yeah Im surprised nobody is talking about this. When everyone first saw Taalas I looked at the design and it had a big legup in physical cache availale compared to most chips. Makes you wonder how much of a benefit there is to the actual "baking" of the model vs just having a large chip with a ton of SRAM (or whatever) soldered close to the edge physically.
I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)
Legend2440 13 hours ago [-]
What you're describing is what Cerberas does.
Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.
Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
Melatonic 13 hours ago [-]
Cerebras chips are massive and do have more on the edge but they dont have any top or bottom cache do they?
jaggederest 13 hours ago [-]
They can't due to power density, I believe - they have to be run in a sandwiched waterblock with massive cooling, as far as I can tell. That's the biggest thing that baked weights gets you - a relatively modest watts-per-square-mm compare to cerebras, where they had to engineer a whole system to get the watts out of the chip
pstuart 8 hours ago [-]
Do you think there's room for reducing power requirements? Obviously shrinking the process is a win, but is the existing implementation a "just make it work" phase that has opportunities to increase computational efficiency?
With that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.
jaggederest 7 hours ago [-]
I mean I think I read their spec sheet that it was 2.5kw on the asic but closer to 25kw with GPUs, for the same number of tokens (different latencies of course), but there's no way that a dedicated from-scratch matmul+memory chip can be less efficient than a GPU.
wmf 13 hours ago [-]
Taalas does not have cache so...
I agree that Groq with multilayer hybrid bonding could be a good idea.
tandr 15 hours ago [-]
Well, technically it is their hardware now...
questionableans 15 hours ago [-]
And their team, if they treat them well.
zxspectrum1982 13 hours ago [-]
1. How come you didn't make your implementation public? You could be a millionaire now.
2. Especially if AMD has the technology to do what Taalas does, it makes a ton of sense for AMD to acquire Taalas: remove them from the market. Make sure nobody else (Intel, Huawei, Alibaba, NVIDIA, etc) acquires them. It could have been a great acquisition for a rebirth of BlackBerry btw.
mikeayles 4 hours ago [-]
It will be, I'm just not happy with my blog post before making it live. The blog will have a live interactive chat and a link to the repo with the HDL.
I don't think anything I did was particularly novel, as I really just wanted to see how fast I could push a commodity FPGA to it's limit.
Scaling to an ASIC or getting into the billions of params is where the real engineering is! This was just a side project for a side project for me while the FPGA was idle
zxspectrum1982 3 hours ago [-]
What FPGA are you using? Why?
What exactly did you implement? A full LLM? A subset of it, which collaborates with something running on CPU or GPU? Which LLM? Why?
What language did you use to implement your thing: VHDL, Verilog, Vitis, something else? Why?
I can think of at least 10 blog posts that I'd write before I write a single line of code. Publish early, publish soon ;-)
mikeayles 57 minutes ago [-]
I'm using the AMD (Xilinx) K26. It's a Zynq Ultrascale+, the successor to the old classic 7000's. I'm running it on the KV260 dev board, because I'm using it for another side project.
The K26 has a quad core A53 core alongside the programmable logic (PL, or Fabric). The A53 is pretty weak, and doesn't have any hardware matmul operations, so despite the KV260 being sold as a 'vision ai starter kit' and the vitis object detection running on the arm cores, they're pretty weak cores for anything AI.
For my use, I need true determinism, so my vision pipeline is all implemented in the PL, and it was pretty disapointing that the vitis libraries are basically just opencv on linux, rather than really pushing the fabric. If I wanted probabalistic AI running on a CPU, then I sure as heck wouldn't choose a quad core A53.
Which led me to have a play with this, I saw the taalas/chatjimmy demo and wondered what I could push the fabric to.
The round trip time to DDR or CPU via AXI meant I had to keep the entire inference engine in fabric. The A53 is simply a pipe that gets a request from my server (which has a cloudflare tunnel to the real world for the live demo in the blog post) and manages a queue. So it feeds a string in, and gets a hopefully longer string back a few uS later.
It's all in verilog, because that's what i'm more used to. I did get Claude Code to do a moderate amount, as it's a side project on a side project after all, but pushing an FPGA to it's limit is definitely not as comfortable for it as it is writing a crud app in TS.
I'm using tinystories, as we are talking about megabytes of URAM/BRAM. If I used the DDR, it definitely would have been a real model, but that wasn't my goal. My goal was to hit 100,000tok/s, and even when I conceded on absolutely everything, with a token prediction size of 1tok, I topped out at 60,000tok/s. ?But increasing the window to make an actually plausible chat (story generator, it doesn't understand questions, you need to prompt it with 'once upon a time...' and it finishes it for example) I managed to break 20k tok/s.
I also created the lemmatised version, which was inspired by Kevin from the office (why use many word when few do trick) and trained a new model, I was expecting the output model to be smaller, but was suprised that it came out the same size, but it ran 30%ish faster. In hindsight it makes sense, the parameter count is fixed by the architecture, not the corpus, so training on compressed text doesn't shrink the model at all. What it does is compress the output distribution. The same story takes ~30% fewer characters to tell, so the effective speed goes up even though the per-token rate is identical. The dumbness is the optimisation.
Fully agree on publish early. The blog post with the live demo (a websocket straight to the board through a cloudflare tunnel, so you're genuinely talking to the fabric) is written and sitting in drafts while I fiddle with it. This thread is my peer pressure, it goes live in the next day or two.
A_D_E_P_T 15 hours ago [-]
This is probably a win-win. The team gets paid, and we get greater assurance that their best ideas and architectures -- which are truly impressive -- are going to see the light of day in actual products.
badatnames 15 hours ago [-]
They were too small for this to be a meaningfully sized purchase for AMD, there's real risk they get sucked into a team that ultimately delivers sqat, not to mention the chances of anything being delivered in an even remotely consumer-priced bracket are definitely out the window
hliyan 10 hours ago [-]
Question: we currently emulate neural networks by performing matrix math in synchronous clock CPU architectures. Would it not be better to abandon synchronization and etch neuron synapses directly in silicon, keeping only the weights variable? I think some researchers are pursuing this, but I forget what the approach is called.
Perz1val 4 hours ago [-]
That'd be either analog computing or you're implementing floating point number math in each neuron. Maybe some kind of PWM calculations would be doable, do we have it researched or would be a new kind of math?
Ey7NFZ3P0nzAe 4 hours ago [-]
I don't understand what you'd etch instead of the weight? What's a neuron synapse to you that's not a weight?
freakynit 9 hours ago [-]
"Neuromorphic chips" .... and I have the exact same question in mind.
woadwarrior01 2 hours ago [-]
I suspect this acquisition was a response to NVIDIA's acquisition of Groq.
Cerebras is already public. AFAICT, there are 8 other startups in the space, some of which have mature products: Etched, d-Matrix, SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and Fractile.
redox99 14 hours ago [-]
Is there any LLM from exactly one year ago that would be worth running?
In Aug 2025 you had
- OpenAI o3
- Opus 4.1
- Gemini 2.5 Pro
- Grok 4
Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.
Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.
mdp2021 13 hours ago [-]
> Is there any LLM from exactly one year ago that would be worth running?
Bad perspective: consider the correction: "when are thresholds of sought quality reached"? Hence: not "is there a 10yo from last year that could compete with the current 13yo", but "will there be a 30(?)yo from last year that could compete with the current 33(?)yo" ('(?)': the scale of yearly growth in the future is uncertain).
redox99 13 hours ago [-]
It's not just about it "being smart enough". It's about there being actual user demand when it needs to compete with the shiny new model.
A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?
singingtoday 10 hours ago [-]
We still run GPT 4.1 for some of our use cases. We want to replace it but are having trouble finding models that are as fast with similar or better intelligence.
redox99 7 hours ago [-]
There's nothing fast about GPT 4.1. It's ~50 tps AFAIK. Of course it doesn't use reasoning, but you can run modern models without thinking as well. GPT 5.6 Sol without reasoning should destroy it in intelligence.
itgoon 7 hours ago [-]
Gemini 2.5 Pro was very good at writing single, somewhat complex functions. Sure, the rest of the loop would still take time, but nearly-instant implementation? Sign me up.
daishi55 13 hours ago [-]
That is fkin wild. o3 was just a year ago? The progress is truly insane.
redox99 13 hours ago [-]
Yeah I had to double check, o3 feels like it was ages ago. But GPT 5 came out Aug 7, so it's only one day off from my 1 year ago cutoff!
badatnames 16 hours ago [-]
Well so much for that dream.
Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
dogomatic 11 hours ago [-]
What’s stopping competition?
reqo 3 hours ago [-]
One advantage that I see in models that are implemented as code is that they can quickly and cheaply be modified using LoRAs. What would the equivalent be in hardware? Another piece of hardware you would attach like adding a graphics card to a computer?
kristianp 10 hours ago [-]
I've been eagerly awaiting their 2nd gen HC2, which uses multiple chips to host a "mid sized reasoning" [1] model. Its due in summer according to the article, I wonder if it will ever be released in that form now.
> I wonder if it will ever be released in that form now.
Yeah, I had the same thought. The key thing for them was the price point at which they could deliver a ~30B model. I would buy one today if it was ~1000$ and could run whatever the best 30B model is today, at those speeds advertised. Even if the model becomes superseded by model.5 in a few months, there's still a lot of things you can do with a "good enough" model for some tasks. And things like maj@x or generate 10 times and choose "at a glance" what you like (think frontend stuff) would be worth it.
No idea if them selling to AMD is good or bad.
luciana1u 2 hours ago [-]
everyone's worried about the model being obsolete before the chip ships. nobody's worried about the chip being obsolete before the model ships.
proxysna 16 hours ago [-]
Really hoped to see their hw out in the wild one day
preommr 13 hours ago [-]
People are missing the point if they think this is useless because frontier models keep changing every few months.
We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software.
I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.
ggm 14 hours ago [-]
Field reprogrammable, it's an FPGA on steroids. Field upgradable.
Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.
XCSme 13 hours ago [-]
Why not have some a device/hardware that programs itself on-boot.
Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.
wmf 12 hours ago [-]
FPGAs already configure themselves on boot.
XCSme 8 hours ago [-]
I asked a LLM after posting my comment, to see if I had a genius idea or not,just for it to tell me the same as you, that's now they work already...
mdp2021 14 hours ago [-]
Can that be done when the whole idea is to store a multiplier into a handful of transistors?
ggm 13 hours ago [-]
I have no idea. It makes my comment a statement posted as a proxy for a question, a question you correctly pose explicitly.
If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor.
If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.
xyzsparetimexyz 13 hours ago [-]
It can just be pcie
perarneng 4 hours ago [-]
On thing with this enormous speed is that you could create multiagent validation and correction and still have it super fast and achieve greater quality thanks to speed alone.
bhouston 15 hours ago [-]
Toronto Canada startup btw.
cmrdporcupine 15 hours ago [-]
Seems to be somehow some kind of offshoot from or connected to Tenstorrent, which is just down the road. Founder looks like he was/is maybe at Tenstorrent and previously associated with Keller?
Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.
kridsdale1 15 hours ago [-]
Works well, I remember driving by the ATI building as a kid.
MarkWayneNewton 16 hours ago [-]
While this design is self-limiting I think its a good approach. It doesn't take an entirely new architecture or infinite memory to produce significant performance improvement.
Legend2440 13 hours ago [-]
This is a new architecture. It's a non-vonn neumann device.
rbanffy 4 hours ago [-]
My understanding is that the chip carries the neural network itself while the weights are loaded on top, so any model that uses that same network topology should be loadable on demand.
bjackman 4 hours ago [-]
No I think you are thinking of Etched/Sohu.
Taalas' approach (at least for their demo'd product) is to bake the whole thing in completely. IIUC the optimiser can even see the weights while generating RTL. It's like there's an "uint8_t weights[] = " in the source code.
fischeti 4 hours ago [-]
They are loaded on top in the sense that they are contained in the upper layer(s) of the chip. So when you want to change the weights, you have to produce fewer masks for fabrication, which reduces cost and time to market.
andix 12 hours ago [-]
It would be quite ironic if this technology would render all those AI data centers practically useless. If the next step are just a much smaller amount of expensive chips, and the bottleneck becomes manufacturing those chips fast. Not building huge data centers and fighting for electrical power.
est 10 hours ago [-]
Waiting for intelligence on a stick, plugin an USB, characters in, characters out.
100% local and no leaks.
nojs 15 hours ago [-]
Can anyone comment on the economics and likely turnaround times of this process, when it’s more mature?
Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?
Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?
2001zhaozhao 14 hours ago [-]
There are always uses for outdated models.
Claude Code is still using haiku 4.5 from ages ago for explore subagents for instance. Not to mention production uses like customer service that only need to be "good enough"
edot 14 hours ago [-]
Just looked this up, no longer true. Explore subagents inherit whatever model the parent is. And you can of course make other subagent configs.
samtheprogram 14 hours ago [-]
That's solely so that you burn more money. It's totally unnecessary to assume the parent model. Sure, it could be upgraded from Haiku if there was a solid reason to, but...
AussieWog93 14 hours ago [-]
I mean, if you could get Opus or even Sonnet 4.5 at 1000+ tok/s exploring the codebase, they would probably change that setting back.
But either way, I think GP's overall sentiment of "delegating intelligence-saturated tasks to an outdated but fast subagent" makes a lot of sense.
alightsoul 14 hours ago [-]
Customer service has really degraded huh. 4 years ago they expected opus performance out of human call center agents
I guess losing some customers due to poor customer service is ok if the price of customer service is right.
cogman10 14 hours ago [-]
2 to 3 months optimistically assuming everything goes smoothly and is fully automated.
6 months or even a year if something goes wrong in the fabrication process and you need to update things.
If they do more standard asic design, it could be a lot longer as the design needs to be validated on an FPGA cluster, which would necessarily need to be very big for something like a LLM. Easily up to 2 years.
There's a reason chatjimmy isn't demonstrating newer models and why they only show of an 8B model.
shangofox 14 hours ago [-]
I mean even if it take a few months, it'll still be out of date. But there was a hypothetical when it came up in Feb, would you want Qwen 3.5 at like 10k tokens per second.
At the time people were no doubt saying yes but now 3.8 is out, is that still desirable?
xienze 14 hours ago [-]
There's soooo much stuff that such a model is still capable of doing in the pursuit of getting a better overall answer. Imagine a powerful research agent that blasts out dozens of the small, cheap models to fetch and summarize one page each. Then the beefy researcher model performs the final analysis.
rldjbpin 4 hours ago [-]
given their specs [1] show ~100 mm^2 of silicon for each billion params (albeit in 6 nm process), what will come first: nuclear fusion or one of these that can handle the multi-trillon param models of today?
> At ~2 nm: 1T plausibly fits comfortably on one 300 mm wafer.
But then again, 300B to 500B models are to this day also very valuable
akie 4 hours ago [-]
You're assuming no improvement on their current technology
perarneng 4 hours ago [-]
"Generated in 0.007s • 14,370 tok/s" - for some use cases this will be awesome
spacedoutman 4 hours ago [-]
I feel like this will be the end of Taalas, AMD has for the most part of its history always chosen the wrong options.
syntaxing 15 hours ago [-]
Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.
encyclopedism 15 hours ago [-]
Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways.
Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.
This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.
2001zhaozhao 14 hours ago [-]
i'm looking forward to Qwen3.8 27B launch to see how much models have peaked at a given size.
it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.
Grosvenor 15 hours ago [-]
> Imagine a cartridge with a physical model on it.
I can finally have my own Dixie flatline. Cool.
mdp2021 14 hours ago [-]
> Dixie Flatline
In case some did not know: also the movie (actually TV series) is finally happening.
very interesting idea. i didnt think of that. i was just assuming youd have an additional one of these in your phone for actual lightning fast local inference
pstuart 7 hours ago [-]
The cartridge could be a small mac-mini type unit connected and powered over thunderbolt. If it included like an m5 or m7 with 64GB of memory and a PCIe5/6 4TB Nvme it would be amazeballs. Hopefully when the bubble corrects and hardware advances and prices reset something like that will become available.
Just even comparing compute from 10 years ago (Apple silicon vs Intel) and it's significant. 20 years it gets crazy. My first computer was an 8 bit 6502 with 64K RAM and a 128K floppy drive (I think, it's fuzzy). Everything amazing now will look quaint in due time.
yassa9 7 hours ago [-]
It is not linear anymore, take in consideration the Moore's law, the curve is nearly saturated now and gains in performance and memroy are not accelerating any more,
BUT there is some hope with new different technologies, like the PHOTONIC chips , doing GEMMs through light particles instead of electrons
VladVladikoff 15 hours ago [-]
Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?
dumberquestions 15 hours ago [-]
I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.
snek_case 15 hours ago [-]
The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.
foltik 14 hours ago [-]
At least in the case of Taalas the weights are physically encoded directly on the chip.
It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.
I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.
wmf 14 hours ago [-]
somewhere = an AWS data center with multiple layers of security and NDAs
They won't sell/rent/license the weights to an end user at any price because they don't trust your security.
syntaxing 9 hours ago [-]
I work in embedded space. Just because it’s in hardware doesn’t mean you can’t “protect” it. Most modern software (regardless if it’s hardware or not) can be cryptophically signed.
bluezly 8 hours ago [-]
[dead]
amazingamazing 15 hours ago [-]
One idea would be to use an open model.
kevin_thibedeau 15 hours ago [-]
Then we can have machine psychologists pull cards when they run amok.
all2 13 hours ago [-]
You have a robot. You need it to be smarter. You buy a new model cartridge (probably a PCIE 9.x). Now you need some domain specific skills. You'd like it to be able to cook, and you'd like it to not dent your walls anymore. You buy 'improved spatial reasoning LORA' card and 'Gordon Ramsey's Chef ULTRA9000' card.
Now your robot can respond sarcastically when you ask for chicken nuggets. Again. It also doesn't dent your walls anymore.
breadislove 14 hours ago [-]
we have not converged at all, if you look at how different the chinese models in terms of architecture you can guess that the labs are experimenting a lot as well. we are seeing all different types of hybrid architectures, different attention methods and so on. Of course on a high level its still a transformer but if you take a proper look we are seeing more divergence then a convergence.
walrus01 15 hours ago [-]
Having a base model ASIC as a physical piece of hardware makes me think of the early days of microcomputer desktop stuff where having a socketed ROM or PROM was a key piece of hardware, and people actually knew/cared what ROM was on their system's motherboard.
Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.
smokel 15 hours ago [-]
The technical aspects of SOTA models are not publicly documented. How do you know if something is converging?
syntaxing 15 hours ago [-]
SOTA American models are not. SOTA Chinese models are. From a physics aspect, closed source models cannot be too far from open source ones in terms of size. There’s only so much you can squeeze out a B100 style cluster even with fancy Dflash style diffusion model for the speculative model.
_aavaa_ 15 hours ago [-]
If we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.
nurumaik 15 hours ago [-]
Will be capable and fast enough for 2-3 weeks until new sota drops
amazingamazing 15 hours ago [-]
If it is capable today why would a new model change this?
thombles 14 hours ago [-]
I think it’s tongue in cheek. When I first got access to Sonnet 4.5 I remember thinking to myself “y’know if they never got any better and I just had access to this forever then that would be pretty okay”. Turns out my expectations have changed since then and I would like a higher baseline now.
singingtoday 10 hours ago [-]
Interesting. I've yet to find a model I consider sufficiently intelligent.
Fable is nice, but still requires a lot of guidance for large scope tasks.
catchnear4321 15 hours ago [-]
if capability is a commodity then the differentiator becomes taste.
FridgeSeal 15 hours ago [-]
Because new stuff instantly makes anything prior bad and incapable and garbage of course! Did you forget the hype-machine speaking notes??? /s
cyanydeez 15 hours ago [-]
if they were still exponentially increasing, they wouldn't be preparing for an IPO. IPO is where companies go to die and founders escape.
cyanydeez 15 hours ago [-]
I don't think there'll be a fine tune card; you'll have the base model vintage whatever year, and then your GPU will do whatever LoRA layers you want it to do; the LoRA will wrangle older dated models into the current of whatever your looking at.
But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA
sgc 12 hours ago [-]
What does it take to go from here to a model on a pcie card or an m.2 card, so I can plug one into my workstation / laptop? Will 'intelligence' become much like a gpu, where most people just live with the performance of whatever they have installed, outside large companies that must have cutting edge, or prosumers that have a incrementally better version than the masses?
Are we a couple years away, a decade away, or something else?
mdp2021 12 hours ago [-]
> What does it take to go from here to a model on a pcie card or an m.2 card
It is already that.
> Will "intelligence" become much like a gpu
As an option among the implementations.
> Are we a couple years away
They could mass produce now, but it makes no sense at this rate of improvements in the models.
num42 10 hours ago [-]
I have used chatjimmy before, it is incredibly fast, waiting for latest SOTA model on the chips in future. Great!
analog31 8 hours ago [-]
Wow, we're heading back to mask-programmed ROMs. I'm feeling young again.
zkmon 9 hours ago [-]
I guess the idea is, gains from inference speed could offset the cost of upgrading the chips to a new model when really required. I think general purpose models would consolidate and release frequency might flatten out, favoring this strategy.
yunnpp 10 hours ago [-]
I would've hoped the company stayed independent instead of being engulfed into a behemoth. I'd like to see more diversity in the hardware ecosystem, but I guess the economics of hardware manufacturing aren't there.
matheusmoreira 9 hours ago [-]
> Once the chips are deployed you’re stuck with that model.
At least we can be sure that's the model we wanted. Service providers could be serving modified versions and nobody would ever know.
ford 10 hours ago [-]
I've been showing people chatjimmy for months - it's incredible. Both reasoning and tool use generation scale with TPS. Imagine 100x more reasoning on a model, or 100x parallel tool uses.
rvz 16 hours ago [-]
Didn't even give them a chance to launch the hardware.
tecoholic 14 hours ago [-]
With web search and tool call a decent current generation model at the speed of the chatjimmy could do a lot. People saying it would be out of date are missing the point. It’s not going to make much sense for frontier companies that’s chasing the SOTA. But for a lot of business use cases if someone can put GLM 5.2 and sell it as a box, it would make so much sense.
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
5555watch 13 hours ago [-]
In my understanding the first Deep Think / Pro models were already very good as they were doing some kind of parallel repeated reasoning, thus were slow and expensive. So if chatjimmy speeds enables a fast deep think level performance, I think that would be great.
equinumerous 14 hours ago [-]
100% agree - you don't need the most up-to-date model to have something that's useful in agentic contexts. They could even produce chips with weights that make all the decision making/logical reasoning and have it delegate to other specialized agents. If it becomes cheap enough to print a run of custom chips, releasing a batch for each major advancement does not seem unreasonable for SOTA companies.
cephei 14 hours ago [-]
There are so many use cases for supremely fast offline models. The first thing that comes to my mind is for real-time video processing or other non-textual content in real time.
They moved from HBM to dedicated silicon and only got a 48x speed up? That is so, so, so much less than I would've expected. Any numbers on how it scales?
redmoonx 14 hours ago [-]
It obviously won’t be continuous delivery but could make sense if the lifecycle of a model (train, deploy, iterate (meaningfully) is about 1-2 years. In that case it fits nicely in the “this year’s model” already established with cars, phones, etc.
ratsbane 8 hours ago [-]
Smart move by AMD. Chatjimmy is very fast and not very good, but I think it might become very fast AND very good.
galaxyLogic 11 hours ago [-]
I think the big news is that AMD is getting into memory-business so they won't be so dependent on Hynix and what have you. Memory is the bottleneck currently.
nolist_policy 5 hours ago [-]
AMD doesn't even have their own fans.
nullc 2 hours ago [-]
Might be an interesting motivation for looped LLMs to cut the gate count down. Perhaps even a collection of mixed programmable layers and baked layers in a loop.
hugmynutus 6 hours ago [-]
HN is rightly pointing out putting a model into an ASIC is kind of dumb.
HN is failing to understand that AMD knows this well.
Taalas has WO2025217724A1 pending and AMD wants that because it is immediately a function block they can sell to anyone doing FP math, since large (mostly) read only memory banks are ideally suited for that micro-code type stuff.
laweijfmvo 13 hours ago [-]
I’ve been using Gemma as my default (via Kagi) because it’s served on Cerebas hardware. The speed is honestly a game changer for day to day queries.
jackdoe 13 hours ago [-]
Can you imagine in few years getting Fable level intelligence at 20k tokens per second?
"You are not prepared"
--Illidan Stormrage
drob518 13 hours ago [-]
So, Kimi K3 in silicon sometime soon?
roughly 13 hours ago [-]
How's that jive with the fact that they're introducing a new model every other week?
drchickensalad 13 hours ago [-]
The new model every week is not necessary at this point really. What if you could run opus 5 for the next couple years at 1/20 the cost?
roughly 12 hours ago [-]
What's interesting about this is that I as a user would find this useful, but I think the AI industry as a whole would find it an absolute goddamn disaster. Opus 5 is a very good tool, but it is not a human-replacement-level intelligence, which means the entire revenue stream the industry's built on - labor replacement - is not met by this, and the only slightly charitable read of the industry's finances is that they're gonna bootstrap their way to creating the labor replacement hypothesis by getting people to spend money on Opus/etc, whereas if the actual product is a 1/20th the cost Opus-on-a-chip, the entire business and financing model that's tying up $N Trillion dollars of investment money goes out the window.
Great for us, looks like a recession as far as the Market is concerned.
jaggederest 13 hours ago [-]
Pipeline the burn into silicon, lower the latency as much as you can, for the 10-100x operation cost it's worth it. Imagine if frontier models cost $5/mtok and the 2nd or 3rd tier models cost $5/billion tokens for 3-month-old models.
yousif_123123 12 hours ago [-]
If things like this get traction, will we need all the datacenters?
downrightmike 11 hours ago [-]
You are mistaken about what the datacenters are for
tripledry 6 hours ago [-]
What are they for?
yigalirani 6 hours ago [-]
what prevents amd to just do what they do without acquiring them?
roschdal 5 hours ago [-]
Is this the singularity?
christkv 5 hours ago [-]
There is a big risk in etching a model into silicon like this. We are still evolving what small models look like and improving their performance. When do you decide to etch one into silicon knowing that right now an improved one can be 3 months away.
bob1029 15 hours ago [-]
I feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.
mdp2021 15 hours ago [-]
The basis of Taalas is "compute in memory" electronics - past Von Neumann's separation of processor and memory.
You need to be able to add|mul where the data (the weights) are stored.
addaon 15 hours ago [-]
NAND hasn't been scaling great lately. It seems like PCM or MRAM would both be better fits.
kridsdale1 15 hours ago [-]
FPGA model storage?
OddMerlin 11 hours ago [-]
Congrats to the Taalas gang.
concraper 11 hours ago [-]
A massive L for Canada
jauntywundrkind 13 hours ago [-]
Core rope memory is back baby!
Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.)
https://youtu.be/3MKRjt59hh4
so qwen3.x-27b on hardware? or better deepseek-v4-flash on hardware .
ilaksh 15 hours ago [-]
I wrote them an email asking for PrismML Bonsai 27b Ternary which is like 6b or something crazy small and would be a lot easier for them to do initially.
mdp2021 14 hours ago [-]
They were specializing their forthcoming system on 4-bit FP - which I understand is a structural decision.
Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.
andrewvl 14 hours ago [-]
It must be a “super model”. What will be if new model released? New chips?
downrightmike 11 hours ago [-]
Chip pops out like a gameboy cartridge. AI not working? Blow on it and jam it back in
galaxyLogic 11 hours ago [-]
"... the chip serve Meta’s Llama 3.1 8B at a blistering 16,960 tokens a second — when announced last February, that was 48x faster than Nvidia's GPUs and 8.5x faster than Cerebras' accelerators. "
tonyhart7 5 hours ago [-]
so in the future I can buy KIMI, GLM or whatever model that get "soldered" directly into GPU ????
so instead of RTX xx70 series, I can buy xxTA that have kimi integrated ??? is that right ??
api 13 hours ago [-]
I've had an endgame idea in mind for a while.
Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges.
You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.
wmf 13 hours ago [-]
Each cartridge costs $1,000. Do you still want it?
anigbrowl 11 hours ago [-]
For fast Kimi K3? You're damn right I do
wmf 10 hours ago [-]
$1,000 only gets you the Qwen 27B cartridge. For Kimi K3 it would be more like $100,000 (and the "cartridge" is the size of a refrigerator).
trollbridge 5 hours ago [-]
I would gladly pay $100,000 for local K3 running at 18,000 tok/sec.
singingtoday 10 hours ago [-]
Yeah. I have 3 max20 plans.
api 13 hours ago [-]
Me? Probably not. A business or a hoster, sure. There'd probably end up being an aftermarket in used cartridges with slightly older but still good models on them.
ur-whale 13 hours ago [-]
Yeah, so https://chatjimmy.ai/ ... the model is crap, but the speed is amazing.
Worth checking out.
empiricus 2 hours ago [-]
Worth wondering why they used a crap model.
jijji 10 hours ago [-]
taalas is great for llama 3.x 8B models, really bad for one board serving Kimi K3, it seems like you would bottleneck at a few hundred tokens no matter what you do.... spreading the big model against multiple cards seems the only way to get into the 1k+ tok/sec range. Another thing taalas is doing is masking the model weights into the silicon itself, not a flashable firmware, which would increase latency....
walrus01 15 hours ago [-]
Imagine the size of chip needed to 'etch' something like Qwen 3.6 27B in size.
golem14 14 hours ago [-]
Interesting thought, because it's a yield question. How tolerant are models today to a few broken weights.
If tolerant, they could churn out many cheaper chips, some perhaps with slight abnormal tendencies ;)
thepasch 14 hours ago [-]
> How tolerant are models today to a few broken weights.
Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses.
I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence.
I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.
walrus01 14 hours ago [-]
Brings to mind the scene in '2001' where Bowman is pulling out individual pieces of hardware that represent the mind of HAL, and it becomes increasingly incoherent as more physical hardware is detached.
I wonder if you had a few percent of problems in the yield, if it would be functionally equivalent to the difference between a unsloth-published Q6 standard size GGUF vs. the nearly perfect precision of an unsloth Q8-K-XL. Or more like Q4 vs Q8 where a lot is lost.
mdp2021 14 hours ago [-]
Not too dissimilar to the first HC1 (6nm 815mm² 53B Transistors embedding an 8b LLM):
> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM
flog 14 hours ago [-]
If someone has that sort of knowledge; how big a chip would be required? Is it possible?
mdp2021 14 hours ago [-]
Well, given the data above, roughly a 220b transistors chip for the HC1 tech.
cubefox 14 hours ago [-]
> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
wmf 13 hours ago [-]
The methods for splitting weights across multiple chips are well established. Groq/Cerebras can't hold a model on one chip either.
pyrolistical 11 hours ago [-]
Umm I have an extra 35, do you have layer 6?
IsTom 14 hours ago [-]
I think it's enough that a single layer fits on each chip if you can daisy-chain them with good interconnects.
octoberfranklin 9 hours ago [-]
They pipeline-parallelize across multiple chips. DeepSeek v4 Pro will be 30 chips.
moralestapia 12 hours ago [-]
Taalas is just a phenomenal startup from Toronto. My dearest congratulations to the founders.
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent advancements that made attention more efficient are probably going to be very useful for them.
And one can say LLMs are not as smart as a human, but a lot of the reasoning humans do for product and service generation isn't smart at all—it's just a bit of fuzzy input/output plus some reasoning rules. And then if you hook a robot up to the LLM, you can get results in atoms instead of bits.
I'm very excited about the future. I also hope it will stop money from flowing to bureaucrats who are incentived to keep the problems open to keep the money flowing, and instead facilitate sharing directly with the people (for example, no money to the state to solve homelessness—instead, spay instead with intelligence output to build a house and provide food as part of taxes.)
Specific traces for specific inferencing will mean that some generations get deprecated. Look at H265.
Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
AcmeAI Carbon
Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely new hardware, your lineup becomes:
AcmeAI Nitrogen (top tier) AcmeAI Carbon (mid tier)
If you just kept pushing the same model down your pricing tier over time you could still extract a lot of value from an old model, even years after it's been set in stone. Working on brand new code/frameworks? Pay to use the newest model. Working on legacy code? Use the lower tier models that will already know your legacy frameworks, pay far less and still get massive throughput. I've worked on a lot of government projects that this would be absolutely brilliant for.
The other side of this is that agent harnesses are NOT set in stone, so even a legacy model with a knowledge cut-off that's years out of date can likely still be helped quite a bit by harness and fetch behaviours that are still developing rapidly. Especially at this kind of throughput.
This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints.
___
Specification HC1
Model Llama 3.1 8B (hardwired)
Process TSMC 6nm
Die size 815mm²
___
So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.
Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others.
[1]https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
At some point the music will stop on training bigger models, and when that happens it will make sense to have ROM weights (or 100% analog circuits given how noise-resistant LLMs are), but we'll know when that is because the investment bubble funding the training of new models will have burst.
The rate of change to the models has to be slower than the hardware roll-out to be worth a hardware solution. If "good enough" happens before then, that just means the user gets a software solution.
Now you sell the same phone with higher price tag.
Ofc if the model has some critical bugs that’s another matter.
Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.
What Siri is missing is more logical solutions and answers for recipes, etc (still suck even with chatgpt integration).
Nope. And not only not a decade ago, right now.
If you have an Android or iPhone, you can give it clear and easy to understand instructions that Gemma 4 could complete[1] if it had tool calls on it, and that 100.00% of Claude, ChatGPT, Grok, Kimi, you name it, could understand and all complete if they had the access.
The phones will fail to complete it. I just tried Siri. I said "hey Siri", waited for Siri to come up, and then I asked one of the exact sentences you replied to: "what's the weather this afternoon?" It thought for around 20 seconds, and said "Something went wrong. Please try again."[2]
I have Wifi, I have mobile Internet, I have free storage space, I have up to date software. What went wrong is that phones have never properly connected agents, not ten years ago, not last year, not this year, and probably not next year.
But don't settle for what Google could do in 1999 by hotlinking the keyword "weather" in any query to the weather being shown in the results.
Tell your phone (any phone): "Please call back the last number that called me that is not an unlisted number, regardless of who it came from."
0 out of any phone will complete that today, tomorrow, a year from now, five years from now, ever, because phone makers are not going to let them do that.
Meanwhile, 100% of all frontier agents could complete it if they had tool calls on the phone. Which they don't, and won't ever, thanks to the duopoly.
Okay, that's a bit dismissive, I would love to be wrong!
[1] after any voice recognition to text - which does work really well on both Android and iPhone! [2] screenshot: https://ibb.co/21rtDnfV
Can you say this to it: "Hey Siri [wait for it to come up] - please send me an email with the temperature right now so I have it for my records." and see if it can complete the task without any backtalk or misunderstanding, and if you get exactly what you asked for. (It's a really clear request.) Should be 1 statement, no clarification, conversation, random search results, ("Here's what I found!"), etc.
A normal frontier model can do that - or Siri can do it if it is properly connected to Claude, ChatGPT, Gemini, Grok, or any other frontier AI - but previously it was never properly connected.
If it can do this task, I might have to look into this again. It counts as a success if it sends yourself any email with the current temperature and you actually get it (it can include whatever other text in the email), and a failure if it talks back, says "here's what I found", says it can't, asks you any question, sends you an email that doesn't actually contain the current temperature, just reads you the temperature and then asks if you want it to send an email, etc. Should be 1 shot.
let me know if it works!
That's the only thing the normie consumer cares for really.
Think it's called Askjimmy or similar.
(Not that I believe it, it writes too well for GPT-3.)
Hosted frontier models from two years ago would be much faster today, too.
Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.
[1] https://taalas.com/the-path-to-ubiquitous-ai/
I could see it being feasible to get a Qwen-3.6-27b type of model done on something like this. Qwen-3.6-27b at 18tok/s would be a game changer.
Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).
But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).
But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!
And then the economy implodes.
I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!
It would certainly be an accelerator for people who know exactly what they want. And it would remove multi tasking, which I‘d appreciate.
This is the "dumber but honest person that works harder" phenomenon, vs "lazy genius".
Sure, in the future full rewrites and stuff like that will be just another "throw money at it" problem, but fundamentally software can get arbitrary complex and we barely know how to write large, maintainable code bases.
Nonetheless, I think testing (and maybe proofs) will have its long-awaited time to shine, as being the "reward function".
Whatever you can cheaply do with AI is not a moat, if there is profit in there there will be quick imitation and competition will eat away those profits.
Models can be replaced easily, harnesses & AI tools too. And if cloud inference gets too expensive there are local models keeping the cloud prices hard capped.
Probably AI won't make anyone very rich.
If inference speed goes up, I can launch the same query 5 times, evaluate the best result and proceed from there. Of course, evaluation is also instant, so in seconds I can get a near perfect solution. Or maybe 10 and I can pick what I like the best.
AI previously provided speed but not quality. As soon as quality reached an acceptable threshold, the speed became the reigning factor.
In my opinion the quality is still much lower, but speed means the cost is significantly lower also.
I mean, if an agent can do half-decent work in less time than it takes the user to prompt them (and "user" in this context is a fast touch-typist like most programmers are), it's obvious it's not the agent that's the bottleneck anymore.
This has always been the case for human project management, and LLMs just aren't at that level yet.
It's more like everyone is speed running to how fast they can convince others that "half decent" is good enough. And for sure, newer models of LLM seem to be getting better at that.
That said, it obviously depends on the project.
A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.
This model had zero information right, while being fast in responding.
Unacceptable.
I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}.
Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Generated in 0.024s - 14,163 tok/s", the page says). Slightly more powerful model while keeping this under a second, and this could easily become a qualitatively different form of autocomplete/text suggestion. Running in the background every couple keystrokes, or every time user stops typing for more than 500ms.
>I just pasted your comment and its whole inheritance chain to it,
Good idea. Only problem is it doesn't work. I just did the same thing with exactly this prompt:
>did the user IOT_Apprentice participate in the thread below and if, number and quote all of their comments. Only just number and quote the comments or write "Did not participate", do not add any commentary. Quote any comments by this user verbatim, exactly as input. Thread:
followed by pasting the thread[1]
And received the answer "IOT_Apprentice did not participate in the thread."[2] in 0.001s, even though they have literally the last comment in my quote and it's clearly legible.
It's particularly insidious because the understanding and thinking that is required to follow my requested answer format exactly is substantial - so based on the fact that it gets the format right and clearly understood the assignment, I would be inclined to believe that it would also be correct!
So to use your example, it's not just autocomplete, it's autocomplete that confidently returns "No matching results" in 0.001 seconds, even though there is a search term matching what you put in, right in the prompt itself that was sent to it. That is much worse than useless.
[1] prompt: https://ibb.co/CKVmRvtd
[2] result: https://ibb.co/BKdRKmyD
> Bruce Lee was born in San Francisco, California, USA on November 27, 1940.
> Bruce Lee's father was a Chinese opera singer
That being said, this is not a good test. It is a language model (a very small one), not an encyclopedia.
ChatJimmy interface is just a tech demo. Without tool calling functionality we can't expect it to be factually correct.
you need to launch 10-15 more terminals, who is waiting these days? :)
Autocorrect that works. Reply suggestions that almost work, just need to be tad more accurate (probably more of a data access issue than model) and a tad faster to look completely seamless. Screenshots with automated text detection and OCR and automatic interpretation (different suggested actions for when something on the picture looks like a web link, phone number, postal address, e-mail, or QR code, or an event poster). That's just a fraction of things I saw showing up on my Samsung phone over the last 6 months.
For over a year now, you could get a much better autocorrect and spell/grammar check, and a translator all in one, if you just pasted your text to a frontier model and asked it to check for errors or translate into target language. Now imagine being able to go through a round of such checks in a 1/100 of a second. You could have this running every keystroke, and suddenly the inline autocorrect/checks would not suck anymore.
Auto-linkifying that can correct for typos and doesn't need careful regex tuning because it understands from context what is meant to be a link or not. That's just one of many obvious things possible once you get local models running fast enough. Tip of an iceberg, and the first step to imagining all the other potential uses is to let go of the two mistaken beliefs people hold on to:
1. That LLMs are about written language. They're not; ever since "multimodal models" became a thing, tokenization extended to visual and audio space, and now textual and visual and aural inputs are all just regular, first-class tokens.
2. That chatting with the models is the only optimal way for end users to interact with AI. That's just artificially limiting yourself to the space of chat-based UI.
Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.
But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.
That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
That also needs server class hardware though. A phone won’t happily service the insane amount of IO, compute, and network that this cascade would require.
Fully interactive realtime NPCs in videogames at scale.
Recommender systems that simulate individual consumers.
Crazy shit
In a way... when it's finance, they should be maybe called Gray'ish Swans?
8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2
8B model (Taalas) = 4 GB ROM = ~800 mm2
I don't think it needs to be on phone per-se. It can keep chugging in cloud - plenty of people use cheaper older models.
And I suspect the growth will slow eventually making taalas interations slower.
I expect this to be around the time when we're finally ready to travel to Mars.
A ~30mm side for the HC1 tech for an 8b model (still unclear the planned HC2)?
It is more that there are multiple reasons why this idea (burning an LLM into silicone and deploying it into a device in people’s pockets) requires huge piles of cash and the kind of engineering chops only a few company posesses.
Of course i would like it if a small upstart would do this, but it doesn’t seem likely as a posibility. They won’t have the funds to fab the IC. They won’t have the funds to train and validate the model before burning it into silicone. They can’t absorb the risk of the first tape out going wrong. They can’t absorb the risk of the model being faulty in some subtle way. They don’t have a device to integrate the IC into. They won’t have the funds to develop one. If they somehow would make a device they don’t have the marketing and sales channels built out to get the device into people’s hands in sufficient numbers to justify the development cost.
Basically this idea feels ruinously expensive. Apple has deep pockets, they already have working well-regarded phones, and an ethos of privacy preserving innovation. This is why this idea feels well suited for them and not many others.
Do i want the winners to keep winning? No. But not many others can pay for a moonshot crossed with a manhattan project. They just can’t.
Completely fixed-function HW can't be used for training, it's inherently a statement that "this model is Good Enough and we are now gonna start just extracting its value instead of extending it". So yeah it's an inference moat but it's not a growth moat.
Makes perfect sense for a company trying to get into the compute business, not companies who wanna be in the creating-ASI business.
Still, I guess/hope they have teams doing it in-house anyway. Just not something they'd wanna make a huge amount of noise about, it doesn't look good for To The Moon valuations.
[0] https://www.dwarkesh.com/p/why-compute-might-get-10x-more-ex...
Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the maximum size that conventional lithography can expose. They claim they could partition the model across many chips, but I'm not sure if they have tested this process or what it means for compute. The basic storage arithmetic is unforgiving: for a one trillion parameters model at four bits it will take 50–100 chips. To service a sizable customer base will take thousands of 100-chip fabs.
That all said, I'm bullish on this technology, and look forward to seeing it evolve.
Eventually someone will have to solve compute in memory at scale.
If the LLM response only takes a few milliseconds, the chip can process hundreds of other requests until the first conversation becomes active again.
Sounds a lot like "640Kb ought to be enough for anybody"
You did not compute that as the cost for a speculative card from Taalas, right?
Yes, a cheap and fast Opus4.6 can drive a lot of value in current context. But if we continue to craft bigger-and-bigger balls of mud, Opus 4.6 may end up hitting its conceptual ceiling and unable to contribute.
Winding the clock back on your statement gives:
> I'd gladly pay for a Claude Sonnet 3.5 in silicon and use it for 1-2 years.
Man, I dunno.
Perhaps in some cases, but the value I personally and professionally got out of LLMs reached a limit a while ago and has since kind of fluctuated between that limit and a bit less.
If the best model was instant, like the demo here, it could certainly provide more value, I guess, but I think the limit I'd quickly hit is the same one as now, which is how much of it do I want to produce, for what reasons?
Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.
(Yes, you could fix a number of masks, e.g. entire logic gates, of course).
*(It's local: private files managing firm oriented. It's blazing fast: it can be placed into recursive, intensive local workflows.)
Text diffusion might be a disruptor here, but let me just say the most cutting edhe form of image diffusion (JiT and DiT) right now is just a big fat stack of alternating attention and MLP matmulls. Not theoretically hard to bake
i have written about this:
"For device makers
Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models. A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."
https://try.works/role-model-the-case-for-a-model-routing-pr...
To answer your question: A large language model itself does not know this (afaik). But chatbots are not "just LLMs" but a whole bunch of systems (and models) around them.
That said, I read the question I am replying to as a rhetorical one. If it was meant as a genuine question, curious about the question of meta knowledge, then I misread. Certainly the question is extremely interesting, for both LLMs and humans! But it's also obviously a very difficult one, as we don't even have a clear theory on how "knowing" works in the base case.
Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory bandwidth.
Bitcoin OTOH has used the same PoW algorithm for a decade. Barring some really exciting discoveries about the nature of computation, new ASICs are not that much more efficient than old ones.
BTC mining is also not exactly competitive anymore; the nature of the PoW algorithm means that it's dominated by a few large players who've set up shop next to a dam and who pay very little for electricity.
New entrants are highly discouraged because the mining rewards are constantly halving, it's hard to find cheap power, and the price of BTC is now so volatile that a yearslong investment is very likely to lose money.
It’s like talking about anything else than Megapixels when everyone was convinced that megapixels must go up in certain periods of the smartphone boom.
The Jalapeño mentioned («Anthropic is not alone in walking this path») in the article is still a classical Von Neumann architecture.
And Taalas' idea makes sense in a perspective of scale - producing a large number of cards; "for internal use" (a lower order of items) means a high production cost.
"Yes, the Wang Corporation, the company that originally developed and marketed the Wang 2200 computer, still exists as a rebranded company under the name PPL (Precision Pencil and Label), but it has undergone significant changes and challenges over the years.
Here's a brief overview of what happened:
In fact, Wang labs was founded in 1951. PPL seems to be a made up entity. But it did generate those "facts" in 0.033 seconds. If people value speed over accuracy then I can write an LLM that is 100x faster than chatjimmy.ai and make big bucks by responding one of N canned responses to any question.I also think that etching models into ASICs may be a bit too inflexible for what OpenAI and Anthropic want.
Indeed, for some kinds of applications involving secure/legal data etc. I can see the consistency of silicon winning out, because it combines performance with immutability and guardrails in hardware. Some chips have write-once PROMs to store password hashes and similar, you could do the same thing with prompt hashing to absolutely force or forbid certain behaviors. A model that can't be updated is also a model that can't be hacked.
If I compare the Pixel 6 Pro I'm using at the moment to current models, they are functionally identical. The only reason to upgrade might be getting a fresh battery and access to firmware updates.
Otherwise I'd be happy to continue using it for the next 10 years.
Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.
We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time.
Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food drive-thrus. And then somebody discovers the model has a fondness for goblins[1], and if you role-play convincingly enough, you can get it to accept payment in shiny buttons and rodent skulls instead of cash.
What do you do then? I guess your options are to try and fix the behavior with a better prompt, or put some kind of filter in front of the model to catch attempted exploits. If the filter is cheap and dumb it probably won't work well enough, and if you use another model as a filter, you've negated the cost and speed benefits of putting the first model in hardware.
Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.
[1]: https://openai.com/index/where-the-goblins-came-from/
I think it's far more likely to see them used in safety critical applications where you need a capable model that can run on low power and doesn't have multiple layers of operating abstractions between the model and the hardware.
Does it though? Isn't that what CPUs are, very fast-not-so-clever computing brain surrounded by layers that protect it?
NVIDIA will probably give us a new GPU when someone competent in the free market decides they want wheelbarrows full of money. Unfortunately, AMD is entirely, incomprehensibly, incompetent, to the point where I can only assume they're colluding with Nvidia, behind the scenes.
For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity.
I'm not good at predicting, but some ideas:
1. All information gets augmented in real time with personalised context.
2. AI interaction seems more like find-as-you-type than a back and forth.
3. AI produces many outputs to pick from. Either the human, AI, or another system can do the deciding.
Even if it's last year's model, speeding up LLM inference could open up all sorts of opportunities.
https://youtu.be/7NfyZhV1dKM?t=53
Imagine this demo but the apps render in real time generating real code.
Obviously not valuable because we have OS today, but could be your companies "WorkOS"
Edit: also consider centralized Room-641A-type surveillance when models summarize and/or flag all calls processed by public telephony
That is a probably a significant jump in search quality for many queries, that people didn't take the time to research properly.
This move undercuts NVIDIA directly.
Maybe AI models may become reduced to a collection of tiles you add to a chips one day, maybe sooner for some areas as you say, motor control for balance, vision systems, speach recognition systems etc, broken down, for robotoics, much is already there and just cost of battery/power holding much back.
It doesn't actually specialise in anything in particular that one can point to.
For this reason you can really transfer them between models.
There are probably limited applications but not zero.
But ”top of the curve reached” feels like the likelier scenario.
I expect AI models chopped up into building blocks where 99.9% of the compute is fixed but glued together with flexible "fine tuning" layers that will adapt them to specific applications. Those kind of chips will run 99% of consumer AI and at some point be integrated into consumer devices.
I'm trying to get the most out of it by redlining my AI subscriptions. Hopefully I'll manage to start a business in my niche. I don't even know if my niche will exist in the future.
You can actually test it out on their website, just imagine 3 x faster and maybe 15% smarter.
Also there are other people innovating in hardware.
Next time, one of those number will be smaller, and the other will likely be bigger. How long before the analysis side gets too overwhelming to bother with? Probably less than 6 years.
It also mostly passes the "schlong" test
https://pastes.io/YcxSi8Fp
"A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"
Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true
It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not
You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.
[0] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity [https://arxiv.org/abs/2604.24827]
It tells me it is a variant of Codex.
If we can get to this speed with reasoning models, man... I can't even imagine the impact.
I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.
It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…
Developing software becomes 95% about intent and requirements. Can’t wait for the next iteration of that.
Because it's not human and not "thinking", it's a mathematical algorithm
The characters in the 3-act Shakespearean play had very little depth, many of the names were similar, and they were not very smart, but the simple plot was cohesive.
But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.
After all, there’s already very useful models even for productivity at 27 or 35B.
Not sure if modern models "think" only by outputting <thinking> blocks, or there is a more complex mechanism at play.
That's pretty much it - a small refinement to "Chain of Thought" prompting, where you tell the model explicitly in the prompt to "Think step by step" or similar, so it writes out more steps before giving a final answer, potentially catching some errors. The "thinking" models are tuned to do that without being prompted to, and to output the "thinking" markers around it, so they can be hidden from the user.
I am curious what the drop in thoughput is for multi-turn answers, instead of one-shot. More in line with the current "agentic" use-cases.
https://huggingface.co/meta-llama/Llama-3.1-8B
As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.
....damn. It's very impressive notwithstanding its limitations.
After a quick search, I found that Ta'ala is actually a Canadian chip startup that produces artisanal, high-end potato chips. They offer a range of unique and creative flavor combinations, often featuring Canadian and international ingredients.
Ta'ala is known for its high-quality, small-batch potato chips made with premium ingredients and care. The company is committed to creating unique and delicious flavor profiles that showcase the best of Canadian ingredients and cuisine.
Is this the Ta'ala you were thinking of?
There’s no difference in the inference implementation, parameter count, or speed.
But yeah, there are a lot of factors, so it's hard to answer, and tokens/s isn't the right question.
(/s!)
Generated in 0.051s • 14,092 tok/s
Impressive...
Given gpt 5.5 was very good to me and gpt 5.6 series seems not boost too much, i kinda like the way bake the model weight to the chip, and connect multiple chip to serve the large scale model and allow respin some parts(ROM like?) to do model weight update, maybe this seems sustainable, the future is exciting
WTF is this?
Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.
I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
What I imagine an on-device model should be doing is just translate natural language to search requests and calls to tools manipulating retrieved data - much like no model currently does calculations and instead they open up calculator and use that instead.
Now, that’s slow and expensive although seems to work quite well (haven’t really evaled this properly, don’t have the time). If inference can be made fast and cheap, multi-model approaches like this would become more viable for more applications.
It’s really a dream for setting up a homelab
What are some examples?
Does it about once a day, that I notice.
I asked it a slightly tricky math problem (I re-asked it the same problem to create the paste, and it did about as well the second time). It was unable to solve the problem, and it’s a small, old model, so…fair enough…but also its answer was pretty incoherent, with stuff like “Since A is an invariant set, it's always possible to find a cave that the fox cannot be in. Therefore, you can always catch the fox in that cave.” (…catch it…in the cave it can’t be in?).
Then, off the top of my head: Claude somewhat recently generated a Spark Job where the worker timeout was longer than the worker heartbeat, so workers would always inevitably be killed when they didn’t heartbeat within their timeout window. (also…neither option needed to be set?) Before I noticed the problem, I asked Claude why the job was taking so long, and it told me the data set was too large. More recently, there was a blog post by John Scalzi I was having a hard time finding, so I posed the problem to ChatGPT, and it came back with a blog post that didn’t include any version of the text I remembered and wasn’t really topically relevant (and maybe I hallucinated the blog post, but it could’ve said “I can’t find it” instead of “here you go”). On another occasion, I was trying to find a particular episode of Bob the Builder for my kids, so I Googled it, and Gemini kept giving me the wrong season and episode number, even after several rounds of “no, s5e6 is ‘that thing’, I’m looking for ‘this thing’.” Turned out the episode wasn’t on Amazon at all (which I had to tell it), and I had to go find it on YouTube.
That said, as I sit here scrolling through my history to see if I’ve forgotten any particularly good examples, I have to admit they do a better job than I’m giving them credit for. But I still wouldn’t have them write my email for me (the one time I tried that, when I was playing with openclaw, it sent a fairly demanding email to someone I didn’t know that well without asking for confirmation, and I had to go apologize and explain that I hadn’t really written the email, which was embarrassing), nor am I particularly excited to have chatjimmy as a permanent resident of my pocket.
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
AI companies constantly update/change stuff, new models come out, new requirements, etc.
But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.
Not sure if that's better or worse than online personalizaed ads...
What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster.
Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors.
815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money.
If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip.
There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports.
Reasoning models are barely usable with contexts that short.
The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM.
Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.
Yet, maybe it can work well enough, so that as a manufacturer, you don't pay $50 for a PI, but only $0.50 for a tiny "hard-coded" chip.
The advantage can be that, as a LLM, as opposed to other types of chips, the use-cases could be more varied, so same chip could be use in different devices (robo vacuums, security cameras, ball-shooting training robots, etc.)
Could this enable a reasonable context size ?
That said, I've been wondering if they could go with multiple smaller ones instead. Like one per layer maybe even?
What are your thoughts on that? You seem to be more qualified than me on that matter.
I already harped on Cerebras, but their approach of just copy/pasting a whole bunch of identical functional blocks, over-provisioning the chip by ~8%, and then just fusing off blocks with defects allows them to effectively have 100% yield on a wafer-sized monolithic chip. This is very desirable, and just another reason I like their approach better.
A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper.
And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.
If they begin etching Fable into silicon now and release it 2-3 years later, i can see the market for it
Also I believe there is both a market for extremely fast local inference with current model performance and that such fast inference would unlock unforeseen usecases. Especially as TPS approaches early computer clock cycles and data rates.
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)
Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.
Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
With that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.
I agree that Groq with multilayer hybrid bonding could be a good idea.
I don't think anything I did was particularly novel, as I really just wanted to see how fast I could push a commodity FPGA to it's limit.
Scaling to an ASIC or getting into the billions of params is where the real engineering is! This was just a side project for a side project for me while the FPGA was idle
What exactly did you implement? A full LLM? A subset of it, which collaborates with something running on CPU or GPU? Which LLM? Why?
What language did you use to implement your thing: VHDL, Verilog, Vitis, something else? Why?
I can think of at least 10 blog posts that I'd write before I write a single line of code. Publish early, publish soon ;-)
The K26 has a quad core A53 core alongside the programmable logic (PL, or Fabric). The A53 is pretty weak, and doesn't have any hardware matmul operations, so despite the KV260 being sold as a 'vision ai starter kit' and the vitis object detection running on the arm cores, they're pretty weak cores for anything AI.
For my use, I need true determinism, so my vision pipeline is all implemented in the PL, and it was pretty disapointing that the vitis libraries are basically just opencv on linux, rather than really pushing the fabric. If I wanted probabalistic AI running on a CPU, then I sure as heck wouldn't choose a quad core A53.
Which led me to have a play with this, I saw the taalas/chatjimmy demo and wondered what I could push the fabric to.
The round trip time to DDR or CPU via AXI meant I had to keep the entire inference engine in fabric. The A53 is simply a pipe that gets a request from my server (which has a cloudflare tunnel to the real world for the live demo in the blog post) and manages a queue. So it feeds a string in, and gets a hopefully longer string back a few uS later.
It's all in verilog, because that's what i'm more used to. I did get Claude Code to do a moderate amount, as it's a side project on a side project after all, but pushing an FPGA to it's limit is definitely not as comfortable for it as it is writing a crud app in TS.
I'm using tinystories, as we are talking about megabytes of URAM/BRAM. If I used the DDR, it definitely would have been a real model, but that wasn't my goal. My goal was to hit 100,000tok/s, and even when I conceded on absolutely everything, with a token prediction size of 1tok, I topped out at 60,000tok/s. ?But increasing the window to make an actually plausible chat (story generator, it doesn't understand questions, you need to prompt it with 'once upon a time...' and it finishes it for example) I managed to break 20k tok/s.
I also created the lemmatised version, which was inspired by Kevin from the office (why use many word when few do trick) and trained a new model, I was expecting the output model to be smaller, but was suprised that it came out the same size, but it ran 30%ish faster. In hindsight it makes sense, the parameter count is fixed by the architecture, not the corpus, so training on compressed text doesn't shrink the model at all. What it does is compress the output distribution. The same story takes ~30% fewer characters to tell, so the effective speed goes up even though the per-token rate is identical. The dumbness is the optimisation.
Fully agree on publish early. The blog post with the live demo (a websocket straight to the board through a cloudflare tunnel, so you're genuinely talking to the fabric) is written and sitting in drafts while I fiddle with it. This thread is my peer pressure, it goes live in the next day or two.
Cerebras is already public. AFAICT, there are 8 other startups in the space, some of which have mature products: Etched, d-Matrix, SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and Fractile.
In Aug 2025 you had
- OpenAI o3
- Opus 4.1
- Gemini 2.5 Pro
- Grok 4
Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.
Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.
Bad perspective: consider the correction: "when are thresholds of sought quality reached"? Hence: not "is there a 10yo from last year that could compete with the current 13yo", but "will there be a 30(?)yo from last year that could compete with the current 33(?)yo" ('(?)': the scale of yearly growth in the future is uncertain).
A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?
Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
[1] https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-la...
Yeah, I had the same thought. The key thing for them was the price point at which they could deliver a ~30B model. I would buy one today if it was ~1000$ and could run whatever the best 30B model is today, at those speeds advertised. Even if the model becomes superseded by model.5 in a few months, there's still a lot of things you can do with a "good enough" model for some tasks. And things like maj@x or generate 10 times and choose "at a glance" what you like (think frontend stuff) would be worth it.
No idea if them selling to AMD is good or bad.
We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software.
I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.
Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.
Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.
If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor.
If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.
Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.
Taalas' approach (at least for their demo'd product) is to bake the whole thing in completely. IIUC the optimiser can even see the weights while generating RTL. It's like there's an "uint8_t weights[] = " in the source code.
100% local and no leaks.
Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?
Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?
Claude Code is still using haiku 4.5 from ages ago for explore subagents for instance. Not to mention production uses like customer service that only need to be "good enough"
But either way, I think GP's overall sentiment of "delegating intelligence-saturated tasks to an outdated but fast subagent" makes a lot of sense.
I guess losing some customers due to poor customer service is ok if the price of customer service is right.
6 months or even a year if something goes wrong in the fabrication process and you need to update things.
If they do more standard asic design, it could be a lot longer as the design needs to be validated on an FPGA cluster, which would necessarily need to be very big for something like a LLM. Easily up to 2 years.
There's a reason chatjimmy isn't demonstrating newer models and why they only show of an 8B model.
At the time people were no doubt saying yes but now 3.8 is out, is that still desirable?
[1] https://taalas.com/products/
> At 6 nm: 1T doesn't fit on one wafer.
> At ~2 nm: 1T plausibly fits comfortably on one 300 mm wafer.
But then again, 300B to 500B models are to this day also very valuable
The https://chatjimmy.ai demo was impressive.
Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.
This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.
it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.
I can finally have my own Dixie flatline. Cool.
In case some did not know: also the movie (actually TV series) is finally happening.
# Neuromancer - Official Teaser ( https://news.ycombinator.com/item?id=49055037 )
Just even comparing compute from 10 years ago (Apple silicon vs Intel) and it's significant. 20 years it gets crazy. My first computer was an 8 bit 6502 with 64K RAM and a 128K floppy drive (I think, it's fuzzy). Everything amazing now will look quaint in due time.
It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.
https://patents.justia.com/patent/20250123802
They won't sell/rent/license the weights to an end user at any price because they don't trust your security.
Now your robot can respond sarcastically when you ask for chicken nuggets. Again. It also doesn't dent your walls anymore.
Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.
Fable is nice, but still requires a lot of guidance for large scope tasks.
But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA
Are we a couple years away, a decade away, or something else?
It is already that.
> Will "intelligence" become much like a gpu
As an option among the implementations.
> Are we a couple years away
They could mass produce now, but it makes no sense at this rate of improvements in the models.
At least we can be sure that's the model we wanted. Service providers could be serving modified versions and nobody would ever know.
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
HN is failing to understand that AMD knows this well.
Taalas has WO2025217724A1 pending and AMD wants that because it is immediately a function block they can sell to anyone doing FP math, since large (mostly) read only memory banks are ideally suited for that micro-code type stuff.
"You are not prepared" --Illidan Stormrage
Great for us, looks like a recession as far as the Market is concerned.
You need to be able to add|mul where the data (the weights) are stored.
Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.) https://youtu.be/3MKRjt59hh4
Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.
so instead of RTX xx70 series, I can buy xxTA that have kimi integrated ??? is that right ??
Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges.
You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.
If tolerant, they could churn out many cheaper chips, some perhaps with slight abnormal tendencies ;)
Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses.
I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence.
I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.
https://www.youtube.com/watch?v=UwCFY6pmaYY
> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
Edit: Lol, downvotes? Stay jelly, meanwhile Talas goes brrr.