Yeah I agree. Small models is the way. You can also use LoRa/QLoRa adapters to “fine tune” the same big model for specific tasks and swap the use case in realtime. This is what apple do with apple intelligence. You can outperform a big general LLM with an SLM if you have a nice specific use case and some data (which you can synthesise in come cases)
Unlike the dotcom bubble, Another big aspect of it is the unit cost to run the models.
Traditional web applications scale really well. The incremental cost of adding a new user to your app is basically nothing. Fractions of a cent. With LLMs, scaling is linear. Each machine can only handle a few hundred users and they’re expensive to run:
Big beefy GPUs are required for inference as well as training and they require a large amount of VRAM. Your typical home gaming GPU might have 16gb vram, 32 if you go high end and spend $2500 on it (just the GPU, not the whole pc). Frontier models need like 128gb VRAM to run and GPUs manufactured for data centre use cost a lot more. A state of the art Nvidia h200 costs $32k. The servers that can host one of these big frontier models cost, at best, $20 an hour to run and can only handle a handful of user requests so you need to scale linearly as your subscriber count increases. If you’re charging $20 a month for access to your model, you are burning a user’s monthly subscription every hour for each of these monster servers you have turned on. That’s generous and assumes they’re not paying the “on-demand” price of $60/hr.
Sam Altman famously said OpenAI are losing money on their $200/mo subscriptions.
If/when there is a market correction, a huge factor of the amount of continued interest (like with the internet after dotcom) is whether the quality of output from these models reflects the true, unsubsidized price of running them. I do think local models powered by things like llamacpp and ollama and which can run on high end gaming rigs and macbooks might be a possible direction for these models. Currently though you can’t get the same quality as state-of-the-art models from these small, local LLMs.
This is nice to see. I wish they’d actually do something about our water companies too.
We all have to feel sorry for the drones?
Relative privation is when someone dismisses or minimizes a problem simply because worse problems exist: “You can’t complain about X when Y exists.”
I’m talking about the practical reality that you must prioritize among legitimate problems. If you’re marooned at sea in a sinking ship you need to repair the hull before you try to fix the engines in order to get home.
It’s perfectly valid to say “I can’t focus on everything so I will focus on the things that provide the biggest and most tangible improvement to my situation first”. It’s fallacious to say “Because worse things exist, AGI concerns doesn’t matter.”
Here’s how I see it: we live in an attention economy where every initiative with a slew of celebrities attached to it is competing for eyeballs and buy in. It adds to information fatigue and analysis paralysis . In a very real sense if we are debating AGI we are not debating the other stuff. There are only so many hours in a day.
If you take the position that AGI is basically not possible or at least many decades away (I have a background in NLP/AI/LLMs and I take this view - not that it’s relevant in the broader context of my comment) then it makes sense to tell people to focus on solving more pressing issues e.g. nascent fascism, climate collapse, late stage capitalism etc.
100% ageed. We seriously need to normalise compassion for colleagues and pushing back on overwork and entitled clients. I’ve been a senior manager in IT for a decade and whether one of my reports’ kids is sick or someone just needs half a day out due to a migraine, I’ll let them do what they need to do. Turns or if you treat your colleagues like actual fucking human beings they actually appreciate you and often return the favour by working late or going above and beyond somewhere else when the crisis has been dealt with.
Some of the horror stories in this thread of people phoning into zoom calls from the doctor’s office are insane. The world is not going to end just because Jerry couldn’t join this morning’s scrum or you had to move a call with a client.
“Oh but if underling #148 takes the afternoon off sick we won’t hit our weekly sales goal”. You know what Karen, firstly we sell receipt processing software, we are not saving any lives here get a fucking grip. Secondly, if you deny them that time off they will be doing the bare minimum for the next quarter while they job hunt and inevitably leave.
I truly resent being forced to partake in this ridiculous system but for as long as I have to, I’m going to have the decency to try to protect my team the best I can.
deleted by creator
She lost that some time in August 2022 I think. Put it in a “safe place”