> VFXs, video game graphics, CAD, modern industrial techniques, telecommunication, signal processing, computational finance, biodinformatics, fluid simulation
> looks inside
> basic linear algebra
Most of math can be boiled down to trying to get Linear Algebra to do more complicated stuff for us. Calculus is all about how to use linear approximations to conclude things about non-linear functions.
But LLMs have one extra, important ingredient which is the activation functions. These are what actually allow for the computations to be non-trivial. The universal theorems for neural networks require them. Activation functions are how we get linear algebra to do the more complex work for us.
There is another really important ingredient: all that human-generated data and all the human labor to curate/annotate/label/etc. None of these models work without it.
I guess a lot of it really is matrix multiplication but things like transformers and attention blocks aren't something that I personally learned on Linear 1
But attention is something that someone who has taken basic linear algebra should mathematically be able to understand. It’s just an application that wouldn’t fit in a math class
Actually, I didn't realise the difference between writing equations on a paper vs actually running them on a computer until I took a "Numerical Linear Algebra" course.
Modern LLMs are anything but "basic" linear algebra. With basic LA the LLMs would take years to respond. Its the clever engineering behind what NOT to multiply that's making modern LLM infra possible
I agree all the numerical stuff is important but you can also understand transformers just fine without actually understanding the numerical implementation details since that’s all abstracted away. I use flash attention every day but couldn’t tell you how it’s implemented since I just don’t need to know that 🤷♂️
No one said that basic linear algebra is sufficient to intuit the training dynamics of large transformers. All I did was agree with the other guy that fundamentally it’s all basic linear algebra with a little calculus on top
All you need for attention (haha, get it?) is literally first year university linear algebra. Maybe the softmax is not, because that's high school math. My issue is understanding why it does what it does.
I find it really interesting how different countries teach this. (I'm from Germany)
In my last school I was teached "y=mx+b", in the school before it was "y=mx+c", at and now I hear of "y=wx+b"...
Standard basic US middle school algebra plotting was y = mx + b too. Then some later classes (e.g. statistics) did y = bx + a using b for two different things just to mess with you I guess?
My pesronal theory is that using AI is a lot like being a CEO or a billionaire. You spend all day talking to a team of obsequious agents that will do anything you ask and will fawn over your brilliance. AI users get it from ChatGPT or grok or whatever, and billionaires get it from their entourage, but it's the same problem, and it rots their brains in the same way.
Similar issue with people who want to just “ban AI” tho right? Like what we’re gonna ban gradient descent? Partial derivatives? The chain rule??
Ban models over some parameter count? Imagine making a mathematical model illegal because it models too many things.
This is the mathmemes subreddit, so at least a lot more people here understand what a model actually IS, but the predominant ban-AI position on other subs is “don’t know, don’t care.”
I’m talking about the position of “ban AI” not “regulate AI from a position of knowledge and nuance.”
I’m personally of the opinion we should socialize ownership of AI entirely.
I am obviously also extremely nervous about current US administration having regulatory ability on AI. I don’t think it’s a stretch to imagine this admin trying to enforce “no pro-trans rhetoric from any model - it’s dangerous!” as one of these types of “regulation.” Similarly with AI that expresses anything that isn’t pro-capitalist sentiment.
Regulation gets a lot messier morally when selfish greedy people are the ones doing the regulating. Against those kind of systems, you could imagine why someone might lean more towards opensource models that don’t need executive branch approval for release (the current situation)
Why do they need to know how AI works / how a ban would function technically-speaking in order to support banning it?
Do people need to know how illicit drugs work in order to support their illegality? Most people definately don’t, they see the effect they have and that’s what matters to them. The exact mechanism behind a ban is fine to be left to subject matter experts.
Because the exact same math that allows for all the slop they hate also allows us to model complicated protein folding problems, solve translation accessibility for people who don’t speak the dominant language very well in their area, model the future of the climate, and like basically everything else?
Gradient descent on trainable parameters is a miracle of applied mathematics. It’s like calling for a blanket ban on all plastics in use everywhere because you are upset about plastic straws.
The slop machine does have legitimate/more defensible use cases (like the translation accessibility one), yes.
But for other kinds of trained models, its not like people were out here advocating for the banning of alpha fold (they barely knew what that was). They clearly aren’t about banning the entire class of model, rather the specific use cases / implementations being marketed right now.
honestly, I prefer to look at how fucking stupid ai users have become.
Some people were fine. Normal average type of person. Now they can barely fucking read.
Some people were stupid, which is fine - not everyone is smart. But now they are basically an active threat to others.
Some people were smart, good for them. Now they refuse to do any mental (or emotional) work, and I mourn the person that they could have been if AI hadn't hijacked their development.
A few people have managed to use AI in actual good ways, but partly because they don't treat it like it has a personality, or because they treat it like the personality is simply the UI of the answer-engine. Just a feature to be ignored in most situations, and actively worked against in others.
They weren't claiming all ai users were smart because Terrance Tao is smart. They are giving an example that at least one ai user is smart because OPs claim can be interpreted as all ai users are stupid.
Second, the person they were responding to is suggesting there's a much greater proportion of reds to blues, without any evidence. I'd bet the intelligence of ai users is pretty average tbh.
We're in a math subreddit. That one blue one should be enough to disprove the original statement "how fucking stupid AI users are". It's called proof by counter example
The downsides have been massively overstated. It has been well documented that Chinese bot farms have flooded US social media with anti-AI propaganda. Can you guess why they would do that?
You're taking one piece of information from a notably unreliable source (Musk-controlled Twitter which very much profits from AI) and extrapolating it into a claim that there's a Chinese post-op. That's basically the definition of a conspiracy theory.
As the article you linked states, we do not know what the exact underlying facts are and certainly not the reasoning.
The one point I will give you is that some of the downsides of AI are certainly being overblown by some accounts. Gen AI has become a very devicive topic where all sides stretch the limited available facts to the extremes to try to argue their points. For example, some of the statistics about AI's water consumption are very misleading (see Hank Green for a great video on it). However, other points, such as the detriments of AIs being used for companionship, the impact on artists, the energy cost of training AI, data centres polluting and disrupting communities, and the massive AI bubble in the economy, really can't be dismissed no matter how you look at them.
First, using emojis in conversations with people who actually have critical thinking skills, which is really to say in any place outside of instagram comment sections, will cause them to justifiably infer that is where you belong.
Second, the exclusive use of universal and existential quantifiers is a convenience that exists only in the carefully-engineered framework of a first order language. Anyone who is critically engaging with their peers can determine immediately that a statement made about 'group X of sufficient size' cannot be about every single member. Why would statistics need to refer to distributions for random variables at all if there were not naturally occurring variation among naturally occuring groups?
•
u/AutoModerator 1d ago
Check out our new Discord server! https://discord.gg/e7EKRZq3dG
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.