It's weird to me how the article in the very first paragraph mentions brute force, but as data problem. As if we hadn't seen a staggering raise in ops and memory bandwidth. Machines have been able to hold the textual history of humans in short-term memory for a while. But the ops they could perform on that have been limited. Not much point telling a 2005 person to "add data". What were they going to do? Wait 20 years to finish a round we do in a week now?
It's very clear to me that the progress we observe in machine intelligence is due to brute processing power. Of course evolution of learning algorithms is important! But the main evolution that drives progress is in the compute. Algorithms can be iterated on that much faster if your generations are that much shorter.
Why are all these AI companies falling over each other to buy the best compute per watt humans have ever produced? Because compute is king and our head was optimized by evolution to be very efficient at probabilistic computing. That's where machines are catching up.
The mark of intelligence is to not need much data at all.
You don't need more data when the data you have characterizes a problem well. More data is simply redundant and resource wasting. In this case, talking like people about things people talk about is covered well by current data sets. Saying we can't get more data is really saying we have collected at least enough data. Probably more than we need.
Lots of room to improve models though:
Using convolution for vision learning didn't create/require more data than training fully connected matrices. And it considerably increased models efficiency and effectiveness on the same amount of data. Or less.
Likewise, transformers have a limited window of response. Better architectures with open ended windows will be able to do much more. Likely more efficiently and effectively. Without any more data. Maybe with less.
Maybe in a few decades we will reach a wall of optimal models. At the rate models are improving now that doesn't appear to be anytime close.
Finally, once we start challenging models to perform tasks we can't, they will start getting data directly from reality. What works, what doesn't. Just as we have done. The original source of our knowledge wasn't an infinite loop of other people talking back to the beginning of time.
michael levin talks about "intelligence at every scale". he has a recent study where he found some of the hallmarks of intelligence in an off-the-shelf sorting algorithm. individual cells by themselves certainly have signs of intelligence, such as memory, attention, the ability to recognize that a strategy has failed and come up with another, etc.
It's weird to me how the article in the very first paragraph mentions brute force, but as data problem. As if we hadn't seen a staggering raise in ops and memory bandwidth. Machines have been able to hold the textual history of humans in short-term memory for a while. But the ops they could perform on that have been limited. Not much point telling a 2005 person to "add data". What were they going to do? Wait 20 years to finish a round we do in a week now?
It's very clear to me that the progress we observe in machine intelligence is due to brute processing power. Of course evolution of learning algorithms is important! But the main evolution that drives progress is in the compute. Algorithms can be iterated on that much faster if your generations are that much shorter.
Why are all these AI companies falling over each other to buy the best compute per watt humans have ever produced? Because compute is king and our head was optimized by evolution to be very efficient at probabilistic computing. That's where machines are catching up.
The mark of intelligence is to not need much data at all.
Important concept for model building:
You don't need more data when the data you have characterizes a problem well. More data is simply redundant and resource wasting. In this case, talking like people about things people talk about is covered well by current data sets. Saying we can't get more data is really saying we have collected at least enough data. Probably more than we need.
Lots of room to improve models though:
Using convolution for vision learning didn't create/require more data than training fully connected matrices. And it considerably increased models efficiency and effectiveness on the same amount of data. Or less.
Likewise, transformers have a limited window of response. Better architectures with open ended windows will be able to do much more. Likely more efficiently and effectively. Without any more data. Maybe with less.
Maybe in a few decades we will reach a wall of optimal models. At the rate models are improving now that doesn't appear to be anytime close.
Finally, once we start challenging models to perform tasks we can't, they will start getting data directly from reality. What works, what doesn't. Just as we have done. The original source of our knowledge wasn't an infinite loop of other people talking back to the beginning of time.
michael levin talks about "intelligence at every scale". he has a recent study where he found some of the hallmarks of intelligence in an off-the-shelf sorting algorithm. individual cells by themselves certainly have signs of intelligence, such as memory, attention, the ability to recognize that a strategy has failed and come up with another, etc.
Lots of good thinking in this article. A few things come to mind before we hit a data wall.
1. Sensor all things
2. Waves upon waves
3. Dynamic or Living Semiotic Graphs. Bring your own terminology.
4. General Artificial Synesthesia.
Language Models do not work like the human brain. Continuing to compare the two like there is an analogy at all is doing far more harm than good.