Data is the fossil fuel of AI

I’ve watched Ilya Sutskever’s 10-year retrospective.

We’ve seen that LLM’m rate of improvement has slowed down significantly.

The problem is that even though compute is growing through better hardware, better algorithms, and large clusters.

Data is not. That’s it. We fed it all into the machine. We’ve only got one internet.

So, the next evolution may come from agents, synthetic data or inference time compute.

It may be something else entirely.

But the “throw more data at the problem” seems to have reached its limit.

Source: Ilya Sutskever: “Sequence to sequence learning with neural networks: what a decade”

Yours, Taj