Increasing Entropy #3
Discoveries of the week, random links, AI internals, and a few thoughts on random topics
The End of Tokens
Some time ago, AI researchers realized that it would be impossible to build AI by teaching it concepts we humans know. Demis Hassabis realized that before founding DeepMind (see The Infinity Machine). The alternative was to give AI raw data and make it learn things itself. Some more time passed before everyone accepted this.
As researchers trained neural networks on more and more data and made them bigger, it became obvious that the more data we give AI to learn on and the bigger the network is, the smarter it becomes. Richard Sutton summed this up in his famous “The Bitter Lesson” essay.
In 2022, OpenAI released ChatGPT and showed us the capabilities of LLMs trained on large amounts of data.
Since then, we’ve scaled LLMs even bigger and invented a great many tools and hacks to make LLMs even smarter and suitable for more kinds of work. Vision transformers were added to make them understand images. We trained them to use our computers using tool calls. We started using them in loops and calling them agents.
LLM architectures have grown bigger and more complex, but the input has stayed the same. In 2022, we started sending text to the OpenAI API and receiving text back, and we still are. We got billed for tokens then, and we still do.
Apparently, in 2025 Albert Gu published two blog posts: H-Nets: The Past and H-Nets: The Future.
I may be describing it roughly, but the idea was to stop splitting input text into tokens before giving it to a model, and instead give the model raw text and let it make sense of it.
This week, Cartesia, the company Albert Gu cofounded, published two articles showing the efficiency gains we get from giving a model raw bytes:
- From Tokens to Bytes I: Scaling Hierarchical Networks
- From Tokens to Bytes II: Learning from Raw Data
What excites me the most is not the efficiency gains but the beauty of the solution.
Tokens have always felt like a weird concept. Intuitively, that’s not how our brains work. Tokens are ugly and hardcoded.
Raw bytes, on the other hand, feel much more natural. This approach could unify input so we would no longer need a separate vision transformer for images or have to invent another way to encode audio.
Training on raw bytes feels so well-aligned with the inductive reasoning Demis Hassabis once believed in and so suitable for scaling.
Be prepared to pay for megabytes instead of tokens.
MRI of the AI Brain and Lobotomy
Sauers observed neuron activations in Qwen3-8B when asking multiple role-playing questions. Then he steered those neurons and made the model think it’s an owl!

x.com/Sauers_/status/2111327320573419523
Turns out steering just one neuron is enough. Highly recommend reading Sauers’ feed from October 17.

x.com/Sauers_/status/2111552823582457916
This went crazy. People figured out how to make Qwen absolutely evil by changing a single neuron.

x.com/chiefofautism/status/2115280104133710092
Apparently, it’s possible to combine two modifications to get Cookie Monster. Hilarious!

x.com/treetowntree/status/2114573600976183723
The biggest surprise for me is how brittle LLMs are. How would we deploy them in space?
* * *
Why Is This Fingerprinting Script Creating Bioweapons? — A smart (and fun) method for preventing your code from being analyzed by AI.
A great explanation of the Exponential Time Hypothesis and the significance of OpenAI and Anthropic claims that they disproved it: x.com/rishabhkio/status/2114892366586823133.
Challenge Is Gone — “Now I’m completely absorbed in thinking about what to do rather than whether I can do it”.
* * *
“How I Design with AI” as a engineer who is not a designer and hates slop by Matt Dailey: part 1 and part 2.
How to scale intent, quality, and artistry with AI — Katie Dill from Stripe in a 20-minute video about designing things without allowing slop to grow.
Also, an update to Amp that adds tools for designing things: Visualize This.
