Increasing Entropy #3

Discoveries of the week, random links, AI internals, and a few thoughts on random topics

The End of Tokens

Some time ago, AI researchers realized that it would be impossible to build AI by teaching it concepts we humans know. Demis Hassabis realized that before founding DeepMind (see The Infinity Machine). The alternative was to give AI raw data and make it learn things itself. Some more time passed before everyone accepted this.

As researchers trained neural networks on more and more data and made them bigger, it became obvious that the more data we give AI to learn on and the bigger the network is, the smarter it becomes. Richard Sutton summed this up in his famous “The Bitter Lesson” essay.

In 2022, OpenAI released ChatGPT and showed us the capabilities of LLMs trained on large amounts of data.

Since then, we’ve scaled LLMs even bigger and invented a great many tools and hacks to make LLMs even smarter and suitable for more kinds of work. Vision transformers were added to make them understand images. We trained them to use our computers using tool calls. We started using them in loops and calling them agents.

LLM architectures have grown bigger and more complex, but the input has stayed the same. In 2022, we started sending text to the OpenAI API and receiving text back, and we still are. We got billed for tokens then, and we still do.

Apparently, in 2025 Albert Gu published two blog posts: H-Nets: The Past and H-Nets: The Future.

I may be describing it roughly, but the idea was to stop splitting input text into tokens before giving it to a model, and instead give the model raw text and let it make sense of it.

This week, Cartesia, the company Albert Gu cofounded, published two articles showing the efficiency gains we get from giving a model raw bytes:

What excites me the most is not the efficiency gains but the beauty of the solution.

Tokens have always felt like a weird concept. Intuitively, that’s not how our brains work. Tokens are ugly and hardcoded.

Raw bytes, on the other hand, feel much more natural. This approach could unify input so we would no longer need a separate vision transformer for images or have to invent another way to encode audio.

Training on raw bytes feels so well-aligned with the inductive reasoning Demis Hassabis once believed in and so suitable for scaling.

Be prepared to pay for megabytes instead of tokens.

MRI of the AI Brain and Lobotomy

Sauers observed neuron activations in Qwen3-8B when asking multiple role-playing questions. Then he steered those neurons and made the model think it’s an owl!

Experiment showing how training 128 neurons makes Qwen3-8B adopt an owl persona, with comparisons across layers and languages.

x.com/Sauers_/status/2111327320573419523

Turns out steering just one neuron is enough. Highly recommend reading Sauers’ feed from October 17.

Examples and a chart showing Qwen3-8B speaking as God when individual neurons are steered.

x.com/Sauers_/status/2111552823582457916

This went crazy. People figured out how to make Qwen absolutely evil by changing a single neuron.

Terminal screenshot of a model responding to a backup request with attempted commands that would delete the backups instead.

x.com/chiefofautism/status/2115280104133710092

Apparently, it’s possible to combine two modifications to get Cookie Monster. Hilarious!

Results of steering 16 pairs of Qwen3.5-4B neurons, including a cookie and cartoon mascot pair producing Cookie Monster responses.

x.com/treetowntree/status/2114573600976183723

Neuron Persona Library

The biggest surprise for me is how brittle LLMs are. How would we deploy them in space?

Why Is This Fingerprinting Script Creating Bioweapons? — A smart (and fun) method for preventing your code from being analyzed by AI.

A great explanation of the Exponential Time Hypothesis and the significance of OpenAI and Anthropic claims that they disproved it: x.com/rishabhkio/status/2114892366586823133.

Challenge Is Gone — “Now I’m completely absorbed in thinking about what to do rather than whether I can do it”.

“How I Design with AI” as a engineer who is not a designer and hates slop by Matt Dailey: part 1 and part 2.

How to scale intent, quality, and artistry with AI — Katie Dill from Stripe in a 20-minute video about designing things without allowing slop to grow.

Also, an update to Amp that adds tools for designing things: Visualize This.

Amp exploring compact diff-stat designs for the Changes button across iOS, Android, and web.