Attention Is All You Need

2026-06-01

Thoughts of the Thinking Variety

A few days ago, while working my math research job, I was struggling with a theorem that I needed to prove. I had no direction on how I was going to prove it. And I really needed to prove it because I had planned on it being proven during the previous week, and it was blocking my progress. My research advisor's expectation of that progress along with the stress and cognitive load of working on another side gig, applying to jobs, working on my portfolio, as well as other basic personal responsibilities all contributed to a sort of analysis paralysis that I was experiencing. So many things were crawling inside my head, yet my mind was devoid of any full thoughts. I was listening to music, shifting positions in my chair, checking phone notifications, sifting through irrelevant parts of my simulation code--anything to escape this stress, while simultaneously worsening it. I truly could not reason clearly about this proof.

"Here's the key insight..." - Claude Opus 4.6

It would be natural to guess from the heading of this section that I may have used an LLM to prove this troublesome theorem. We all have our weak moments. And in my despair, I did think about it. But I didn't do it. Something deep inside of me couldn't see myself succumb to the easy way out. I would've robbed myself of the understanding of the problem. I would've tried to convince myself that I understood the generated proof at a level that I really didn't. Instead, I rolled my chair to my writing desk, turned off my music, sat up straight, took out a pen and paper, dove into the problem, and proved it myself. Turns out it was a 3-line proof. It took less than 5 minutes. Afterwards I went outside in the sun for a few minutes to just bask in the comedy of what just happened and the freedom from that week-long blocker (to be fair, I usually only work this job on weekends). I not only achieved an understanding of the problem that was only possible by thinking of the proof myself, but the experience also got me thinking about the big picture of what just happened...

Attention Economy

The reason that I struggled with the proof was not its difficulty, but my lack of attention. In particular, I could not focus my cognitive effort into the task at hand which caused me to miss the blatantly simple solution. In his book, The World Beyond Your Head: On Becoming an Individual in an Age of Distraction (2015), Matthew Crawford says this about attention:

"In the main currents of psychological research, attention is treated as a resource--a person has only so much of it. Yet it does not occur to us to make a claim for our attentional resources on our own behalf."

Crawford goes on to describe what he calls attentional commons:

"There are some resources that we hold in common...we take them for granted...I think the absence of noise is a resource of just this sort. More precisely, the valuable thing that we take for granted is the condition of not being addressed. Just as clean air makes respiration possible, silence, in this broader sense, is what makes it possible to think."

Here, Crawford frames attention as a resource that is stifled by external noise, which I think is accurate. However, I think that it is useful, especially as the 'age of distraction' has accelerated, to also frame attention as a resource that is stifled by more internal, personal mechanisms.

Nowdays, I feel less distracted by advertisements, notifications, or general external noise and more distracted by my own internal noise and urges telling me that I need to think about xyz thing. And in this way, 'claiming our attentional resources on our own behalf' is more of a form of self-mastery and less of a protection of a common resource.

Large Language Models and Learning

In 2017, a team of Google Research and Google Brain researchers published a now-famous paper titled Attention Is All You Need. The paper introduced a new kind of neural network architecture that they called the Transformer. Previously with sophisticated neural networks, especially those designed for natural language processing, components like recursion or convolutions (not important to know that means) were used, as well as an 'attention' mechanism which allowed the model to learn which parts of its input are more important than others. This new Transformer architecture did away with those other components and showed that models actually performed better with only an attention mechanism (as well as some other developements, but this was the main one), hence the name of the paper. Unbeknownst to the Google researchers, this Transformer architecture would underpin OpenAI's first GPT (Generative Pre-trained Transformer) model in 2018, which we now know as the start of the Large Language Model revolution.

If you are in education, know someone in education, or have been educated recently, then you are probably under the impression that kids just don't learn like they used to. As a recent college graduate, the sheer amount of students in my classes that I could tell knew almost nothing about the subject they're studying is worrying. I talk to highschool teachers and hear essentially the same story. But it isn't just in education. It's happening to people in all walks of life: professionals, artists, researchers, etc. And I get the feeling from my own experiences that the anecdote I shared about the math proof is somewhat telling: that what we're missing is attention. When was the last time you sat down and just wondered about something? Read every word of an interesting article and applied it to yourself? Or internalized a subject to such a deep level that you will never forget it? This kind of learning and understanding requires attention. It's not something you get from a LLM-generated summary (or any summary for that matter). It's not something you get from watching a YouTube video. You can't even get it from passively reading. You get it from becoming intimate with a subject. You get it from a silent, unadulterated, independent focus on doing. You have to make--you have to explore--to understand.

We're at an ironic point in history where humans are losing control over their attention, while machines are, in a sense, learning to harness their own. So how do we take back our attention?

The most immediate answer is to identify the noise-makers. The attention-siphons. I've personally figured out that I am bad at moderation in most everything I do, so I put hard limits on certain attention sinks. For example, my most basic hard limit is a 30-minute screen time cap on specific apps on my phone. I also have no social media on my phone. If I need to see social media for some reason, I am forced to use my computer. And my cookies are cleared after I close my browser, so I have to re-login every time. But nefarious attention sinks can be less stereotypical than something like social media. Maybe you spend too much time watching podcasts or playing games. Maybe you're incredibly busy and stressed, and stretching your attention thin. You might be surprised how much higher-quality your other work becomes when you drop some unnecessary projects. But of course, it's a trade-off.

Once you gain more control over your attention, you should become more intentional with where your attention actually goes. Maybe you set aside time for yourself to simply think. Or maybe you become more honest with yourself and challenge your understanding of something: force yourself to practice. Most concretely of course, you should create, build, and explore things in-depth to gain understanding. However far you decide to take this, you should keep in the back of your mind that attention is all you need.