By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
World of SoftwareWorld of SoftwareWorld of Software
  • News
  • Software
  • Mobile
  • Computing
  • Gaming
  • Videos
  • More
    • Gadget
    • Web Stories
    • Trending
    • Press Release
Search
  • Privacy
  • Terms
  • Advertise
  • Contact
Copyright © All Rights Reserved. World of Software.
Reading: Now That You Have Attention, Context is All You Need: the Next Challenge to Solve In AI | HackerNoon
Share
Sign In
Notification Show More
Font ResizerAa
World of SoftwareWorld of Software
Font ResizerAa
  • Software
  • Mobile
  • Computing
  • Gadget
  • Gaming
  • Videos
Search
  • News
  • Software
  • Mobile
  • Computing
  • Gaming
  • Videos
  • More
    • Gadget
    • Web Stories
    • Trending
    • Press Release
Have an existing account? Sign In
Follow US
  • Privacy
  • Terms
  • Advertise
  • Contact
Copyright © All Rights Reserved. World of Software.
World of Software > Computing > Now That You Have Attention, Context is All You Need: the Next Challenge to Solve In AI | HackerNoon
Computing

Now That You Have Attention, Context is All You Need: the Next Challenge to Solve In AI | HackerNoon

News Room
Last updated: 2025/05/22 at 5:41 PM
News Room Published 22 May 2025
Share
SHARE

This is my sequel to the famous Transformers paper titled “Attention is all you need”. The technology described in that paper has led to almost every major breakthrough in AI in the past seven years, everything from ChatGPT to Google’s video generation model, Veo and everything in between.

The original paper was focused on helping computers (AI models) better understand and generate language. It describes how computers can derive meaning from a sentence by reading it all at once and using a mechanism called “attention” to decide which words are important.

For example, in the sentence “The cat that chased the mouse was hungry”, the model can connect the words “cat”, “mouse”, and “was hungry” to get meaning from the sentence while ignoring everything else.

Visualisation of how attention works. Credit: Google Brain team Visualisation of how attention works. Credit: Google Brain team 

The implementation of “attention” led to the creation of the powerful LLMs we have today.

As people have gotten more used to LLMs and made it a part of their lives, I have noticed that there’s still one challenge to maximising the usefulness of AI tools and models…

Context.

Context is the surrounding circumstances and background information that helps explain something.

It guides how you carry out tasks at work, it’s sometimes what makes a joke funny, it’s why Google search won, it’s everything.

A big challenge with AI models/tools at the moment is that they don’t always have context while carrying out certain tasks, and this impacts the quality of work done or answers provided.

Knowing what to take into consideration, how much impact it should have and when it’s no longer useful is the true test of intelligence.

Going back to our example sentence The cat that chased the mouse was hungry”, if we know how AI models apply attention, we can get more out of them by providing that information efficiently.

I’ll go as far as saying even if AGI is achieved, it isn’t relevant unless it can handle context well.

At the moment, various AI tools have tried to solve this in different ways. Memory and session awareness, RAG (Retrieval Augmented Generation), fine-tuning, MCP, and even creating prompt engineering guides.

However, I don’t think the problem has been properly solved yet because a lot of the current approaches put the burden of context on the user. This is not ideal because:

  1. It’s not always easy for humans to provide context because we sometimes don’t fully remember all the information and metadata we have, and use to make certain decisions.
  2. It negatively affects the user experience of AI tools if the end-user has to decide what to add, when etc. It’s like saying a joke but having to explain it to everyone every time.

I see a couple of ways that AI tools will try to solve the challenge of having enough context about how a person thinks of things:

  1. Constant presence: General-purpose AI models could start to provide ever-present tools similar to the Limitless AI pendant to constantly track places you visit, things you see, conversations you have, reactions you have to things and apply that in answering questions or performing tasks for you. This could also just be the next evolution of smartphones.

  2. Data import: In this approach, AI tools will provide mechanisms for users to import and sync their data from various platforms into their favourite AI tool. You’ll be able to sync your activity on X (things you see, like, post or repost), your work tools (emails, presentations, meeting recordings), your shopping data (items you’ve liked or bought on Amazon, Asos, etc)

  3. Implants: In the far future, I also think AI tools could explore a neuralink type brain implant that allows people ‘extend their brain’ and ‘get the full power of AI’. With this, they could get more context to be able to answer questions and even eliminate delays with using AI tools. Why do you need to think of a question, unlock your phone, open the ChatGPT app and type the question when you can just think of a question and get the answer?

The ethics around these approaches is a conversation on its own. But whichever way they decide to go, the next thing will be to figure out when to apply that context and how much weight it should have. Some platforms try to do this through fine-tuning, weights and pre-processing requests AKA Reasoning. I absolutely think this is the way to go.

However, I think it would be interesting to see dynamic weights. i.e AI tools using different weights and providing answers in a different way based on what they already know about you. Some sort of personalisation. Similar to how Google sometimes tailor answers to location and search history, etc.

I’m not sure what the next phase of AI looks like, but I‘m convinced that a strong understanding and application of context will be an important part of it. I’m excited to see how players in the space handle it.

Sign Up For Daily Newsletter

Be keep up! Get the latest breaking news delivered straight to your inbox.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Share This Article
Facebook Twitter Email Print
Share
What do you think?
Love0
Sad0
Happy0
Sleepy0
Angry0
Dead0
Wink0
Previous Article M&S to face disruption from massive cyber attack for another two months
Next Article Apple Watch with cameras reportedly canceled
Leave a comment

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Stay Connected

248.1k Like
69.1k Follow
134k Pin
54.3k Follow

Latest News

Hurry! One Month Left to Win from $5000 in the Web3 Development Writing Contest | HackerNoon
Computing
Finnish tech group ONEiO unveils UK expansion – UKTN
News
Firefox Maker Mozilla Discontinues Pocket and Fakespot to Focus on Browser
News
Mantle And Republic Technologies Forge Strategic Partnership For Institutional mETH Integration | HackerNoon
Computing

You Might also Like

Computing

Hurry! One Month Left to Win from $5000 in the Web3 Development Writing Contest | HackerNoon

4 Min Read
Computing

Mantle And Republic Technologies Forge Strategic Partnership For Institutional mETH Integration | HackerNoon

6 Min Read
Computing

ThreatBook Named a Notable Vendor In Global Network Analysis And Visibility (NAV) Independent Report | HackerNoon

4 Min Read
Computing

INE Security Partners With Abadnet Institute For Cybersecurity Training Programs In Saudi Arabia | HackerNoon

5 Min Read
//

World of Software is your one-stop website for the latest tech news and updates, follow us now to get the news that matters to you.

Quick Link

  • Privacy Policy
  • Terms of use
  • Advertise
  • Contact

Topics

  • Computing
  • Software
  • Press Release
  • Trending

Sign Up for Our Newsletter

Subscribe to our newsletter to get our newest articles instantly!

World of SoftwareWorld of Software
Follow US
Copyright © All Rights Reserved. World of Software.
Welcome Back!

Sign in to your account

Lost your password?