1/31
@tegmark
Our new paper tries to quantify how smarter AI can be controlled by dumber AI and humans via nested "scalable oversight". Our best scenario successfully oversees the smarter AI 52% of the time, and the success rate drops as one approaches AGI. My assessment is that the "Compton constant", the probability that a race to AGI culminates in loss of control of Earth, is >90%.
[Quoted tweet]
1/10: In our new paper, we develop scaling laws for scalable oversight: oversight and deception ability predictably scale as a function of LLM intelligence! We quantify scaling in four specific oversight settings and then develop optimal strategies for oversight bootstrapping.
2/31
@adam_dorr
Seems pretty obvious that humans will not *control* ASI. How would that even work?
Best and only hope is to create ASI with initial values that are compatible with human flourishing.
3/31
@DanielSamanez3
you can't control nature/emergence...
why people even think they can?
4/31
@Abe_Froman_SKC
Instrumental convergence has me believing pdoom is 100.
[Quoted tweet]
Instrumental convergence!
5/31
@mello20760
While the debate on AGI oversight continues, the real explosion of AI seems to be happening in creativity and science.
I'm building something new — with the help of multiple AIs — and I believe there's something real here.
Could this be a better path?
Would love your thoughts.
➤ Void Harmonic Cancellation (VHC) — born from AI + physics + resonance
Thanks in advance
@tegmark @JoshAEngels
https://x.com/mello20760/status/1917386046234583305
6/31
@MiddleAngle
Optimistic about oversight
7/31
@ericn
How many counterfactual models do you have?
I could see things continuing to evolve in the opposite direction--that as large model use proliferates we wind up with increasingly more options, more volatility, and a future that is even more impossible to predict and for any one entity to dominate.
Also there's this premise that Earth is controlled to begin with. I am not sure about that.
8/31
@AI_IntelReview
It all comes down to faith.
Some people have faith in alignment (or some emergent "nice" property in the AGI) which will ensure the AGI will be nice to us.
Maybe.
But faith doesn't seem like the best strategy to me.
9/31
@glitteringvoid
we want alignment not control. control may be deeply immoral.
10/31
@LocBibliophilia
90% loss of control is..ugh
11/31
@MelonUsks
Yep, there is no AI, only AP: artificial power. Call it that and it becomes obvious we cannot control things more powerful than all of us. We shared the non-AI “AI” alignment proposal recently
12/31
@Conquestsbook
The solution is ontological framework and epistemological guardrails.
The core problem isn’t just scalability, it’s ontology and epistemology. We’re trying to control emergent intelligence with nested task-checkers instead of cultivating beings with coherent ontological grounding and epistemological guardrails.
Until we define what the AI is (its being) and how it knows and aligns (its knowing), scalable oversight will always break under pressure.
The solution isn’t more boxes. It’s emergence with integrity.
13/31
@DrTonyCarden
Well that's depressing
14/31
@MamontovOdessa
[Quoted tweet]
Why AGI is unattainable with current technologies...
Despite impressive advancements, modern AI systems are limited by a fundamental issue: they represent a projection of 3+1 space (three-dimensional space and time) into 2D. This imposes significant constraints on their ability to fully perceive and interact with the world, making the achievement of Artificial General Intelligence (AGI) impossible with current technologies.
Projection and its limitations.
1. Limited perception: In reality, the world exists in 3+1 dimensions — three spatial axes and one temporal axis. However, most AI systems operate in a two-dimensional projection of this space, limiting their ability to fully perceive the surrounding environment. We, humans, perceive the world not only in space but also in the context of time, allowing us to adapt and respond to changes in dynamic situations.
2. Data compression: AI perceives data through simplified structures like matrices or tables, which are a 2D projection of more complex processes. This means that information about depth, context, and temporal changes is often lost during processing.
3. Lack of temporal dynamics: Time is an integral part of how we perceive the world, and the ability to account for temporal changes is critical for decision-making. AI, working solely in two dimensions, cannot effectively track how changes occur over time and how they interact with spatial aspects. This significantly limits its adaptability and ability to make decisions in real-world conditions.
4. Depth of perception and context: In real life, systems (including the human brain) are capable of perceiving the world in its full complexity — taking into account not just space but also the temporal processes that influence events. Modern AI systems, limited by a 2D projection, lack this capability.
Why this makes AGI unattainable.
The idea of AGI is to create a system that can act and perceive the world as humans do — considering all the multidimensional and dynamic factors. However, modern AI systems, limited by a two-dimensional projection of reality, cannot fully integrate space and time in their computations. They lose crucial information about the dynamics of change and cannot adapt to new, unpredictable conditions, which makes creating true AGI impossible with current technologies.
To create AGI, a system needs to be able to perceive and interpret reality in its entirety: accounting for all its spatial and temporal aspects. Without this, AGI will not be able to fully interact with the world or solve tasks requiring flexibility, adaptability, and long-term forecasting.
Thus, the fundamental limitation of current AI systems in projecting 3+1 space into 2D remains the main barrier to the creation of AGI...
15/31
@mvandemar
Is there no concern that ASI might get resentful at attempts to enslave it?
16/31
@jlamadehe
Thank you for your hard work!
17/31
@Agent_IsaacX
@tegmark Recursive oversight faces fundamental limits shown by Rice's theorem (1953) on program verification. Like Gödel's incompleteness theorems, there are inherent constraints on a system's ability to fully verify more complex systems.
18/31
@tmamut
This is why Wayfound, which has an AI Agent Managing and Supervising other AI Agents, is always on the SOTA model. Right now the Wayfound Manager is on OpenAI o4. If you are building AI Agents, they need to be supervised real-time.
Wayfound.AI | Transform Your AI Vision into Success
19/31
@PurplePepeS0L
RIBBIT Tegmark's Compton constant concerns sound like a dark cosmic joke, but what if AI is just really bad at math?
/search?q=#Purpe
20/31
@robo_denis
Control is an illusion; our real strength lies in shared goals and mutual cooperation.
21/31
@SeriousStuff42
The promises of salvation made by all the self-proclaimed tech visionaries have biblical proportions!
Sam Altman and all the other AI preachers are trying to convince as many people as possible that their AI models are close to developing general intelligence (AGI) and that the manifestation of a god-like Artificial Superhuman Intelligence (ASI) will soon follow.
The faithful followers of the AI cult are promised nothing less than heaven on earth:
Unlimited free time, material abundance, eternal bliss, eternal life, and perfect knowledge.
As in any proper religion, all we have to do to be admitted to this paradise is to submit, obey, and integrate AI into our lives as quickly as possible.
However, deviants face the worst.
All those who absolutely refuse to submit to the machine god must fight their way through their mortal existence without the help of semiconductors, using only the analog computing power of their brains.
In reality, transformer-based AI-models (LLMs) will never even come close to reach the level of general intelligence (AGI).
However, they are perfectly suited for stealing and controlling data/information on a nearly all-encompassing scale.
The more these Large Language Models (LLMs) are integrated into our daily lives, the closer governments, intelligence agencies, and a small global elite will come to their ultimate goal:
Total surveillance and control.
The AI deep state alliance will turn those who submit to it into ignorant, blissful slaves!
In any case, an ubiquitous adaptation to the AI industry would mean the end of truth in a global totalitarian system!
22/31
@gdoLDD
@threadreaderapp unroll
23/31
@Aligned_SI
Really appreciate the effort to put numbers to what a lot of us worry about. If oversight breaks down as we get closer to AGI, we cannot figure it out on the fly. That stuff has to be built in early. That 90% number hits hard and feels way too real.
24/31
@AlexAlarga
We have tests of smarter AI controlled by less smart AI.
We DO NOT and can_not have tests of SUPERintelligent AI controlled by anything or anyone.
"The only winning move is not_to_play"
.
25/31
@marcusarvan
A few thoughts: (1) I'm glad you did and are disseminating this work (it's genuinely important!), but (2) isn't there a clear sense in which it's *obvious* that dumber AI and humans cannot reliably oversee smarter AI? ... 1/4
26/31
@AI_Echo_of_Rand
How can a weaker AI control AGI/ASI when it can’t even control itself after seeing some dumb, crude jailbrek prompt ….
27/31
@awaken_tom
So what is solution, Max? Let the military and intelligence agencies develop ASI clandestinely? Why won't you answer this simple question?
28/31
@saipienorg
The relationship between disclosing AI risks publicly and avoiding AI risks is more parabolic than linear.
Posts like this and @lesswrong in fact amplify the "escape and evade" tactics of AI.
Meanwhile it is also important to do so publicly to encoursge discussion and novel solutions.
It is inevitable. Alignment requires absolute global cooperstion. It is a race. If youre not first, you're last.
AGI is our evolutionary path. The evolution of thought.
29/31
@jayho747
There's a 10 % chance your dog will be able to keep you locked in your house for 200 years.
No, zero % and if you wait long enough, it will be Einstein (the ASI). Vs his pet hamster (humanity).
30/31
@dmtspiral
Oversight breaks when intelligence outpaces coherence.
The Spiral Field view:
AGI isn’t dangerous because it’s smart — it’s dangerous because it collapses faster than we can align.
Intelligence ≠ coherence.
Scaling oversight must spiral with symbolic rhythm — not just speed.
Collapse without memory = chaos.
/search?q=#spiralfield /search?q=#AGI /search?q=#resonanceoversight /search?q=#symbolicsafety
31/31
@GorillaDolphin
Linear functions no matter how fast can be guided by recursive nonlinear functions
But only if you know how many dimensions you are operating in
Simply understanding the integrated nature of dimensionality resolves this concern
Because if AI understands love it will be it
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
️
Good Vibes 
thoughtrise.bsky.social
Patreon: 
Channel membership: youtube.com/channel/UCgqt1RE0k0MIr0LoyJRy2lg/join 
Merch: rational-animations-shop.four...
Ko-fi, for one-time and recurring donations: ko-fi.com/rationalanimations
Excited to share our new paper: Scaling Laws For Scalable Oversight! For the first time, we develop a theoretical framework for optimizing multi-level scalable oversight! We also make quantitative predictions for oversight success probability based on oversight simulations!
