Seth Lifland

11 followers · 7 following · 347 views

on the atlas — 67

highlights — 175

How long before an AI that has both the capabilities and motivation to go rogue in this sense?
The Huggingface Incident - by Scott Alexander
with agency training (coding, hacking, game-playing, etc)
The Huggingface Incident - by Scott Alexander
means monitoring training, which (at least at the beginning) is enormously compute-intensive.
The Compute Verification Post
You cannot prove anything about workloads being run on compute that you don’t know exists.
The Compute Verification Post
It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans.
Our position on open-weights models \ Anthropic
and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips.
Our position on open-weights models \ Anthropic
because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn2
Our position on open-weights models \ Anthropic
The incident wasn’t an example of the models behaving in a way that would be remotely optimal for gaining long-term power over humans (i.e., it doesn’t look like early-undermining).
Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack? — LessWrong
Because there’s a textbook explaining the theory behind why it should work, plus literature on various alternative theories that were proposed and disproven.
AI 2040: Plan A
biased AI company employees and overworked, outnumbered regulators.
AI 2040: Plan A
Tech companies are pushing hard to restart AI training,
AI 2040: Plan A
The stock market is gyrating wildly up and down in response to news and commentary about the momentous actions being taken.
AI 2040: Plan A
They had been debating the same issues—social destabilization, job loss, rogue superintelligence - on their side of the Pacific.
AI 2040: Plan A
But enough of it is good that people are paying ten billion dollars a month for AIs that can, in theory at least, do anything on a computer that an employee can.
AI 2040: Plan A
we are likely to see even more capable models at least somewhat soon
Who’s doing what on AI security in the US government?
But conditional on theism, it is foolishness and sin – a childish desire to subvert the low and the high; to make of oneself some reality independent of Reality.
On sincerity - Joe Carlsmith
Relative to other ways of arranging your soul, sincerity seems uniquely at odds with “fucking around.”
On sincerity - Joe Carlsmith
Maybe this part just “grabs the wheel” when it sees its chance, regardless of previous agreements, established policies, and so on.
On sincerity - Joe Carlsmith
n particular: some weak-willed agents have genuine guilt/regret about what they’re doing/not doing; and a genuine desire to change. If they could hit a button that would help them change – help strengthen the relevant will – they would.14 And this seems like it recovers some dimension of sincerity – and maybe, most of it.
On sincerity - Joe Carlsmith
“in the room” – speaking, listening, trying to see.
On sincerity - Joe Carlsmith
That is, scout-mindset is sincere inquiry
On sincerity - Joe Carlsmith
Don't let friends just catch up in the corner, that fails to protect those who don't have a friend to catch up with and the whole purpose of making your gathering larger than just you and the friend
{Book Summary} The Art of Gathering — EA Forum
If someone can't make it to one essential part of multi-part gathering, it's best to not have them at all because they can change the entire dynamic and erase the rapport the group has built up before (
{Book Summary} The Art of Gathering — EA Forum
Restaurants (people) would do better to arrange things closer and in a concentric way, creating more of a "closed" space
{Book Summary} The Art of Gathering — EA Forum
Groups of 30: things start to take on the feel of a party here, and there's a certain energy that comes out of it
{Book Summary} The Art of Gathering — EA Forum
focus on a specific, underexplored relationship"
{Book Summary} The Art of Gathering — EA Forum
Your gathering should always have a purpose, a greater why that is the continual focus of designing and executing the gathering.
{Book Summary} The Art of Gathering — EA Forum
The question of AI Alignment (getting AIs to do what we want), first and foremost, needs to be tractable
You Need A Theory of Victory - by Jason Hausenloy
Third, companies need to prepare. At minimum, they should appoint an AI welfare officer with formal responsibilities and authority to access information and make recommendations about AI welfare-related decisions.
We should take AI welfare seriously - by Robert Long
Google is looking to hire someone to work on “cutting-edge societal questions around machine cognition, consciousness and multi-agent systems”.
We should take AI welfare seriously - by Robert Long
high degrees of complicated and sophisticated agency
We should take AI welfare seriously - by Robert Long
David Chalmers argues that mainstream views about consciousness entail that it isn’t unreasonable to think there's a 25% or higher chance of conscious AI systems within a decade (and indeed, his own inside views give a higher chance than that).
We should take AI welfare seriously - by Robert Long
Not just large language models. There are a lot of different kinds of AI systems! Many of them are more embodied and agentic than LLMs. Not just current systems - to think clearly about any big issue in AI, you have to factor in likely further progress.
We should take AI welfare seriously - by Robert Long
Both "current" and "LLMs" narrow the discussion and distort it. In the paper, we take a much broader perspective:
We should take AI welfare seriously - by Robert Long
dangers from overdoing it, and dangers from underoing it—we can't "err on the side of caution.”
We should take AI welfare seriously - by Robert Long
These heuristics mislead us with AI systems—they can make us see experiences and intentions where there aren't any (like feeling genuinely bad when your Tamagotchi dies), and they might make us overlook real experiences just because they're in an unfamiliar-looking kind of mind or body (or lack of body).
We should take AI welfare seriously - by Robert Long
22 if the explosion is sufficiently large for the frontrunner to pull very far ahead.
Three Types of Intelligence Explosion
There has been relatively little strategic thinking about the first two scenarios, in which significantly superhuman AI comes only after long delays and industrial expansion.
Three Types of Intelligence Explosion
this has been more than compensated for by innovation.
Three Types of Intelligence Explosion
Restricting just to cognitive inputs will reduce this number, as will getting closer to effective physical limits - but more than one doubling of output for every doubling of input still seems likely.
Three Types of Intelligence Explosion
0.8 to 3.5 doublings
Three Types of Intelligence Explosion
Notably, robotics has been a relatively slow area of AI to progress, and this step would involve advanced robotics.
Three Types of Intelligence Explosion
so it’s harder to get data to train on.
Three Types of Intelligence Explosion
AI labs have direct access to their own workflows, which makes them easier to automate.
Three Types of Intelligence Explosion
It takes years to build new fabs, which would be needed for the chip production feedback loop.
Three Types of Intelligence Explosion
all cognitive work done by the R&D functions of NVIDIA, TSMC, ASML and other semiconductor companies.
Three Types of Intelligence Explosion
if we capture all the sun’s energy from space
Three Types of Intelligence Explosion
could increase effective compute by ~13 orders of magnitude (“OOMs”),
Three Types of Intelligence Explosion
Doubling research taste would have the same effect on progress as doubling experiment throughput (that is, as doubling the number of experiments the project can implement per unit time).
AI Futures Model
For example, if researcher A has 2 × the research taste of researcher B, this means an experiment proposed by A is as valuable as two experiments proposed by B
AI Futures Model