But to many commentors he's a now gatekeeping AI-hating Luddite clinging to a dying profession out of bitterness and envy because his position is more nuanced than "throw AI at everything and turn off your brain".

And yet as a society we typically consider these technologies a net positive these days, because alongside all the political instability and violence that followed them at first, some people were figuring out how to do beneficial things with them along the way that did more good than bad. Because the issue really was that the influx of more voices and ideas shifted the power dynamic, required relearning how to communicate, and in the end be better as a society.

The issue was figuring out how to "domesticate" these wild new communication channels. One successful example of which was the invention of scientific journals and papers.

Shirky predicted the internet would probably lead to a few decades of political instability too (about fifty years was his guess), and we definitely seem to be in the middle of that process.

Now, I haven't checked his stance on LLMs. I also don't know if I would quite call them a medium for mass communication like the others (as used today, they take humans out of the loop rather than let more voices join the public discourse), but I feel like they are similar enough to otherwise fit the pattern.

Tao's stance similarly feels about wanting to domesticate this wild animal before we get mauled by it. And to stick with the metaphor, I feel that most of the time the AI industry is trying to bamboozle us with spectacular rodeo displays instead.

That is probably the wildest misinterpretation of sth I have ever read here. Misalignment is used as in the goals and interests of ai COMPANIES are not the same as the ones of the mathematical community. It is very proper use of the word, and if anything imo the properest, as it refers to actual people and institutions to which actual self-ascribed goals can be defined, as in contrast to hypothetical superintelligence. AI doomers do not own some trademark on the word "alignment".

- a rare low likelihood sequence of coincidences

- deliberate frontier lab behavior, setting intellectual interrupts on LLM "aha"-moments, so that when some mathematician explains for the umpteenth time the approach they want to take "stop paraphrasing my approach, start the actual calculation!" and when the chatbot eventually groks it, they can scoop in, possibly with live dash-board and interrupt priority levels ranging from "ensure this sample gets into the new dataset" to "Millenium Prize Scoop Opportunity Imminent, call Sam"...

There is always an alignment problem, incentives to ideals, behavior to incentives, ...

The word is being used entirely correctly, and I'm sure the nod to the AI usage is quite deliberate.

This is exactly the assumption that Tao is smuggling in with "misalignment" talk and then refusing to elaborate on any further. Is the issue that AI companies are willfully refusing to provide mathematical insight that they could provide (because they have diverging underlying "goals" to those of human mathematicians) or are they merely working under a capability gap, where current AIs can awkwardly settle major open questions but are not smart enough to provide the kind of understanding and insight that the community of human mathematicians relies on? These are two very different problems and by foregrounding the word "misalignment" in his letter so openly (as opposed to talking about AI capability to provide valued insight), Tao is picking the more adversarial reading with zero proof or motivation.

AI can be used to advance/deepen understanding, or it can be used to superficially go settle a whole bunch of open problems in a field without helping really in understanding them. It all depends on who uses the AI and why. Essentially, it is exactly the same concept as using the AI as a course tutor vs having it do your homework. Or using the AI to write millions of lines of code that nobody can actually read, vs keeping overview of what is going on.

In math it is probably worse because there is no objective function to maximise. Some people here think that the objective function of mathematics is to prove things, which is actually wrong. Mathematicians are not only maximising an objective function, they are also defining the objective function they need to maximise (they are defining which problems to study). The problem with AI/ML is that it can be pretty good when the goal is to maximise a set objective function, but not to set intentions and goals themselves. I would not call that a "capability gap" because we can actually get to have very useful and smart AI systems without ever reaching that point.

Again, nobody is accusing anyone of deliberately trying to harm mathematics. It's about misaligned objectives. Eg. Tao wrote the following before the Navier-Stokes announcement (referring to exactly the project OpenAI was undertaking):

> At this point, I would not be surprised if one could batter out such an extension by pouring an enormous amount of compute and AI assistance at such a task. But such an exercise does not particularly hold my interest; I am far more interested in digesting the proof methods and extracting out the key new insights uncovered by this approach.

And then went further to say that such activity could be actively harmful to the field. (All this can be read on Mathstodon: https://mathstodon.xyz/@tao).

The misalignment is that this activity that he and 24 other Fields medalists think is actively harmful to their field is deemed by OpenAI and others to be worth ploughing vast financial, human and compute resources into.

Though it doesn't take even an outsider to question the matching here, just look what the only guy who settled a millennium prize what he thought about the community. Or ask some actual PhD, postdoc, or even better someone who dropped along the way, how does it feel to go through this community. Not sure the goals exposed in this article are really helped with the community as it is. One again, this doesn't mean everywhere encompasses the same issues and everyone is acting badly on is own.

But if we want to accept happy pink shiny depiction of a community, we should be giving the same generosity to other communities à priori. All the more as AI and mathematical community do share a large set of common individuals.

The online comments I’ve read that side with the letter explain that the problem is AI proofs are inscrutable and useless, but the reality is using computers to brute-force things has long been part of proofs in one way or another, and AI proofs range in legibility up to 100% (like pointing out a proof exists in a long-lost paper, or writing something that is basically correct and just needs a human reviewer to fix it up). People are probably defending the declaration inaccurately, but I think that’s because the real argument or arguments are unclear.

Is it that students won’t learn math if there’s ChatGPT? That could be discussed.

To a non-mathematician (MIT physics and CS ‘06), the letter sounds like, “We have a fun job. Sometimes there are no practical applications of the work, so it’s just kind of like a sport, er I mean science. If someone solves a problem that has stumped mathematicians for decades or centuries, we worship them as a great mathematician. It’s a status thing. So we don’t like some guy with a computer coming along and solving our problems. The way things unfold with all the ideas coming from humans, over time, maybe it’s slower, but it’s nice.”

I don’t think anyone can really stop someone from using a computer capable of solving unsolved problems to solve unsolved problems, and there are always going to be mathematicians who DO think it’s fun to try to figure out what a computer is doing (if the computer isn’t already explaining it in English, which it generally can), and make progress that way, and it obviously will lead to mathematical advances in human understanding, from my point of view. So the whole thing is a non-issue that no one can do anything about anyway.

That's said, I'm a non-mathematician, but a computer scientist. Mathematics was always my weak part, because solving problems which doesn't mean anything doesn't motivate me, and when I'm not motivated, I fail.

Without digressing so much, I want to say that, some of the things in mathematics and mathematics adjacent sciences baffle me. Many mathematicians don't understand the proofs of others, yet they accept it since it checks out within the rules of mathematics. Moreover, many engineers don't understand the formulae they work with. I have developed a very performant Boundary Element Method evaluator, yet I don't know the reason of taking Gaussian Integrals over the surface. The only answer I got is "because the method works that way".

So, if computers can make the proofs now, and we have no curious students pecking their professors to understand how these works, the whole mathematics as a science could wither and die. Because besides it being a sport, it's the very language which can model and explain anything, but it's very hard to master and understand due to that nature.

Our professors warned us against using too much Mathematica, to make us learn things the hard way and make the knowledge permanent. Now, if we offload science to a matmul engine, it's possible that we lose the connection and never able to close the gap in some cases.

Again, it boils down to "This machine has no brain, use yours (or lose it)".

So let's just quickly agree that the actual quote is “However, the push by AI companies to solve mathematical problems as a benchmark is detrimental to the science of mathematics, and to the mathematical community.” And that in this, "as a benchmark" is load-bearing.

We wouldn't see nearly the same amount of contempt from researchers had OpenAI picked a research-friendly approach.

What they did: Hear a rumour about the problem being solved by other researchers, then rush to scoop them (unethical), then, when they actually go talk to them, they try to oust an author (also unethical), and when they finally decide to share their own work, do so in the least useful way possible.

What they could have done: Upon hearing the rumours, connect with the researcher in question and propose that they join efforts instead; set up a joint project to test if the machines are useful in any way, and if that's not appreciated, back down again. And instead of dumping only an undigested paper* and a Lean proof, do the digestion prior to publishing anything (as Buckmaster was in the process of doing). If their own lack of competences was keeping them from digesting it, then again, reach out to the researchers to understand if anyone would be willing to do so.

In the second of those two worlds, we wouldn't be seeing nearly the amount of outrage that we are seeing right now.

*: Here, “digestion” is the process of turning an AI slop paper into something humans can read. LLMs can indeed sometimes (if much more rarely than marketing material from the large LLM companies will suggest) produce correct proofs, but they are often written in bizarre ways – they'll use lingo that doesn't exist, seem overly pretentious, dwell on extremely easy steps while glossing over the hard ones. Currently, a real researcher will take that output and transform it into something that others can understand, use, and build upon. This is not so different from what happens when using it to write software, although as someone who does both, I will say that the amount of digestion needed for proofs tends to be orders of magnitudes larger than for code. This meme is quite accurate: https://mathstodon.xyz/@tao/117068266071803252

My point is just that I feel a little optimistic that the human culture around math ("ideas we disseminate in talks, private discussions and careful writeups, connecting them to the previous ideas of others" and so on, to quote the declaration) is robust even against relatively irresponsible use of AI (i.e., an onslaught of proof-slop).

And I guess we'd better try to be optimistic, because even if the declaration results in some realignment between the community and big AI companies, the models capable of this work are not always going to be exclusively under the control of those aligned parties.

That said, I think the declaration is great and I support it--let's see what comes out of it.

Yes, the trailing edge will catch up quickly.

I wonder how long until P vs NP falls.

Lee Sedol said in an interview that "losing to AI, in a sense, meant my entire world was collapsing. ... I could no longer enjoy the game. So I retired", and I think there will be folks in the mathematical community who would feel the same when the solutions pages to hard problems are suddenly available.

But on the other hand, people learned a lot from chess engines. After decades of chess computers beating humans, there was still a renewed interest in watching Leela beat Stockfish, with many people trying to understand the strategy Leela used.

If your happiness comes from grinding on a problem and making progress, the prospect of having to dig through a corpus of AI-generated proofs might be hard to swallow. But if you're willing to do that, you will still find beautiful things that only so many people can truly appreciate.

Carlsen is bored by studying engine lines.

The popularity is boosted by YouTubers because chess is very suitable for somewhat higher class content.

I'm not sure we'd want that world for math. Positions will be cut just like archaeologist positions are cut now.

There was a renaissance during Covid and due to 'The Queen's Gambit' where it gained much more mainstream popularity, but... Chess AI was already far far (like 1000+ Elo) ahead of human players at that point.

The thing is... chess is humans playing (communicating) with humans and that's what keeps it interesting. Check out the view counts of chess AI tourneys vs. human tourneys.

I'm not sure what the equivalent would look like in the math field, but it probably involves a lot of mathematicians losing their jobs and the quality of human-produced math decreasing overall.

The quality of the math in general would be fine, since in this scenario cpus will keep producing it. The quality of cpu-cpu chess games is quite high, beyond human understanding in many cases.

Chess is a weird example because it doesn't really have any utility beyond itself. Even pure math sometimes ends up having use in the strangest places. Although if no one understands the frontier math (because no one is getting paid to), I'm not sure it even matters what the quality of the cpu math is?

It's a bit like a tree falling in a forest. If an LLM proves a theorem but no one understands it, did it make a sound?

Presumably AI will be connect the dots to the applications. As the declaration says, this isn't just about math. Human understanding is losing economic value. You can understand stuff on your own time, I guess.

The standard justification for pure math to holders of purse-strings is something like "it might lead to a useful application down the road, like crypto, who knows". That looks pretty inefficient now. We have to entertain the possibility that AI can develop the math needed for any application we put to it. Eg if number theory didn't exist, we could have asked AI for a way to transit messages securely and it would maybe come up with fermats little theorem as part of its solution or maybe come up with an approach we can't conceive of right now seeing as most of us are constrained to available number theory. Like how in the last year when I give an LLM a programming project I see it doesnt even bother with of the many software libraries I and others have written and just codes up the calls it needs on the fly or finds some other ad hoc solution.

But in future most proofs will be for consumption by other AI models in the pursuit of yet other proofs.

It's kind of surprising so many mathematicians act surprised by this given this was clearly where automated proof assistants would lead. I guess they assumed they'd always be the ones guiding them.

What is the purpose of that?

Its like art being produced for AI to consume. What is gained from that?

And at some point AI will start suggesting - or doing - physical experiments.

But if AI is to recursively self improve understanding and evolving its own foundations, which are clearly mathematical, is essential. There is no need for humans to grasp what is going on in that loop.

Yes.

Friends and I often work on Putnam problems and this series:

The (Almost) Impossible Integrals, Sums, and Series by Cornel Ioan Vălean

But it's far from the only reason people in general do and have done math.

Also I'd argue that chess's utility is ultimately the same as pure math, particularly in esoteric fields. These things are highly unlikely to ever lead to any sort of real world breakthrough or application. The main benefit is an outlet for human logic, creativity, and exploration - which significant self improvement possible along the journey for players.

Though I think even that's probably too socially utilitarian. I think ultimately the 'real' drive is the same in both fields - it's fun and personally rewarding.

That's fine, and no amount of machine excellence will keep you from enjoying recreational chess or recreational math.

There's two points about this I am assuming 1) Mathematics actually has a significant subjectivity to it and is community oriented and not just climbing a never ending list of theorems that exists in the universe 2) A lot of mathematical research work is inside of a subfield and isn't directly motivated by applications. Sometimes it is but e.g. people don't work on obscure theorems about elliptic curves because of a dire need for that but more because the community found it interesting.

If your goals are understanding the game, self improvement, building thinking skills-- this is the best chess has ever been. It's only if your goal is to beat every opponent you can find that chess is in a bad place.

Even if you don't blunder anything, you'll still find yourself in a worse position without any clue as of what went wrong and why.

Whereas when playing humans, they can usually explain their approach and when they noticed errors in your play.

It's the same for Magnus Carlsen. Even with Queen odds, Stockfish is literally unbeatable for the best players in the world. It's just too strong at evaluating all kinds of random tangent moves (and ensuing positional advantage) which no human player can possibly pay attention due to the time required.

Stockfish vs any human is like Carlsen vs other players by about 3-5 orders of magnitude[0]. It's that stark.

[0] A wild pun appears.

EDIT: To avoid having to respond to each responder, fair comments about Queen odds. Maybe I was thinking Rook odds? Also, I kinda lumped Stockfish in with all the other engines, but I realize there are other engines with different properties ofc.

Leela odds networks, on the other hand, are an entirely different beast. I cannot beat Leela queen odds, much less rook or minor piece odds, and even GMs struggle against Leela knight odds.

Without odds though, yeah, Stockfish is just incomprehensibly strong by human standards. All top chess engines are, but Stockfish moreso.

But sometimes, these GMs can flag it, which counts as a win (especially when it's proxied by a cheater). Sometimes they can also explain the idea that cost them the game, so they've learned something maybe.

Whereas us scrubs literally cannot do anything at all for reasons completely beyond our understanding.

Computer moves are typically much more concrete than human moves: a human will play based on pattern matching ("intuition") and can only make explicit calculation of a small fraction of possibilities, after which decisions are guided by guesswork. The computers are unbeatable in practice because they can calculate concretely in seconds what might take an expert human long intensive study to notice, and they don't make the same kinds of oversights humans can make.

But if you stop and explore a particular position for an extended time, and if you have an intermediate level of chess skill, you too can probably often (usually?) figure out why it's doing something. Sometimes understanding the computer's reasons takes searching multiple branches of a tree several unlikely looking moves deep, but the collection of threats the computer was preemptively thwarting, traps it was setting, etc. are comprehensible to humans with enough effort, especially in games between the computer and a human.

The frustrating thing about playing against the computer is that it notices and thwarts every plan you might come up with, before you make up the plan yourself, and it doesn't make (human-apparent) mistakes, so the game ends up feeling hopeless. Nothing you try works on it, and if your idea is even slightly inaccurate it will be exploited.

There is a real, undeniable possibility of AI becoming better at mathematics in the same way that it became better at chess and Go, and in such a scenario, one may expect the community's response to be comparable.

Tech companies are as much the topic of this post as AI, I think that's the immediacy.

There's a single answer to that question: yes, math very much has an impact without humans in the loop.

Math has a lot of applications, and those applications don't care whether eg the new faster matrix multiplication algorithm was found and proven correct by a machine or a meatbag.

The main situation I can think of where better calculations have a really visible effect is video and image compression, and that stuff is very far away from mathematical proof territory.

If you tell someone they can't use a matrix multiplication with better asymptotic performance than n^2.3755 from 1990 they're going to shrug and not care.

You give your AI the problem, like 'make this computer program faster and more reliable', and it'll go off and do the math necessary to make it so.

As the US has offshored manufacturing, the number of patents issued for those processes has fallen. Innovation occurs where the foundational understanding is applied; they arise from a desire to do the required work more efficiently.

Similarly in math, attempting to solve a problem leads to new questions. IF you actually do the work.

You need to have done enough of the work to know what the correct next question are, or you need to rely on AI for everything.

For all we know, P vs NP will never be resolved neither by human nor AI, and the world will just keep turning.

I am saying that applications supplied and supply an inexhaustible amount of good problems and questions to consider. Purely theoretical concerns also supply some questions, but even if that well dries up for some reason, applications persist.

I’ve watched quite a lot of YouTube videos where two machines compete, so you may not be completely right here

In other videos he's called out the influence that this and similar games have had on human players in recent years, particularly around square denial and thorn pawn strategies.

Top chess engine championship is pretty fun to watch.

Chess is only valuable as an entertainment. So no one really gains anything from computers becoming really good at chess.

But with math, everyone in the world would gain from computers becoming really, really good at it.

Take the Navier-Stoke problem for example. Knowing that there are solutions that "blow up" probably doesn't have a lot of practical applications. Such solutions couldn't happen in a real system. But the process of finding that proof could result in increased understanding of how turbulence works, or new techniques for solving non-linear partial differential equations (which has a lot of applications in science and engineering).

Sure. And AIs can use ideas from AI published proofs in one domain to inspire other domains just fine. Nothing changes here.

There's lots of problems like that. Eg if we prove P != NP, that won't have much of an immediate effect either.

However, there's also plenty of problems whose solutions will have practical effects, some even immediate.

Sure. But do you know which ones they are? Or do we discover later that they were valuable?

Your argument would be 100x more convincing if you gave an example.

I will try: a super-compressor that made my 100Mb web app into a 5 kb binary bundle would immediately speed up my work. Can/will AI move human understanding or machine capabilities on this front?

A browser without security vulnerabilities would be wonderful. I think LLMs are already helping with this a lot, but a lot of complexity remains.

A right to privacy in society would be amazing (see the UN Declararion of Human Rights). AI is eroding this.

So I tried but I’m not very impressed with my list. Do you have one?

This has nothing to do with mathematics.

> So I tried but I’m not very impressed with my list. Do you have one?

Look into operations research. Or narrower, you can look at improvements in linear programming solvers and mixed integer linear programming.

(These are examples of areas that have seen mathematical improvements in applications recently. I don't think good AI has been around for long enough to contribute much to progress there, yet.)

AI’s solutions are like the answers section to practice problems at the back of a textbook. Answer is 42, so what?You have to attempt the problem yourself, that’s the whole point of the exercise.

As an engineer I’m happy to use AI for math. If I publish a paper that way, very common these days, I think it’s still problematic. My paper would include something I didn’t come up with and I don’t really understand.

I think this is a good time to properly discuss these things because AI is coming for everything. Mathematics and Software were just the first two.

Yes, you might not get new human capabilities. But your applications still work better.

I think the error you're making is that you're assuming these systems have the same mathematical capability as humans (or better). But that's not the case, nor would an informed prediction be that they surely will get there soon enough if technological evolution keeps apace. That would be akin to believing a hiker will reach the moon if they will just keep ascending the mountain. "But look, they are making such good progress!"

I'm not convinced of that.

It is one possible outcome but there are also other possible outcomes.

There is no bound on the amibitions of AI. AI is set to replace anything done by people, and there won't be any room left for people. There isn't any task done by humans that AI won't be better at.

This is not a tenable outcome.

We should never have built machines with agency, rather than optimization processes that operate as subroutines of humans.

We haven't quite crossed that bridge, but we do appear to be standing on it.

I am much more than my job. If I receive basic income, please replace my job ASAP!

And if I don't receive a basic income in a world where AI is capable of replacing all human jobs, then the problem is not AI, but the powers that be.

Theyre never going to use that power to provide basic income for everybody. You cannot wave a wand to make them do that. Theyre going to use that power to accumulate even more resources and raw materials and impoverish/expel the "useless" labor.

Realistically our leverage over the powers that be is mostly about our labor and our ability to withdraw it.

That's not necessarily true. Technology gives more power to everyone. The rise of factories historically did not give more power to the powers that were at the time (the aristocracy), but instead lifted the masses from poverty (after some initial turbulent period).

> Theyre never going to use that power to provide basic income for everybody. You cannot wave a wand to make them do that. Theyre going to use that power to accumulate even more resources and raw materials and impoverish/expel the "useless" labor.

Contrary to leftist propaganda, the rich are not some cartoon villains and psychopaths that abuse people just for fun. The reason why the rich currently exploit the poor is because doing so provides them with significant material gains. Once they can obtain the same or even better material gains by "exploiting" robotic labor instead of human labor, the logical outcome is not further abuse and exploitation of other humans, but simply indifference.

> Realistically our leverage over the powers that be is mostly about our labor and our ability to withdraw it.

That is (partly) true, but only in the current economic system. Widespread human-level AI changes the equation, and not necessarily in favor of those who are currently rich. You are applying capitalist and socialist analysis to a system that transcends those terms (AI post-scarcity economy).

Did not give more power to everyone. At best you could say that it shifted power from landed gentry to industrialists. Even that switchover was less of a change than you'd think.

The practical upshot of the beginning of industrialization was very negative. The enclosure movement stripped families of their land to push them to work in the factories where they would never go willingly. The famines in Ireland and Ukraine were both triggered by redirecting grain to export in order to fund domestic industrial expansion.

The most brutal two wars in human history were that brutal precisely because industrialization made it possible.

>instead lifted the masses from poverty (after some initial turbulent period).

What lifted the masses from poverty wasnt the factories it was the labor movement which occurred in response to horrific working conditions and the reliance those factory workers had on mass labor (specifically coal mining which was incredibly labor intensive and a key economic chokepoint).

The whole of that is glossed over by right wing libertarian propaganda but that last part is particularly underemphasized.

>Contrary to leftist propaganda, the rich are not some cartoon villains and psychopaths

Do you look at peter thiel and elon musk or the robber barons and see anything else?

The few billionaires ive encountered personally were no less sociopathic but they kept it hidden better. Power corrupts. Immense wealth corrupts. That isnt a leftist thing, that's a human thing.

>The reason why the rich currently exploit the poor is because doing so provides them with significant material gains. Once they can obtain the same or even better material gains by "exploiting" robotic labor instead of human labor, the logical outcome is not further abuse

False. It just shifts the focus of their exploitation from human labor to natural resources.

They'll fight over oil and minerals and gas and water resources and at best leave us to rot (homelessness will skyrocket) and at worst they'll find some excuse to exterminate those of us they particularly dislike (Gaza serves as a model here).

The world economy's reliance on human labor has been the best inducement to peace there is. It's no coincidence that all of the countries in the world with lots of natural resources and no industry are the biggest shitholes and vice versa.

>That is (partly) true, but only in the current economic system. Widespread human-level AI changes the equation, and not necessarily in favor of those who are currently rich.

Human level AI (assuming it ever happens) will simply make the fight over natural resources that much more intense because labor will matter that much less.

You're living at the tail end of a relatively golden period in a country where labor was the economic bottleneck and natural resources were relatively plentiful. Venezuela and Angola and Iraq are models of what happens when that equation is reversed.

Nobody gives a shit about appeasing the people who live in those countries, their labor is virtually worthless. They are a model for how the rest of us will be treated in a world where human labor loses its value.

Compare the standard of living in 1850 vs. 1950. Even of relatively poor people. I rest my case. Technology is the main force that improves human wellbeing. There are of course also other factors, but they are less important.

> The practical upshot of the beginning of industrialization was very negative. The enclosure movement stripped families of their land to push them to work in the factories where they would never go willingly. The famines in Ireland and Ukraine were both triggered by redirecting grain to export in order to fund domestic industrial expansion.

Yes, that's what I mean by "initial turbulent period". Perhaps the same will happen with AI, but it will be worth it in the end. Don't give up prematurely!

> What lifted the masses from poverty wasnt the factories it was the labor movement which occurred in response to horrific working conditions and the reliance those factory workers had on mass labor (specifically coal mining which was incredibly labor intensive and a key economic chokepoint).

It was not one or the other. It was both. The rise from poverty would not be possible if factories were not developed. And I am not saying that in the AI world we would not have to fight for our rights. Of course we would. But the problem is not AI, just like historically the problem were not the actual machines in factories.

> False. It just shifts the focus of their exploitation from human labor to natural resources.

Exploiting more natural resources is the only way to increase standard of living of humanity. I am OK with that. Resources don't have feelings, and ecology is not more important than human wellbeing.

> The world economy's reliance on human labor has been the best inducement to peace there is. It's no coincidence that all of the countries in the world with lots of natural resources and no industry are the biggest shitholes and vice versa.

The reason why some countries become shitholes is mostly ideological (extremist political or religious ideologies take hold of the population). Every shithole country is non-democratic (communist, totalitarian, fascist, theocratic etc..). This is not a problem of natural resources. It is a problem of people, their education, their beliefs/ideology, or, as capitalists say, "human capital" is the main problem here. AI could help here too, especially with education.

But yes, if people themselves are largely ignorant and extremist, no amount of resources and human-level AI robots will help them make a well-functioning society. You could drop masses of AGI robots into Afghanistan tomorrow, and people will just use them to kill or oppress each other more effectively, instead of using them to start building an AGI utopia...

And if I don't receive enough, then again: the problem is not AI, but powers that be. And there are various solutions for that... and none of them are helped by me being anti-AI.

The limit is inflation.

Do you really think making labor obsolete and giving humans back their time is such a bad thing?

In the short term, AI is disempowering the vast majority of people in favor of a very small subset. In the also way too short term, AI is disempowering all people.

I believe we will distribute the fruits of AI's labor at the very least to the extend where everyone can live comfortably.

No one has an interest in putting large parts of society on the streets in some sort of dystopia.

Wow.

Only on HN.

Also apparently not from the US? Or Russia?

This does not reflect what I see in my government.

Historically though, technological improvement has lead to large increases of living standards for the overwhelming majority of people, so I think that past trends support the utopian view more than the doomer view.

I am not assuming humans don't remain in control, but rather, I am not axiomatically assuming there's any inherent reason we will.

We have precedents of people warning about existential risks in the past, when the warnings turned out to be false, or overblown. In some cases, such overblown warnings led to serious negatives for society (demonization of nuclear power).

Those counterfactuals are important, though. We do have a history of averted disasters, albeit not as large. Ozone hole, Y2K, think about things that seemed overblown at the time, and consider whether they were actually overblown or whether there was a concerted effort to successfully avert them.

Its not a counterargument to basically anything, except as to avoid having to deal with actual evidence.

It is an argument that seems almost tailor made to have to ignore mountains of evidence against you.

In any other contexts the supposed "rationalists" would be fully in agreement that having evidence matters, and that not having any works against you.

So, in order to fight against this severe issue with their arguments, they have to invent a reason as for why the entire concept of evidence itself doesn't apply to them and they get to ignore normal evidentiary requirements.

No, the argument is instead that the person claiming that humanity is going to be destroyed is making a fairly extraordinary claim and that requires fairly extraordinary evidence.

Or, in other words, we have tons of evidence already as for why the world ended is a fairly far out there prediction, given all the crazy people making these predictions keep turning out to be wrong.

So, you can make your extraordinary claim if you want, but really the burden is entirely on you to prove your extraordinary claim, and everyone else is free to remain on the default and completely normal end of the prediction spectrum, of believing that the world isn't going to end.

And when people do tricks like this, they are running away from the fact that they are making a wild completely out-there prediction, and hiding behind that by trying to come up with reasons as for why evidence doesn't matter and actually the burden of proof is shifted to those who have the default and boring prediction of the world not ending.

I'd guess horses today have on average a better life than back then.

Stockfish does not steal research or scoop researchers.

The concentration of computing resources and capital should be examined by the math community.

- I think competitive open models are just an artifact of the AI race we're witnessing right now. What's the incentive for a company to spend billions researching, developing, and training a model, only to release it for free? Leading to the next point.

- Even if open models are good enough to be competitive, how are we going to run them? Doing so locally is next to impossible and I don't see that changing. The capabilities of models that you can run locally will always get better, of course, but the level of quality that is considered essential for work will always stay pinned at "near-frontier". Datacenters will always have better optimization and economies of scale, the industry will consolidate over time and eventually we'll end up with a handful of companies that operate the hardware serving 95% of all inference needs.

I'm not convinced that we can reach the fantasy world that they're trying to sell without killing the entire planet, but if we somehow do I don't see how we can avoid the world turning into a dystopian hellscape. We would need extremely radical interventions to avoid that scenario, such that these models and the hardware to run them would be owned and governed democratically, i.e. the end of capitalism.

And a lot of people are using trailing edge models just fine already.

> [...] but the level of quality that is considered essential for work will always stay pinned at "near-frontier".

Why? When we'll finally all write our software in Lean and prove it correct and prove it fast, it won't matter that a slightly more clever model could have found a slightly nicer proof or whatever.

Just like today people happily use Python for many programs, even though rewriting in C might give you a performance boost. Good enough is often good enough.

Frames could literally cost 1 dollar (only happened after china entered this market), but good luck finding ones like this with good lenses.

You need to buy from one supplier who intentionally offers cheapest frames for 50+ dollars and those frames look like crap. The ones that look better (even if same plastic) cost 500+ dollars - and all due to price gouging.

For lenses I am not sure, but suspect something similar.

Fielmann in Germany has been offering very reasonably priced glasses and frames for ages.

If that's your only objection: in a few years you can prove Rieman's hypothesis on your smartphone, no need for any trillionaires to give you permission. Does that make any change to your argument, or did it not actually matter?

For the latter: I assume that having a whole data centre will always be an advantage. I am saying that for a fixed target, like proving the Rieman hypothesis from scratch, the required hardware will shrink.

And, yes, the Rieman hypothesis hasn't been proven yet. So to take your fears into account, replace my example with something they've already done, like constructing a solution to the Navier-Stokes-problem.

Yes, but to what point the hardware will shrink? There are several orders of magnitude of difference in what in your mind AI will become and what more conservative people believe. You take your view as granted...

What makes you think these companies (and I'm not a fan of all their motives) are not interested in developing research? The motives may be self-serving, but it is undoubtedly and objectively accelerating research.

Part of the point of the letter is that it is quite possible to act in a way that is a net negative to research. The most obvious case is when the companies violate ethical standards in research.

The subtler case, the one for maths in particular, is what happens when you fail to follow well-established patterns for making maths research productive. Tao himself spelled out how that can look in https://mathstodon.xyz/@tao/117207856734787448 (which notably came before any of the news on Navier–Stokes).

Sorry, but this is backwards. Mathematicians are the ones coming to the table asking for money from the taxpayer. This is the context we are speaking in. Mathematicians speaking with the taxpayers who fund them. It's not a great strategy to get offended or speak from a high horse. If the taxpayer is not getting it, patiently explain why you think this art project should remain funded at the same level instead of spending it on something else.

I don't like nuance here. I think progress is really measured by what humans are able to do and understand, not machines. It is significant if we find problems we struggle to solve. That tells us something. What does it take for humans to solve these problems is related.

The best analogy I can give is if you wanted to climb Mt. Everest you might ask someone for guidance. Would it be better to ask someone who has climbed Mt. Everest or someone who took a helicopter ride up near the top and then went to the peak? This is like the AI versus human gap to me. The helicopter is like using AI to generate a proof. The person who actually climbed Mt. Everest has firsthand knowledge of the experience. Same thing for a difficult proof. The struggle people have is actually valuable here. Likewise, we know people are actually capable of climbing Mt. Everest but if they had only ever rode a helicopter to the top, the knowledge of climbing it would not exist, and surely that is meaningful knowledge given the risks.

So if we rely on AI for proofs I think we lose a sense of what is difficult and why. We lose a sense of what human achievement is. Surely climbing Mt. Everest means more than taking a helicopter up? For students, why bother grinding through all the material of climbing Mt. Everest and then attempting it if the helicopter ride is how things are done now? This would have the affect of destroying knowledge.

(please do not nitpick the analogy because it's the best but perhaps a clumsy way to describe my thoughts)

> From my memory I think they said it took 88 hours to solve a Millenium Problem versus the decades of time humans have put into it.

Keep in mind those ~88 hours were spread across ~10,000 simultaneous agent instances.

So, roughly 880,000 hours of compute.

Assuming a fifty-year career, and forty-hour workweeks, a human mathematician's career is about 100,000 hours of "compute".

I suspect that with six good mathematicians spending their whole careers primarily focused on it, and working together closely, Navier-Stokes might well have fallen already.

The perverse incentives of academia mean this has never occurred.

The perverse incentives of industry mean OpenAI intentionally scooped researchers who were getting close (granted, with AI help).

I'm not trying to dismiss the achievement - if the proof turns out to be solid, it's quite impressive (though much less so if the training data included the recent human breakthrough, which seems pretty plausible).

I'm just pointing out that "88 hours" is a very misleading way of framing this.

> The perverse incentives of academia mean this has never occurred.

This. Mathematicians in their most energetic years are trying to get tenure or land a tenure-track job. They are disincentivized to go all-in on ultra high risk, high-reward problems. The potential downside is just too forbidding. It's much safer to develop a research program in a mainstream field that affords many opportunities for partial progress that can translate to a robust publication record.

There were more than six doing that and it's essentially why it was ripe for AI to finish it off. But the finishing off was quicker than anyone expected

What teams of people spent an entire career working together, focused entirely on Navier-Stokes or problems they suspected were related, without any "publish or perish" concerns?

I realize tenure is a thing, but my limited understanding is that a significant amount of time is spent earning it, once you account for Ph.D. program and the years of needed to be granted it.

I'm just trying to point out that economically, we have not yet reached the point where AI mathematics research is a no-brainer hands-down win, no consideration required.

It might already be a win, and certainly the ability to compress those 900,000 hours of effort into an actual week of linear time is mind-boggling and potentially a huge game-changer for all kinds of open research questions.

It's not obvious that human math research is dead yet.

Maybe soon, but not yet.

Depends on if I want to go by helicopter myself.

> why bother grinding through all the material of climbing Mt. Everest and then attempting it if the helicopter ride is how things are done now?

I think you answered this yourself earlier:

> I think progress is really measured by what humans are able to do and understand

People want to make this progress. Therefore people will "grind Everest" as a mathematical community, and that is maybe not so hugely different from a lot of previous mathematical work.

There's still ample room for creativity: simplifying, generalizing, asking new questions humans are interested in, ...

Building a machine that solves Millennium problems is pretty cool too. You wouldn't know it from reading these stories, though.

It would be more like Lem's novel where it completely disappears from the human horizon: https://en.wikipedia.org/wiki/Golem_XIV

Perhaps indeed a better understanding of what intelligence really is would allow this sort of Uplift (as in Brin's books).

if a particularly intelligent sixth grader was highly motivated to understand chromatic homotopy and had a highly capable private teacher available to her 24/7, she might within a year get to the point where she could apply it by herself to figure out some simple but nontrivial topological properties. (nobody has tested this. maybe it would require at least three years instead of one.)

if a particularly intelligent chimpanzee was for some reason highly motivated to understand chromatic homotopy, no matter how many years the most incredible teachers spent explaining it to her, she would never comprehend anything about it.

that's the pessimistic scenario: humans will be to future AI like our closest evolutionary relatives are to us. or even more pessimistic: we will be to future AI like insects are to us.

Dr. Tao said the same thing. Somehow, this letter came through. He wants to conduct Math competitions where participants who don’t have formal credentials can contribute to mathematical research through AI.

Title: Terence Tao - SAIR Competitions and the Future of Experimental Mathematics

https://www.youtube.com/watch?v=rB9YOi3lb7w

and this:

Daniel Litt - Working with LLMs to do high quality math

https://www.youtube.com/watch?v=0wL8NlhxXcU

And embarrassingly they used him for a "coal miners should learn math" moment that just benefits the AI industry.

He has severely reversed course in the past week. Without concrete propositions it remains to be seen how much of the new resistance is for show.

AI is a tool. It speaks languages I don't (Math, Science, Code). I would love to participate in a Math competition without a hint of any formal advanced math training because my experience so far tells me I will do well.

I wrote a fancy polygon decomposition algorithm in university (pre-AI) which my professor didn't seem very impressed by because it was missing some sort of mathematical rigor. Yet everything I threw at it worked! Even he couldn't find a counter example.

It took a while for me to find some failing cases but it turned out they did exist.

But hey, maybe all I was missing is an AI-written lean proof.

Apparently he has since changed his mind.

He doesn't see value in scrolling through unsolved problems asking an AI to please solve them. In his view, this is a fundamental confusion about what mathematical research is for. Knocking down unsolved problems without developing the community's understanding of them is like prompting Claude to go through a Jira board, write code for all the open tickets, and then close them without merging or deploying the code.

Yet that's exactly how the field works. A new grad student is tasked with finding a suitably difficult problem from a list of unsolved problems. The sweet spot is obscure, so that fewer people are working on it, but not too obscure that no one knows about it. It works the same way in theoretical physics and theoretical Comp Sci, and I speak from insider knowledge. The rosy view of mathematicians in the media is largely a product of marketing.

I understand the perspective: The journey of a PhD thesis is a learning experience greatly beneficial to the student. Yet that journey is funded by society (esp. for domestic students) and society benefits from the results. The average person benefits when progress is made.

Society doesn't benefit from results in research mathematics because it mostly consists of pure mathematics, which is completely useless for society.

Most math textbooks have solutions in the back. That didn't wreck peoples ability to learn math, did it?

So we are left to speculate about why solutions in appendix are fine, but LLM solving open problems is not fine.

The idea that when LLMs produce solutions, people won’t try to understand them and won’t learn from it, is obviously not true. Terry Tao himself spent time digesting and simplifying LLM proofs.

So again we’re left to speculate what the actual problem is.

Math understanding will increase with LLMs. Not just professional mathematicians but amateurs.

I think you’re struggling to understand what Tao and his cosignatories are saying because you’ve acquired a very specific kind of “AI-pilled” mindset from social media, where taking AI seriously implies accepting LLMs should be used at any time for any purpose. They’re saying in great detail that LLM solutions are unhelpful when presented in a particular way, but you can’t help but hear them saying that LLM solutions aren’t helpful at all, even as you rightly point out that this makes no sense and is inconsistent with their observed behavior.

   (A) it furthers human knowledge
   (B) it gets used in applied sciences, engineering, etc.