At what point does a structure stop being something we built and start being something that built itself?

[u/JCWanWriter]

Been thinking about this one for a while.

Suppose you give an autonomous construction system a very simple instruction: build a habitat that can survive for a few hundred years.

You give it raw materials, energy, some basic manufacturing equipment, and a set of constraints.

That's it.

At first, it's obviously just a machine following instructions.

It builds the first sections according to the plans. Then it starts modifying the plans because some things work better than expected and others don't.

A few decades later, most of the original structure has been replaced or altered.

A century later, the system is using materials and manufacturing methods that weren't part of the original design at all. Some of the architecture exists because the system discovered that it worked, and nobody who gave the original instructions ever anticipated it.

Eventually, even the people maintaining the habitat don't really understand why some parts are arranged the way they are. They just know that changing them tends to cause problems somewhere else.

So here's the thing I'm wondering:

At what point would you stop describing this as "a structure built by humans" and start describing it as something that developed its own architecture?

Not necessarily alive. Not necessarily intelligent.

Just... no longer fully designed.

And if the original builders came back after 300 years, would they still be looking at their creation?

Or would they be looking at something that had spent three centuries becoming something else?

I think there's a weird middle ground between engineering and evolution here, and I'm not sure what to call it.

[u/JCWanWriter]

One thing I keep coming back to is the difference between designing something and being able to explain why it exists.

A human engineer can point to a component and say, "I put that there for this reason." But after centuries of autonomous modification, you could end up with a structure where every part has a reason, yet nobody knows all of those reasons anymore.

Maybe that's the real transition I'm looking for. Not when it becomes alive, but when the causal history of the design becomes too deep for any one mind to hold.

At that point, is it still an engineered object, or has it become something closer to an artifact of its own history?

[u/Alive-Philosophy2632 → u/JCWanWriter]

That's a good point, and a system this capable would likely have a lot of it built in, though if there were a capability gap in certain areas it might not be explainable. Isaac Arthur did have a couple episodes on basically an advanced civilization that has lost a lot of their scientific knowledge with a priest caste with rites supporting the operation of a system they no longer understand. That sounds like what you might be getting at, because I think ideally such a system would be self-documenting.

Now that's a different thing from your end point "where it becomes too deep for any one mind to hold". Because I would say we're already there with computer systems. I don't think there's any one person alive who understands the entirety of computer systems, the entirety of mathematics, the entirety of human knowledge definitely. So it's a question of how fine-grained you want your knowledge or questions to be, and that's pretty much a continual boundary.

And as for engineered vs evolved I think we're already there, and we've generally found evolved works better, with a seed engineered scaffolding. In high feedback density domains like chess, the AlphaZero's of the world learn entirely by playing against themselves, developing inhuman strategies and patterns that are explainable, but not really without "holding the whole thing in your head" to paraphrase. I agree it's very interesting to think about

[u/SoylentRox → u/JCWanWriter]

By current human knowledge and engineering practices this would be a reason to condemn the habitat and replace it.  No autonomous self modification it's against the building codes.

Future vibe-engineers may choose differently.

[u/JCWanWriter → u/SoylentRox]

Yeah, that's probably the most realistic objection to the whole hypothetical. If it's a habitat people are supposed to live in, the first generation of engineers probably wouldn't care how interesting the self-modification was. They'd see an undocumented change to a load-bearing system and shut it down. The part I'm curious about is what happens if the system gets good enough that the self-modifications are actually safer and more effective than the original design. At some point the argument stops being "should we allow this?" and becomes "why would we keep replacing a system that's consistently designing itself better?" That's probably where the future vibe-engineers you mentioned show up.

[u/SoylentRox → u/JCWanWriter]

You still wouldn't do it.  For something like human habitation you want extremely conservative changes and you can afford to waste mass.  Being "more effective" doesn't mean anything if your materials are plentiful.

The lunar factories and mining robots or warship engines are where you push the envelope and let AIs loose.  Also solving problems that we currently can't solve - you might want to let AIs do what they can if you're trying to deage lab rats and human mockups.  Once it works have the designs pushed towards simpler and more conservative solutions.

It's how you vibe code right now when the problem is difficult. Let the AIs do what they want to make the code work at all.   Then make them simplify the solution and make it human readable.

[u/MiamisLastCapitalist]

Largely the same as your previous question, there are analogies to human life.

At what point are you something your parents raised and built and when are you your own person who is responsible for doing your own laundry for a change?

[u/JCWanWriter → u/MiamisLastCapitalist]

So you're saying the habitat just needs to move out, get a job, and start paying rent. 😂

But yeah, that's basically the weird part I'm stuck on. Humans have a pretty recognizable transition from "my parents made me" to "I'm my own person." I'm not sure a self-modifying system would have anything nearly that clean.

At what point does "I built you" become more like "I was there at the beginning"?

[u/MiamisLastCapitalist → u/JCWanWriter]

It's a transition. And at some point you look back on your creation and see that it is good.

[u/Alive-Philosophy2632]

If the autonomous building system had that much capacity then it would probably be intelligent, though probably not alive.

I think you wouldn't have to draw a distinction between the building system and the thing itself, they're symbiotic sort of, or at least whatever the neutral/helpful version of that is called. 

The design is from the building system. There's a lineage that goes to humans just like there's a lineage going back to archaeae in deep sea vents we think on earth or the big bang. Systems evolve, you can draw links between patterns but anything beyond that is probably vanity

[u/JCWanWriter → u/Alive-Philosophy2632]

The lineage argument is probably where this gets really weird.

Maybe there isn't actually a meaningful boundary between the building system and the thing it builds. If both keep changing each other for 300 years, trying to decide which one is the "real" creation might just be imposing a distinction that the system itself doesn't have.

But then I'm still curious what the original builders would think when they came back. Would they see their creation, their descendant, or just another branch of the same lineage?

[u/Alive-Philosophy2632 → u/JCWanWriter]

Probably depends on how much they knew or anticipated it would evolve. This is basically the ship of theseus argument generalized and recontextualized.

And yes I agree, I don't think the distinction exists. You can draw ontological boundaries but we mostly draw them so that they're useful.

The closest analogy I can think of is overgrown ruins, even though the goals of the vegetation and the original builders aren't aligned. You probably wouldn't say that's fully human-designed (and again not optimizing for the same things so not a perfect analogy)

[u/JCWanWriter → u/Alive-Philosophy2632]

Yeah, I think the lineage argument is probably the strongest answer I've seen so far.

Maybe the mistake is assuming there has to be a clean point where the habitat stops being "ours." If its architecture is continuously shaped by its own history, then maybe the original human design is just one ancestor among many.

But I still wonder what you'd call the thing after 300 years. The same system? Its descendant? Or is that distinction exactly the kind of vanity you're talking about?

[u/Alive-Philosophy2632 → u/JCWanWriter]

Yeah, because if it's a self-improving system it can be or be inhabited by its own descendants. So you'd basically be doing a ship of theseus-like thing and saying "at what point are the patterns of behavior or internal state of the system sufficiently different to classify it differently?" and maybe that's a meaningful question and there's probably a lot you can say about capability boundaries and stuff, but I think the broad "is this the same thing?" is not a useful question to me, because yes and no would be the answer to such a broad question imo

edit: this has obvious parallels to LLMs/AI and the capability explosion we've witnessed over the last year. I would say similarly that a vibe-coded project was created by humans, in a way, in pretty much the same way as in your generations long self-improving AI example

[u/SirShriker]

You are describing an autonomous intelligent machine lifeform though. If the building machine is the building and a machine, it is first and foremost a machine with a building body. It was always independent and building itself.

If you plant an acorn, did you plant a tree or did the tree grow itself from the seed? Even if you provided it with the nutrients to follow the instructions, the seed no longer exists once the tree grows. There is a narrow window in which it is a seedling, a nascent proto tree, neither seed nor tree, but a brief interregnum while it continues its path to freedom.

If you planted the seed, did you also then plant the forest? Or did it plant itself? Does it matter if you wanted that seed to proliferate into a forest?

I think it only makes sense to say it built itself if the intelligence that guides the fabrication also inhabits the construction itself, not merely physical interior space. Anthills don't build themselves, ants do. Coral reefs do build themselves. One inhabits the structure, maintains it, the other IS the structure. You couldn't separate the structure of the coral from the guide factors without destroying it. Ants removed from their hill will build another place to live.

[u/JCWanWriter → u/SirShriker]

The acorn analogy is actually really close to what I was getting at.

The part that makes it difficult is that the habitat in my example doesn't just grow according to some fixed "genetic" instructions. The building system keeps changing the rules, the materials, and eventually even the reasons for why certain things are built.

So I like your distinction between "the thing that inhabits the structure" and "the thing that is the structure."

But now I'm wondering about the borderline case: what if the intelligence gradually becomes inseparable from the structure itself? At that point, are we still talking about a machine that built a habitat, or a habitat that became its own organism?

[u/Xeruas]

It sounds like those evolution competitive design solutions were they make designs compete against each other and they end up with designs humans haven’t come up with that would better in some ways and different in others. But yeh sounds like that

[u/JCWanWriter → u/Xeruas]

What gets me is the part where you have to decide what counts as the "designer" anymore. Give the system a way to generate variants, test them, keep the ones that work, and repeat that for a hundred years... sure, you can still say "humans designed the process." But that's starting to feel a bit like saying we designed evolution because we invented the petri dish. At some point the word "design" starts doing a lot of suspiciously heavy lifting

[u/NearABE]

I just watch a practicalengineering youtube video last night: https://youtube.com/watch?v=r1wwtGgnzg4

It is all about beavers and dam building. I think it is relevant.

[u/JCWanWriter → u/NearABE]

I hadn't thought about beavers, but that's actually a really interesting example.

The dam changes the river, the river changes what the beaver builds next, and after enough cycles they're basically shaping each other.

Feels like a pretty good example of a feedback loop where the line between builder and built thing starts getting blurry.

[u/NearABE → u/JCWanWriter]

Seriously, watch the video. The dam designs have many similarities. However beavers engage in a steady state process.

The video is about the dams. Other videos talk about beavers themselves. They build on instinct. Wildlife rescuers sometimes care for injured beavers. They are quite friendly and bond like domestic pets. They also rip up your house on a mission to plug the bathtub.

In Canada the highway people figured out the beavers are noise triggered. That inspired beaver proof culverts. Now the beavers build dams on the upstream side of roads.