When the Systems We Created to Serve Us Begin to Tell Us How We Must Live
Human beings create tools because we are trying to accomplish something.
We create knowledge so that we can better understand reality.
We create money so that value can be exchanged, stored and accounted for more easily.
We create employment systems to organise productive activity and distribute income.
We create educational systems to help people learn.
We create bureaucracies to coordinate complicated societies.
We create technologies to extend what human beings are capable of doing.
And now we are creating and deploying artificial intelligence to extend certain forms of human reasoning, creation, analysis and action.
There is something worth remembering in all of this.
We created them.
Not necessarily any single one of us.
Not necessarily deliberately in their present forms.
Many evolved over centuries.
But they are human systems and human instruments that emerged because they served—or were intended to serve—human purposes.
The danger is that after living inside a system for long enough, we can forget this.
The tool becomes normal.
Then indispensable.
Then unquestioned.
Eventually, instead of asking whether the tool is still serving its purpose, we begin reorganising ourselves around the requirements of the tool.
And sometimes we even accept harm because:
“That is how the system works.”
Perhaps that sentence should make us more uncomfortable than it does.
Money Was Supposed to Help Us Exchange Value
Consider something simple.
There is food.
Someone needs food.
The food may otherwise spoil.
The person who needs it does not have enough conventional money.
Perhaps that person is unemployed.
Perhaps they tried to find work and could not.
Perhaps their community simply has very little money circulating through it.
The food exists.
The need exists.
Human capacity exists.
Perhaps the person can provide something useful in return.
Yet exchange fails because the recognised medium connecting those things is absent.
That does not mean money has failed.
Money remains one of humanity's most useful technologies.
But the situation should at least make us ask:
What was money for?
If money helps human beings exchange value, excellent.
But if the absence of money prevents useful value from moving between willing people even where resources and needs coexist, perhaps the answer should not always be:
“Nothing can be done because there is no money.”
Perhaps sometimes the better response is:
“Is there another way of achieving what money was supposed to help us achieve?”
That is not an argument against money.
It is an argument for remembering that money is the instrument, not the objective.
Knowledge Can Do Something Similar
Knowledge should expand our ability to understand reality.
But even knowledge can become something we serve.
A person spends years learning how something works.
They become highly qualified.
They master the accepted frameworks.
They understand the rules.
Then reality changes.
A new possibility appears.
And instead of asking whether existing knowledge should be extended, reconsidered or applied differently, the response becomes:
“That isn't how it is done.”
There is a strange possibility here.
Someone can become so knowledgeable within a system that the knowledge which should have expanded their thinking begins defining the boundaries within which they permit themselves to think.
This is why the problem is not simply one of ignorance.
Nor is it a problem only for wealthy people, powerful people or poorly educated people.
“Knowledged” people can become system-bound too.
Expertise is extraordinarily valuable.
But expertise should help us interrogate reality.
Reality should not be forced to obey our expertise merely because we spent a long time acquiring it.
We should apply knowledge.
We should be careful that inherited knowledge does not begin applying us.
Employment May Be Another Example
Employment helps organise human contribution.
People perform useful work.
Businesses and institutions obtain the labour they require.
Workers receive income.
Income gives people access to goods and services.
For a long time, these relationships have been tightly connected.
But imagine artificial intelligence and robotics eventually allow society to produce substantially more while requiring substantially less human labour.
That could represent an extraordinary productive achievement.
Yet we could produce a bizarre outcome if we forget what our systems were meant to accomplish.
Machines produce abundant goods.
Businesses require fewer workers.
People lose employment.
People therefore lose income.
People without income cannot purchase the goods.
And society concludes:
“Unfortunately, they cannot have them. They do not have jobs.”
At that point we should ask a difficult question.
Did humanity become poorer?
Or did our productive capability become richer while our mechanism for distributing access failed to adapt?
The answer matters.
Because if the physical capacity to provide something exists but access collapses because an old institutional relationship no longer fits the new productive reality, then blindly preserving the relationship may not be wisdom.
It may simply be familiarity.
Systems Can Begin Defending Themselves
This pattern appears in smaller ways everywhere.
An examination is created to measure learning.
Eventually teaching becomes organised around passing the examination.
A performance metric is introduced to help understand whether workers are doing useful work.
Eventually workers begin organising their behaviour around improving the metric—even when doing so makes the underlying work worse.
A bureaucracy is created to deliver a public service.
Eventually somebody desperately needing that service encounters a process so rigid that satisfying the procedure becomes more important than helping the person.
A workplace system is introduced to help managers make better decisions.
Eventually employees hear:
“The system says no.”
Nobody seems quite sure why.
Nobody feels authorised to challenge it.
The tool has quietly moved position.
It is no longer merely assisting the decision.
Human beings are arranging themselves around its output.
This does not require an evil mastermind.
It does not require a conscious machine.
It does not even require anybody intending harm.
A system can produce harmful outcomes simply because enough people become accustomed to obeying its internal logic without periodically returning to the question:
What was this supposed to achieve?
And Now We Have AI
Artificial intelligence makes this question more urgent.
Not because AI is uniquely evil.
Quite the opposite.
AI may be unusually good at doing what we ask it to do.
And that creates an interesting danger.
Suppose an organisation has a badly designed process.
AI makes it faster.
Suppose a company measures the wrong thing.
AI optimises it brilliantly.
Suppose an institution has an unfair decision-making structure.
AI allows it to process one million decisions instead of ten thousand.
Suppose a bureaucracy has forgotten why one of its rules exists.
AI enforces the rule perfectly.
We have improved the system.
But have we improved the outcome?
This is why efficiency cannot be the final question.
Whenever somebody says:
“AI can make this more efficient,”
perhaps another question should immediately follow:
“More efficient at achieving what?”
Because making the wrong objective dramatically more efficient does not necessarily constitute progress.
Sometimes it merely allows us to produce the wrong outcome faster.
Human in the Loop Is Not Enough
There is already considerable discussion about keeping humans involved in important AI decisions.
That is sensible.
But merely placing a human somewhere in a workflow does not guarantee human control.
Imagine:
AI recommendation.
Human clicks approve.
Next recommendation.
Approve.
Next.
Approve.
The human technically remains “in the loop.”
But where is the judgment?
Where is the authority to disagree?
Where is the understanding necessary to recognise when the system is wrong?
Where is the willingness to say:
“This outcome contradicts what we were trying to accomplish, hopefully, something positive and helpful.”
Meaningful human oversight requires more than human presence.
It requires human agency.
The person needs sufficient understanding to question the system, sufficient authority to challenge it and sufficient institutional permission to override it.
Otherwise we may preserve the appearance of human control while quietly transferring practical authority to the machine.
This Is Not an Argument Against Systems
Civilisation could not function without systems.
Money is useful.
Institutions are useful.
Expertise is useful.
Metrics are useful.
Employment is useful.
Rules are useful.
Artificial intelligence can be extraordinarily useful.
The lesson cannot be:
Destroy the systems.
It should be:
Remember why they exist.
A good system should remain accountable to the human purpose that justified creating it.
And when circumstances change, we should retain the intellectual freedom to ask whether the system should change with them.
That requires something surprisingly difficult:
the ability to distinguish the objective from the mechanism currently being used to achieve it.
If the objective is education, the examination is a mechanism.
If the objective is useful economic exchange, money is a mechanism.
If the objective is productive human contribution and access to resources, today's employment arrangements are mechanisms.
If the objective is better decision-making, AI is a mechanism.
Mechanisms matter enormously.
But they should not quietly become sacred.
Perhaps This Is the Awareness We Need
Every generation inherits systems created by generations before it.
Because we encounter them already functioning, they can begin to feel less like human inventions and more like laws of nature.
But they aren't.
And every new technology gives us another opportunity to forget.
Artificial intelligence is powerful enough that forgetting could become particularly consequential.
So perhaps alongside AI literacy, AI safety, AI ethics and AI regulation, we need something more basic:
awareness of our relationship with the tool.
We should use AI.
AI should not use us.
We should apply knowledge.
Knowledge should not prevent us from seeing beyond what is already known.
We should use money to facilitate value.
The absence or structure of money should not automatically prevent us from recognising value that plainly exists.
We should build institutions to serve human purposes.
Human beings should not be harmed merely to preserve the internal logic of institutions whose purposes we have forgotten.
This does not provide an easy formula for deciding when a system should change.
There will be disagreements about what its original purpose actually was.
Purposes themselves can evolve.
Constraints are real.
Scarcity is real.
Rules often exist for reasons that become apparent only when somebody removes them.
So questioning a system is not the same thing as assuming the system is wrong.
It is simply refusing to surrender the right to ask why.
Perhaps that is the awareness worth carrying into the age of artificial intelligence:
A tool can become extraordinarily powerful without becoming our purpose.
And perhaps one of the simplest questions we can preserve as technology becomes more capable is also one of the most important:
What was this for?
If we can still answer that—and still change the system when its behaviour begins contradicting the answer—then perhaps we remain on the correct side of the tool.
Read the series introduction: The Asymmetric Technology Ledger →