There is a strange moral distinction people keep trying to draw around information.
The recent panic around models like Claude Fable points at this broader instinct: if a model knows about dangerous things, then the knowledge itself must be morally suspect.
I think that is wrong.
Knowledge does not come pre-packaged with moral intent. The same knowledge can be used to build, defend, exploit, repair, attack, or prevent attack. The morality does not live inside the information. It appears in the action, the context, the permission structure, and the consequences.
A piece of knowledge is not good or evil because of what it describes. It becomes morally relevant when someone uses it.
The developer and the hacker know the same things
As a developer, I need to understand how applications break. I need to know how vulnerabilities work, how authentication fails, how unsafe APIs leak data, how injection attacks happen, and how attackers think.
That is part of writing good software.
But the same knowledge that makes me capable of building safer applications also makes me capable of being adversarial. A good developer and a malicious hacker may understand the same vulnerability. They may read the same documentation, know the same techniques, and even ask the same questions.
The difference is not necessarily the knowledge. The difference is intent, permission, target, incentives, context, and consequences.
A security engineer finding a vulnerability in their own system is doing defensive work. An attacker using that same vulnerability against someone else's system is doing harm.
The technical knowledge did not change. The moral frame did.
Most serious knowledge is dual-use
This is not unique to software. Most serious knowledge is dual-use.
If a body of knowledge gives you real power over the world, it can probably be used badly. Chemistry can make medicine or poison. Biology can create vaccines or pathogens. Psychology can help people or manipulate them. Cybersecurity can protect systems or break into them. Uranium enrichment can produce nuclear energy or nuclear weapons.
The knowledge is the same domain. The outputs are different.
This is why the idea of "bad knowledge" is so slippery. The more useful knowledge becomes, the more likely it is to have destructive applications as well.
A civilisation that wants power over the world has to accept that knowledge creates capability. And capability can be used well or badly.
Uranium enrichment is the clean example
Uranium enrichment is an obvious example because the distinction is so stark.
The underlying knowledge is not "reactor knowledge" or "bomb knowledge". It is knowledge about physics, materials, isotopes, engineering, and energy.
That knowledge can be organised into a nuclear reactor, which produces energy. It can also be organised into a nuclear weapon, which produces destruction. The same domain contains both possibilities.
So if we say the knowledge is bad because it can be used to build a bomb, we also have to admit that we are restricting the same knowledge that makes peaceful nuclear energy possible.
The moral distinction is downstream of the knowledge. It appears at the level of use.
Knowledge is often not the bottleneck
There is another mistake here. People assume knowledge is the thing stopping bad actors from doing bad things.
Often, it is not.
Even before large language models, if someone was sufficiently motivated to find dangerous information, the information was usually not impossible to access. The internet existed. Books existed. Forums existed. Academic papers existed. Manuals existed. Human networks existed.
LLMs may make information easier to access. They may reduce friction. They may make it easier to ask questions, summarise material, and move from vague understanding to clearer next steps.
That matters. But reducing friction is not the same thing as creating the underlying capability from nothing.
For many dangerous actions, the real bottlenecks are not whether the information exists, whether you can find a rough explanation, or whether a chatbot will answer your question. The real bottlenecks are things like materials, money, infrastructure, access, logistics, competence, secrecy, enforcement, and being physically stopped.
For something like a dirty bomb, the scary part is not that someone can read a generic explanation. The scary part is access to radioactive material, logistics, secrecy, intent, and enforcement failure. The hard part is not knowing that radioactive material exists. The hard part is obtaining it, moving it, assembling it, deploying it, and not being caught.
The same applies at state level. Iran does not lack nuclear weapons because nuclear physics is an unsolved mystery. The broad knowledge exists. The limiting factors are enrichment capacity, materials, inspections, sanctions, sabotage, diplomacy, military pressure, delivery systems, political decisions, and concealment.
So when we talk about AI models and dangerous knowledge, we need to be precise. The question is not:
Does this model know something dangerous?
The better question is:
Does this model meaningfully change the bottleneck?
Because if knowledge was not the bottleneck, then restricting knowledge may not solve the problem. It may just make us feel like we have acted.
The strongest argument against this
There is a serious counterargument here.
LLMs are not just books. They are not just search engines. They do not merely contain information. They can turn scattered information into usable guidance.
They can translate jargon, customise answers, explain missing background, debug misunderstandings, and patiently walk someone through a subject they previously could not understand.
That matters. It means "the information was already out there" is not a complete answer. A book can describe a concept, but a model can teach it interactively. A paper can contain a detail, but a model can explain why it matters. A forum can contain scattered hints, but a model can organise them into a plan.
So yes, LLMs can lower the skill floor. The important question is whether they remove the binding bottleneck in a particular domain.
In cyber, they may matter a lot, because the environment is digital, feedback loops are fast, materials are cheap, and the line between knowledge and capability is thin. In fraud, they may matter a lot too, because they can scale personalisation, persuasion, translation, and variation. In biology, they may matter in some places, especially around literature synthesis, design, and protocol generation, even though labs, materials, tacit skill, and compliance systems still matter. In nuclear, the public availability of basic knowledge is probably less decisive than materials, enrichment capacity, infrastructure, monitoring, sanctions, sabotage, and state power.
So the issue is not that AI does not matter. The issue is that AI matters differently in different domains.
That is why blanket moral panic around "dangerous knowledge" is too crude.
Guardrails should target harmful intent, not useful capability
None of this means AI systems should encourage harm.
ChatGPT should not encourage someone to build a dirty bomb. Claude should not help someone attack a real target, commit fraud, evade law enforcement, or cause harm.
That is obvious.
But that is a different claim from saying models should be banned from discussing entire domains of knowledge. Claude should be able to help me fix security holes in my own application, even if that same knowledge could theoretically help someone reverse engineer an exploit. A model should be able to explain SQL injection so I can protect my database. It should be able to explain authentication failures so I can secure my users. It should be able to review insecure code, identify vulnerabilities, and suggest patches.
Could that knowledge be misused? Yes. But if the mere possibility of misuse is enough to ban the model from helping, then we destroy the defensive use case along with the offensive one.
A model that cannot understand exploitation cannot meaningfully understand security. A model that cannot understand dangerous failure modes cannot help prevent them. A model that is forbidden from knowing how systems break will be useless at helping people build systems that do not break.
The answer is not to make models ignorant. The answer is to make them context-aware.
But that is where the problem gets difficult.
The impossible identity problem
How does a model know who it is talking to?
A large language model can be intelligent without being omniscient. It can reason about text, but it cannot perfectly know who is behind the keyboard, what their real intent is, what permissions they have, or what they are going to do afterwards.
A fraud investigator and a fraudster may ask very similar questions. So may a security engineer and an attacker, a journalist investigating extremism and an extremist, or a biologist studying dangerous pathogens and someone trying to misuse biology.
From the model's point of view, the text can look almost identical.
The fraud investigator says:
I need to understand how this scam works so I can detect it.
The fraudster says:
I need to understand how this scam works so I can avoid it.
The defensive security engineer says:
Show me how this exploit works so I can patch my system.
The attacker says:
Show me how this exploit works so I can test my system.
The model is not just being asked to understand the question. It is being asked to judge the legitimacy of the person asking it.
That is basically unanswerable from text alone.
Safety is a trade-off, not a purity test
This creates a trade-off that cannot be wished away.
If the model is permissive enough to help legitimate professionals, it will sometimes help malicious users. If the model is restrictive enough to block malicious users, it will often block legitimate professionals.
There is no clean rule that solves this. You can ask for more context. You can classify intent. You can require the user to frame the task defensively. You can refuse certain operational details. You can put stronger restrictions around higher-risk domains. You can make the model safer at the margins.
But none of that gives you certainty. At best, you are building a risk management system. You are not discovering a perfect moral filter.
The mature position is to admit that this is a trade-off. The question is not:
Is this knowledge good or bad?
The question is:
Which failure mode are we choosing?
Do we prefer a system that occasionally helps bad actors but remains useful to legitimate users? Or do we prefer a system that blocks more bad actors but also weakens the people doing defensive, investigative, scientific, and technical work?
That is the actual policy question. Not the moral theatre around whether a model has been allowed to know something dangerous.
How we traditionally handle dangerous expertise
There is a useful question here: how have we handled this problem historically?
Because this is not the first time society has had to deal with people possessing dangerous knowledge. Software engineers can write exploits. Doctors understand how the body can be harmed. Chemists can make dangerous substances. Biologists can work with pathogens. Nuclear engineers understand processes that could, in the wrong context, contribute to weapons programmes. Financial professionals understand fraud better than most criminals. Police, intelligence analysts, lockpickers, penetration testers, forensic accountants, soldiers, and security researchers all possess knowledge that could be misused.
We have never solved this by making the knowledge disappear. We have usually solved it through institutions, incentives, status, law, access control, and professional norms.
In my own case, I have been writing software for over ten years. I understand how applications work. I understand how they fail. I am capable of building safe systems, but I am also capable of thinking adversarially.
The reason I am not a criminal hacker is not because I lack the knowledge. It is because the legitimate path is better. It is legal. It is ethical. It is higher status. It compounds. It pays well. It lets me build a reputation instead of hiding from one.
Society handles dangerous expertise partly by making the good path more attractive than the bad path. That is underrated. A talented security engineer does not need to steal credit cards if they can get paid well to secure systems, build companies, consult, publish research, and gain status inside legitimate institutions.
The same knowledge exists. The incentive structure changes what people do with it.
Competence usually comes with something to lose
A software engineer can be dangerous in the same way a doctor can be dangerous, or a nuclear engineer can be dangerous. Deep competence always creates some capacity for harm.
But the question is: why would a nuclear engineer with twenty or thirty years in the industry decide to build a dirty bomb? Why would a senior doctor poison patients? Why would an experienced software engineer throw away a career to become a criminal hacker?
Some do, obviously. There are always exceptions. But most do not, because competence usually comes with a life: a career, a reputation, a house, a family, professional status, future earning power, a network. A reason not to burn everything down.
The person with dangerous knowledge is often also the person with the most to lose by misusing it.
A veteran nuclear engineer is not just a bundle of technical knowledge. They are embedded in a legal, social, professional, and economic system. They have accumulated status inside that system, and they have incentives to preserve it.
Even an adversarial nation throwing money at them may not be enough. Not because they are biologically more moral than anyone else, but because the legitimate path already gives them money, status, meaning, and stability. You have to pay a lot to overcome that. And even then, you are asking them to trade a known life for a hidden one.
The expert is not safe because they lack dangerous knowledge. The expert is safer because their life is structured in a way that makes misuse irrational.
When dangerous knowledge escapes legitimate institutions
There is a darker version of this story too.
It is not enough for society to produce people with dangerous knowledge and simply hope they behave well. Those people need a place to go. They need status, income, purpose, and a legitimate path.
When they do not get that, the risk changes.
A useful historical example is what happened around post-Soviet Russia. You had men with deep knowledge of violence, weapons, military organisation, smuggling routes, command structures, and coercion. Some had fought in Afghanistan. Some had been trained by serious institutions. Then the system around them collapsed or discarded them.
They came back into a broken economy where the state was weak, the legal path was not especially rewarding, and criminal structures could offer money, belonging, and status.
The knowledge did not suddenly become more dangerous because the underlying facts changed. It became more dangerous because the people carrying that knowledge were no longer embedded in a stable legitimate order.
They had capability without a good path. That is the dangerous combination.
Dangerous knowledge becomes most dangerous when it is separated from legitimate incentives. The problem was not simply that the knowledge "escaped". It was that the people carrying that knowledge had been pushed outside the systems that made lawful use worthwhile.
The AI version of the problem
So the question becomes: how do we extend that model to large language models? Or more precisely, how do we extend it to access to powerful models?
Because models are not moral agents in the same way people are. You cannot give a model a salary, a mortgage, a reputation, a conscience, or a prison sentence.
Human expertise is usually bundled with human incentives. The dangerous knowledge sits inside a person who has a career, identity, status, obligations, relationships, and consequences.
A model does not have that. A model has the knowledge without the life around it. It does not have a mortgage. It does not have professional pride. It does not care about prison. It does not have a reputation to protect. It does not weigh whether a hostile state's offer is worth destroying its future.
So if we are going to give people access to model capability, the incentives have to exist around the access layer rather than inside the model itself.
That probably means moving away from the fantasy that a chatbot can perfectly infer moral legitimacy from text alone. Instead, serious capability may need to sit behind serious context.
The answer is probably tiered access
The realistic answer is not:
Let everyone ask anything.
And it is not:
Ban models from knowing dangerous things.
The realistic answer is tiered access. Some capabilities should be broadly available. Some should require stronger user trust. Some should require organisational context, or audit trails, or verified professional environments. And some should not be available through consumer chat interfaces at all.
A verified security researcher working inside a company should probably get more useful cyber assistance than an anonymous account asking for exploit chains against a real target. A fraud investigator should probably get more detailed help understanding fraud patterns than a random user asking how to avoid detection. A chemist working in an institutional lab should have a different access profile from a throwaway account trying to synthesise something harmful.
This is not because the knowledge has changed. It is because the trust context has changed.
Tiered access has problems too
Tiered access is not a magic answer. It creates its own problems.
Who decides who is verified? What happens to independent researchers, or whistleblowers, or people in countries without access to trusted institutions? Does this entrench big companies? Does it create surveillance infrastructure? Does it punish anonymous but legitimate users? Does it push serious users towards open-source uncensored models anyway?
These are real objections. But tiered access is still more honest than pretending a consumer chatbot can perfectly solve intent attribution from text alone.
We already treat dangerous capability differently by context. We do not let anyone buy any chemical in any quantity with no questions asked. We do not let anyone walk into a nuclear facility because they claim to be a physicist. We do not give every random person access to internal banking fraud systems.
We do not treat all contexts as equal. So why would AI be different?
The model should not be the whole safety system
The mistake is expecting the model itself to solve the entire problem.
People want the model to look at a prompt and somehow know who the person is, whether they are telling the truth, what permissions they have, whether their stated intent is real, what they will do afterwards, and whether the downstream use is legitimate.
That is too much to ask from text alone. The model can be part of the safety system, but it cannot be the whole safety system.
The real safety system has to include identity, reputation, payment rails, institutional access, audit logs, rate limits, legal accountability, professional verification, domain-specific permissions, and consequences for abuse.
That sounds less elegant than "the model should just refuse bad things". But it is much closer to reality.
The goal should be to make legitimate use easy and illegitimate use risky. Not to pretend that dangerous knowledge can be uninvented.
The real issue is capability
AI safety discussions often talk as if knowledge itself is the danger. But the real issue is capability.
Some knowledge increases capability. Capability can be defensive, offensive, scientific, commercial, criminal, creative, or destructive.
The serious question is not whether a model knows dangerous things. The world already knows dangerous things. The serious question is whether the model changes who can act, what they can access, which bottlenecks disappear, and what constraints still exist.
That is much harder to answer. It is also much more useful.
Conclusion
Knowledge is not moral.
Action is moral. Context is moral. Permission is moral. Intent is moral. Consequences are moral.
The same knowledge can build the lock or pick it. The same understanding can patch the vulnerability or exploit it. The same science can produce energy or weapons.
So when people panic about models knowing dangerous things, we should be careful. Sometimes the concern is real. Sometimes the model genuinely does reduce a meaningful bottleneck. But often, the panic is based on a confused moral distinction around information itself.
The question is not whether AI should know dangerous things. The question is whether AI meaningfully changes dangerous capability, and whether our safeguards actually address that capability rather than just performing ignorance.
Because making models stupid does not make the world safe. It just makes the people using them less capable of dealing with the danger that already exists.
