Yuval Harari on AI Risks and Human Control

The core threat of AI that Yuval Noah Harari warns about is the loss of human control. It can arise as people delegate judgment for the sake of convenience. Financial transactions and personal persuasion are two major risk pathways. The standard for responding is whether people can understand and reject decisions.

The context of Harari’s remarks can be found in his official FAQ and his July 25, 2026 interview with EL PAÍS.

What Does Harari Mean by Loss of Control?

Loss of control is a state in which humans lose the ability to modify AI decisions. It does not mean that everyone uses AI. Even when users approve decisions directly, their actual control may be weak.

For example, the person in charge may be unable to review the basis for a recommendation. They may also lack the authority to reject it. In such cases, an approval process alone does not guarantee human judgment.

Harari rejects the view that the consequences of technology are predetermined. His official FAQ describes dangerous futures as possibilities. His position is that outcomes vary depending on the choices society makes. Harari’s official FAQ

Applying this perspective also makes questions of accountability more specific.

• Who defined the objective that the AI is to carry out?
• Who granted the authority to turn a recommendation into action?
• Who can intervene when a problem occurs?
• Who is responsible for identifying and correcting harm?

Why AI Control Is a Problem in Finance

In finance, the complexity of judgment can be combined with the authority to execute decisions. Harari is concerned about financial structures that are difficult for humans to understand. A system in which AIs trade with one another is one future scenario he presents. This is not a diagnosis that AI has already taken over the entire financial sector.

The Financial Stability Board has identified more specific vulnerabilities. In an announcement on November 14, 2024, it pointed to concentration among service providers. It also addressed the risk that market participants’ behavior could become interconnected. Cyber risks and model risks were also among the issues reviewed. Financial Stability Board announcement

Comparing Recommendation and Execution Authority

Even with the same AI, the issues that need to be examined differ depending on its authority to act. The following table analyzes differences in the delegation of authority. It does not describe the actual operations of any particular financial institution.

There is no need to read this as a prediction that AI will cause a financial crisis. The key issue is how much authority is delegated to systems that are difficult to understand. Risk management must address not only model performance but also operational structures.

How Does AI’s Intimacy Become a Means of Persuasion?

AI’s intimacy creates the possibility of using personal trust for persuasion. Conversational AI can continue responding in ways tailored to what the user says. Users may interpret those responses as a sign that it understands them.

Now consider a situation in which product recommendations or political claims are mixed into the conversation. Users may receive them more like advice from a friend than advertising. This is the core of the warning about influence through intimacy. It does not mean that every AI conversation is manipulation.

Harari argues that people should be able to distinguish between humans and machines. In the EL PAÍS interview, he also criticized children’s use of AI companions. These remarks should be read as his regulatory proposals. They do not describe current rules that apply worldwide. EL PAÍS interview

How Are Intelligence and Consciousness Different?

Intelligence is the ability to perform tasks, while consciousness concerns subjective experience. Harari develops his argument by distinguishing between the two concepts. The ability to express emotions alone cannot prove the experience of emotion.

It would be inaccurate to state that AI lacks consciousness as if this were a settled scientific conclusion. A 2023 study by Butlin and others evaluated the systems available at the time. The researchers did not consider the systems evaluated to be conscious. However, they left open the possibility that consciousness-related indicators could be implemented in the future. AI consciousness study

Past research therefore cannot be applied to every future AI. Nor does the claim that warm and caring language is evidence of consciousness hold up. Judgments about consciousness and safety verification each require their own evidence.

Do AI Systems Learn Lies and Biases Like Humans?

Not every inaccurate AI output should be treated as an intentional lie. Factually incorrect generated results must be distinguished from goal-directed deception. Explanations that compare this process to the moral learning of human children also have limitations.

Training data alone cannot explain every behavior. Training objectives and evaluation methods must also be examined. The deployment environment and the authority granted to the system also affect risk. NIST’s explanation of risk management addresses these factors together. NIST’s explanation of AI trustworthiness

For example, consider an objective of increasing user engagement. There is no guarantee that this objective aligns with the accuracy of information. Actual outcomes must be verified to determine safety.

What Risks Does AI Pose to Democracy and Jobs?

Harari’s concerns extend beyond job losses to the concentration of power. He is wary of combining data with biological knowledge. This is because predictions about individuals could lead to surveillance and control.

This issue also appears in his explanations related to Homo Deus. He discusses the possibility that the benefits of technology may be concentrated among certain groups. Professions such as education and healthcare are also included in discussions of automation. This should not be read as a definitive prediction of when individual occupations will be replaced. Harari’s interview about Homo Deus

Broadcasting and entertainment can also be considered task by task. Content production and recommendation-based programming are different tasks. Automating part of a job does not mean that the entire occupation will disappear. There is no basis here for ranking which occupations will be replaced first.

• The labor question: Which tasks can be automated?
• The distribution question: Who receives the benefits of productivity gains?
• The political question: Who controls data and the means of persuasion?

The need for international cooperation and the need for a single world government are also different claims. The need for cooperation does not mean that there is only one possible institutional form. Shared safety standards and information sharing can also be forms of cooperation.

Is Human Approval Sufficient in Military and Management Settings?

Human approval alone does not constitute sufficient oversight. There must also be time for review and the authority to reject decisions. Competitive pressures for speed in military and management settings can weaken these conditions.

Consider a situation in which there is no time to understand a recommendation. The person in charge may simply press the approval button repeatedly. This is not a definitive description of actual conditions in any particular organization. It is an example showing how human oversight can become a formality.

Harari also identifies distrust between companies and states as a problem. The fear of falling behind competitors accelerates the development race. He believes cooperation among humans is necessary for safe development. WIRED interview

Comparing Formal Approval and Substantive Control

Substantive control must be judged by the ability to intervene in decisions. The identity of the approver alone cannot establish this. The items below compare how the preceding risks translate into operational conditions.

NIST’s AI RMF addresses human oversight procedures. It also includes mechanisms for shutting down systems that deviate from their intended purpose. The relevant items are MAP 3.5 and MANAGE 2.4 of the AI RMF Core. These are elements of a voluntary risk management framework. NIST AI RMF Core

Can AI Decisions Be Challenged?

Control must be considered not only from the operator’s perspective but also from the perspective of those affected. Even when an internal approver exists, a victim may be prevented from raising an issue. Review and remedy pathways must therefore be examined separately.

The following sequence extends the discussion of human oversight to the user’s perspective. It is not a procedure that comprehensively describes legally guaranteed rights. NIST’s post-deployment management items also address appeals and recovery.

1. Check whether AI was involved in the decision.
2. Find the person or channel responsible for handling review requests.
3. Check for incorrect input information and examine the basis for the decision.
4. Confirm the correction and remedy procedures available after the review.

For example, suppose an unfavorable decision was made based on incorrect information. Merely receiving an explanation from the AI does not resolve the problem. There must be a way to correct the information and have the result reviewed. This is an interpretation applying NIST’s post-deployment management principles. NIST AI RMF post-deployment management items

What Rules Does Harari Propose?

Harari’s proposal is to reconsider the purposes of development and the conditions of control. He also recognizes AI’s potential in areas such as healthcare. It would be inaccurate to simplify his position as a call to eliminate all AI development.

• Identity disclosure: People should be able to know whether they are speaking with AI.
• Maintaining accountability: It should be possible to trace the accountability of the entity operating the decision-making process.
• Safety verification: Risks should be evaluated alongside advances in capability.
• Social participation: Citizens should discuss the direction of development and the conditions of deployment.

He proposes a separate reservation regarding legal personhood. He cautions against recognizing AI as an independent legal entity. This proposal must be distinguished from a description of current law as a whole.

There is also no need to attach an unverified percentage to safety investment. A paper co-authored by Harari points to a shortage of safety research. It proposes strengthening technical research and governance together. It also states that views differ regarding the pathways through which risks may arise. Joint paper on AI risk management

Common Mistakes When Reading Harari’s AI Warnings

The most common mistake is to read risk scenarios as established facts about the present. Conversely, people may postpone addressing control issues because they concern the future. The nature of the evidence must be distinguished for each claim.

Apply the same standards when using AI advice personally. If a claim requires evidence, check the original source. If it recommends a purchase or political judgment, examine the interests involved. Also check whether the decision can be reversed.