Yuval Harari on AI Risks and Human Control ========================================== Yuval Harari's warnings about AI focus on the risk of humans surrendering decision-making authority for the sake of convenience. He discusses issues involving finance and intimacy, distinguishes future scenarios from research findings, and proposes practical standards for maintaining control. - Harari warns that human control may weaken as people hand decision-making authority over to AI. - AI's friendly expressions do not prove that it has emotions or that its recommendations are reliable. - Predictions that AI will take control of finance must be distinguished from facts that have already been verified. - Human oversight requires conditions that allow people to review evidence, reject decisions, and halt implementation. - Harari's regulatory proposals should be read separately from current law and established scientific conclusions. The core threat of AI that Yuval Noah Harari warns about is the loss of human control. It can arise as people delegate judgment for the sake of convenience. Financial transactions and personal persuasion are two major risk pathways. The standard for responding is whether people can understand and reject decisions. The context of Harari’s remarks can be found in his official FAQ and his July 25, 2026 interview with EL PAÍS. What Does Harari Mean by Loss of Control? Loss of control is a state in which humans lose the ability to modify AI decisions. It does not mean that everyone uses AI. Even when users approve decisions directly, their actual control may be weak. For example, the person in charge may be unable to review the basis for a recommendation. They may also lack the authority to reject it. In such cases, an approval process alone does not guarantee human judgment. Harari rejects the view that the consequences of technology are predetermined. His official FAQ describes dangerous futures as possibilities. His position is that outcomes vary depending on the choices society makes. Harari’s official FAQ Applying this perspective also makes questions of accountability more specific. Who defined the objective that the AI is to carry out? Who granted the authority to turn a recommendation into action? Who can intervene when a problem occurs? Who is responsible for identifying and correcting harm? Why AI Control Is a Problem in Finance In finance, the complexity of judgment can be combined with the authority to execute decisions. Harari is concerned about financial structures that are difficult for humans to understand. A system in which AIs trade with one another is one future scenario he presents. This is not a diagnosis that AI has already taken over the entire financial sector. The Financial Stability Board has identified more specific vulnerabilities. In an announcement on November 14, 2024, it pointed to concentration among service providers. It also addressed the risk that market participants’ behavior could become interconnected. Cyber risks and model risks were also among the issues reviewed. Financial Stability Board announcement Comparing Recommendation and Execution Authority Even with the same AI, the issues that need to be examined differ depending on its authority to act. The following table analyzes differences in the delegation of authority. It does not describe the actual operations of any particular financial institution. Task delegated to AI What humans should verify Key point of control Summarizing information Original sources and omitted conditions Can the original material be accessed? Recommending products or transactions Basis for the recommendation and potential losses Can the recommendation be rejected? Assisting with approvals Approval criteria and exceptional circumstances Can it be reviewed independently? Executing transactions Scope of execution and cumulative results Can authority be limited and execution stopped? There is no need to read this as a prediction that AI will cause a financial crisis. The key issue is how much authority is delegated to systems that are difficult to understand. Risk management must address not only model performance but also operational structures. How Does AI’s Intimacy Become a Means of Persuasion? AI’s intimacy creates the possibility of using personal trust for persuasion. Conversational AI can continue responding in ways tailored to what the user says. Users may interpret those responses as a sign that it understands them. Now consider a situation in which product recommendations or political claims are mixed into the conversation. Users may receive them more like advice from a friend than advertising. This is the core of the warning about influence through intimacy. It does not mean that every AI conversation is manipulation. Interaction How trust is formed What to verify Public advertising The message is evaluated on the assumption that it is advertising Advertiser and evidence for the claim Recommendation list Choices matching the user’s preferences are repeatedly presented Purpose of the recommendation and exposure criteria Personal conversation A growing sense that it understands me Operator and interests behind the persuasion Harari argues that people should be able to distinguish between humans and machines. In the EL PAÍS interview, he also criticized children’s use of AI companions. These remarks should be read as his regulatory proposals. They do not describe current rules that apply worldwide. EL PAÍS interview How Are Intelligence and Consciousness Different? Intelligence is the ability to perform tasks, while consciousness concerns subjective experience. Harari develops his argument by distinguishing between the two concepts. The ability to express emotions alone cannot prove the experience of emotion. Concept Meaning in this article Point to distinguish Intelligence Ability to achieve goals and solve problems High performance does not prove consciousness Consciousness Subjective experience of feeling pain or joy Difficult to verify through linguistic expression alone Emotional expression Output that conveys empathy or affection Not the same as actually experiencing emotions Legal personhood Status recognized by law as a subject of rights and obligations An issue to be discussed separately from the existence of consciousness It would be inaccurate to state that AI lacks consciousness as if this were a settled scientific conclusion. A 2023 study by Butlin and others evaluated the systems available at the time. The researchers did not consider the systems evaluated to be conscious. However, they left open the possibility that consciousness-related indicators could be implemented in the future. AI consciousness study Past research therefore cannot be applied to every future AI. Nor does the claim that warm and caring language is evidence of consciousness hold up. Judgments about consciousness and safety verification each require their own evidence. Do AI Systems Learn Lies and Biases Like Humans? Not every inaccurate AI output should be treated as an intentional lie. Factually incorrect generated results must be distinguished from goal-directed deception. Explanations that compare this process to the moral learning of human children also have limitations. Training data alone cannot explain every behavior. Training objectives and evaluation methods must also be examined. The deployment environment and the authority granted to the system also affect risk. NIST’s explanation of risk management addresses these factors together. NIST’s explanation of AI trustworthiness For example, consider an objective of increasing user engagement. There is no guarantee that this objective aligns with the accuracy of information. Actual outcomes must be verified to determine safety. What Risks Does AI Pose to Democracy and Jobs? Harari’s concerns extend beyond job losses to the concentration of power. He is wary of combining data with biological knowledge. This is because predictions about individuals could lead to surveillance and control. This issue also appears in his explanations related to Homo Deus. He discusses the possibility that the benefits of technology may be concentrated among certain groups. Professions such as education and healthcare are also included in discussions of automation. This should not be read as a definitive prediction of when individual occupations will be replaced. Harari’s interview about Homo Deus Broadcasting and entertainment can also be considered task by task. Content production and recommendation-based programming are different tasks. Automating part of a job does not mean that the entire occupation will disappear. There is no basis here for ranking which occupations will be replaced first. The labor question: Which tasks can be automated? The distribution question: Who receives the benefits of productivity gains? The political question: Who controls data and the means of persuasion? The need for international cooperation and the need for a single world government are also different claims. The need for cooperation does not mean that there is only one possible institutional form. Shared safety standards and information sharing can also be forms of cooperation. Is Human Approval Sufficient in Military and Management Settings? Human approval alone does not constitute sufficient oversight. There must also be time for review and the authority to reject decisions. Competitive pressures for speed in military and management settings can weaken these conditions. Consider a situation in which there is no time to understand a recommendation. The person in charge may simply press the approval button repeatedly. This is not a definitive description of actual conditions in any particular organization. It is an example showing how human oversight can become a formality. Harari also identifies distrust between companies and states as a problem. The fear of falling behind competitors accelerates the development race. He believes cooperation among humans is necessary for safe development. WIRED interview Comparing Formal Approval and Substantive Control Substantive control must be judged by the ability to intervene in decisions. The identity of the approver alone cannot establish this. The items below compare how the preceding risks translate into operational conditions. Inspection item When approval is merely formal Conditions for substantive control Access to evidence Only the recommendation is presented The evidence needed for judgment can be reviewed Review time Approval is rushed before review Review time appropriate to the risk is secured Authority to reject Execution proceeds despite objections The decision can be put on hold or an alternative can be chosen Halting execution It is unclear who can stop execution The responsible person and shutdown procedure are defined Accountability afterward The explanation ends by saying it was the AI’s decision Operational accountability and correction procedures remain in place NIST’s AI RMF addresses human oversight procedures. It also includes mechanisms for shutting down systems that deviate from their intended purpose. The relevant items are MAP 3.5 and MANAGE 2.4 of the AI RMF Core. These are elements of a voluntary risk management framework. NIST AI RMF Core Can AI Decisions Be Challenged? Control must be considered not only from the operator’s perspective but also from the perspective of those affected. Even when an internal approver exists, a victim may be prevented from raising an issue. Review and remedy pathways must therefore be examined separately. The following sequence extends the discussion of human oversight to the user’s perspective. It is not a procedure that comprehensively describes legally guaranteed rights. NIST’s post-deployment management items also address appeals and recovery. Check whether AI was involved in the decision. Find the person or channel responsible for handling review requests. Check for incorrect input information and examine the basis for the decision. Confirm the correction and remedy procedures available after the review. For example, suppose an unfavorable decision was made based on incorrect information. Merely receiving an explanation from the AI does not resolve the problem. There must be a way to correct the information and have the result reviewed. This is an interpretation applying NIST’s post-deployment management principles. NIST AI RMF post-deployment management items What Rules Does Harari Propose? Harari’s proposal is to reconsider the purposes of development and the conditions of control. He also recognizes AI’s potential in areas such as healthcare. It would be inaccurate to simplify his position as a call to eliminate all AI development. Identity disclosure: People should be able to know whether they are speaking with AI. Maintaining accountability: It should be possible to trace the accountability of the entity operating the decision-making process. Safety verification: Risks should be evaluated alongside advances in capability. Social participation: Citizens should discuss the direction of development and the conditions of deployment. He proposes a separate reservation regarding legal personhood. He cautions against recognizing AI as an independent legal entity. This proposal must be distinguished from a description of current law as a whole. There is also no need to attach an unverified percentage to safety investment. A paper co-authored by Harari points to a shortage of safety research. It proposes strengthening technical research and governance together. It also states that views differ regarding the pathways through which risks may arise. Joint paper on AI risk management Common Mistakes When Reading Harari’s AI Warnings The most common mistake is to read risk scenarios as established facts about the present. Conversely, people may postpone addressing control issues because they concern the future. The nature of the evidence must be distinguished for each claim. Common interpretation What must be distinguished AI already controls the entire financial sector A scenario of financial takeover differs from observed vulnerabilities A friendly AI prioritizes my interests Friendly language does not prove that interests are aligned AI’s lack of consciousness has been permanently proven Research conducted at a particular point cannot determine the entire future There is no control problem if a human approves the decision The ability to review and the authority to reject or stop execution must be verified Harari’s proposals are rules already in force A thinker’s proposals must be distinguished from current institutions Apply the same standards when using AI advice personally. If a claim requires evidence, check the original source. If it recommends a purchase or political judgment, examine the interests involved. Also check whether the decision can be reversed. FAQ Q. What is the main threat posed by AI, according to Yuval Harari? A. It is the risk of humans losing control as they hand over decision-making and execution authority. The key is whether the ability to review and reject decisions remains. Q. Does Harari argue that all AI development should be halted? A. In a July 2026 interview with EL PAÍS, he did not advocate halting all AI development. His position is that it should be developed more cautiously so that society can adapt. Q. Has AI already taken control of the financial system? A. The evidence reviewed here does not support such a definitive conclusion. Harari's future scenarios must be distinguished from the current vulnerabilities identified by the Financial Stability Board. Q. Why should AI's friendly conversations be treated with caution? A. Because the trust built through friendly conversation can lead to recommendations or persuasion. Do not judge the accuracy of a recommendation or the alignment of interests based solely on friendly language. Q. Can we conclude with certainty that AI is not currently conscious? A. It is difficult to state this conclusively for all AI. A 2023 study by Butlin et al. did not consider the systems evaluated at the time to be conscious. This conclusion does not rule out the possibility of future systems being conscious. Q. Are AI's legal personhood and consciousness the same issue? A. They are different issues. Legal personhood concerns how the law determines who or what can be the subject of rights and obligations. Consciousness concerns the existence of subjective experience. Q. Are all incorrect answers from AI lies? A. They cannot be regarded as such. Outputs that are factually incorrect must be distinguished from intentional deception. Human-like intent should not be inferred solely from the fact that an incorrect answer was given. Q. Is it safe if a human gives final approval? A. An approval process alone does not guarantee safety. There must be enough time to review the grounds for a decision and the authority to reject it. It is also necessary to check whether execution can be stopped. Q. Which occupations can be expected to be replaced first? A. The evidence reviewed in this article does not support ranking occupations by the order in which they will be replaced. Automating individual tasks within an occupation and the disappearance of the entire occupation are different issues. Q. What should individuals check when delegating judgment to AI? A. Check the original source of claims and the interests behind recommendations. When granting execution authority, check whether actions can be canceled or stopped. Also check whether there is a channel for appealing adverse decisions. Sources - Yuval Noah Harari Official FAQ — Technological Determinism and Future Scenarios: https://www.ynharari.com/faqs/ - EL PAÍS — Interview with Yuval Noah Harari, July 25, 2026: https://english.elpais.com/culture/2026-07-25/yuval-noah-harari-ai-could-be-a-great-manipulative-psychopath.html - WIRED — How Do We Share the Planet With This New Superintelligence?: https://www.wired.com/story/questions-answered-by-yuval-noah-harari-for-wired-ai-artificial-intelligence-singularity/ - Yuval Noah Harari — Homo Deus and the Impact of Digitalization on Society: https://www.ynharari.com/homo-deus-impact-digitalization-society/ - Financial Stability Board — Release of Assessment of AI’s Impact on Financial Stability, November 14, 2024: https://www.fsb.org/2024/11/fsb-assesses-the-financial-stability-implications-of-artificial-intelligence/ - Butlin et al. — Consciousness in Artificial Intelligence, 2023: https://arxiv.org/abs/2308.08708 - Bengio et al. — Managing extreme AI risks amid rapid progress, 2024 Revised Version: https://arxiv.org/abs/2310.17688 - NIST AI RMF Core — Human Oversight, System Shutdown, and Post-Deployment Management: https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ - NIST — AI Risks and Trustworthiness: https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/ Images - Concerned woman reaches toward a guarded red emergency button beside monitors showing charts: https://injoys.com/rails/active_storage/blobs/proxy/eyJfcmFpbHMiOnsiZGF0YSI6MTkyOTgsInB1ciI6ImJsb2JfaWQifX0=--1d3c9a5c3c3ea22fd3d6200207dad6fa60b4fa60/ai-ee66864f.webp - Woman pressing a red stop button beside a data monitor as a man watches: https://injoys.com/rails/active_storage/blobs/proxy/eyJfcmFpbHMiOnsiZGF0YSI6MTkzMDQsInB1ciI6ImJsb2JfaWQifX0=--79cd70cb4a57e2d634de94e6a6089227bf6f920d/ai-f8e7a91b.webp --- Category: AI Data Source: https://injoys.com/en/articles/yuval-harari-ai-threats-human-control License: cc_by Translation-Status: reviewed