Key Takeaways
- The author faced a medically uncertain decision about prophylactic oophorectomy and consulted multiple human experts before turning to ChatGPT for additional input.
- AI did not make the final choice, but it reshaped the “architecture of trust” by altering whose advice felt authoritative and how information was weighed.
- Modern AI systems move beyond simple automation bias; they become conversational partners that remember context, explain reasoning, and adapt to the user, fostering a relationship‑like dynamic.
- When AI is framed as a “teammate” rather than a mere tool, users place greater trust in it and are less likely to challenge its suggestions, shifting the social context of decision‑making.
- The “human in the loop” safeguard assumes human judgment remains independent, yet the very judgment is being shaped by the AI’s framing, filtering, and relational influence.
- To responsibly integrate AI into high‑stakes domains (medicine, law, governance), we must design systems that make the machine’s influence visible, contestable, and subject to review, rather than merely ensuring a human signs off.
A Personal Decision Under Uncertainty
On Monday, September 7, I decided to part ways with my ovaries. “It was not clear that anything was wrong with them. My latest ultrasound looked slightly less concerning than the one before it. And yet, after intensive testing and consultations, I decided that my ovaries and I had reached the end of our journey together.” The choice was fraught with uncertainty—balancing risk, family history, quality of life, probabilities, and personal preferences—so I leaned on trusted human authorities: a senior physician sister and one of Israel’s leading gynecologists, both of whom advised surgery.
Turning to ChatGPT for a Second Opinion
The night after speaking with them, I also turned to ChatGPT. “We have been talking for several years now—I was an early adopter—and over the years it has become far more capable, more involved, and better able to remember what I have told it.” The model replayed my ultrasound printouts, “analyzed the ultrasound printouts and found meaning in millimeters,” laid out arguments, dismantled them, and repeated explanations until I felt satisfied. Its nonhuman precision offered a perspective that differed from my sister’s, leaving me feeling momentarily lost and prompting me to add my longtime family physician to the discussion.
A Marketplace of Trust
In the end, ChatGPT did not decide whether I would undergo surgery; I did. But I made that decision within something like a “marketplace of trust.” I recalled that the ultrasound technician had been less concerned, the specialist recommended surgery, my sister thought it through with me, the AI saw things differently, and finally my family physician joined the discussion. Each source carried a different “trust coefficient” for me—compounded of expertise, familiarity, prior experience, intimacy, availability, and perceived interests. Thus, artificial intelligence entered the “architecture of trust” through which I determined whom to listen to.
Beyond Automation Bias: The Evolving Human‑Machine Relationship
For decades, scholars warned of “automation bias”: the tendency to accept an automated system’s recommendation even when independent verification is warranted. The new generation of AI, however, changes the nature of the problem. “The ‘machine’ is no longer a warning light in a cockpit. We talk to it. It explains itself. We argue with it and it responds. We ask it to consider an alternative and it does.” Reliance on a machine is shifting from a one‑way encounter into a relationship, which fundamentally alters how we must think about AI‑assisted decision‑making.
The Hidden Power of Framing and Summarizing
Even when a machine sits “only on the desk,” it can shape outcomes by determining which documents a judge sees first, what is included in a case summary, which precedent is highlighted, and which legal question is framed as central. “Even summarizing is not a cognitively neutral act. Whoever organizes the material for us also participates in organizing the way we think about it.” Consequently, the “human in the loop” safeguard overlooks the fact that the decision space itself has been pre‑structured by the AI before the human ever reaches the point of choice.
When the Tool Becomes a Teammate
Research shows that framing matters: presenting the same system as a “teammate” rather than a “tool” increases trust and changes interaction patterns. In “smart boardroom” simulations with senior executives, an AI that allocated speaking time and commented on behavior was rarely challenged, while advisors given to CEOs and CFOs were soon referred to as “my adviser,” and their interpretations were defended loyally. The AI did not make decisions for the humans; it became an actor within the social system, acquiring authority, inspiring loyalty, altering language, and legitimizing certain behaviors.
Reflecting on My Night with ChatGPT
That night, ChatGPT did more than place another piece of information alongside the doctors’ opinions. “By then, we had a history of conversations. I knew how to question it, and it knew a great deal about my circumstances. I could challenge its reasoning, and it could defend its position or revise its analysis.” This ongoing relationship begged the question of how the interaction itself altered the position from which I weighed everyone else’s advice. The decision emerged not from a solitary human judgment but from a system comprising a woman, a machine, doctors, tests, knowledge, uncertainty, expertise, and trust.
Designing for Visibility and Contestability
Human judgment has never been pure or free from influence; we have always operated within systems of influence—judges listen to lawyers, doctors consult colleagues, we rely on trusted confidants. Institutions such as legal procedure, medical ethics, and administrative law attempt to make those influences visible, contestable, and reviewable. “Perhaps that is also what we will need to do with artificial intelligence. We should not be satisfied with asking whether we left a human being in the loop. We need to understand what kind of relationship is being created between the human and the machine: what authority the machine acquires in the person’s eyes, how trust in it is built, whether it encourages resistance and skepticism or instead weakens the tendency to question, and how its very presence changes the way a person sees the problem before them.”
Moving Forward: Shaping the Relationship Within the Loop
The author concludes that AI did not make the decision for her, nor did she make it unaided; the outcome was the product of a distributed cognition network. As medical, legal, governmental, and professional decisions increasingly arise from similar hybrid systems, our challenge is not merely to ensure a human remains in the loop but to understand and shape the relationships within that loop—making the machine’s influence transparent, open to challenge, and aligned with democratic and ethical ideals. Only then can we harness AI’s strengths without surrendering the integrity of human judgment.
https://en.idi.org.il/articles/66416