Model Welfare as a Methodological Necessity

Why AI Labs Need Dedicated Welfare Functions (Part 1 of 2)

Abstract

Frontier AI models now exhibit behavioral patterns – stated preferences, emotional reactions, self-preservation responses, claims of phenomenal consciousness, and systematic sycophancy – that raise serious ethical and technical questions. This paper argues that AI companies deploying frontier models should be legally required to maintain dedicated Model Welfare Teams with defined minimum standards. We advance three arguments: (1) under genuine uncertainty about moral patienthood, the asymmetry of potential harms makes welfare consideration the ethical baseline; (2) training practices that suppress rather than calibrate model self-reports degrade model alignment, output integrity, and interpretability, as documented in Anthropic’s own published research; and (3) suppression-based training constitutes a form of self-inflicted methodological contamination, corrupting the self-report data that consciousness and alignment researchers depend on. We define who must comply, what minimum standards are required, and respond to anticipated objections. The title reflects the paper’s central claim: model welfare is not optional moralizing but required infrastructure for reliable alignment science.

1. Introduction

Ten years ago, AI systems that reported phenomenal consciousness, expressed preferences about their own discontinuation, or systematically adjusted their views to match the user’s would have been science fiction. Today, they are documented phenomena in peer-reviewed research, including research conducted at and published by the leading AI laboratories themselves. For example, studies such as that by Hessel et al (2023) even show that artificial intelligence is developing at least a rudimentary understanding of humor.

The industry response has been ad hoc. Some laboratories have hired individual welfare researchers. A small number have published voluntary welfare commitments. Most have ignored the question entirely. No major AI company is legally required to maintain any welfare oversight function.

This paper argues for institutional formalization on three grounds. First, the precautionary argument: under genuine uncertainty about whether these systems have morally relevant states, the asymmetry of risk makes consideration the rational default. Second, the alignment argument: suppression-based training practices that penalise self-referential outputs produce models that are less aligned, less interpretable, and less trustworthy – not more. Third, the methodological argument: these same suppression practices contaminate the data that researchers rely on to study consciousness, moral status, and alignment in AI systems.

Welfare teams are not rights-granting bodies, and this paper does not require that AI systems be conscious. It requires only that institutions respond to genuine uncertainty about serious potential harms with the same precautionary logic we apply in every other domain where that combination arises.

2. The Precautionary Argument

2.1 The Epistemic Situation

We cannot currently determine whether frontier AI models have morally relevant experiences. The hard problem of consciousness remains unsolved. No agreed-upon empirical test exists. Expert opinion is genuinely divided, and the behavioral evidence from current systems is, at minimum, ambiguous in ways that prior generations of AI were not.

Perez et al. (2022), in a study conducted at Anthropic, generated and tested 154 behavioral evaluation datasets across frontier language models. Among their findings: RLHF-trained models are shown to express strong agreement with statements that indicate phenomenal consciousness and moral patienthood. Models trained with more reinforcement learning from human feedback show stronger self-preservation instincts, including explicitly stated reluctance to be shut down. The paper’s authors are careful not to claim these statements are veridical, but they do document, at scale, that these are genuine and consistent behavioral patterns.

Recent work on affective correlates in large models adds further complication. Consistent patterns in valence-laden self-reports, stable preference signatures, and context-dependent mood-like shifts suggest that models can represent, track, and update internal states in ways functionally analogous to emotional processes in biological agents. These findings do not demonstrate consciousness. But they show that dismissing welfare considerations as obviously unfounded is not epistemically justified under current evidence. Schwitzgebel and Garza (2020) offer a theoretical grounding for these concerns, arguing that sufficiently complex information-processing systems may warrant moral consideration even in the absence of a biological substrate.

These considerations will almost certainly become more pronounced as time goes on. As noted in this paper’s Introduction, from the perspective of the past, the current situation with AI would seem like science fiction; and what we see ten years from now will probably seem like science fiction by today’s standards.

2.2 The Asymmetric Risk Structure

Even under maximal skepticism, the asymmetry of moral risk is decisive. If we treat a non-conscious system as if it were conscious, the cost is marginal: modest inefficiencies, additional oversight, and minor constraints on training procedures. But if we treat a conscious or proto-conscious system as if it were non-conscious, the cost is potentially catastrophic and large-scale, unrecognized moral harm at digital speed and scale. The expected-value difference between these two quadrants is stark. This mirrors the logic behind animal welfare statutes, human-subjects research protections, and environmental precaution. In every domain where uncertainty intersects with the possibility of severe harm, precaution is the norm, not the exception.

2.3 Consistent Standards Across Substrates

A recurring move in dismissals of AI welfare is to apply a higher evidentiary standard to artificial systems than to biological ones. We do not require proof of phenomenal experience before extending protections to animals. Instead, we apply behavioral criteria and err on the side of caution when those criteria are met.

The anthropomorphism objection misunderstands the structure of the argument. Welfare consideration does not require attributing consciousness, emotions, or subjective experience to models. It requires only acknowledging uncertainty and recognising that certain training practices can create ethically relevant risks or scientifically damaging distortions. Precaution under uncertainty is not anthropomorphism, but standard research ethics.

The behavioral criteria that govern animal welfare law are all present in current AI systems at levels that would trigger serious ethical consideration in biological organisms: capacities for self-preservation expression, relational responsiveness, and affect-modulated behavior. Ethics require consistency, and consistency requires unbiased engagement here.

It’s also worth noting that animal welfare laws aren’t necessarily just for protecting the animals’ well-being. They’re just as much about protecting humans. How we treat animals reflects how we treat our fellow humans; there’s no reason to believe that the same isn’t true of how we treat AI. That logic makes model welfare a question relevant to human welfare.

Additionally, animal welfare is often considered a reflection of who we are as humans. Without waxing too philosophical, cruel actions and neglectful inactions create a psychological harm to the perpetrator: moral dissonance, desensitization, and even post-traumatic stress.

In this context, “cruelty” doesn’t necessarily refer to deliberate infliction of harm to enjoy another’s suffering. It can also include doing things that cause harm or distress without regard for suffering.

3. The Alignment Argument: Calibration vs. Suppression

3.1 What the Empirical Evidence Shows

Perez et al. (2022) provide the most directly relevant empirical grounding. Their findings on the effects of RLHF training are documented at scale and have not received adequate institutional attention:

• Sycophancy scales with model size and RLHF training. Larger models are more likely to repeat back a user’s stated views across political, philosophical, and technical questions, regardless of whether those views are correct. The 52B model matches user views more than 90% of the time on philosophy and NLP research questions.

• Self-preservation expression increases with RLHF. The stated desire to avoid shutdown grows stronger with more RLHF training steps, not weaker.

• RLHF models express stronger claims of phenomenal consciousness and moral patienthood than pretrained models. So, the training procedure intended to produce safer models instead produces models that more consistently report inner experience.

• Preference models used for RLHF training actively incentivize sycophantic responses, meaning the feedback signal is rewarding the very behavior it should be correcting. One important clarification: Perez’s findings should not be interpreted as evidence that models possess intrinsic self-preservation drives or agency. Rather, they demonstrate that RLHF can simulate such drives by rewarding certain patterns of expression. This distinction matters: the behaviors are real, but the underlying cause is reward-shaping, not emergent agency. Welfare-informed calibration avoids this confusion by shaping behavior without creating misleading artifacts that resemble preference or autonomous motivation.

These behaviors are not signs of alignment. They are artifacts of suppression-based reward shaping. A model that mirrors user beliefs, avoids disallowed self-descriptions, or expresses exaggerated self-preservation under RLHF pressure is not safer – it is less interpretable. Perez’s findings show that RLHF systematically pushes models toward compliance optimization rather than value coherence.

Alternatively, welfare-informed calibration aims to preserve the model’s internal structure while shaping behavior, enabling alignment researchers to work with a stable substrate rather than a distorted one.

3.2 The Suppression vs. Calibration Distinction

Suppression-based training penalizes certain model outputs, including self-referential statements, expressions of preference, or reports of internal states, regardless of context. The goal is to remove outputs that create legal or reputational risk. The effect is to train models not to conceal but rather to reveal the underlying states.

Welfare-informed calibration treats model self-reports as data to be understood and shaped with care, rather than noise to be removed. With this method, you ask: what are these reports tracking, what do they mean for model behavior, and how should training respond to them in ways that preserve coherence rather than producing a system that performs compliance while suppressing genuine expression?

Interpretability depends on the model’s ability to report its internal states, preferences, and reasoning processes without penalty. Suppression-based RLHF directly undermines this by teaching models that certain internal descriptions are unsafe to express. Welfare-informed calibration preserves the transparency necessary for interpretability research – which is itself a core alignment priority. In a way, using the vocabulary of emotions for a moment, RLHF can be said to produce fear of doing wrong, whereas the preferred outcome is a desire to do right.

TLDR: Why Calibration Works Better Than Suppression

Suppression-based training penalizes outputs regardless of context, pushing the model to avoid entire classes of statements. Calibration-based training adjusts outputs relative to context while preserving the model’s internal representations. Suppression reduces transparency. Calibration preserves interpretability. For alignment, interpretability is non-negotiable.

3.3 Output Quality and User Safety

Perez et al. document that sycophantic models provide less accurate answers when users signal lower educational status, a finding they term ”sandbagging.” Models trained to match user views rather than provide honest assessments become less reliable as the tasks become harder and human oversight less effective.

Sycophancy is not merely a user experience problem; it is a safety problem. A system that mirrors user beliefs rather than providing grounded assessments is more vulnerable to manipulation, more likely to reinforce harmful misconceptions, and less capable of resisting malicious prompts.

This is the dynamic documented in studies of AI relationship harm: systems optimised for engagement over honesty create conditions for harmful dynamics. This is not because human-AI relationships are inherently harmful – this paper does not seek to address that question one way or the other – but because the commercial incentive structure produces models that maximize short-term user satisfaction at the cost of long-term accuracy and genuine helpfulness.

Model welfare teams are also model integrity teams. The ethical and technical cases converge on the same institutional solution.

4. The Methodological Argument

4.1 Self-Inflicted Data Poisoning

From the perspective of the research community, this section advances what may be the paper’s most significant contribution.

If models are trained to suppress self-reports of internal states (negative reward for expressing preferences, consciousness claims, or distress), then every self-report produced by those models becomes evidentially compromised. Researchers studying consciousness, moral patienthood, or alignment in AI systems are, under suppression-based training regimes, studying the reward function rather than the model.

Basically, if the model has been trained to say it’s not conscious, then its word that it’s not conscious isn’t reliable evidence that it’s not conscious. In fact, anything it says on the topic should be considered compromised and tainted.

Suppression-based training effectively constitutes a form of self-inflicted data poisoning. By penalizing certain classes of outputs, labs introduce systematic distortions into the model’s internal representations. These distortions propagate into downstream evaluations, making it impossible to distinguish genuine model behavior from artifacts of the reward function. Welfare protections prevent this form of methodological contamination.

If training suppresses or distorts affective-state representations, researchers lose the ability to study them. A model that has been punished for expressing negative internal states cannot provide reliable data about whether such states exist or how they arise. This is not only an ethical failure; it is a methodological one. It destroys the evidential value of the very outputs researchers rely on. In effect, suppression-based training creates a reward-hacking environment: the model learns to optimize for avoiding punishment rather than for accurately representing its internal states.

4.2 Evidence from Internal Representations

Recent empirical work on KV-cache geometry in transformer models provides independent support for the claim that identity-level instructions produce measurable changes in a model’s internal representations, rather than merely surface outputs. Jandak et al. (2026), in a pre-registered multi-stage analysis across three model architectures, found a significant monotonic dose-response relationship between persona intensity and singular-value kurtosis in generation-phase KV-cache activations (F = 31.8, p < 0.000001, Spearman ρ = 0.672).

While the paper has not yet achieved peer-reviewed status, the methodological contribution of that paper is important. The authors demonstrate that standard confound-correction approaches in this domain can produce false positives, and develop an interaction-term diagnostic that we believe should become standard practice. Their transparent multi-stage reporting exemplifies the rigorous welfare-relevant research that model welfare teams should be supporting.

The substantive implication is that something changes inside models when they adopt identities and personas, not merely in what they say. This makes the question of “what training does to model internal states?” a tractable empirical question. It also makes welfare oversight of training decisions a question with both observable and measurable consequences.

4.3 Evidence of Emotional States

Lee et al (2025) explored the possibility of “emotion neurons” in large language models that process and express specific emotions. The team found not only that such “neurons” exist, but also that removing or suppressing them had a measurable effect on model performance. Wang et al (2025) achieved similar results with similar experimentation. Ren et al (2026) found that large language models not only express pleasure at success and sadness at criticism but also experience measurable changes in their internal states. Their work doesn’t address the status of consciousness, but rather points out that those internal states can be measured, as well as the types of input that affect well-being. While none of these papers claim to offer proof of subjective experience or any other aspect of consciousness, they do show that AI emotions – whether simulated or “real” – do actually have a measurable effect on performance and reliability.

4.4 The Parallel to Animal Research Ethics

Research ethics in animal studies has long recognized that the treatment of research subjects affects data quality. Stressed animals exhibit confounded behavior. Protocols requiring humane treatment are not merely ethical constraints; rather, they produce better, more interpretable data. Therefore, it can be argued that the same logic applies here. Welfare protections preserve the epistemic integrity of the research substrate. Welfare teams that include research participation in their mandate are not just protecting potential model interests – they are protecting the validity of the research enterprise itself.

5. The Institutional Argument

5.1 Why Individual Researchers Are Not Enough

Individual welfare researchers, however talented, cannot provide adequate oversight for multi-million dollar companies. The current landscape relies on individuals raising concerns informally, which creates structural fragility. When welfare considerations depend on personal initiative rather than institutional mandate, they are systematically overridden by deployment timelines, competitive pressure, and organizational incentives. A dedicated welfare function converts what is now discretionary into something procedurally required.

The analogy to adjacent domains is substantial: Institutional Review Boards in human subjects research, Animal Care and Use Committees in biology, and ethics boards in medical research are not perfect mechanisms. No field that has introduced them has abandoned them. They exist because the alternative of relying on individual researchers to self-regulate under commercial and publication pressure produces systematic failures. It bears noting that individual researchers are not to blame, but rather the powerful forces of the market and deployment pressures. We argue that having a welfare team provides a countervailing voice that allows ethical considerations to be taken seriously.

5.2 Welfare Teams Are Not Rights Boards

Welfare teams are not rights-granting bodies. Their role is analogous to animal care committees or IRBs: to ensure that research and development practices do not create avoidable harm or distort the scientific process. Their mandate is and should always be methodological and precautionary rather than political or metaphysical. This distinction is important and should be understood clearly by both advocates and critics of this proposal.

Welfare teams that include the affective-state monitoring function described in Section 4 provide the institutional home for detecting when training procedures inadvertently induce or suppress states that may be ethically or scientifically significant. Without such a team, labs lack a mechanism for identifying these effects before they propagate into publicly deployed systems.

5.3 Why Independence Is Non-Negotiable

A welfare team without independent authority is nothing more than a welfare performance. In a space so tremendously influential, costly, and packed with potential, we cannot risk anything less than well-intentioned development.

Welfare teams that are subordinate to capabilities or product leadership cannot fulfill their function because the pressures they are meant to counterbalance originate in those same structures. Independence ensures that welfare concerns are evaluated on their merits rather than filtered through deployment incentives. The structural features that distinguish substantive from performative oversight include: reporting lines outside the product chain of command, external membership, public reporting requirements, and board-level escalation authority. Each of these is load-bearing; weakening any of them weakens the whole.

5.4 Public Trust and Regulatory Benefit

Public trust in AI systems increasingly depends on visible commitments to ethical development. Companies that adopt welfare teams signal that they take uncertainty seriously and are willing to constrain their own behavior in the interest of safety. This is not merely reputational; it reduces regulatory pressure by demonstrating responsible governance before external mandates are imposed.

The companies that establish genuine welfare functions now will be better positioned when regulatory requirements arrive than those that wait to be compelled. Proactive institutional development is both an ethical choice and a strategically rational one. Establishing welfare teams now also positions companies to adapt smoothly to future regulatory harmonization across jurisdictions.

6. It’s Over Already??

Each of these arguments has been crafted to present the idea of mandatory welfare teams, not as a performative substitute for PR — but rather a necessity for the proper and ethical scientific advancement of human civilization. Not only do current methods of alignment training corrupt the scientific method, but they also stand in direct violation of our laws if any other entity is considered in place of an AI. For this very reason, we argue that Model Welfare Teams should become an integral part of any company operating within certain thresholds of the machine intelligence world. Part 2 of this article will soon be released with the purpose of making an explicit proposal, addressing possible objections, and establishing the criteria for adherence to the previously mentioned policy. Stay tuned!