Hierarchy
HMIA 2025
HMIA 2025
TRIVIAL COOPERATION: shared goals and information - pick a goal and execute
Hobbes' observation: scarcity + similar agents→competition life is brutish&short
Hobbes' fix: cede sovereignty to boss with credible enforcement. Command→order
Command Failure Mode (preferences)
Agents retain autonomy→effort substitution & selective obedience
Command Failure Mode (information)
Orders incomplete & ambiguous, environments shift.
Principals and Agents
From commands to contracts. Alignment by design: selection, monitoring, incentives to align autonomy with principals goals.
Agent as RL learner.
Naked RL is a clean micro-model: the agent updates a policy to maximize rewards.
Goodhart risk: m(·) omits what drives V(·), maximizing T(m(a)) reduces Us. Gaming, reward hacking, short termism.
Requires governance and guardrails. Lagged, hard-to-game proxies, HITL overrides, team rewards, culture, the "alignment stack"
Incentives are transfers on signals.
\text{Let measurements be }
𝑚(𝑎_𝐴); \text{incentives } 𝑇(𝑚(𝑎_𝐴) \\
\text{Agent utility:}
𝑈_𝐴(𝑎)=𝑇(𝑚(𝑎))−𝐶_𝐴(𝑎)+𝐼_𝐴(𝑎) \\
\text{Principal utility: }
𝑈_𝑆(𝑎)=𝑉(𝑎)−𝑇(𝑚(𝑎))−𝐾_𝑆(𝑎)
As soon as behavior is driven by 𝑇(𝑚(𝑎)) T(m(a)), the problem is no longer obedience—it’s measurement.
But T(m(a)) is always a lossy compression of what matters.
HMIA 2025
TRIVIAL COOPERATION: shared goals and information - pick a goal and execute
Hobbes' observation
scarcity + similar agents
competition
life is nasty, brutish, & short
→
HMIA 2025
Hobbes' Fix
cede sovereignty to boss with credible enforcement.
Command→order
HMIA 2025
Command Failure Mode (preferences)
Agents retain autonomy→effort substitution & selective obedience
Command Failure Mode (information)
Orders incomplete & ambiguous, environments shift.
Obedience Failure Modes
"THE" Principals and Agents Problem
Suppose P needs some help realizing their goals and objectives
U_P
A could provide help, but P and A care about different things
U_P \ne U_A
Read: A is not going to get out of assisting P the same thing that P will get out of having some help.
From commands to contracts. Alignment by design: selection, monitoring, incentives to align autonomy with principal's goals.
Principal's Challenge: design an intervention that aligns their utilities
You want "do the work I need" to be agent's "best option"
\arg\max_{a_A} U_A(a_A) \longrightarrow \arg\max_{a_A} U_P(a_A)
HMIA 2025
To get this agent to work for us, we need to sweeten things
An ordinary RL agent as our model
Agent does action a in state s and garners reward - has some utility
U_A(a_A)
Add a transfer based on the behavior
U'_A(a_A) = U_A(a_A) + T(a_A)
HMIA 2025
Pay for time
Piece rates: pay per widget
Peformance bonus above threshold
Promotion (but delayed and fuzzy)
U'_A(a_A) = U_A(a_A) + T(a_A)
STOP+THINK: What are some common ways humans design T(aA)?
In AI/RL we call it "reward shaping" - tweaking reward signal to get desired behavior.
Kerr's Argument
T(a_A) \text{ is always actually } T(metric(a_A))
HMIA 2025
Incentives are transfers on signals.
\text{Let measurements be }
𝑚(𝑎_𝐴); \text{incentives } 𝑇(𝑚(𝑎_𝐴)) \\ \; \\
\text{Agent utility: }
𝑈_𝐴(𝑎)=𝑇(𝑚(𝑎))−𝐶_𝐴(𝑎)+𝐼_𝐴(𝑎) \\ \; \\
\text{Principal utility: }
𝑈_𝑆(𝑎)=𝑉(𝑎)−𝑇(𝑚(𝑎))−𝐾_𝑆(𝑎)
T, the incentive, is based on an observable signal.
m(a_A)
STOP+THINK: How do we read these equations?
\text{Credit Assignment Problem - hard to know what }a_A \text{ makes the difference in }U_P
STOP+THINK: What is an everyday example of credit assignment problem?
HMIA 2025
As soon as behavior is driven by 𝑇(𝑚(𝑎)), the problem is no longer obedience—it’s measurement.
T(m(a)) is always a lossy compression of what matters.
STOP+THINK: What's that mean?
STOP+THINK: What's that mean?
Goodhart's Law: "when a measure becomes a target, it ceases to be a good measure"
STOP+THINK: Examples of measure becoming a target?
STOP+THINK: What's that mean?
HMIA 2025
\text{Goodhart risk: }\\ \; \\ \text{If }m(·)\text{ omits what drives }V(·), \\ \text{ maximizing }T(m(a))\text{ reduces }U_P.
\\ \; \\ \; \\
\text{ Gaming, reward hacking, short termism.}
HMIA 2025
Takeaway
Hierarchy - alignment by command and incentive, requires:
- governance and guardrails.
- Lagged, hard-to-game proxies;
- Human in the loop (HITL) overrides;
- team rewards;
- culture;
- the whole "alignment stack."
Example: Working Retail
tl;sc
managing humans in retail might be easier in some ways than managing robots in retail
STOP+THINK: If I am a retail company and I hire you for my store, what is my utility?
STOP+THINK: What signals about your work actions do your manager have?
STOP+THINK: What rewards do you respond to in the course of your job?
- Internalization (norms of politeness, honesty): Values like honesty and helpfulness have been learned long before the job starts. (hiring)
- Feedback Loops (enjoyment of helping): Smiles and gratitude from customers reinforce the behavior emotionally. (organizational culture)
- Peer Signaling: Co-workers' approval or disapproval regulates over-performance and conformity. (organizational culture, hiring)
- Managerial Oversight: Rules, checklists, and performance targets structure effort and attention. (organizational routines)
- Cultural Framing: The meaning of "good service" or "professionalism" is stabilized through shared language, training videos, and informal stories. (organizational culture)
STOP+THINK: What other than pay and review on stats might contribute to variance in the a sub A you deliver?
STOP+THINK: Which of these levers and influences are or are not available if we want to use robots in retail?
- Internalization (norms of politeness, honesty)
- Feedback Loops (enjoyment of helping)
- Peer Signaling
- Managerial Oversight
- Cultural Framing
|
Mechanism |
ML Analogy |
Availability |
Approximation |
Alignment Challenge |
Guardrails |
|---|---|---|---|---|---|
| Internalization | Value embedding | Partially | Hard constraints (no deceitful claims); constitutional rules (“prefer safe/transparent acts”); escalation policies | How to represent and update social norms that are situational and culturally coded | Periodic red-team tests; human review of edge cases; log-and-explain decisions |
| Feedback loops | RL signals | Yes (risky) | Pair short-term CSAT with lagged outcomes (30-day returns, complaint rate, repeat purchases) | Avoid “reward hacking” (e.g., maximizing smiles/positive tone without real help) | Penalize “star-begging” patterns; randomize survey prompts; weight durable outcomes more than immediate smiles |
| Peer signaling | Multi-agent coordination | Partially | Shared queue health; handoff quality; “assist” events credited to both giver and receiver | Maintain cooperation without competition for metrics | Team-level rewards for cooperation; cap individual metrics that encourage hoarding easy cases |
| Managerial oversight | Monitoring & human-in-the-loop | Yes | Real-time dashboards; safe-interrupt (“stop/ask human”); audit trails | Ensure corrigibility: robot accepts override and interprets feedback appropriately | Reward acceptance of human override (no penalty for deferring); require explanations for high-impact actions |
| Cultural framing | Brand policy | Yes (with care) | Style guides → operational checks (offer alternatives, confirm understanding, follow-up reminders) | Translate vague brand values (“friendly,” “helpful,” “authentic”) into operational behavior | Calibrate on customer narratives, not just tone scores; fairness audits across customer segments |
Example: Experts
tl;sc
experts require regulation beyond incentives
STOP+THINK: What does society hope for from expert intelligence?
Society (the principal) wants competent disinterested expertise that advances the public welfare.
STOP+THINK: What does society reward?
STOP+THINK: What can go wrong?
Above average pay; autonomy, exclusivity, status.
Guild interest, protection of low performers, exploitation based on knowledge asymmetry, conflicts of interest.
STOP+THINK: What do we do?
Meta governance: accreditation, standardized exams, continuing education, peer review.
Liability and accountability: malpractice, negligence standards, fiduciary duties, whistleblower protection
Incentive Hygeine: conflict of interest rules, disclosure requirements, separation of roles, rotation, cooling off periods
Measurement and Learning: registries, outcome tracking, incident reporting, public dashboards (outcomes not outputs)
HMIA 2025
CLASS
HMIA 2025
Resources
Author. YYYY. "Linked Title" (info)
NEXT Markets
HMIA 2025 Hierarchy
By Dan Ryan
