Innovating AI Governance From Principles to PRactice
March/April 2021
Reference Decks
Governance Exemplars
Regulatory Tools
Analogy Candidates
*shared with GlobCert
Bibliography
Governance
Criminal Justice
immigration
MoFo
PanPlat
BigPlex, Audit, Globcert
Table of Contents
Innovating Governance for Artificial Intelligence
A multi-day workshop in which interdisciplinary teams grapple with concrete challenges in AI/ML governance with the goal of producing project prototypes that can be piloted in the field in the near term.
Teams of six will consider one of five problems curated collaboratively with sponsor organizations. The workshop begins with team-building and knowledge sharing exercises after which participants will research and "excavate" the problem by examining analogs and mapping the organizational environment in which the problem and solution reside. Then teams will build on insights derived from their research through several iterations of ideate-create-critique. At the end of the workshop each team will pitch a project prototype to the sponsoring organizations.
RA Tasks
- Clean up bibliography - complete info, tags, etc.
- Finish transferring from Jacob's paper to governance deck
- Polish analog decks
Innovating Governance for Artificial Intelligence
What kind of workshop? Borrowing from an IDEO infographic, this workshop has elements of five different goals.
Discovery: we will endeavor to generate innovative approaches to solutions by a process that fosters insight and creative thinking
Agenda Setting: we want to help small groups align and strategize around a possibility
Ambition Foundationing: Establish relationships and connections and engage in consensus building that unlocks seemingly intractable challenges
advancing knowledge Build partnerships, shared leadership, and commitment to collaborative action to achieve impact at scale
launch Collectively generate influential content
Innovating Governance for Artificial Intelligence
Workshop Workflow
Becoming
a Team
Practice solutioning
Practice pitch+Catch
Organizational Empathy Maps
Problem Analogies
Version 0
Research
Insights
Intentional
Prototype
pitch+Catch
iterate
Innovating Governance for Artificial Intelligence
The "Stipulations"
This workshop is about using an interdisciplinary design process to prototype novel governance mechanisms for specific AI/ML governance challenges.
In these times there is a need for technical, political, and philosophical discussions about what kinds of problems AI/ML represent, how much regulation is rights, and what principles should guide the development and deployment of these tools.
This workshop is not THOSE conversations. We ask participants to start from the premise that 1) the awesome potential benefits of AI for humans depend on building new institutions capable of regulating its development and deployment and 2) that this task may require as much, or more, innovation and ingenuity as the engenders technology itself.
Five Problems, Five Teams
Transforming a 2 Day Workshop in Italy...
|
|
|
partire |
|
|---|---|---|---|
|
arrivare |
|||
|
|
|
Day 1
Day 2
Morning
Afternoon
Evening
to
an
exercise
in
remote collaboration spread
over
Six
weeks
| 22 |
23 |
24 | 25 | 26 |
|---|---|---|---|---|
| 29 |
30 | 31 | 1 | 2 |
| 5 |
6 | 7 |
8 | 9 |
| 12 |
13 | 14 | 15 | 16 |
| 19 |
20 | 21 | 22 | 23 |
| 26 |
27 | 28 |
29 | 30 |
| Description | Brief overview of purpose and description |
| Duration & Timing | |
| Host(s) | Who from the SRI team will be speaking during the segment |
| Configuration | How the 'room' will be organized (e.g. break outs, main room, etc.) |
| AV/Tech Reqs | What tools will be used during the segment (e.g., screen sharing on Zoom, slides.com, Google forms, Miro board, polls, etc.) |
| What's on screen? | Link to the slide being shown on the screen |
| Team Materials? | List (with links) of materials being used by participants |
| Facilitator Instructions | Script (or link) for relevant facilitator instructions |
| Host Script | Script (or link) for relevant host comments |
| Outstanding Decisions |
Black Box Template & Field Descriptions
| Exercise Title | name of the workshop segment |
To Do List
- Pre-workshop exercise instructions and materials
- Kickoff Script
- About
Kick-Off Session
Welcome to the Workshop
1. Welcome
Kick-Off Session - March 23 (12 - 2 EST)
| Description | Overview of what we're trying to achieve, what we hope participants will get out of it, etc. - norm-setting |
| Duration | 0:20 |
| Host(s) | Gillian |
| Configuration | Main Zoom Room |
| AV/Tech Reqs | Zoom, polling thing? |
| What's on screen? | Slides for Gillian - not started |
| Team Materials? | N/A |
| Facilitator Instructions | N/A |
| Host Script | Yes -TBD |
| Outstanding Decisions | Discuss the message we want to convey here. |
Kick-Off Session - March 23 (12 - 2 EST)
| Exercise Title | Welcome to the Workshop |
|---|
Goals:
Methods: ideas borrowed from design, innovation, etc.
Ground Rules: because we encourage out of the box thinking we ...
Support:each team has a "band manager" dedicated to its success
Kick-Off Session - March 23 (12 - 2 EST)
| Description | Introduction to the process (goals, methods, ground rules, support system, etc.) |
| Duration | 0:20 |
| Host(s) | Dan |
| Configuration | Main Zoom Room |
| AV/Tech Reqs | Slides in screen share? |
| What's on screen? | Slides? Zoom background slides? |
| Team Materials? | N/A |
| Facilitator Instructions | N/A |
| Host Script | TBD |
| Outstanding Decisions | What, if anything, is being shared on screen? |
Kick-Off Session - March 23 (12 - 2 EST)
| Exercise Title | About the Workshop |
|---|
Problems worth solving don't yield to solo or siloed efforts, but the alchemy of successful interdisciplinary teams is elusive. Great teams can take months or years to gel, but we can do a few things at the start to move us in that direction.
We begin with an exercise we call "Forming the Band." The object of the exercise is to get to know one another a bit via imagining our team is a band and to come up with a name for the band and to decide what (whimsical) "instrument" each of us plays.
We will start out in breakout rooms with our teams and introduce ourselves briefly - name, affiliation, geography, a first fun fact. Then we'll send everyone to pre-assigned breakout rooms with one teammate. Your task in the break out room is to find out a bit more about one another like to do - professionally, intellectually, and/or avocationally.
After this we return to the full team breakout room and each of us will introduce the other to the rest of the group and then the group sets its mind to the task of coming up with a name for the band and an instrument (or action - my band once has a lead ranter instead of a lead singer). The next slide has a google form on which to enter the band's information.
| Description | Ice-breaker/initial team building exercise for problem groups |
| Duration | 0:45 |
| Host(s) | 1. Dan (introduction) 2. Band Managers (break out rooms) |
| Configuration | 1. Main Zoom Room (intro) 2. Breakout Rooms - Problem Groups |
| AV/Tech Reqs | Breakout Rooms, Screensharing, Slides.com (embedded Google form) |
| What's on screen? | 1. Intro slides to explain the process? 2. Screenshare from Band Manager of Google Form |
| Team Materials? | Yes - Form with Band stuff |
| Facilitator Instructions | Yes - TBD |
| Host Script | Yes - TBD |
| Outstanding Decisions |
Kick-Off Session - March 23 (12 - 2 EST)
| Exercise Title | Forming the Band |
|---|
When your team has settled on its name and lineup fill in the information on this google form so we can share with the other teams.
Each team will properly "introduce the band" during our opening session on the Xth.
To further get to know our team mates and to stretch our brains a little before we get down to to proper solution building next week we will convene in pairs for quick lightning rounds of "Imagining Governance." We will pose a governance problem paired with a governance solution and ask you to imagine something like "what if there were something like a World Trade Organization for regulating facial recognition software?" How might that work? Even if it feels like quite a stretch, the idea is to identify ways in which this might actually be do-able. What would it look like? What parts of the problem would be addressed? The goal is to spend some time thinking about the question and then get on a call with two teammates to talk about it and jot down some ideas on a MadLibs-like form that we'll provide.
But all we need to do TODAY is to confer with our team members and put the calls on our calendar. Team members 1-3 will meet, and team members 4-6 will meet.
| Description | Give participants their assignments for the pre-workshop exercises, give them time to schedule the 30m meetings. Teams split in half, each assigned to one combo. |
| Duration | 0:15 |
| Host(s) | Facilitators |
| Configuration | Breakout Rooms - Problem Groups |
| AV/Tech Reqs | Breakout Rooms, Screensharing, Calendly? |
| What's on screen? | TBD |
| Team Materials? | Calendly? |
| Facilitator Instructions | Yes - Give out assignments and facilitate the scheduling |
| Host Script | Yes - TBD |
| Outstanding Decisions | What is the best way to do the scheduling? What Zoom links are we using? How are people getting calendar invites? |
Kick-Off Session - March 23 (12 - 2 EST)
| Exercise Title | Schedule the Pre-Workshop Exercise |
|---|
Scheduling
But all we need to do TODAY is to confer with our team members and put the calls on our calendar. Team members 1-3 will meet, and team members 4-6 will meet.
NEED: calendar graphic, instructions on how to schedule, doodle? then we put links here?
What if there were something like a __________ for regulating ______________?"
MORE TO COME
| Description | Participants will be paired with other team members to engage in some preliminary examination of governance problem and solution pairings to encourage deeper thinking about the problem and broader thinking about solutions. |
| Duration | 0:30 (x2) |
| Host(s) | N/A |
| Configuration | Two 1:1 meetings between members of the same team |
| AV/Tech Reqs | Zoom (participant accounts) |
| What's on screen? | N/A |
| Team Materials? | Yes: (1) problem and solution background info; (2) mad-libs style form |
| Facilitator Instructions | N/A |
| Host Script | N/A |
| Outstanding Decisions |
Pre-Workshop Exercise - March 24 - April 1 (participant scheduled)
| Exercise Title | Lightning Governance Solutions |
|---|
Immigration agencies want to use AI/ML to speed up, increase accuracy, increase consistency. But they want to know that they aren't making matters worse or introducing new biases.
What if there were
MoFo trying to ....
What if there were
Workshop Day 1
Workshop Day 1
Workshop Day 1
1 Opening Remarks
| Description | Quick welcome and overview of the day, reminder of norms, etc. |
| Duration | 0:15 |
| Host(s) | Dan, Gillian, Jamison? |
| Configuration | Main Zoom Room |
| AV/Tech Reqs | Screensharing for the day's agenda? |
| What's on screen? | Agenda? |
| Team Materials? | N/A |
| Facilitator Instructions | N/A |
| Host Script | Yes - TBD |
| Outstanding Decisions |
Workshop Day 1 - April 7 (11:00 - 3:00 EST)
| Exercise Title | Day 1 Introduction |
|---|
1. Write a sentence about AI governance.
2. Pass to the left.
3. Draw what you read.
4. Pass to the left.
5. Write what you see.
6. Pass to the left.
7. Draw what you read.
8. Pass to the left.
9. Write what you see.
0. The Reveal.
3 Visual Telephone
| Description | Starting with a prompt, participants alternate between drawing what they read and describing what they see. Serves as an ice breaker and builds team collaboration |
| Duration | 0:30 |
| Host(s) | Band Managers, Dan |
| Configuration | Breakout Rooms, Main Zoom Room |
| AV/Tech Reqs | Screen share, Google Slides |
| What's on screen? | Each 'telephone' deck |
| Team Materials? | Telephone slide decks |
| Facilitator Instructions | Yes - TBD |
| Host Script | Yes - TBD |
| Outstanding Decisions | Are we doing this one or scrapping it? |
Workshop Day 1 - April 7 (11:00 - 3:00 EST)
| Exercise Title | Visual Telephone |
|---|
R2
R1
R3
R4
R5
R6
M2
M1
M3
M4
M5
M6
E2
E1
E3
E4
E5
E6
B2
B1
B3
B4
B5
B6
C2
C1
C3
C4
C5
C6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
BR1
BR 2
BR 3
BR 4
BR 5
We create 5 separate slide decks that alternate between draw/text slides. Band Manager shares new link in the chat every two minutes for each team member to take their turn. One player is doing the work, but the teams can talk about it together
R2
R1
R3
R4
R5
R6
M2
M1
M3
M4
M5
M6
E2
E1
E3
E4
E5
E6
B2
B1
B3
B4
B5
B6
C2
C1
C3
C4
C5
C6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
S1
S2
S3
S4
S5
S6
Host shares full presentations:
Back to Main Zoom Room...
4 Introduce the Band
| Description | Teams introduce themselves to the rest of the group using the material from kick-off |
| Duration | 0:30 |
| Host(s) | Dan |
| Configuration | Main Zoom Room |
| AV/Tech Reqs | Screensharing? |
| What's on screen? | Band images |
| Team Materials? | N/A (SRI to prepare?) |
| Facilitator Instructions | N/A |
| Host Script | Yes - TBD |
| Outstanding Decisions | Do we do this during workshop day 1 or incorporate during the kick-off? |
Workshop Day 1 - April 7 (11:00 - 3:00 EST)
| Exercise Title | Introduce the Band |
|---|
5 Team Norms
| Description | To speed up team building, teams look through a selection of norms, discuss, and agree on up to 6 to adopt. |
| Duration | 0:30 |
| Host(s) | Dan, Band Managers |
| Configuration | Main Zoom Room, Breakout Rooms |
| AV/Tech Reqs | Screensharing, Slides.com, Google Slides |
| What's on screen? | Instructions, Slides.com page with summary of chosen norms |
| Team Materials? | Norm Decks, Slides.com workbook |
| Facilitator Instructions | Yes - TBD |
| Host Script | Yes - TBD |
| Outstanding Decisions |
Workshop Day 1 - April 7 (11:00 - 3:00 EST)
| Exercise Title | Team Norms |
|---|
5 Team Norms
Heads Down
- Go to your deck of team norm cards
- Ponder the norms
- Select a norm to pitch
- Move the slide to the top of the deck
- Rejoin your team mates for a quick pitch and catch
5 Team Norms
Pitch and Catch
5 Team Norms Pitch and Catch
6 Five Things We All Should Know
Each of us has a "wheelhouse," a "precinct," a zone where we know what's going on. Although this workshop is interdisciplinary by design, we want to start out by asking folks to gather with their disciplinary comrades to think about what their profession or discipline brings to the challenge of AI/ML governance from their area of expertise.
Start out "heads down" for 5 minutes - what five ideas, concepts, or insights from your own area of expertise should everybody here be familiar with? Jot them down in the boxes below. And then we'll convene with our colleagues.
1
2
3
4
5
6 Five Things We All Should Know
Join your colleagues in a breakout room and compare notes. Proceed "round robin" taking turns each "nominating" one idea at a time. Record them as you go. ______
6 Five Things We All Should Know
Join your colleagues in a breakout room and compare notes. Proceed "round robin" taking turns each "nominating" one idea at a time. Record them as you go. ______
6 Five Things We All Should Know
Join your colleagues in a breakout room and compare notes. Proceed "round robin" taking turns each "nominating" one idea at a time. Record them as you go. ______
6 Five Things We All Should Know
Text
1
Five Things
| Description | Experts talk amongst themselves and reach consensus about what the most important contributions from their field/area of expertise are to the workshop. All groups share with the full room. |
| Duration | 1:00 |
| Host(s) | Dan |
| Configuration | Breakout Rooms (Professional Groups), Main Zoom Room |
| AV/Tech Reqs | Slides.com, Google Docs/Slides |
| What's on screen? | TBD |
| Team Materials? | Yes - TBD |
| Facilitator Instructions | Yes - TBD (incl. which facilitators go to which groups) |
| Host Script | Yes - TBD |
| Outstanding Decisions | What are the groups? Does this still work with the majority of participants being from sponsors? |
Workshop Day 1 - April 7 (11:00 - 3:00 EST)
| Exercise Title | Five Things |
|---|
Pitch and Catch
on
Lightning Designs
| Description | Revisit the topics from the pre-workshop exercises. Practice pitching and catching the 'mad libs', getting in the mindset of responding to pitches with iteration forwarding ways. 'Yes, and' thinking. |
| Duration | 1:10 |
| Host(s) | Band managers? This may be logistically confusing |
| Configuration | Breakout Rooms |
| AV/Tech Reqs | Screensharing, anything else? |
| What's on screen? | Results of the pre-workshop activities |
| Team Materials? | TBD - is this just a verbal exercise, or are there notes being taken? |
| Facilitator Instructions | Yes - TBD |
| Host Script | Yes - TBD |
| Outstanding Decisions |
Workshop Day 1 - April 7 (11:00 - 3:00 EST)
| Exercise Title | Pitch & Catch on Lightning Designs |
|---|
Global Certification Mark
Precision
Medicine
FDA
for
AI
Covid
Global Certification Mark
FDA
for
AI
Precision
Medicine
Covid
Global Certification Mark
FDA
for
AI
Global Certification Mark
FDA
for
AI
Precision
Medicine
Covid
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
PEOPLE
PROBLEMS
GOVERNANCE
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Climate
Climate
Climate
Immigration
Cities
Cities
Cities
BigPlex
Pandemic Platform
Global Certification Mark
Mo Fo
Global
Device Audit Program
Global
Device Audit Program
Global
Device Audit Program
Global
Device Audit Program
Regulatory
Market
Regulatory
Market
Regulatory
Market
Regulatory
Market
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Climate
Climate
Climate
Immigration
Cities
Cities
Cities
BigPlex
Pandemic Platform
Global Certification Mark
Mo Fo
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Climate
Climate
Climate
Immigration
Cities
Cities
Cities
BigPlex
Pandemic Platform
Global Certification Mark
Mo Fo
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Climate
Climate
Climate
Immigration
Cities
Cities
Cities
BigPlex
Pandemic Platform
Global Certification Mark
Mo Fo
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Governance
mind stretching
10min
20min
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Stakeholder Empathy Part II
10min
20min
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Climate
Climate
Climate
Immigration
Cities
Cities
Cities
BigPlex
Pandemic Platform
Global Certification Mark
Mo Fo
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
WORK IN TEAMS
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Climate
Climate
Climate
Immigration
Cities
Cities
Cities
BigPlex
Pandemic Platform
Global Certification Mark
Mo Fo
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
pitch catch
pitch catch
pitch catch
pitch catch
pitch catch
Pitch &
CATCH
10min
20min
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
pitch catch
pitch catch
pitch catch
pitch catch
pitch catch
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
SRI+
pitch catch
pitch catch
pitch catch
pitch catch
pitch catch
Pitch
THIS ITERATION
CATCH
with ITERATION FORWARD FEEDBACK
10min
20min
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
pitch catch
pitch catch
pitch catch
pitch catch
pitch catch
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Global Certification Mark
Precision
Medicine
FDA
for
AI
Regulatory
Market
Global
Device Audit Program
Climate
Cities
Covid
Global Certification Mark
FDA
for
AI
Regulatory
Market
Global
Device Audit Program
Precision
Medicine
Climate
Covid
Global Certification Mark
FDA
for
AI
Regulatory
Market
Global
Device Audit Program
Global Certification Mark
FDA
for
AI
Regulatory
Market
Global
Audit Program
Precision
Medicine
Climate
Covid
Precision
Medicine
Climate
Covid
Ali
Bri
Che
Dov
Eli
Fra
Git
Hal
Ion
Jia
Kia
Lev
Mia
Noa
Ogi
Pia
Ali
Bri
Che
Dov
Ion
Jia
Kia
Lev
Eli
Fra
Git
Hal
Mia
Noa
Ogi
Pia
Ali
Bri
Che
Dov
Eli
Fra
Git
Hal
Ion
Jia
Kia
Lev
Mia
Noa
Ogi
Pia
Ali
Bri
Che
Dov
Ion
Jia
Kia
Lev
Eli
Fra
Git
Hal
Mia
Noa
Ogi
Pia
Lightning Design Done Pre-Workshop
Cities
Global Certification Mark
Regulatory Market
FDA for AI
Global Audit Program
Precision Medicine
Climate
Covid
Precision Medicine
Climate
Covid
Ali
Bri
Che
Dov
Eli
Fra
Git
Hal
Ion
Jia
Kia
Lev
Mia
Noa
Ogi
Pia
Ali
Bri
Che
Dov
Ion
Jia
Kia
Lev
Eli
Fra
Git
Hal
Mia
Noa
Ogi
Pia
Ali
Bri
Che
Dov
Eli
Fra
Git
Hal
Ion
Jia
Kia
Lev
Mia
Noa
Ogi
Pia
Ali
Bri
Che
Dov
Ion
Jia
Kia
Lev
Eli
Fra
Git
Hal
Mia
Noa
Ogi
Pia
Cities
Covid
Precision Medicine
Cities
Climate
Covid
Precision Medicine
Climate
Cities
P
C
P
C
P
C
P
C
R1
R2
R3
R4
Homework
Workshop Day 2
Workshop Day 2
-
Research
- Analogy
- Organizational Empathy
- Iteration 0
Workshop Day 2
Learning by Analogy
How is our governance challenge the same as and different from instances of governance that we take for granted?
Example
Workshop Day 2
Learning by Analogy
The two easiest mistakes to make when thinking about governance and new technologies are starting from the premise that "this changes everything" or from the premise "this changes nothing." One leads to reinventing the wheel and failing to exploit experience; the other to naively expecting grand dad's tool box to be helpful in starship repair.
In this exercise we are invited to ride the wave between these extremes by thinking deliberately about how our very new governance challenge does and does not share characteristics with technologies that societies have worked out governance institutions for in the past.
Learning by Analogy
How is our governance challenge the same as and different from instances of governance that we take for granted?
Team 1
Team 2
Team 3
Team 4
Workshop Day 2
Learning by Analogy
TEAM 1
CHALLENGE SYNOPSIS
Learning by Analogy
TEAM 2
CHALLENGE SYNOPSIS
Learning by Analogy
TEAM 3
CHALLENGE SYNOPSIS
Learning by Analogy
TEAM 4
CHALLENGE SYNOPSIS
4 Mapping the Stakeholder Environment
Workshop Day 2
Stakeholders include the regulated and the regulator and all the other organizations and groups that win or lose depending on how governance turns out. Stakeholders can motivate change or stand in its way. Stakeholders care about different things and see the world from different perspectives and measure what's good in divergent ways.
It's hard to build things with diverse stakeholders in mind; but it's impossible to build things without them. Sometimes you need to have them "at the table," sometimes you just need to be mindful of where they are coming from. But in either case, you have to have them on the map.
This exercise has three parts. First we individually list the entities we think of as stakeholders in connection with the governance problem we are considering. Second we map these to identify the ones most relevant to our project's success. And third we sketch "organizational empathy maps" for these most relevant stakeholders.
4 Mapping the Stakeholder Environment
Workshop Day 2
What corporate entities are stakeholders?
What social groups are stakeholders?
What government agencies are stakeholders?
What social movement organizations are stakeholders?
What NGOs are stakeholders?
What ... are stakeholders?
What have we left out?
Take five minutes heads downs to brainstorm on stakeholders relevant to this governance challenge.
4 Mapping the Stakeholder Environment
Workshop Day 2
Compare notes with your team members. Do some entities appear on most people's lists? Discuss among the team and select 6 highest priority stakeholders.
- ________
- ________
- ________
- ________
- ________
- ________
4 Mapping the Stakeholder Environment
Workshop Day 2
Next we will construct "organizational empathy maps" for each of our highest priority stakeholders.
An organizational empathy map captures what makes an entity tick: what is its connection with this issue? through what lens does it see the world? what is its bottom line? what is its honey? what is its kryptonite?
what incentivizes
this stakeholder?
stakeholder kryptonite?
stakeholder lens?
what is success for
this stakeholder?
stakeholder
name
Learning by Analogy
SAMPLE
Governance Challenge: the use of predictive algorithms for determining whether one is released pending trial, required to post bail and how much,or held without the option of bail
Title Text
Learning by Analogy
SAMPLE: bail algorithms :: weights and measures
objective ground truth available in one but not the other
commerce
vs. justice
(real) prediction
vs.
observation
things vs people
constitutional questions
both involve "standards"
both involve measurement
long legal
history
identical machines used over and over all across the land
both are about numbers with material consequences
Similarities
Dissimilarities
Learning by Analogy
SAMPLE: bail algorithms :: weights and measures
objective ground truth available in one but not the other
commerce
vs. justice
(real) prediction
vs.
observation
things vs people
constitutional questions
both involve "standards"
both involve measurement
long legal
history
identical machines used over and over all across the land
both are about numbers with material consequences
Similarities
Dissimilarities
Homework
Asynchronous Research Work
Intentional Prototypes allow you to interview the world about your ideas
Prototypes give the world a chance to talk about your idea.
Iteration,Pitch,Catch 1
Iterate,Pitch, Catch
Homework
Asynchronous Research Work
Iteration,Pitch,Catch 2
Iterate,Pitch, Catch
Homework
Asynchronous Research Work
Pitch Session
Pitch Session
Challenge
Too Bigplex to Regulate
How might big AI/ML models be fitted with an audit layer that made it possible for citizens and regulators to verify compliance with legislated standards of acceptable behavior?
Challenge
AI at the Border
What could we build that would facilitate the deployment of AI/ML to make our immigration systems less arbitrary and more aligned with our values and human rights?
Challenge
How to Stay good while doing well with data
What kind of data governance regime could progressive organizations adopt so that they can fully exploit the power of the data they collect without betraying their commitments to users and a healthy internet?
Challenge
Governance and BIg surveillance for public health
Can we build a generic governance model that would facilitate development of large scale syndromic surveillance systems that draw on data derived from diverse sources such as consumer applications (e.g., transactional records of cough syrup purchases) and private platforms (e.g., social media surveys or chatbots)?
Challenge
Stamping AI as TRUSTWORTHY
HOW MIGHT WE CREATE AN ENTITY THAT COULD DEPLOY A UNIVERSALLY RECOGNIZED FRAMEWORK THAT VALIDATES AND CERTIFIES AI TOOLS AND TECHNOLOGIES AS RESPONSIBLE, TRUSTWORTHY, ETHICAL AND FAIR?
Challenge
Too Bigplex to Regulate
Big AI/ML models are of such large scale and complexity and are created by companies with so much human capital and compute that any conventional attempt at regulation is left in the dark and in the dust.
But what if there were an "audit layer" built into such models, a kind of API that would let developers expose the models in a manner that made verifying compliance with legislated standards of acceptable behavior a tractable task?
X?
But what if there were an "audit layer" built into such models, a kind of API that would let developers expose the models in a manner that made verifying compliance with legislated standards of acceptable behavior a tractable task?
What if instead of wishing that regulators were more sophisticated or that companies create ethics bureaucracies we built a standardized AI/ML model audit layer with an API that would allow regulators, advocates, and citizens to verify compliance with social standards?
What if...
Instead of wishing regulators were more sophisticated or that
companies create ethics bureaucracies
we built a standardized AI/ML model audit layer
with an API
that would allow regulators, advocates, and citizens
to verify compliance with social standards?
Title Text
Something like this...
Becoming
a Team
Practice solutioning
Practice pitch+Catch
Organizational Empathy Maps
Problem Analogies
Version 0
Research
Insights
pitch+Catch
iterate
Intentional
Prototype
HOW MIGHT WE... produce a first draft of realistic, technically grounded requirements for an "audit layer" solution that companies would have incentives to build and use?
Workshop Product
LET'S describe the functions: what would an "audit layer" have to do if it were to "solve" the problem of "too big and complex to regulate"?
LET'S describe the organizational operating environment of an "audit layer" solution - how do the most relevant players think? What would make building an audit layer attractive for them (versus a regulator needing to actively incentivize them to build it)?
LET'S sketch some measures of effectiveness - against what kinds of objectives and outcomes could we measure our solutions?
LET'S plan how to get started: mapping out what would make an audit layer useful and something that could solve problems being faced by tech companies and meet specific regulatory needs from specific regulatory partners.
Workshop Product 1: Functions
To "solve" the problem of "too big and complex to regulate" an "audit layer" solution would need to:
Workshop Product 2: Objectives
We measure solutions against...
Workshop Product 3: Environment
How do the most relevant players think? What would make building an audit layer attractive for them (versus a regulator needing to actively incentivize them to build it)?
Workshop Product 4
Plan for moving forward...
What tech companies as initial partners? What's the pitch?
What regulatory partners to start with? What's the pitch?
Where do these pieces come from?
Functions
Objectives
Environment
Strategy
Ecce Rem
Proof of Concept
Organizational
Empathy
Maps
Proof of Concept
Too Bigplex to Regulate: Outputs
THIS COMES FROM ORGANIZATIONAL EMPATHY MAPPING EXERCISE
Operating Environment of Solutions
What organizational entities are the "users"? Governments, government agencies, companies, industries, professions, communities of practice, NGOs, consortia of NGOs, social movements. How do they think? What would make building an audit layer attractive for them?
Objectives
What are some measures of effectiveness - against what objectives could we measure solutions?
Functions
What would an "audit layer" have to do if it were to "solve" the problem of "too big and complex to regulate"? This functional statement will demarcate a boundary around all possible solutions.
Prototype
Demonstrate what would make an audit layer useful to solve problems being faced by tech companies, and to address the specific regulatory needs from specific regulatory partners.
THIS COMES FROM THIRD PROTOTYPING EXERCISE
THIS COMES FROM FIRST PROTOTYPING EXERCISE
THIS COMES FROM ORGANIZATIONAL EMPATHY MAPPING EXERCISE
THIS COMES FROM SECOND PROTOTYPING EXERCISE
Intentional Prototyping
Ecce Rem
Proof of Concept
Real World
Intentional Prioritization
The Output
OUTTAKES
About the Deck
Becoming
a Team
Practice solutioning
Practice pitch+Catch
Organizational Empathy Maps
Problem Analogies
Version 0
Research
Insights
pitch+Catch
iterate
Intentional
Prototype
Excavate the Problem
Extract the Insights
Iterate the Solution
Modes of Interaction
- Instructions
- Heads Down Creative Time
- Pair Work
- Team Turn Taking
- Team Work
- Pitch and Catch
- Present to the Room
- Informal Random Socializing
Pitch and Catch
Pitch and Catch
"Pitch and Catch" refers to a mode of presentation and feedback that we'll use throughout the workshop. In a pitch and catch session the pitching team makes a brief presentation about its work in progress and then the catchers' job is to offer "iteration advancing feedback." That is, feedback about how to move the project along in its next iteration. The catchers understand their responsibility is to improve the pitchers' work as much as possible. So, a pitch and catch session LOOKS like "Shark Tank" or "Dragons' Den" but the spirit is much more "yes and" than it is trial by fire. Catchers compete to give the most valuable advice not the snarkiest critique. This does not mean, I should add, that we avoid critique; it's just that we don't value telling someone they are barking up the wrong tree unless we can help them see which right tree would be more fruitful.
We'll typically do pitch and catch as a series of short rounds so that by the end each team has pitched and everyone has played the role of catcher several times.
2 Lightning Governance Solutions
Pairs of team mates spend 30 minutes brainstorming about how we might introduce governance solution X to governance problem Y.
Pre-Workshop Exercises
Pre-Workshop Exercises. Participants will be paired with team members to engage in some preliminary examination of "governance problem/governance solution" pairings to encourage deeper thinking about the problem and broader thinking about solutions.
Consider this Problem
Some background about ML in criminal justice.
Recidivism is the tendency of a convicted criminal to reoffend. The prediction of recidivism plays a role in considerations of bail before trial and parole after punishment. Computational assessments of recidivism risk have been around for almost a hundred years (Donohue 2019). Today “fourth generation” tools use machine learning, can pull data directly from government databases and can generate a prediction for any offender. Because of the massive amounts of data involved, it may be hard to see what led to the system's recommendation (Donahue 2019). Although not designed for sentencing, these tools find their way into that part of the process where it is felt the risk scores are useful information. Some are troubled by the explicit use of statistical data about others to inform judgments about an individual, some by the lack of transparency of proprietary algorithms, some by the fact that computer generated numbers can distort the human decision making process.
Consider this Solution
Some background about NTSB.
The National Transportation Safety Board (NTSB) is an independent agency (meaning it is not under a cabinet secretary) within the US government. It is headed by a five member presidentially appointed board, has about 400 employees (engineers, safety experts, human factors, attorneys, investigators, etc.) and a budget of about 100 million. NTSB tasks include accident investigation, safety recommendations, assisting victims, and it is a court of appeal for FAA and Coast Guard license suspensions. The Board has no enforcement power so it has to make recommendations to agencies that do. Much of the board's work - accident investigation, for example - is carried out using the "party system" whereby it designates entities as parties to the investigation that can provide needed technical or specialized expertise are permitted to serve on the investigation.
What if there were an NTSB for AI/ML in CJ?
TWO MORE EXAMPLES