YT YouTube2Text

YouTube2Text → Transcripts → Debugging Bug Bounties in Cyberspace: From Vulnerability Discovery to Algorithmic Harms Redress

Debugging Bug Bounties in Cyberspace: From Vulnerability Discovery to Algorithmic Harms Redress — Transcript

by Columbia SIPA · 20,677 words · 3,358 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:10hi everyone my name is jason healy i run
0:13the cyber programs at columbia
0:15university school of international
0:17public affairs and i've just been so
0:19excited for this event
0:21with some fabulous colleagues
0:23to talk about
0:26these issues so this is a range as part
0:29of the nijilo road and digital futures
0:31forum it's been organized with the
0:33support of the saltzman institute in
0:34warren peace studies my thanks to vir
0:36patan vikram singh for for helping put
0:38this together
0:40um i'm not going to keep any time
0:42because i want to hear what the
0:42panelists have to say so we're going to
0:44turn right uh to matt gertzen he's a
0:47researcher i just published a great
0:48report um with data in society and let's
0:51turn it over to matt
0:54thank you jason it's a pleasure to be
0:56here uh thank you for inviting me to
0:59moderate this event
1:00i have had the great privilege to watch
1:04uh both of the reports that we're going
1:05to be discussing today kind of in their
1:08development phase over the past couple
1:10years um and uh it's just it's such a
1:14great uh thrill to see them out in the
1:17world and uh and as a backdrop for this
1:20discussion on the subject of
1:22bug bounty programs and their uh history
1:25and their contemporary application today
1:28so we're going to have a
1:30pretty standard format where i'm going
1:32to give a little bit of context for the
1:34event
1:35and then
1:36our panelists will each present a little
1:39bit about the papers that they've that
1:42they've just published
1:43um i think that'll probably take us
1:45about uh 30 40 minutes something like
1:47that and then we'll move into a
1:50you know semi-structured uh conversation
1:52where i'll ask some ask questions i'll
1:55also keep a track on the the q a section
1:58for any questions from the audience and
1:59intersperse those as appropriate
2:02and uh we'll
2:03also allow the panelists to you know
2:06jump in and address questions that
2:08come in from the audience
2:10if they if they see anything that
2:12particularly grabs their interest
2:14um
2:15that's about it i think um so i'll
2:17briefly introduce
2:19each of the people here so first we have
2:21ryan ellis who is an associate professor
2:23of communications studies at
2:24northeastern university and an affiliate
2:26of data and society research institute
2:29ryan's research and teaching focuses on
2:31topics related to communication law and
2:33policy infrastructure politics and cyber
2:36security he's the author of letters
2:38power lines and other dangerous things
2:40the politics of infrastructure security
2:42and the editor with vivek mohan of
2:44rewired cyber security governance
2:47and then we have yuan stevens who is a
2:49legal and policy expert focused on
2:51information security data protections
2:53and human rights she works towards a
2:55world where powerful actors and the
2:57systems they build are held accountable
2:59to the public especially when it comes
3:00to vulnerable and marginalized people
3:02she brings years of international
3:04experience to her work as a researcher
3:06having examined the impacts of
3:07technology on vulnerable populations in
3:09canada the us and germany uan is a
3:11research affiliate of data and society
3:13research institute and a collaborator at
3:15the center for media technology and
3:17democracy at mcgill university she
3:19previously worked at harvard
3:20university's berkman client center for
3:21internet society during her studies and
3:23joint degree in civil and common law at
3:26mcgill university
3:27camille francois is a lecturer at the
3:29columbia school of international public
3:31affairs and the co-lead of the
3:33algorithmic justice lease community
3:35reporting of algorithmic systems harm's
3:38crash project her work spends several
3:41aspects of cyber security from
3:43developing industry-leading programs
3:45focused on protecting vulnerable users
3:47to detecting information operations
3:49currently the global director for trust
3:51and safety at nyan tech
3:54she was previously chief innovation
3:55officer in grafica where she built and
3:57led a team dedicated to exposing and
3:59mitigating information operations across
4:01platforms and prior to that a principal
4:03researcher at google camille has advised
4:06his governments and parliamentary
4:07committees on both sides of the atlantic
4:09and investigated russian interference in
4:11the 2016 u.s presidential election
4:14on behalf of the u.s senate select
4:16intelligence committee
4:17and finally we have josh kenway who's a
4:19policy analyst at paypal working on
4:21corporate governance for technology and
4:24cyber security and until mid-2021
4:27he was a research fellow with the
4:28algorithmic justice league where he was
4:30part of the community reporting
4:32again the crash project i just described
4:35prior to joining paypal josh was an
4:36associate of the cyber threat alliance a
4:39non-profit organization that enables the
4:41sharing of information on cyber threats
4:43among cyber security companies
4:44governments and civil society
4:46organizations so we have a very
4:48impressive cast of characters here with
4:51a vast breadth of knowledge
4:54and i'm very excited for this um
4:56so i think we can move over to the next
4:59slide really quickly um if that's all
5:01right
5:02and
5:03i am
5:04the co-author alongside uh my frequent
5:08collaborator and former advisor
5:10gabriella coleman of the recent data
5:12society report wearing many hats
5:15and that
5:17report kind of looks at the
5:19early history of hackers
5:22professionalizing throughout the 1990s
5:24and it really forms a perfect kind of
5:27precursor pre-history to a lot of the
5:29topics we're going to discuss today
5:31so i'm going to give a very brief
5:33introduction to the material we cover in
5:35that report which was also released i
5:37think about three weeks ago by downtown
5:38society
5:40um
5:41and basically in in our report we look
5:43at
5:44um the emergence of two very important
5:47phenomena by which
5:48hackers um you know people who break
5:51into
5:52uh secure
5:54computer systems
5:56um
5:57sought to kind of change the the public
5:59public conversation
6:01and the process by which security issues
6:03were thought about and addressed
6:06so one of the issues we really take on
6:08seriously in that report is the
6:11practice of what was called full
6:13disclosure and it still exists in some
6:14forms today
6:16and full disclosure basically involved
6:18hackers finding vulnerabilities and
6:20security systems
6:22and
6:23publishing them
6:24publicly to the world
6:26making them known
6:27and in doing so
6:29you know putting pressure on
6:31vendors like microsoft was one of their
6:34big targets and also
6:36you know institutions of all sorts to
6:38take those issues very seriously
6:41address them patch them do whatever they
6:43could to
6:45ensure that other hackers
6:47perhaps of more malicious intent could
6:49not exploit them
6:51to harm
6:52and um
6:54this
6:55practice was you know very common among
6:57a variety of mailing lists
6:59one of the ones we talk about in the
7:00report most
7:02intensively is called bug track
7:06and
7:07you know over time
7:09people working for companies would come
7:11to these lists engage with these people
7:13take the issues seriously
7:15before that often
7:16vendors would pretend that those issues
7:18didn't exist because there was really no
7:21way to hold them accountable
7:25another phenomenon we look at is
7:26something we call security by spectacle
7:28which is the way that hackers sometimes
7:30staged
7:32um you know spectacular
7:34media focused um actions like releasing
7:38tools that made exploitation of
7:40vulnerabilities even even more easy
7:42in order to kind of exacerbate
7:45the threat to exacerbate the potential
7:47for harm and thus motivate vendors to
7:49move more seriously
7:51um or more quickly now by the end of the
7:541990s many of the hackers that
7:56participated in these practices actually
8:00became employees of a lot of the
8:02companies that they had formerly
8:04antagonized microsoft in particular
8:07went on a hiring spree
8:10hiring many many prominent hackers and
8:12also
8:13um you know consulting with a security
8:16firm called at stake which
8:18also uh you know hired many of the
8:20hackers that have been involved in full
8:22disclosure research
8:23and so there's kind of a you know from
8:25the early 1990s to the to the early
8:282000s there was kind of a water a sea
8:30change where hackers suddenly were often
8:33working for the very companies that they
8:34had been
8:35um uh
8:37exposing vulnerabilities in
8:39now
8:40this is important context because
8:43bug bounty programs the topic of which
8:45we're going to discuss today was the
8:46major kind of counter trend
8:49uh in a lot of that kind of profession
8:52like
8:53hiring of standing professional security
8:56researchers and bug down he's basically
8:58created a casualized market
9:01by which
9:02companies could uh solicit the
9:05submission of vulnerabilities in a very
9:08private directed way and pay piecemeal
9:11uh to the workers who did that and we're
9:12going to hear a lot more about the
9:14dynamics of that in just a second as i
9:17turn it over to i believe uh ryan alice
9:20ryan is uh is the is starting
9:23um
9:24so yeah take it away and um i'm excited
9:27for this thanks everyone
9:29all right great thanks matt i appreciate
9:30it hello everybody welcome from my
9:33basement in boston massachusetts i'm
9:34sorry we can't be together but i'm glad
9:36we can connect this way
9:38so i thought what um i would do with you
9:40ann my co-author on this new report
9:41bounty everything hackers in the making
9:43of the global bug marketplace which we
9:44published with dad and society a couple
9:46weeks ago was provide you with like the
9:48cliff notes overview of the report it's
9:50a long report these are all very
9:52detailed excellent long reports but i
9:54thought it'd be useful to have like a
9:55quick summary so you ann and i are going
9:57to spend a little time just walking
9:58through maybe 10 minutes what the report
10:00argues sort of top line conclusions
10:02so a little bit of context here um for
10:04our report we were really curious about
10:06bhagnani programs we wanted to
10:08understand them where they came from
10:09where they're going and most importantly
10:11what they mean for the workers who
10:13participate in them so we went out and
10:14we interviewed over 40 different folks
10:17who participate in the market that's
10:18hackers who find and sell flaws people
10:20who do triage and run and manage these
10:22programs as well i think a couple of the
10:24folks we spoke with might be listening
10:26today and i can't wait for them to chime
10:28in and tell us what we got wrong
10:30at the end during q a um but those
10:32stories really informed our report and
10:34what we found was
10:35that bounty programs offer a lot of
10:38fantastic opportunities there's pleasure
10:40to be found in the market there's new
10:41career opportunities they stabilize they
10:43provide opportunities for recognition
10:44for hackers that have been missing they
10:46provide rewards to hackers as well
10:48however
10:50some of the more celebratory accounts of
10:51the market seem to miss what the labor
10:53dynamics actually are and the ways in
10:55which bug bounty programs in some ways
10:57have created unanticipated risks risks
11:00both for the workers themselves who are
11:02participating in this market and in some
11:04ways broader security risks for all of
11:06us when they're not designed correctly
11:08or implemented appropriately bug bounty
11:10programs can actually be
11:12counterproductive and can undermine
11:13security
11:14so by going out and speaking with these
11:16people we got a little bit better
11:17insight into how the market operates
11:19what makes it tick and we're going to
11:20share all that with you today
11:22um so the report does and sort of walks
11:24through a couple of key topics first we
11:26provide a sort of overview of what bug
11:28bounty programs are and i'll do that in
11:29just a moment
11:30then we talk about where they came from
11:32their history starting in the mid 1990s
11:34with netscape
11:35then we look at the motivations for
11:37people who participate in this market
11:38and the risks that they face
11:40and then finally we conclude with some
11:41recommendations about how we can maybe
11:43imagine a future where bug bounty
11:44programs serve not only um the interests
11:47of the programs that run them but
11:48workers and society at large
11:50so very quickly um i'm going to give a
11:52quick overview of what bug bounty
11:54programs are in case any of you are
11:55unfamiliar this is new to you
11:57so bug body programs are fairly simple
11:59and straightforward there are ways in
12:01which hackers sell bugs to programs and
12:03platforms
12:05they can be operated in a few different
12:06ways they can be open or closed open
12:08anyone can submit closed also only
12:10invitation only
12:12additionally they can be run by the
12:14vendor themselves so microsoft or google
12:16or facebook might run their own bug
12:17bounty program or they can be run by a
12:19platform like hacker one or bug crowd
12:21that runs them for them now these
12:23programs were once upon a time like
12:24really
12:25novel
12:26and the province of high-tech companies
12:28now it seems like everybody has one
12:30united airlines department of defense we
12:31always are updating the slides my
12:33favorite recent one is lululemon the
12:34clothing uh the clothing company
12:37so they become pretty common
12:39but for the folks that working in them
12:40as we'll see um they can have
12:42unanticipated risks and hazards so
12:44briefly i'll walk through the history if
12:45you go to the next slide and tell you
12:46where they came from
12:47how they started out
12:49so bug bounty programs sort of emerge
12:52from two competing different interests
12:54on the one hand there's something of a
12:55pr stunt a way of
12:58sweeping under the rug bad press so to
13:00understand that sort of thrust and where
13:02they come from we have to go back to it
13:04seems like yesterday to me and maybe
13:05some of the other folks on the call but
13:06from some other people might be ancient
13:08history it's the mid 1990s
13:10and netscape so netscape was at the time
13:12in 1995 one of the largest and most
13:15stunning successes on wall street they
13:17just gone public in their browser
13:18netscape navigator you see the little
13:20logo there on the left hand side that's
13:21sort of aqua n
13:24um dominated the market it was the first
13:26most successful widely used web browser
13:29netscape had gone
13:31public in a stunning display and had
13:34sort of dominated
13:36wall street expectations and was a real
13:38star however they had two serious
13:40problems at the time the first problem
13:42was they didn't really make any money
13:43which is always a problem
13:44the second problem which was equally
13:46annoying to the folks who ran that scape
13:48was that every time a security
13:50researcher or hacker found a new flaw in
13:52netscape's software it seemed to make
13:54headline news and we're not just talking
13:56like in the computer security press
13:57we're talking the new york times the
13:58wall street journal the boston globe npr
14:01repeatedly security researchers were
14:03finding bugs disclosing them to the
14:05public through what matt described as
14:06full disclosure and making headline news
14:09this was a serious problem a serious
14:11headache for netscape
14:12and in the fall of 1995 they decided to
14:15do something about it
14:16they decided to launch what they
14:18described at the time as a bugs bounty
14:19program
14:20it was a way in which hackers would not
14:22be sued their voices wouldn't be
14:24restrained in that way they'd be invited
14:26to submit their bugs to netscape
14:27privately and netscape would pay them
14:29t-shirts a couple hundred bucks
14:32in exchange for submitting their bug to
14:34them directly it was a brilliant idea it
14:37was a way of short-circuiting and
14:38cutting off the sort of headaches bad
14:41press that was associated with full
14:42disclosure while borrowing some of the
14:44ethos of the free and open source
14:46software movement of collaboration
14:48inviting hackers in
14:50netscape's sort of gambit was pretty
14:52successful the press attention quickly
14:54changed and for a while this no longer
14:56was a big problem for netscape so one of
14:58the things that this story tries to
14:59bring out and the reason why i think the
15:00history is interesting not just like is
15:02a historical curiosity is it shows us
15:05that power and control in some ways were
15:07baked into the model of bugs from day
15:09one it was a way of restraining and
15:11counteracting full disclosure if we go
15:13to the next slide that'd be helpful
15:17but bug bug body programs didn't just
15:19emerge from the desire of companies to
15:21sort of
15:22enclose disclosure and sort of cut off
15:24the sort of free flow of information
15:25that hackers were celebrating at the
15:27time i had a different genesis as well
15:29in the early 2000s hackers were starting
15:32to
15:33demand more respect
15:34more recognition more legal protections
15:37and money so here we have a great sign
15:40scrawled on a piece of cardboard no more
15:41free bugs and that's dino de xavi and
15:44alex sauron charlie miller is sort of
15:46the third of the three musketeers and
15:48they got up on stage in 2009 in a big
15:50conference at cansec west a security
15:52conference and said that no more would
15:54they be submitting bugs free of charge
15:56they wouldn't release them to the public
15:57they weren't going to lease them to
15:58vendors unless they got paid
15:59this is an important moment hackers were
16:02standing up and articulating a new
16:03desire and a new sort of conception of
16:05their work as work from now on they said
16:08you're not going to get this from free
16:09you got to pay us
16:11companies like google and then facebook
16:13and microsoft were listening and they
16:15basically agreed
16:16they instituted bug bounty programs as a
16:18way to recognize the work hackers were
16:20doing the serious and important
16:21contribution contributions they were
16:23making providing them also with some
16:25legal protections and a pathway to gain
16:27some recognition and it was very
16:28important and useful
16:30what happened next was bug value
16:32programs did just stay in that little
16:33corner of the world of the high-tech
16:34world they started to spread out
16:36significantly and the way they did so
16:38was by being adopted by bounty companies
16:41platforms like hacker one and bug crowd
16:43they would take this model and spread it
16:45seemingly everywhere so you can chart it
16:47from netscape to microsoft and google to
16:50the lululemons of the world
16:52and when that happened bug bounty
16:53programs no longer were just a way for
16:55covering up bad pr they were no longer a
16:58way to just simply provide recognition
16:59for security researchers they became a
17:02way of transforming hacky into gig work
17:04and that transformation is very
17:05important so i think we'll go on to the
17:06next slide and i'll turn it over to my
17:08co-author yuan yuan i think this is
17:10where you jump in yes thank you thank
17:12you everyone for having me super i'm
17:14glad to be here and talk about our
17:15research on bug bounties so i wanted to
17:17talk about who bug bounty workers are
17:20and what motivates them
17:22we found in our work that bug bounty
17:23work draws on a young and global
17:25workforce
17:27of people working hard to find security
17:28flaws in systems so when bug-bounding
17:31programs became predominant in
17:32particularly in 2010 and beyond
17:35um
17:36uh it attracted a global workforce
17:38reports from bug money programs and
17:40platforms hacker one and bug crowd
17:42provide a window into this labor market
17:45we found from looking at the reports
17:47from 2019 2020
17:49that a large majority of workers are
17:50under 30 and under 20 and many are
17:53students hacker ones report from 2020
17:56shows as well that 40 of hackers spend
17:5820 plus hours a week hacking to find
18:00bugs which is a lot of time
18:02and despite bug bounty platforms
18:04advertising about high worker wages
18:06research by ryan and others shows that
18:08only a small handful of hackers earn the
18:10bulk of these bounty payouts and his
18:12work ryan um is in this book new
18:15solutions for cyber security and feel
18:17free to add to anything i've missed on
18:18that ryan because i know you did such
18:20important research on that no i think
18:22you got it you and that's great thank
18:23you
18:25yeah so what that means though is that
18:26there's stratification there is uh you
18:29know a lot of money to earn but a small
18:30amount of people are earning a lot of
18:32money and the majority of people are
18:34earning small amounts of money so the
18:36promises of bugbendi uh bug menu work as
18:39really lucrative don't come true
18:40necessarily for a lot of the people
18:43so as mentioned the bugbending workforce
18:44is international and hacker one and bug
18:46credit reports um say that upwards of
18:49seventy eighty percent of people
18:50disclosing flaws to their platforms are
18:52based outside of the u.s
18:54these platforms have also reported that
18:5610 to 20 of their registered hackers are
18:58based in india who make up a significant
19:01portion of this workforce
19:03and the last thing we found too is that
19:05most of the companies with bug money
19:06programs are in the us showing that
19:08these companies are relying on a young
19:09workforce outside of the country with
19:12many people coming from the global south
19:14so to go to the next side i wanted to
19:16talk about what motivates big bounty
19:18workers
19:20we have a text heavy slide here showing
19:21uh what motivates people but i you know
19:24i'm just pulling some highlights from a
19:26report and the first thing i wanted to
19:27say is that there is no single
19:29motivation for hacking or engaging in
19:30bug bunny work motivations often overlap
19:33and work by gabriela coleman shows that
19:35hackers constitute a constellation of
19:38loosely tethered and evolving
19:39subcultures with shifting members morris
19:41and rights but what we found is that
19:43many people who do have bug banner work
19:45do it full-time some people do it
19:47part-time and many do on the side for
19:48extra spending money and it's important
19:50to acknowledge as well that several
19:52hackers we spoke to appreciated the
19:54flexibility of bug binding work in terms
19:55of working hours and choosing what to
19:57hack on and many of you engaging in bug
19:59bunnies as an on-ramp to more secure
20:01work there's even this entire industry
20:04of training people to do bug bound new
20:06work you can get certificates you can
20:07you can go to um sort of informal
20:09schools and and do training for this
20:11which means that there's another even
20:12there's another you know market emerging
20:14on top of the market for flaws
20:16in terms of the actual motivations as
20:18well beyond money and and and
20:21remuneration my new hackers we spoke to
20:23said they found hacking fun and saw
20:25their efforts similar to solving a
20:26puzzle for them they think that um bug
20:29benny work is a type of technical work
20:31where they can improve their skills and
20:32get better at what they do i might come
20:34as one hacker we spoke to who said it's
20:37the intellectual satisfaction about you
20:38know just finding a bug and exploiting a
20:40program and making it do something it
20:42wasn't intended to do the puzzle solving
20:44aspect of it i think is pretty
20:46satisfying
20:48other hackers we spoke to find security
20:50work and hacking and bug money work you
20:52know thrilling or engaging there could
20:54be a rush with finding something that is
20:56serious and that can be exploited one
20:58hacker we spoke to described the ability
21:00to send negative amounts of money on a
21:02cryptocurrency platform and therefore
21:03receiving positive amounts as something
21:06that came with a rush because he
21:07realized that this thing he had found
21:09could be exploited for for serious harm
21:11potentially
21:12many others that we spoke to as well
21:14found the work satisfying from a moral
21:15standpoint they they think of what their
21:17do is as hackers as a public good and as
21:20serving their communities so for alyssa
21:22herrera another hacker we spoke to she
21:24she's motivated to quote help further
21:26protect users and help further the
21:28standard of security end quote but i
21:30want to say as well that there are
21:31parallels to bug bounty platforms and
21:33other gig work platforms like uber as
21:36analyzed by alex rosenblatt who used to
21:38be at data in society research institute
21:39and is now at uber
21:41platforms can glamorize work as heroic
21:43and for the public good they can take
21:45advantage of workers desires to do good
21:48and work in the world and you know they
21:49pay people less they take advantage of
21:51them potentially because they frame
21:53their work as community service
21:55lastly for almost all the hackers we
21:57spoke to big brand new work provides
21:59them with a sense of community a sense
22:00of belonging and of being known
22:02sometimes in a large and sometimes at a
22:04small scale as well and this is
22:05something that can also be exploited and
22:07can be used to gamify bug body work
22:10which leads very well into some of the
22:11risks of this labor
22:14and and to the next slide as well
22:21thanks jan so as you had mentioned folks
22:23participate in this market for a variety
22:24of different reasons for a significant
22:26subset this is their job it's not for
22:28beer money it's not just a hobby it's
22:30work and for many of them as yuan just
22:33mentioned it's a way of hope that they
22:35will use this as a stepping stone to a
22:37career where they can engage in
22:39full-time work
22:40it's really interesting to talk to these
22:42folks to hear about their frustrations
22:43and their pleasures
22:45one of the things we see here echoed
22:47with other forms of good work is that
22:49risks are disproportionately shifted on
22:51the workers themselves rather than the
22:52organizations so the creation of bug
22:54bounty programs as they have been
22:56designed now puts workers in some ways
22:59at risk
23:00there's legal in protections are varied
23:02some programs offer safe harbors others
23:04do not so there's legal risks that are
23:06still there we also see that there's
23:09significant risks of simply
23:10uncompensated time hackers are only paid
23:13when they're the first one to find a bug
23:15if you're the second one too bad
23:17so there's this race to be first which
23:19leads to enormous amount of
23:20uncompensated time for the people
23:21participating in this market
23:23additionally we see that the bug value
23:26programs themselves wield enormous power
23:29they define what counts
23:31they're sort of no real way to challenge
23:33that review or no meaningful ways to
23:35challenge that review in many cases
23:37so what this means is in other words you
23:38could spend hours and hours looking for
23:40bugs find new submissions submit it only
23:42be told actually it's not a valid issue
23:44or it's out of scope or it's a duplicate
23:46and the hacker themselves has put all
23:47this time in with very little return now
23:50this is fine if we think about hacking
23:52as
23:53a calling it's fine if we think about
23:55hacking as a hobby when we think about
23:57it as work what we start to see is the
23:58risks disproportionately fall on the
24:00side of the workers
24:02we also see there's challenges around
24:04access how you get access to the most
24:06lucrative corners of the market which
24:07are live hacking events or private
24:09programs is often up for grabs it's not
24:12clear it requires workers to invest more
24:14and more time with the hopes of getting
24:15one of those invitations
24:17and lastly i'll say the risks here
24:18aren't just for the hackers themselves
24:20they're for all of us
24:21one of the big worries and one of the
24:23things we tease out in the report is
24:24this idea that in some cases folks are
24:27trying to use bug bounty programs not as
24:28an added layer of security but as a
24:30replacement for in-house security work
24:32this is like a very grim irony
24:35hackers are participating in these
24:36markets with the hopes that someday
24:37they're going to land a job doing
24:39full-time security work however on the
24:41other side of the table we see folks who
24:43think that this model bounty work
24:45digital piece work as we described in
24:46our report is going to replace those
24:48very jobs they're trying to seek and so
24:50this idea that the jobs themselves might
24:51disappear is something that is very
24:53worrying not only for the workers
24:55themselves but also for the rest of us
24:57because if we're going to defer to and
24:59rely on this sort of model of
25:00maintenance and security work we're
25:02going to miss so much we're going to
25:04live in a world that perpetuates bugs
25:06rather than fixes them at the root cause
25:08now our report sometimes has a bit of a
25:11doom and gloom to it but we're academics
25:13of course there's doom and gloom but
25:14what would we you know what else would
25:15we do but at the end we end with some
25:17very hopeful and i think encouraging
25:19recommendations and so you ann i'll turn
25:21it back over to you to conclude our sort
25:22of brief summary and talk about some of
25:24the more hopeful ideas we have for how
25:26this market might be reformed in a
25:27positive way i'll turn it back over to
25:29you
25:30yeah thanks so much ryan and we do have
25:32recommendations because our report
25:34highlights a lot of the risks but we do
25:37also wanted we wanted to find solutions
25:39and begin mapping these solutions
25:42in order to better secure the working
25:45conditions of people doing this work and
25:46for security in general
25:48the first recommendation that we have
25:50and that came out of our work is that
25:52bug bounty programs should be just one
25:53layer of an organization's larger
25:55security posture
25:56security posture would look like how
25:59able an organization is to respond to
26:01bug reports
26:02how how much they can actually patch the
26:05system and
26:07security in a sense would refer to the
26:09protection of data the protection of
26:10systems the protection of intellectual
26:12property and many other things um but as
26:15biology programs are used for more uh
26:18complicated problems potentially and for
26:20socio-technical problems to build off
26:22work by matt gertzen here um
26:25you know an organization's posture will
26:28involve being able to handle reports for
26:30algorithmic bias and for problems like
26:32that and and it's really important that
26:35you know regardless of how and when bug
26:37money programs are used that this work
26:39never be a replacement for full-time
26:41infrastructure work in fact you will
26:44need um
26:46uh you know you will need a team of
26:48people who respond to reports and who
26:50fix
26:50problems that are raised and as we saw
26:52with the case of netscape it's far too
26:54easy to use bug bendy programs for pr
26:57um and to draw on uh bag bunny expert
26:59and cyber security expert katie masuris
27:01it's really easy as well for companies
27:03to use bugboundy as botox as she calls
27:05it where bounty programs would be used
27:08to cover up um
27:10systems and and security postures that
27:12are in fact uh not necessarily ready to
27:15handle
27:16uh the receipt you know the reception of
27:17flaws but then also where there are many
27:19flaws to be found and that and um
27:23and you know uh for for basically in
27:25short bug money programs um can look
27:28really good but it's really important
27:29that they have resources paired with
27:32them that brings me to my second point
27:34that bug many programs would require a
27:35huge amount of time effort skills and
27:37organizational resources in order for
27:39programs to be effective workers we
27:41spoke to consistently spoke about the
27:43less than ideal working conditions and
27:45the uncertainty they faced related to
27:47slow response times non-payment despite
27:50work being done which is indeed a part
27:52of piecework for example if you're a
27:54journalist and you publish an op-ed or
27:55if you publish a piece and no one wants
27:56to publish it that is a normal a normal
27:58thing in in journalism but it doesn't
28:00mean that this can't be improved and it
28:03also and you know workers we we spoke to
28:04said that there were valid flaws that
28:06they found but maybe they'd be
28:07duplicates or maybe that the company
28:09would say this is not a flaw but they
28:10still they'd patch it anyway and it's
28:12also a clear lack of recognition
28:15we found that hackers said for the work
28:17that they do
28:18the third thing is that hackers also
28:20need better legal protection in order to
28:22safety to safely disclose
28:23vulnerabilities there are indeed certain
28:25carve outs for security security
28:27research and anti-hacking laws both in
28:29canada and the us but good faith
28:30security researchers need legal
28:32protection that does not rely on the
28:34goodwill of organizations and companies
28:36not to pursue legal action in another
28:38project of mine i found that a promising
28:40legal approach is that in the
28:41netherlands where hacker intent and the
28:43steps that the hacker took to disclose
28:46are part of the decision-making process
28:47for prosecutors before hackers face
28:50criminal liability for disclosure so for
28:52example you may have seen the story of
28:54trump
28:54when he was president and he had a
28:57really weak password and and you know a
28:59dutch hacker had actually found out what
29:01this password was try to disclose it to
29:03trump's security team no one took this
29:05hacker seriously the hacker went public
29:07in the us it's quite possible and
29:09potentially even you know probable that
29:11this person would have faced serious
29:13legal risks but in the netherlands what
29:14the prosecutor did there because there
29:16was a main prosecutor and and they have
29:18um a fairly uh centralized system there
29:21for prosecution prosecutorial decisions
29:23the prosecutor there decided that you
29:25know this person had exhausted the
29:26recourse they had taken the steps
29:28necessary which would mean that they
29:30wouldn't be responsible criminally for
29:32disclosing this information to the
29:33public because they intended to fix the
29:35system
29:36the fourth thing is that as ryan
29:38mentioned organizations and platforms
29:40that run bunk body programs often serve
29:42many of the same functions as employers
29:44do and they play a pivotal role in
29:46deciding how bug pound bug reports are
29:48handled
29:49bug bounty programs easily perpetuate
29:51the gig work norm of renting workers to
29:53draw on the work again of alex
29:54rosenblatt but many workers crave
29:56stability beyond this precarious
29:57piecework
29:58we think that classifying hackers and
30:00other bounty workers as employees and
30:02non-independent contractors would open
30:04up opportunities for these workers to
30:06secure workplace legal protections and
30:08benefits
30:09to go to the next slide as well
30:11the fifth thing we wanted to say is that
30:13regardless of
30:14the you know employment relationship our
30:16research found that organizations need
30:18to be transparent
30:19about how they measure workers
30:21performance and what their triage and
30:22dispute resolution processes are for bug
30:25reports workers we spoke to told us they
30:27didn't completely understand how people
30:28were invited to things like private
30:30events where an elite group of people be
30:32invited to hack first then get bigger
30:34payouts and be featured
30:36and and private events where again you
30:39would be a trusted entity you'd be a
30:40trusted person to hack first
30:42um and you'd have many many perks along
30:44with that free flights to things and
30:46events and networking and all that kind
30:48of and all that and indeed
30:50um it's it's very normal in an
30:52employment context that you are measured
30:54and you don't always know how you're
30:55measured but that doesn't mean that
30:57there shouldn't be a better standard for
30:59performance metrics in the world in the
31:00world of hacking and for bug bounty
31:02workers we also believe that all workers
31:04would benefit if there were increased
31:06clarity around how these perks
31:07promotions and prestigious invites were
31:09doled out and particularly with respect
31:11to triage decisions so if you say i have
31:13a flaw i'm going to submit this it's
31:15it's really important to be clear about
31:17what your process is for triage and for
31:20handling dispute resolution because if
31:21you just ghost a worker you're not going
31:23to build goodwill with people
31:25finally we also really want to urge
31:27people to reconsider the approach that
31:29uses a global pool of insecure workers
31:32to maintain business models centered on
31:34rapid iteration and perpetual beta
31:36that's because this model and this
31:38approach can perpetuate the existence of
31:40security flaws because you rely on a
31:42market to be there because your business
31:43model functions upon and rest upon the
31:46idea that people will find flaws in
31:48systems and therefore you profit off of
31:49bugs and this approach as well can
31:51solidify stratification across racial
31:54wines
31:54what we found is that the bugbending
31:57model can create the ideal conditions
31:59for exacerbating labor inequalities in
32:01it work and can also create forms of
32:03predatory inclusion to draw on the work
32:05of tracy mcmillan cotton that absorb
32:07vulnerable workers into hacking for
32:09wages in an extractive labor
32:11relationship when bounty programs and
32:13platforms fail to address the impacts of
32:15their working conditions on certain
32:17communities such as racialized workers
32:19and these programs are not necessarily
32:20inclusive but can be exploitative and i
32:22think it's important that um combating
32:25racialized labor inequalities would be a
32:27part of this journey and part of the
32:29direction of bounty programs in general
32:31and this would require rethinking how
32:33workers are integrated into an
32:35organization and on what terms
32:38so those are recommendations a little
32:39bit doom and gloom there at the end
32:41still but we are indeed hopeful at the
32:44end and i hope this summarizes well what
32:46our report has touched on and
32:48um welcome any and excited to answer any
32:51questions you have about our report as
32:52well
32:59awesome thank you you ann uh also i just
33:01wanted to note uan
33:03shouted out uh some research that i've
33:05done on the concept of social technical
33:06security thank you for that i also want
33:09to acknowledge that
33:10gabrielle lim and elizabeth watkins uh
33:14have been co-authors in some of the work
33:15i published on that or be integral to
33:17those ideas
33:18so thank you for referencing that and
33:21who's up next josh or camille i can take
33:24it from here
33:26all right it's such a joy to be able to
33:28present our work after iwan ryan and
33:32matt because we had the great pleasure
33:34of working on similar topics at the same
33:36time and it was very reassuring to know
33:39that we were not alone in the rabbit
33:40hole and we benefited greatly from their
33:43insight their research and their work
33:45and so i'm here today with my colleague
33:47josh and we're representing the broader
33:49team behind a report called bug bounties
33:52for algorithmic harms which was just
33:54published by the algorithmic justice
33:56league and on this slide you can see our
33:59co-authors which are sasha deb and joy
34:02and this report looks at how people who
34:06work on minimizing algorithmic harm so
34:08the field of algorithmic harms can learn
34:10from infosec practices particularly from
34:13the bounties but also from other types
34:15of vulnerability management programs and
34:18we are particularly interested in what
34:21lessons they are for
34:22vulnerability reporting and
34:24vulnerability disclosure
34:26um
34:27you can read the full report at agl.org
34:31bugs
34:32it's a bit longer than a hundred pages
34:34and we're not gonna try to recap
34:36everything about the report uh we're
34:39gonna try to take you through a few
34:40vignettes to highlight bits and pieces
34:42here and there
34:44josh show it to you
34:47thanks kim uh and likewise uh you know
34:50thanks uh to the folks of colombia for
34:52for having us today and and to the
34:54awesome presenters beforehand uh looking
34:56forward to this discussion uh afterwards
34:58so let's start with a little bit of you
35:00know history behind uh this report back
35:02in 2017 uh when dr joy balawini uh
35:06founder of ajl exposed how facial
35:10recognition technologies or or frts fail
35:14more on on women
35:15and on darker skinned people and and
35:17most of all on uh women with darker skin
35:21she and her research collaborators um
35:23were met with a very adversarial
35:26reaction uh from from the industry that
35:28they were scrutinizing uh frt vendors
35:31responded by attempting to discredit the
35:33research and the researchers
35:36um but eventually they had to backtrack
35:39and and in particular you know this this
35:41uh
35:42this backtracking came about as a result
35:44in part of a nist study that confirmed
35:47the findings
35:48of their research and this is an example
35:50of how
35:52there are you know parallels uh in in
35:55what's going on right now in the fight
35:57against algorithmic harm
35:59um to the early history
36:01of of infosec or of sort of cyber
36:04security as an established discipline uh
36:07and as as ryan and yuan just just
36:10discussed you know the you look back
36:13sort of the the 1990s in the early 2000s
36:16and companies were constantly attempting
36:18to to discredit to sue you know even
36:20file uh criminal charges uh against
36:23hackers just for finding and sharing
36:25security vulnerabilities including uh
36:28with the sort of intent to to fix those
36:31but before we go any further let's
36:33clarify what we mean by algorithmic harm
36:35um so an algorithmic harm occurs uh we
36:39we sort of speculate at ajl or not
36:41speculate it's it's uh you know based on
36:43the organization and the research that
36:45comes from it but this is our working
36:47definition uh it occurs when an
36:49organization or an individual uses an
36:51algorithmic system to automate
36:52classification prediction
36:54recommendations or scoring
36:56in a process that harms people in some
36:58way
36:59algorithmic harm can involve loss of
37:01freedom or opportunity violation of
37:03rights or physical safety social stigma
37:06or affronts to dignity and and even loss
37:09of life
37:10um and these days people often talk
37:12about racial or gender bias in training
37:14data and that is certainly
37:16a part of this problem and a part of the
37:18causes of this problem but algorithmic
37:21harm is not just about biased data
37:24it can arise at or as a product of any
37:26stage in the life cycle of an
37:29algorithmic system
37:30or an ai system um and during data
37:34collection and classification sure but
37:36also in model development and testing or
37:38after uh deployment in the context of
37:41use by real human beings uh next slide
37:44please
37:46and this isn't just hypothetical this
37:49isn't theoretical people in the real
37:52world experience algorithmic harm in all
37:54sorts of forms every single day
37:57for example in 2020 the aclu
38:01filed suit on behalf of robert williams
38:04who's pictured here
38:06who was falsely arrested in front of his
38:08wife and two daughters
38:10due to the failure of facial recognition
38:13technology deployed by the detroit
38:15police department he was mistakenly
38:18wrongfully identified as someone who had
38:20committed a theft
38:22uh algorithmic harms can be
38:24life-changing
38:26um so the idea of rewarding folks
38:29who might be well positioned to help
38:31prevent them
38:33or provide redress for them makes
38:35a lot of sense in the abstract similar
38:38to how rewarding hackers for discovering
38:40vulnerabilities
38:41makes sense but in both cases
38:44and again as you know you've already
38:46heard the devil is in the details
38:49uh next slide please
38:51so to help us uh more fully understand
38:53the draws and the drawbacks of bug
38:56bounties and whether they might really
38:58be useful for algorithmic harms as some
39:00had speculated previously we turned to
39:03sort of fellow practitioners and
39:04researchers some familiar faces up there
39:06on the screen right now
39:08who were kind enough to share their
39:10wisdom expertise and ideas with us
39:13for this report next slide and back over
39:15to you cam
39:16and so as we promised we're gonna go
39:18through a few vignettes on the sort of
39:20history of uh bug bounty on a little
39:23journey we are going to start with a
39:26somewhat wacky historical bounty as a
39:29way to highlight some of the central
39:30themes of our research but also as a way
39:32to introduce our design lovers
39:35then we're going to jump ahead to the
39:37moment at which
39:39traditional infosec bounties start to
39:41encompass a greater range of
39:43socio-technical issues that's happening
39:46around 2018 which is a pivotal year for
39:49that then we're going to look at
39:51twitter's bias bounty challenge from
39:53last year at defcon and finally we're
39:56going to close with a look ahead on what
39:58is happening right now with proctoring
40:01software and what this may suggest for
40:03the future of algorithmic harm's bounty
40:07all right next slide and that's the
40:08heavy one
40:09okay so a key contribution of this work
40:13we hope and again we're here to be kept
40:15honest and to get uh criticism and
40:17feedback and comments but we wanted to
40:19really unpack the wide variety of the
40:21bounties and associated mechanisms for
40:24reporting and disclosing vulnerabilities
40:26and abstract some key programmatic
40:29differences which is what we call the
40:30design levers
40:32and how these levers are configured for
40:34particular programs is really going to
40:37shape what they do for transparency for
40:40accountability for community building
40:42and for some of these aspects that we
40:44really cared about looking into these
40:46programs to really understand how is it
40:49that we can best adapt it to the
40:50practices that we care about
40:52and so in this uh exercise we are
40:56building on previous work including work
40:58from ryan in the article that you went
41:01cited and this work had noted that these
41:03bug bounty programs tend to vary by
41:06how they define market access
41:08program duration and compensation and so
41:11we're adding on these levers to talk
41:13about whether public disclosure is
41:16guaranteed on a pre-established time
41:18frame
41:19as you can guess this is going to be
41:21extraordinarily important for both
41:22transparency and for accountability
41:25we're also looking into how a given
41:27program is managed is it fully in-house
41:30or is it to some degree outsourced for
41:32instance to hacker one to bug crowd or
41:34to a platform like this
41:36what is officially considered in scope
41:38and what level of access are researchers
41:41actually given
41:42whether a program is voluntary or
41:45adversarial in other words has the
41:47target organization consented to
41:50receiving vulnerability reports that
41:52dimension is one that's really important
41:54in our work because we have found and
41:56we're going to return to this that the
41:59adversarial programs haven't really
42:01found the right way to succeed in the
42:04space
42:05and with this we can try to make those a
42:08little bit less abstract in applying
42:11this into a very old bounty josh over to
42:14you
42:17yeah so not at all to preempt or
42:20contradict the sort of origins of
42:22bounties as previously discussed this is
42:24clearly an out of left field historical
42:27example
42:28um but what the uh this challenge lock
42:31which uh is up on the screen right now
42:34uh which is from the 18th century what
42:37it what it sort of shows is uh
42:40you know how the idea of of uh exposing
42:44flaws in security um you know it has a
42:47has a long history as a means of
42:49providing redress so this lock was
42:51manufactured by joseph brahma it was a
42:54locksmith
42:55from the united kingdom
42:57uh it had almost 500 million possible
43:00combinations uh of course most copies of
43:03this lock sold weren't uh inscribed uh
43:07in the way that you can see here rather
43:08this particular lock was created to sit
43:11in brahma's shop front as a kind of
43:13advertising
43:14you know to the effect of i'm so
43:16confident in strength of this lock that
43:18i'll pay you if you can pick it now
43:21there's a familiar idea
43:23brahma's lock remained unbreakable and
43:25the bounty was uncollected for decades
43:2861 years
43:29actually until 1851
43:32when another locksmith an american
43:35alfred charles hobbs succeeded in
43:36picking brahma's lock after over 50
43:38hours of tinkering over the course of
43:40two weeks
43:42next slide please
43:43so considering this early very early
43:46security bounty through the lens of our
43:48design levers
43:50we can observe that the child is
43:52voluntary rather than adversarial since
43:54the locksmith offered the challenge
43:56compensated in the form of a one-time
43:58bounty and well compensated um you know
44:01200 guineas was the prize which is a
44:04little bit over 20 000 in today's
44:06currency uh and just as with uh you know
44:09the the many of the bounties that we see
44:11today
44:12um hobbs being the first one to break it
44:15was the winner of the prize uh
44:17subsequent you know lock picking
44:19wouldn't have have earned another prize
44:22in terms of disclosure you know hobbs
44:24reportedly performed the feed in front
44:25of journalists which is about that's
44:27about as full disclosure as
44:29it's possible to get
44:31and around the same time hobbs was
44:33actually also making the case for
44:35publishing weaknesses in lock design
44:37specifically in his 1853 book
44:39construction of locks and safes hobbs
44:41wrote that quote the spread of knowledge
44:43is necessary to give fair play to those
44:45who might suffer by ignorance this was a
44:47bounty with open participation anyone
44:49could participate and the duration was
44:51ongoing really ongoing like 61 years
44:54ongoing and lastly regarding scope and
44:56access focused on sort of picking the
44:59lock so there was physical access there
45:01um and and complete access you need to
45:04break the lock i suppose physically and
45:06inspect what was inside and the details
45:08of how it worked for public so
45:11what does this show again none of this
45:13is sort of new conceptually at a high
45:16abstract level and the idea of sort of
45:19compensation for finding uh flaws in
45:22systems of assurance whether it's
45:23security assurance or whatever you know
45:25these these potentially can apply in
45:27various contexts
45:29and you already know what happens next
45:31uh you know what
45:33pause 100 plus years and what happens
45:36next is bug bounties come to infosec uh
45:38next slide and back to you camille
45:41and so here we're going to take a
45:42ginormous sleep forward and skip the
45:46wonderful and fascinating development of
45:48how and when bug bounties come to
45:51infosec
45:52not only because
45:54our co-researchers here have done a
45:56great job at documenting it in their
45:57reports and have given some of this
45:59history on this panel
46:00and we're gonna sort of
46:03regroup in the early 2010. so at this
46:06point uh we're already seeing the
46:08widespread use of bug bounties and it's
46:10often used in combination with
46:12vulnerability disclosure programs and
46:14with pen testing we've already seen the
46:16rise of major bug bounty programs um
46:19and platforms like hakka one rug crown
46:21and yes we hack to sort of centralize
46:23them and of course we're after the first
46:26bug bounty programs by the u.s
46:27government like hack the pentagon
46:30at this point some of the largest
46:32players in tech use these platforms
46:34these intermediary platforms to solicit
46:37and triage reports and as yuan discussed
46:40they are some upside for the hackers
46:42here for instance they use those
46:44platforms you offer a more consistent
46:47user experience they offer access to
46:49many programs in one place they offer a
46:51repository of fast reports to learn from
46:54and often a community those are aspects
46:57that we were really interested in in
46:59thinking about the emergence of a
47:01younger field like algorithmic harms
47:03trying to think about what is the role
47:05that those templates that those previous
47:07reports that this community can play in
47:10bringing about a community of practice
47:13however as we discussed at the beginning
47:15of this conversation those are also the
47:18heady early days of the bug bounty
47:20everything hype
47:22with some wise researchers cautioning
47:24that bounties would not work unless the
47:27organization offering them are deeply
47:29committed to secure development
47:31practices throughout the entire product
47:34life cycle and that is an insight that
47:36we think translates well in the
47:38algorithmic harm space where there is
47:40often a lot of emphasis on the training
47:45data when we consider algorithmic harms
47:47often we hear people say oh if the
47:49algorithm is wrong it's because the
47:50training data was biased that can be a
47:53part of it but of course it's not the
47:55whole explanation and if we want to
47:57meaningfully tackle algorithmic harms we
47:59have to think about the entire life
48:01cycle
48:02so long story short uh bounties have
48:04never been silver bullets and when we
48:07fast forward again we can arrive in 2018
48:11which is the cambridge analytica moment
48:14i think we can do next slide here
48:17after cambridge analytica we're quick
48:20and we can also do next slide sorry
48:23we um see very quickly facebook and
48:26shortly after that google
48:28announced a bug bounty for data and api
48:31abuse now that's really interesting
48:33because of course this is uh quite
48:36similar to a book bounty it's kind of
48:38managed the same but when you look into
48:40the details of it it's substantially
48:42different because it really comes and
48:44stretch into those socio-technical
48:46issues and at the end of the day privacy
48:49abuse is meaningfully functionally
48:52different than a security bug
48:55and so the other thing that we learn in
48:57this story and in this moment is the
48:59pr values of big bounties as band-aids
49:02in a crisis which uh euan and ryan
49:05reminded us had a long history and the
49:07last thing in 2018 that we thought was
49:09particularly interesting is one other
49:12organization out there at least with a
49:15bounty that seeks to surface algorithmic
49:18harms and that's rockstar game
49:20so rockstar
49:22is putting up this new bounty we can go
49:24next slide in response to claim of false
49:27positive band punishments from gamer who
49:30have faced bans at the hands of
49:32rockstar's teeth flagging algorithm
49:35and so the company sets up this add-on
49:37to his traditional security bounty
49:39promising a ten thousand dollar reward
49:42for anyone who could successfully
49:44identify a
49:45reproducible incorrect ban in either
49:48grand theft auto or in red dead so if
49:51you're out there playing these games
49:53know that the bounty is still up
49:55so what are we learning overall from
49:56this expansion of bug downies
50:00to data and api abuse and then to cheat
50:02flagging algorithms that happens around
50:052018
50:06the first one is bug branding programs
50:09can be applied to socio-technical
50:11challenges
50:12beyond security vulnerability and more
50:14importantly they have already started
50:17getting there right so our conclusion
50:19was also bug boundary programs are
50:21coming to a socio-technical issue near
50:23you we see this trend already underway
50:26we see this trend potentially
50:28accelerating now some of this is good
50:30because there's indeed lessons from
50:32cyber security that we can stretch into
50:34a new domain not only for instance
50:36thinking about how to better protect
50:38researchers who do this type of research
50:41and this is where legal safe harbor can
50:43come and play a meaningful role
50:45but we also realize that some of this
50:49really is again meaningfully different
50:51and you can't just copy-paste this model
50:53you have to rethink what is it that
50:56you're trying to address who are you
50:58trying to address it with and some of
50:59the things that we cover in the report
51:01is also many times you need a different
51:04community of researchers to bring a
51:06different perspective on these
51:08socio-technical harms and we can pick
51:10this up in the discussion together but
51:12we've also found that often the
51:14community of researchers that is most
51:16traditionally engaged in these
51:17traditional but bounties program do not
51:20have the wide breadth diversity and
51:23inclusion
51:24that we would want in order to
51:26meaningfully tackle some of these other
51:27socio-technical issues the last thing of
51:30course that gets confirmed in this
51:32moment in time is that bug bounty
51:34programs for algorithmic harms makes
51:36business sense to do for some specific
51:39companies at specific moments for
51:41instance if you need to address a pr
51:43concern or if you need to address um
51:47a customer concern
51:48so with this next video and over to you
51:51josh
51:52thanks okay so recently for defcon 2021
51:56uh twitter announced a one-week
51:58algorithmic bias bounty challenge this
52:01program was created
52:02by the company's machine learning ethics
52:04transparency and accountability or meta
52:07team they were the first meta before
52:10facebook decided on a rebrand
52:12and they did this bounty in partnership
52:14with hacker one it focused on an image
52:18cropping algorithm that users had
52:20previously um
52:22expressed uh you know feelings that it
52:24was biased
52:26in ways that reinforce racism and sexism
52:28uh and in 2020 these twitter users had
52:31performed a participatory audit sharing
52:34screenshots of image crop fails
52:36on the social media platform
52:38which you can sort of see
52:40here in the before picture
52:43in-house researchers from twitter later
52:45published research confirming these
52:46users findings and through the defcon
52:48challenge twitter offered an opportunity
52:50for third-party researchers to again
52:52come in and scrutinize this model
52:54this time with bounties for the top
52:56three submissions
52:58and at the same time the company also
52:59produced a scoring rubric for
53:01algorithmic bias and harms
53:04um next slide please uh we were thrilled
53:07at ajl to see this happen and we think
53:09that twitter did a lot of things right
53:11uh in setting up this bounty we also
53:13think that this case study though
53:14illustrates the difficulty of applying
53:16bug bounties um to the problem of
53:18algorithmic harms for example their
53:21scoring rubric gave more points for
53:23problems that affected um the most
53:25people even though that implies
53:27de-prioritizing
53:28small groups of people who are at risk
53:30of suffering some of the worst kinds of
53:32algorithmic harm um twitter didn't
53:34provide any scores publicly so it's hard
53:36to assess how sort of useful the rubric
53:38was in practice
53:40but we're sort of really excited to see
53:42where that kind of of framework um can
53:44go from here uh in addition while it's
53:47you know great to see these kinds of
53:49programs emerging in response to
53:50controversies in all the cases that we
53:53looked at
53:54um of these sort of more uh
53:57socio-technical bounties so google
53:59facebook and twitter you know the
54:00original reporters of the issues that
54:02precipitated uh the emergence of these
54:04bounties who first put in the work to
54:06document and expose the harms aren't
54:09ultimately rewarded uh aren't under
54:11these programs um but on the other hand
54:14you know there's an important change
54:16management lesson here
54:18um and in particular for those of you
54:20who sort of work in this space and how
54:22to how to you know make progress uh
54:25under difficult internal conditions
54:27um twitter saw an opportunity where the
54:29stars were aligned in favor of doing
54:31something novel
54:33um and in our interviews we heard again
54:35and again the importance of finding the
54:36right pilot um to sort of get the ball
54:39moving forward for instance with the
54:41hack the pentagon program we heard from
54:42lisa wiswell about how critical the
54:45pilot was in ultimately motivating the
54:47proliferation of these programs across
54:49different government agencies
54:51and we think several factors here were
54:52key to minimizing the risk to the
54:54company and therefore willingness to
54:56sort of undertake this this pretty uh
54:58novel approach so first the sort of
55:01harms from the image cropping algorithm
55:03had already been exposed by users so the
55:05reputational damage had already been
55:07incurred by the company
55:08second twitter had already published an
55:10examination of the model's flaws and was
55:13already decommissioning the algorithm
55:16mitigating further risk of public
55:17criticism
55:19and third the cropping algorithm itself
55:21was open source rather than proprietary
55:23so even by opening it up uh they weren't
55:26exposing any uh ip
55:29and you can see here as well on the
55:30screen the sort of configuration of the
55:32program and it's quite unusual and again
55:33happy to sort of come back around to
55:35that in in the q a um but with that uh
55:38back over to cam
55:41all right thanks for watching the next
55:43one
55:43next play yes let's pivot to um who else
55:47could use bounties for algorithmic harms
55:50so this last vignette we wanted to spend
55:52a little bit of time on what happened
55:53during the pandemic where many school
55:56many universities switched very rapidly
55:58to remote learning we know
56:01as educators as students how
56:03difficult this was for everybody
56:05involved and as part of this transition
56:07we also saw the rapid adoption of
56:10e-proctoring systems to monitor students
56:13remotely
56:14now there are a lot of known and
56:16documented problems with the systems for
56:18instance a lot of them use facial
56:21recognition technologies that perform
56:23less well on students with darker skin
56:25to the research that josh initially
56:28mentioned that joy for instance and her
56:30colleagues have published years ago that
56:33part is well documented
56:35we've also seen at least one researcher
56:37ex-librium on their blog proctor ninja
56:40reverse engineer the widely used remote
56:43proctoring system proctorio to find that
56:47proctor io was using a facial
56:49recognition training library not meant
56:51for production environment and known to
56:54perform poorly on darker skin
56:56so people who are subject to these
56:57technologies have been speaking up like
57:00the students activists at encode justice
57:02and they have been turning to both uh
57:05participatory audits online to say look
57:08this is what i'm seeing on my screen are
57:10you seeing the same i think this is a
57:11problem and two more traditional reverse
57:14engineering techniques to go and
57:16document where those problems are coming
57:17from so if we take a step back again to
57:21the
57:22history of book boundaries that we
57:24looked at the other thing that really
57:26stayed with us is that the few attempts
57:28at truly adversarial programs did not
57:32last very long or did not succeed widely
57:35there's a notable exception for programs
57:38who are vulnerability disclosure
57:40programs that live within large and
57:42well-funded corporations for instance
57:44project zero at google but beside these
57:47this idea of adversarial bounties in a
57:50way or another form have not really find
57:53their final form so at agl
57:56we were left with a simple idea
57:59josh over to you
58:01that simple idea is adversarial bounties
58:04for algorithmic harms
58:07and these might be configured we thought
58:09in the following way with adversarial
58:11reporting and clues in the name
58:14compensation bounties although certainly
58:16cognizant that there are
58:18um situations in which other forms of
58:21compensation
58:22would be more appropriate depending on
58:24the nature of the work
58:26delayed full disclosure to ensure
58:28transparency and and drive
58:29accountability public participation to
58:32allow folks from
58:34different uh communities different
58:36research backgrounds uh and and
58:38including sort of impacted folks to
58:41um to sort of participate to contribute
58:44and to provide their um their their
58:46their forms of expertise um program
58:49management would be third party as in a
58:51platform but not uh we think a platform
58:54uh like
58:56hacker one or bug crowd or yes we hack
58:59um that is you know for its business
59:00model dependent on
59:03the custom of target organizations uh
59:06rather this would be an independent
59:07third-party platform
59:09and the thought initially is you know
59:11perhaps time limited focused on
59:13particular sectors or spaces or problems
59:15would help to sort of
59:17you know
59:18scope this to be feasible and and and to
59:21sort of
59:31really drive attention towards but also
59:34uh you know being adversarial um in
59:37nature uh you're not going to have
59:38access to the sort of full inner
59:40workings of the systems under scrutiny
59:42or the organizations that produce those
59:44systems so if something like this sounds
59:46exciting to you and you're interested in
59:48participating
59:50we invite you to sign up for ajl's
59:52mailing list
59:53if you have ideas for targets of
59:55adversarial algorithmic harm bug
59:56bounties please let us know
59:58if you run your own adversarial
1:00:00algorithmic harm bounty and we've missed
1:00:02it uh we'd love to hear about it um and
1:00:06i think we can move to the last slide
1:00:07just to you know point you all towards
1:00:09our report one more time it's at ajl.org
1:00:13bugs there's design lessons in there
1:00:16there's case study on the twitter
1:00:17program all sorts so thanks and with
1:00:19that i think we can go back over to matt
1:00:21and open up the q a
1:00:24wonderful thank you everyone um i've
1:00:28learned i've read the reports and i'm
1:00:30still learning from from hearing all
1:00:32that
1:00:33at the process um so i've got like a
1:00:36stack of questions about an inch thick
1:00:38and i don't know which ones to ask first
1:00:41frankly but
1:00:43one of the things that i'm immediately
1:00:45thinking about on the heels of that kind
1:00:46of lines up with some of the questions
1:00:48we're already seeing in the chat so
1:00:50maybe it's uh maybe it'll be interesting
1:00:52to dig into that
1:00:53um but
1:00:55before before i get before i do that
1:00:58there's this one like really point blank
1:00:59question i want to ask all of you which
1:01:01is
1:01:02who needs to read these reports and what
1:01:05should they do when they read them i
1:01:06mean that's a hard question but i think
1:01:09you know any any slice of that you can
1:01:11take on i would love to hear it you know
1:01:17i'm happy to go first i mean i think one
1:01:20place i would love the report to get
1:01:22read is for the folks running bounty
1:01:23programs generally so that would be
1:01:24hacker one bug crowd and also folks who
1:01:27are interested in setting up their own
1:01:28bounty programs i know
1:01:30every day it seems like there's a new
1:01:31bounty program that spins out from you
1:01:33know public sector to private sector to
1:01:36universities
1:01:37and i think just like a pause and
1:01:38thinking about some of the
1:01:39recommendations that we make at the end
1:01:41of our report
1:01:42would be so helpful so that's that's one
1:01:44place where i certainly hope it could
1:01:45get read
1:01:48yeah to add to that i think that i know
1:01:49that the
1:01:50federal trade commission in the us has
1:01:53actually been trying to crack down on
1:01:54the differences between the uh
1:01:57remuneration that is promised by
1:01:59platforms and then the remuneration that
1:02:01people receive
1:02:02i can imagine that it would be of great
1:02:04interest
1:02:05to the federal trade commission to
1:02:08identify
1:02:09um another
1:02:11place and opportunity
1:02:12in which uh workers
1:02:15are being treated differently than they
1:02:16are then you know they're then
1:02:19in terms of the promises that are being
1:02:20given to them i would want lawmakers to
1:02:23read our report too because
1:02:25uh both in terms of
1:02:27workers who are in need of protection
1:02:28but also because of the legal risks that
1:02:30hackers do face particularly the u.s
1:02:33and because many of the companies that
1:02:34they hack on would be in the us and
1:02:35indeed do pay for bug bounty programs i
1:02:37want lawmakers to better protect hackers
1:02:40as workers as hackers and um and also to
1:02:43address one point you know that ryan has
1:02:46made a few times which i love is that
1:02:47what do we do if bug money platforms
1:02:49like hacker one and bug crowd leave and
1:02:52and and
1:02:53they run out of funding in an entire
1:02:56swath of the industry and different
1:02:58industries are relying on these programs
1:02:59i think that
1:03:01you know alternatives are needed so that
1:03:02we don't rely on these companies to
1:03:04provide such important infrastructure
1:03:06and work regarding security
1:03:09i'll take a last pivot from here and say
1:03:11that on our end we we wrote it for a
1:03:14wide variety of of audience researchers
1:03:17to practitioners we try to summarize
1:03:19some practical lessons in a design
1:03:21companion but there are two audiences
1:03:23that we had in mind particularly the
1:03:25first one is um public interest
1:03:27technologists and and civil society
1:03:30organizations for them to look at these
1:03:32uh programs as potential ways to
1:03:35continue this this work of uh uh you
1:03:38know founding algorithmic harms and the
1:03:40second one is
1:03:41it's it's apparent in the history of bug
1:03:43bounties that public institutions have
1:03:45also played an interesting role in
1:03:48putting out these programs and shaping
1:03:49the norms around how they're run and
1:03:51we're interested in seeing if some
1:03:54relevant government agencies would also
1:03:56consider adversarial bounties for
1:03:59algorithmic harms
1:04:04josh did you want to add anything or do
1:04:05you think that uh
1:04:07can i get something evident
1:04:09all right
1:04:10so there's been a couple questions in
1:04:12the chat um kind of honing in on
1:04:16the difference between bug bounty
1:04:18programs and hackathons
1:04:20and
1:04:20generally the the con the idea of like
1:04:23community building in these programs
1:04:25and one of the things i find super
1:04:27interesting um
1:04:29was the discussion of
1:04:31um
1:04:32[Music]
1:04:33kind of like public explorations of
1:04:36algorithmic
1:04:38vulnerabilities or biases on twitter and
1:04:40that was something that you know people
1:04:42kind of
1:04:44did or and organized on their own
1:04:46and then
1:04:47twitter kind of took the took the ball
1:04:50running from that and
1:04:52uh you know
1:04:53uh hosts the def con event
1:04:55but one of the things you know um
1:04:58that the bounty everything report talks
1:05:00about is how the the early you know
1:05:03netscape
1:05:05bugs bounty program
1:05:06uh was very much like an attempt to kind
1:05:09of control the narrative and
1:05:11and co-op
1:05:12you know co-op things i think you know
1:05:14the the report
1:05:16the language it uses is you know to
1:05:18blunt negative attention
1:05:20and and kind of
1:05:21enclose the this this market so that it
1:05:24could be controlled right so are there
1:05:26are there in your
1:05:29reports did you find like is there some
1:05:32type of balance between companies that
1:05:34are able to like take this information
1:05:36from bounties
1:05:38programs running them versus the value
1:05:40of people doing it from the outside in
1:05:42kind of an uncontrolled way and being
1:05:45able to demand accountability in a
1:05:46public way
1:05:48what is like is there a proper balance
1:05:50between that or
1:05:51do are both needed or is there is it
1:05:53possible to have a bounty program that
1:05:56can really build that kind of community
1:05:57and
1:05:58uh and also serve that function if if i
1:06:01ask the question well
1:06:05i have some thoughts and i might even
1:06:07answer another question while providing
1:06:08these thoughts but someone has asked for
1:06:10our take on federal vulnerability
1:06:11disclosure policies and publicly funded
1:06:13bug money programs and i have done a
1:06:16work that looks at the canadian
1:06:17government's use of vulnerability
1:06:19disclosure programs which is we
1:06:21highlighted are bug money programs minus
1:06:24money
1:06:24and
1:06:26i think it is possible to have bug
1:06:29let's say vulnerability disclosure
1:06:30programs where people aren't paid
1:06:32and what that means is that
1:06:36there could be a sense of unfairness
1:06:37because you're not getting compensated
1:06:38for your labor but by paying people for
1:06:41bug bounties
1:06:42that is bugs and bug reports they submit
1:06:45then that means that you're creating a
1:06:47market and you're turning a person into
1:06:49a laborer i think that there is a time
1:06:52and a place for not paying people
1:06:53because
1:06:54particularly from a
1:06:56state and government perspective to pay
1:06:58people would be to turn the thing you're
1:07:01paying for into a type of market
1:07:03um so i i think that there is a lot of
1:07:06value actually in in
1:07:08vulnerability disclosure programs that
1:07:10are run by governments where they pay
1:07:11people in things like swag or they just
1:07:14they say they pay you in in recognition
1:07:17and i it's hard to answer because you
1:07:19want to also respect the commander the
1:07:21hacker demand for payment as i did in
1:07:25kansas west but i also think that there
1:07:27is benefit in having programs um where
1:07:29you don't almost taint the relationship
1:07:31with money by turning things into a
1:07:33market
1:07:40did anyone else want to jump in on that
1:07:42right yeah sure i think um
1:07:45i got your question like is there a
1:07:46right balance between community building
1:07:49transparency like that's such an
1:07:50important question i saw it pop up in
1:07:52the chat as well
1:07:53people really enjoy working in this
1:07:55market right it's a thrill they find
1:07:57friends they find meaning in their work
1:07:59and so those things are important to
1:08:00acknowledge and not discount one of the
1:08:02tricky things though is how
1:08:04the desire to be part of that community
1:08:06can be sort of turned on its head and so
1:08:08getting access to that community whether
1:08:09it be invited to private programs or
1:08:11invited to like lavishly funded live
1:08:14hackathons and live events in las vegas
1:08:17that are sponsored by platforms and
1:08:18companies
1:08:20the desire to do that then drives
1:08:21engagement and it pushes often
1:08:23uncompensated work and so it really is
1:08:25like a double-edged sword i think the
1:08:27trick that like gig work always is is
1:08:29that it sets people with different
1:08:30motivations
1:08:32into a pool and pushes them against each
1:08:33other in some ways whether they want to
1:08:35be pushed against each other or set
1:08:36against each other or not it's the idea
1:08:38you have people here who consider bounty
1:08:39programs as beer money versus people who
1:08:42see it as their way to a career and or
1:08:44even their full-time job it creates
1:08:46these really strange and difficult
1:08:47dynamics that can make the sustaining of
1:08:49a community very difficult and so that's
1:08:51one of the things that i report try to
1:08:52get out is not to dismiss the fact that
1:08:55people find pleasure and community and
1:08:56friendship and engagement in these
1:08:58things they do but showing how it gets
1:09:00complicated when it's mixed in with
1:09:01these other dynamics
1:09:04and i will say we have in the q a a
1:09:07wonderful question by uh one of our
1:09:10co-authors uh sasha who i think is here
1:09:13with us and sasha is pointing at some of
1:09:16the issues with disclosure too right so
1:09:19how can we help mandate more systematic
1:09:22disclosure and uh you know collectively
1:09:25i think all of our report acknowledged a
1:09:27difficult relationship between the
1:09:29bounty programs and disclosure where a
1:09:32lot of the by default settings is to
1:09:35prevent the disclosure of the flaws and
1:09:38vulnerabilities that are found so i'm uh
1:09:41highlighting this question by by sasha
1:09:44and and if you allow me to do that
1:09:46getting it back on the on the stage
1:09:55one thing that's really interesting and
1:09:57that's a to follow up on that is the
1:09:58question of like
1:10:01when bug bounty programs like go wrong
1:10:03and how they can be used to catch and
1:10:05kill right so we have examples on our
1:10:07report about uber which i think is a
1:10:08very well known story now where the cso
1:10:10was brought up on federal charges
1:10:12because they essentially tried to cover
1:10:13up a data breach through their bug
1:10:14bounty program like don't do that that's
1:10:16not what bug bounty programs are
1:10:17supposed to be for we also have stories
1:10:19from john deere and others that are
1:10:20trying to use their bug bounty program
1:10:21as catch and kill so the question here
1:10:23about
1:10:24how we think about disclosure and how do
1:10:26you create a world where bug bounty
1:10:28programs can be used to help get flaws
1:10:30out in the world rather than to cover
1:10:32them up is so important
1:10:34and it's tricky it's difficult i mean i
1:10:36think the federal case around uber is
1:10:38going to make people very nervous
1:10:40but the other thing that we can do
1:10:43is hackers have power here they have
1:10:44real power by which programs they decide
1:10:46to participate in and looking at the
1:10:48terms of service and picking and
1:10:50choosing based on those that allow for
1:10:51disclosure versus those that are going
1:10:53to require ndas and so the hackers
1:10:55themselves have maybe sometimes more
1:10:57power than they might realize
1:10:59one of the things that we try to
1:11:00emphasize is that this work is very
1:11:02difficult you hear these like eye
1:11:03popping numbers that there's tens or
1:11:05hundreds of thousands of people signed
1:11:06up to participate in bug bounty programs
1:11:08which is true but there's a much smaller
1:11:11group of people who are incredibly
1:11:13effective and successful and they have
1:11:16real power to shape how this market
1:11:17works as well if they are willing to
1:11:19take it if they're willing to make those
1:11:20choices and make those choices publicly
1:11:22so i think that's one way in which we
1:11:24can push for
1:11:25um maybe not regulators themselves
1:11:27pushing for it but the hackers and
1:11:28participants themselves can help be sort
1:11:30of that forcing function around
1:11:31disclosure
1:11:34i mean that leads nicely into one of the
1:11:35things that we were sort of thinking
1:11:36about in the context of how to make
1:11:38adversarial programs work a little bit
1:11:41better than they have historically which
1:11:42is
1:11:44you know these
1:11:46barriers
1:11:48of varying
1:11:50um firmness
1:11:52these sort of legal threats
1:11:54uh that exist
1:11:56to disclosure
1:11:57um and that whether they are
1:12:00you know whether they are realistic in
1:12:02terms of the hacker will end up
1:12:04incarcerated or whether they are just a
1:12:07a looming threat
1:12:09um i think it's been pretty well
1:12:10documented at this point that both of
1:12:12those create a chilling effect uh on
1:12:14what people research and what happens to
1:12:17that research when they find things and
1:12:19so this is this is one of the key points
1:12:21that we want to sort of put out there in
1:12:23the world with respect to potential you
1:12:25know um sort of adversarial platform
1:12:27successors to the likes of of we own
1:12:30token bug bounty is is we need to have
1:12:33support in place um that can't be just
1:12:36reliance on legal safe harbor because
1:12:38that's not going to be offered by
1:12:39organizations who don't want to have uh
1:12:42the flaws uh in and about their systems
1:12:45exposed um they're not going to offer
1:12:47that and so what are the alternatives
1:12:48that are out there sure there are some
1:12:50uh you know whether it's juan
1:12:53spoke earlier about you know
1:12:54prosecutorial guidance to sort of take
1:12:56away
1:12:57that particular threat in places where
1:12:59that can be feasibly achieved and where
1:13:01uh governments understand these issues
1:13:03well but then there's also just a lot uh
1:13:06of potential benefit i think to be
1:13:08gained by having intermediating
1:13:10organizations that can say we have your
1:13:12back
1:13:12uh whether it's resourcing whether it's
1:13:15legal guidance that's provided ahead of
1:13:17time
1:13:18level setting on the playing field of
1:13:20what they are going to have your back on
1:13:23um can can probably go quite a long way
1:13:26i mean if legal safe harbor can solve as
1:13:28many problems as it has in the bug
1:13:29bounty space
1:13:30then presumably having a well-resourced
1:13:32intermediary that truly does have the
1:13:34back of the people doing the scrutiny um
1:13:36could could could support as well
1:13:39um that's obviously not going to get off
1:13:41to all of the challenges of of sort of
1:13:44um cutting out some of the misaligned
1:13:46incentives that exist in this ecosystem
1:13:48right now
1:13:49um but it would certainly help address
1:13:51some of the sort of knock-on effects
1:13:54i don't know if any others have thoughts
1:13:56on that
1:14:00maybe i can add a thought that um uh
1:14:03i've been thinking you know it's a bit
1:14:04in my head for a while um back before
1:14:07ryan and i were writing on the poor but
1:14:09we're doing the research we had this um
1:14:12event that day in society um where we
1:14:14had feedback on on
1:14:16on the work we were doing thus far matt
1:14:17was there um other people i i as well
1:14:20who works at the society and she
1:14:22actually pointed out that there is that
1:14:24she saw similarities between hackers and
1:14:26screenwriters and she said you know you
1:14:28might want to look into screenwrite
1:14:30writer's guilds and writer's guilds and
1:14:32i actually haven't looked into that
1:14:33unfortunately but i think there's a lot
1:14:35there and i say that for two reasons the
1:14:37first is that um there's this notion of
1:14:39labor where you're doing labor it feels
1:14:41it feels like play um and there and i
1:14:44actually think of hacking as a creative
1:14:45field um where you are very creative in
1:14:48your work often and you produce outputs
1:14:51like much like you would when you are
1:14:53creating film or tv
1:14:55and what what i can see is that the
1:14:57screenwriters guild emerged and it acts
1:14:59as a sort of union and it protects many
1:15:01many workers and so there is a question
1:15:03here about what are recommendations for
1:15:05hackers and i think that
1:15:06it would be amazing if groups of hackers
1:15:09had you know formed
1:15:11unions and and
1:15:13guilds that would protect them as
1:15:15workers and i would advocate for
1:15:17them to receive certain compensation or
1:15:20to be acknowledged in certain ways and i
1:15:21think that that movement would be really
1:15:23needed and would be really beneficial
1:15:26for protecting the rights of hackers as
1:15:27hackers and as workers
1:15:35okay
1:15:36that leads me to uh something i was
1:15:39thinking about like one of the
1:15:42really top-line recommendations of the
1:15:44agl report in particular is
1:15:47the need especially as we move into
1:15:51the idea of bounties for
1:15:53algorithmic harms or social technical
1:15:55harms
1:15:55of
1:15:56increasing the diversity in the pool of
1:15:59the people who are actually
1:16:01participating in these programs so that
1:16:03they can surface
1:16:04harms that might be visible to them from
1:16:06their subject position that might not be
1:16:08visible to others or others might not
1:16:10even think to look at
1:16:11um but it seems to me like is there is
1:16:13there a bit of a wicked problem between
1:16:15that kind of imperative and also the
1:16:18imperative to
1:16:19improve job security and pathways to
1:16:22secure employment for the existing pool
1:16:25of laborers i mean there's got to be
1:16:26some way to balance that kind of tension
1:16:29of wanting to draw in as much as many
1:16:31people from as many diverse perspectives
1:16:33with as many skill sets as possible and
1:16:35also make sure that these people are not
1:16:37not being treated to the worst effects
1:16:39of casualized labor
1:16:41um
1:16:42and i guess that gets back to some
1:16:44extent to the idea of how people who are
1:16:47voluntarily doing this as communities
1:16:49aren't necessarily expecting to be
1:16:50rewarded but they should be as well so
1:16:52i'm just curious if if you have thoughts
1:16:54on that it's a very hard problem i'm
1:16:55sorry to to put it out there but
1:16:58if anyone knows hopefully it's you guys
1:17:06i mean i can i can i can sort of
1:17:07speculate i think
1:17:09you are right it is a it is a wicked
1:17:11problem
1:17:12and
1:17:15if we look across
1:17:16our technology ecosystem more broadly
1:17:19right now
1:17:20um
1:17:22the challenges of of building diverse
1:17:25inclusive communities absolutely with
1:17:27respect to background um
1:17:31but also with respect to sort of you
1:17:33know professional expertise research
1:17:36interests and methods and so on like
1:17:38this is not something that we have
1:17:41got answers to today i do think there
1:17:43are places um
1:17:44you know that are looking at how to do
1:17:46this
1:17:47uh
1:17:48and and and making progress
1:17:50my
1:17:51um sense based on our research is that
1:17:56there's not going to be a sort of
1:17:58one-size-fits-all solution with respect
1:18:01to sort of the institutions that can
1:18:02facilitate community building let alone
1:18:05the programs that those um institutions
1:18:08might offer to
1:18:10um attract in participants to protect
1:18:14them in the various ways they need to be
1:18:15protected and to compensate them where
1:18:17they are producing
1:18:19something of value
1:18:22but the question of who gets to
1:18:23determine what is valuable and what um
1:18:27kinds of work and contributions deserve
1:18:29to be compensated i mean this ties into
1:18:32what are your templates what are your
1:18:34impact scoring frameworks who
1:18:35contributed to
1:18:37um to building those the idea that sort
1:18:40of certain work has value in this space
1:18:42and and others that's a political
1:18:44question and it's a social question a
1:18:45cultural question and so you know you
1:18:48sort of have to have
1:18:49i think an open-mindedness to
1:18:52um to to to work out what salute you
1:18:55know to trial and error to work with
1:18:58stakeholders and communities to figure
1:18:59out what solutions are going to work for
1:19:01them best and and the idea that we're
1:19:02going to come in and a panel today and
1:19:04be like yes if you do xyz you will have
1:19:06a diverse community of practitioners on
1:19:08your platform no
1:19:10you're right we're not um and and that's
1:19:12okay
1:19:18thanks did anyone else want to wait on
1:19:20that or should we move on to a lighter
1:19:23lighter
1:19:26affair okay
1:19:28um
1:19:29so that also that uh
1:19:32touches on a couple other directions we
1:19:34can take this one is exploring the
1:19:36idea of the security development life
1:19:38cycle a little bit more
1:19:40and
1:19:41one is exploring the kind of role that
1:19:44these kind of existing institutions in
1:19:46the bug county
1:19:48space place so let's see maybe i'll go
1:19:50with the latter one first so as as
1:19:53josh's answer kind of just suggests like
1:19:56some of these existing big players in
1:19:59the you know bug bounty
1:20:01infosec space are starting to also
1:20:04move into
1:20:06you know providing
1:20:07their existing infrastructure platform
1:20:10for
1:20:10more algorithmic harm type issues
1:20:13um
1:20:14is there what are the kind of you know
1:20:17pros and cons of seeing that happen
1:20:20versus
1:20:22having you know new organizations kind
1:20:24of come up and try to try to start you
1:20:26know develop
1:20:27protocols for these types of bug many
1:20:29programs from scratch suited to you know
1:20:32algorithmic carbs in particular
1:20:37yeah that's a great question matt i'm
1:20:38going to try not to answer we wrote 100
1:20:40pages about this
1:20:42but it was sort of one of the key
1:20:44questions which is you know what does it
1:20:46take to meaningfully take those programs
1:20:48and stretch them to those
1:20:49socio-technical issues to your question
1:20:52on like things that immediately come to
1:20:53mind as trade-offs when you use
1:20:56platforms who've been managing programs
1:20:58for a long time like they can accompany
1:21:01those you know those those companies and
1:21:03like how do you do triage how do you
1:21:05develop an impact scoring uh how do you
1:21:08make sure that you recruit for your
1:21:10program how do you pay people and so
1:21:12they make that transition uh much easier
1:21:16now
1:21:17the other thing of course that comes
1:21:18with that is you know you also don't
1:21:21have adversarial programs by definition
1:21:23here you have programs that are hosted
1:21:26on platforms by targets who have agreed
1:21:29and who are participating and as a
1:21:32result you're also targeting
1:21:34the um sort of existing community of
1:21:37people who participate in these
1:21:39platforms and we can talk a little bit
1:21:41more uh there was an interesting uh you
1:21:43know set of questions in the chat about
1:21:45the lack of diversity uh of the current
1:21:48communities who most often engage with
1:21:50these types of programs so again if we
1:21:52go through all of our our libraries we
1:21:54can um we can spend quite a bit of time
1:21:57trying to figure out like all right is
1:21:58this a good first step right like we
1:22:00definitely think that it is we think
1:22:02it's important to have these programs
1:22:04we're grateful that we have a little bit
1:22:05of transparency of course as researchers
1:22:08we always want more transparency i think
1:22:10josh said it we really wanted to see how
1:22:12those uh scoring framework had been
1:22:15applied to the different submissions so
1:22:17you know using these platforms is an
1:22:19interesting first step in stretching
1:22:21these programs but then you also end up
1:22:23in situations where um you know some of
1:22:26the harder questions of what needs to be
1:22:29done differently for these models to
1:22:32succeed on issues that are fundamentally
1:22:34different
1:22:35doesn't doesn't really get i think the
1:22:36full treatment that it sometimes
1:22:38deserves
1:22:45one interesting like follow-on point
1:22:47that recalls this i don't think it made
1:22:48it into our report but one of the
1:22:49interesting conversations we had was
1:22:51talking to someone who had set up a bug
1:22:52binding program inside a large public
1:22:54agency an organization and when they
1:22:56talked about it that one of the measures
1:22:58of success that they had in their mind
1:22:59was that it allowed them to argue
1:23:01internally to their managers and their
1:23:03higher-ups that things needed to change
1:23:05and so beyond the value of like a
1:23:07particular bug or particular submission
1:23:09it allowed them to say like we need to
1:23:10change our contracting policies we need
1:23:12to change
1:23:13sort of how we do development and
1:23:14testing all these other things and so it
1:23:16occurs to me that thinking about the
1:23:18value of like an experiment like the
1:23:19twitter
1:23:20and other socio-technical experiments
1:23:22some of the value might be that they
1:23:25enable folks who are already trying to
1:23:26work for change inside to have the power
1:23:29and the resources and something they can
1:23:31point to that allows them to
1:23:33advocate for those changes more
1:23:34effectively internally that's not like
1:23:36that certainly wasn't the way i thought
1:23:37about bug bounty programs going into
1:23:39this but certainly it was an interesting
1:23:40wrinkle that sort of camille hearing you
1:23:42talk about it made me think about that
1:23:43as well
1:23:46just to be caring like we totally agree
1:23:48we think there's a lot of value in that
1:23:50right like we've seen over and over
1:23:52practitioners talk to us about like book
1:23:54bounties as a way to accelerate change
1:23:56and then wait to sort of demonstrate
1:23:58that more transparency is possible to
1:24:01demonstrate that more scrutiny can can
1:24:04lead to good outcomes so there's
1:24:06definitely um
1:24:07you know change management of value to
1:24:10to these book family
1:24:14programs yeah i mean doubling down on
1:24:17that i think if you asked even some of
1:24:19the executives at bug bounty platforms
1:24:21they know
1:24:23that the organizations who are doing
1:24:24this
1:24:25the most impactfully and effectively
1:24:28shore in dolls and sense terms for these
1:24:30companies but those who are doing it
1:24:32really well
1:24:33they're doing root cause analysis on
1:24:35vulnerabilities which not every
1:24:37organization does some some say they do
1:24:39they they don't um
1:24:41you know they are looking for the
1:24:42reasons why these things occurred and
1:24:44that then does tie into matt's first
1:24:47question which we you know i guess maybe
1:24:49we can pivot back to of of you know what
1:24:51what does the life cycle look like and
1:24:53and and where do these lessons apply
1:24:56um
1:24:58and are you willing to go beyond
1:25:01you know
1:25:02technical fixes to look at
1:25:04organizational questions processes
1:25:07controls
1:25:09culture
1:25:10that form
1:25:12the sort of underlying bedrock of why
1:25:15security is a relentless uphill battle
1:25:19um
1:25:21i think if we look across it you know
1:25:23for algorithmic harms you know for
1:25:25example to it's it's it's as cam said
1:25:28you can't map across perfectly
1:25:31an an algorithmic you know a sort of
1:25:33algorithmic
1:25:34harms
1:25:36an algorithmic system life cycle that
1:25:38has the right components in place to
1:25:41address all the myriad harms is
1:25:43is going to include a data governance
1:25:45life cycle uh it's got you know it's
1:25:49it needs to be more than just industry
1:25:51setting what this life cycle looks like
1:25:53you have to bring in community
1:25:54organizations so that you can understand
1:25:56what are the risks on the back end when
1:25:57the products are being used
1:25:59and what are the different risk domains
1:26:00there are security risks in ai which
1:26:02also implicate fairness
1:26:05as well as sort of inherent
1:26:07characteristics of the products being
1:26:08deployed um and and you know how do you
1:26:11tie these pieces together
1:26:13and how do you learn from reports in
1:26:15ways that are meaningfully impactful
1:26:16across that life cycle i think is
1:26:19is a wide open question um for sort of
1:26:22ai practitioners scholars
1:26:25communities of all
1:26:27shape and sizes to sort of help you know
1:26:29work through
1:26:30um but that's where the real value comes
1:26:32from is is can you tie this back into
1:26:35what is going wrong the organizational
1:26:37route with people with processes
1:26:40you know there are humans behind and in
1:26:43front of technology
1:26:44and map perhaps that's an area where we
1:26:46can turn the question back to you and
1:26:48say you know what are sort of some of
1:26:50the
1:26:51you know what are some of the lessons
1:26:52here for security bug bounties
1:26:55of how to address the sort of
1:26:57socio-technical characteristics of these
1:26:58problems more effectively than they do
1:27:00today
1:27:02yeah so well that actually tight i'm
1:27:05going to answer
1:27:06that by asking the next question that i
1:27:09was formulating in my head for you
1:27:11because
1:27:12you know one one thing my understanding
1:27:15of early
1:27:16uh you know software development
1:27:18security
1:27:19development life cycle
1:27:20type of
1:27:22practices and processes
1:27:24was that one thing that was very
1:27:26important was like vulnerability
1:27:28databases
1:27:29and i think this this gets to some of
1:27:32the questions that are in the chat and
1:27:34kind of synthesize them together because
1:27:36one of the things disclosure allowed
1:27:39was for you know different people to
1:27:42gather
1:27:43various vulnerabilities classify them by
1:27:46different types see when
1:27:47they remained you know
1:27:50in existence
1:27:52uh it allowed engineers or developers to
1:27:55have an awareness of where other people
1:27:57had gone wrong so that when they start
1:27:59building their software they can
1:28:00incorporate those
1:28:02you know known problems and address them
1:28:04preemptively
1:28:05so one thing like i've wondered is like
1:28:08what
1:28:09is there anything like a vulnerability
1:28:11database being developed for algorithmic
1:28:14harm socio-technical harms
1:28:18how you know how integral is that to an
1:28:20effective stl
1:28:22you know type of thing type of framework
1:28:24for
1:28:25algorithmic systems what are the bare
1:28:28you know what are the things stopping
1:28:29that from happening how do bug bounty
1:28:31programs relate to that i'm just curious
1:28:32if any of you have thoughts on that if
1:28:33you know of any any projects already at
1:28:36you know in an ex
1:28:39existent on that front
1:28:43yeah i'm so glad that you brought this
1:28:44up because this is something that we got
1:28:46really excited about in thinking about
1:28:48what is it you know what would it mean
1:28:50to translate the
1:28:52central and public uh databases for
1:28:55vulnerabilities to the space of
1:28:57algorithmic harms i think there are a
1:28:59lot of um
1:29:01ways in which this makes sense and not
1:29:03only because you also see some of the
1:29:07same problems of having underlying
1:29:09pieces of technologies or underlying
1:29:12databases that are used by multiple
1:29:15projects and it can help the
1:29:16transparency and it can help
1:29:18the accountability that being said i
1:29:20think here and we already have some
1:29:23examples of some of these databases
1:29:25appearing uh josh can talk about a few a
1:29:28few of them but i will say i think the
1:29:30the difficulties you have to circle
1:29:32around
1:29:33what is the topic that you're aiming to
1:29:35cover
1:29:36back to your last question a metaphor
1:29:39that we really liked and and used is
1:29:41what uh katie mussoris calls the
1:29:43digestive systems that are needed for
1:29:46bug bounties right it's the point that
1:29:47josh was explaining if you are opening
1:29:50yourself to receiving bugs you need the
1:29:52digestive systems internally to actually
1:29:55process those bugs and you need the
1:29:57teams on the other side to impact them
1:29:59and to address them and when we take the
1:30:01entire space of socio-technical harms we
1:30:03realize that those digestive systems in
1:30:05industry are so scattered and different
1:30:07right so the people who will address for
1:30:09instance some of these algorithmic harms
1:30:12on the machine learning side are very
1:30:13different people that some of the other
1:30:16type of issues that should be routed
1:30:18through the trust and safety teams or
1:30:20the anti-cheat teams and so i think that
1:30:22the first question is like
1:30:24yes this is a super promising idea we
1:30:26would love to see more people working on
1:30:28this what would it mean to create those
1:30:29centralized
1:30:31databases
1:30:32and i think that in order to succeed at
1:30:34it you would have to be super specific
1:30:37on which actual type of harm are you
1:30:40trying to circle around and does this
1:30:41type of harm have kind of i don't want
1:30:44to call it unified digestive system
1:30:45because it sounds like a bizarre
1:30:47metaphor now but like do do we have some
1:30:49form of agreements on where does it go
1:30:52and where does it need to be digested
1:30:54um
1:30:55josh over to you i know that you and i
1:30:57have had long long conversations around
1:30:59this
1:31:00yeah and i'll try not to i'll try not to
1:31:02be as long-winded on this one as i was
1:31:03on the last i mean i think there's
1:31:05there's some nascent projects out there
1:31:07i know that in the course of sort of
1:31:08other crash project research streams
1:31:11um and community engagement work we came
1:31:13across
1:31:14a partnership on ais artificial
1:31:16intelligence incident database
1:31:19um
1:31:20but i think that you know you hit the
1:31:22nail on the head with what uh what is it
1:31:24that we are taxonomizing or planning to
1:31:26taxonomize planning to itemize here
1:31:30incidents are tracked separately from
1:31:32vulnerabilities in the cyber security
1:31:34space
1:31:35um we shouldn't necessarily assume you
1:31:38know that harms versus
1:31:40um
1:31:43the sort of sources of harm should be
1:31:45uh tracked
1:31:47collectively together they should tie
1:31:49together but but you know that that is i
1:31:51think uh something to consider
1:31:53um
1:31:55i would just suggest that you know
1:31:58double down on the more visibility we
1:32:00can get here the more thoughtfully we
1:32:02can start organizing these
1:32:04these things and understanding how to
1:32:05prevent them um in a in a sort of
1:32:08structured way
1:32:09um
1:32:10and you know if there's
1:32:13if there if there's i mean there's many
1:32:15lessons in the history of bug bounties
1:32:18uh for today's security practitioners um
1:32:20but perhaps there's there's a key one
1:32:21there which is i i don't think we are
1:32:23getting the the value out of this as a
1:32:25you know across
1:32:27across the sort of security space as we
1:32:30could be from bug bounties from
1:32:32vulnerability disclosure programs
1:32:33because
1:32:35even when things get fixed they
1:32:37generally don't get disclosed i mean
1:32:38we've read through hundreds of program
1:32:41terms from hacker one
1:32:42and it was a small minority that
1:32:45afforded any kind of you know guarantee
1:32:48of subsequent disclosure upon for
1:32:49example patching or 90 days or 120 days
1:32:53and this is what we're getting at with
1:32:54these you know independent
1:32:55intermediaries and what they can achieve
1:32:57google project zero they say you know x
1:32:59days later we are going to release this
1:33:01vulnerability with details you know and
1:33:04and and they do and they can do that
1:33:06because
1:33:07you know they are empowered in this
1:33:09ecosystem um to to sort of speak in that
1:33:12way
1:33:13um and and i think
1:33:15what can we learn from sort of that
1:33:17model and to bring back and to scale up
1:33:20the the sort of learnings that we get
1:33:22out of out of vulnerability disclosure
1:33:24and
1:33:25um really actually start to get after
1:33:27some of the um you know the fundamental
1:33:30misaligned economic incentives in tech
1:33:33space when it comes to security i i
1:33:35think this this is promising but we have
1:33:37to figure out how to get past the nobody
1:33:39wants to talk about x problem
1:33:43i wanted to chime in too um i think
1:33:46camilla you really highlighted well how
1:33:48if you're going to have a database it
1:33:49means it's harder in some ways when you
1:33:51when it comes to augmented harm and bias
1:33:54and accountability because systems are
1:33:56so different
1:33:57and what harm means in a certain context
1:34:01is going to be different in another
1:34:02context i did want to share two links i
1:34:04didn't know about this incident
1:34:07ai incident um database which is super
1:34:09interesting so i'll just share a link to
1:34:11that for anyone interested thank you for
1:34:12mentioning that josh and then the very
1:34:14sexy topic of vulnerability databases um
1:34:18also brings me to the the first link i
1:34:20shared um
1:34:21i i would think that there is much value
1:34:23in having a database of ai incidents but
1:34:25acknowledging the issues that you raised
1:34:27camille and then i also can't help but
1:34:29think of your work mat where in a way um
1:34:32uh email lists where hackers would
1:34:34disclose flaws they found was served as
1:34:36a sort of database it's just that it
1:34:38does it was harder to search for it
1:34:40wasn't you know information wasn't
1:34:41necessarily indexed in the same way
1:34:43um and i think before there would be um
1:34:46a really useful database of algorithmic
1:34:49harms i think it would need to you know
1:34:52there need to be research on how are
1:34:53these vulnerability programs or
1:34:55databases being used in cyber security
1:34:57more traditionally understood
1:34:59you know who visits them how does that
1:35:01inform other people's work and then
1:35:03you'd want to figure out you know what
1:35:05taxonomies like you mentioned josh would
1:35:07make sense in the algorithm the carb
1:35:08space
1:35:09um because otherwise if you just create
1:35:11a database it could just exist in the
1:35:13ether at the same time i'm also
1:35:15conscious of other work that i feel like
1:35:16you've um done to matt where you've
1:35:19tried to catalog
1:35:20the um affordances of things like social
1:35:23media platforms and the harms that can
1:35:24arise but there can also be features as
1:35:26someone mentioned in the q a
1:35:28and i think those kind of databases are
1:35:30really valuable as well and i know that
1:35:32there's actually um a u.s freedom of the
1:35:34press tracker where journalists are
1:35:36trying to track how
1:35:38they've been treated by governments by
1:35:40entities in terms of harm and that the
1:35:42experience and the work that they do and
1:35:43i think of that as a kind of database as
1:35:45well that you know serve as inspiration
1:35:47but i think more research would be
1:35:49needed and especially in order for this
1:35:50kind of database to be useful when it
1:35:52comes to ai harms
1:35:57yeah they're oh sorry ryan go ahead no
1:36:00just to reiterate and emphasize um one
1:36:02of the things we found in our interviews
1:36:05is like the common source of frustration
1:36:06for people working on the infosec bug
1:36:08bounty side is like what counts as a
1:36:11valid flaw is so deeply contested and so
1:36:14i think it's attractive as outsiders
1:36:16think well in the technical world flaws
1:36:17are clear the socio-technical world
1:36:19squishy like they're squishy all the way
1:36:21down
1:36:22and so thinking about common taxonomies
1:36:25databases like it is so important
1:36:27because it's such a recurring point of
1:36:28friction even on the technical side that
1:36:30when you move into the world of
1:36:31socio-technical harms it's going to be
1:36:34so open to
1:36:36competing interpretations and
1:36:37contestation that anything you could do
1:36:39to have like baselines and agreed upon
1:36:40metrics or agreed upon frameworks that
1:36:42aren't just um defined by one
1:36:44organization or one institution would be
1:36:46really helpful right because it's like a
1:36:48recurring point of friction that drives
1:36:50the participants absolutely batty with
1:36:52good reason so i think that just
1:36:54underlines the point of like why this is
1:36:55both needed and also the the hazards the
1:36:57risks of thinking about like where would
1:36:59it live where would it sit and how could
1:37:00it be developed
1:37:02yeah i just want to echo echo this point
1:37:04by ryan i think um you know and there's
1:37:06a great question in the chat too about
1:37:08have we given some thought about the
1:37:09ways in which uh algorithmic harms and
1:37:12cyber security bugs are different we
1:37:14wrote an entire section on this i can i
1:37:16can send the
1:37:18specific portion of the report but i
1:37:19think that what ryan is saying is so
1:37:21important right we have a tendency to
1:37:23say like oh isn't it nice on the cyber
1:37:25security side that everybody agrees on
1:37:27whether it's a bug and whether you can
1:37:29fix it and you say this to a to a hacker
1:37:31or to someone who routinely participates
1:37:33in a bug bounty and they just laugh you
1:37:35out of the room right this
1:37:37so much dispute so much uh interesting
1:37:40and fascinating questions on what's in
1:37:42scope out of scope what's the feature
1:37:43what's a bug what's fixable what has
1:37:46been fixed uh what is duplicative and i
1:37:48think this is also part of the what
1:37:50we're hoping to learn right like um none
1:37:53of this is is trivial all of this is
1:37:55actually quite complicated and so it's
1:37:57interesting and important to see those
1:38:00mechanisms that have been set up to
1:38:02address these um
1:38:04these questions and which way have they
1:38:06worked and in which ways have they
1:38:07failed right this is why we really
1:38:09enjoyed reading iwan and ryan's uh
1:38:12examination of um
1:38:15dispute resolution mechanisms in bug
1:38:18bounty programs and in which ways uh do
1:38:21they help clarify
1:38:23uh what what what really we were trying
1:38:25to solve with uh with the reporting and
1:38:27vulnerability management
1:38:33i mean they can just add to that
1:38:34actually um i i think that it's just
1:38:37these are all such good points and maybe
1:38:39um that perhaps it could it could make
1:38:41sense then for there to be many
1:38:43databases of ai harms and that maybe
1:38:46certain communities decide for them what
1:38:47is what is a harm to them just as
1:38:49journalists are creating this database
1:38:51in the us and they're going to be doing
1:38:52like i know people are doing this in
1:38:54canada where they're saying here are the
1:38:55harms we face or hear the risks we face
1:38:57we want to catalog this for transparency
1:38:59for accountability and to and so that
1:39:01people can learn from us and in terms of
1:39:03freedom of information requests and even
1:39:05being and harassed by police or or or
1:39:07arrested by them for doing their
1:39:09journalistic
1:39:10work and
1:39:11and i think that it could make sense for
1:39:13communities themselves to decide what
1:39:15constitutes harm and i can't help but
1:39:17think of sasha's work on this topic of
1:39:18participatory design and how the human
1:39:20um the people who are most impacted by
1:39:24um the harms of a system might be the
1:39:25best place to decide what what we talk
1:39:28what we mean when we say harm or or
1:39:30weakness or security or even exploit
1:39:34i will paste the link to sasha's
1:39:36wonderful book design justice for
1:39:38everybody in the chat
1:39:42yeah that's
1:39:43great sorry go ahead josh
1:39:46i was just going to put an exclamation
1:39:47point on that i mean you look into the
1:39:49way that cvss the common vulnerability
1:39:52scoring system works and there is a base
1:39:54score a sort of temporal score you know
1:39:56and then an environmental score which is
1:39:58supposed to figure in context
1:40:04it doesn't work perfectly
1:40:06but there is at least an acknowledgement
1:40:09even in that sort of you know uh
1:40:11component of the metric that it matters
1:40:14where a vulnerability sits in a system
1:40:16and who is interacting with that system
1:40:19who is authorized and able to access
1:40:21that system and and sort of what are the
1:40:24you know the sort of
1:40:25more socio-technical elements of it you
1:40:27know what data is flowing through this
1:40:29vulnerable system you know those things
1:40:31really matter whether you can
1:40:34access something that you shouldn't be
1:40:36able to access is sort of secondary to
1:40:38what are the effects of that access and
1:40:42we definitely do identify some
1:40:44distinctions between algorithmic harms
1:40:47as a sort of outcome
1:40:49versus vulnerabilities as a mechanism
1:40:51towards an outcome that can be harmful
1:40:54and there's there's some stuff going
1:40:56back in there but
1:40:58these
1:41:01i think
1:41:02a lot of the issues that we've seen in
1:41:04security in the last 25 years
1:41:06um have at their root some in some part
1:41:09um a failure to acknowledge
1:41:11uh that
1:41:13this is these are socio-technical
1:41:14problems and
1:41:16if we could start getting after that and
1:41:18if that comes about as a result of you
1:41:20know other socio-technical issues being
1:41:21taken more seriously i think that would
1:41:22be a really great thing uh for the for
1:41:25the sort of so for cyber security
1:41:26practitioners and and ultimately the
1:41:29sort of users and and subjects of tech
1:41:33just want to
1:41:34quickly pick up on that because i think
1:41:36josh is talking about something that's
1:41:38really important and that matt your
1:41:40writing has really helped eliminate
1:41:41which is cyber security itself is really
1:41:44dealing with its own borders and
1:41:46expanding as a field infosec is
1:41:48expanding into a right what are all
1:41:50those technological issues that are
1:41:52adjacent which ones are infosec which
1:41:54ones are not we've seen some of these
1:41:56discussions fascinating with privacy as
1:41:59we said api abuse i'm you know closed to
1:42:02to my work i think i've really seen this
1:42:04in the way infosec tackled information
1:42:07operations as some of these were indeed
1:42:10conducted by traditional apt actors in
1:42:13the field was very familiar with and
1:42:14some of that felt very much close to
1:42:18close to the field another dimension of
1:42:20it felt very far from it so i think that
1:42:22what we're talking about is also a field
1:42:24that's very much in movement very much
1:42:27negotiating its own boundaries uh and
1:42:30were our small hope was that you know on
1:42:33the socio-technical side we could
1:42:35extract some lessons for
1:42:38people working on algorithmic harms but
1:42:40perhaps too there are some lessons from
1:42:42people who work closer to the
1:42:44socio-technical side of things that can
1:42:46help infosec and overall the cyber
1:42:49security field in its own transition to
1:42:51better understanding what's um
1:42:54on its borders on the frontiers of the
1:42:56field sorry that was very abstract
1:43:01now this is one i mean i love that this
1:43:04this got into such abstract kind of
1:43:06epistological questions because i think
1:43:08that's really
1:43:09where a lot of these
1:43:11problems are at which is interesting and
1:43:13i mean that's not just from our
1:43:14discussion these are also the types of
1:43:16questions that are repeating over and
1:43:18over
1:43:19in the chat and it's just reminding me
1:43:21that you know even
1:43:22passwords were even controversial on
1:43:26you know shared mainframes because some
1:43:28people
1:43:29saw that as
1:43:30entity
1:43:32you know opposed to the philosophy of
1:43:35shared resources and
1:43:36there's always going to be
1:43:38i mean maybe maybe buffer overflow
1:43:40attacks and things like that are an
1:43:42example of something that probably
1:43:44everyone agreed was a
1:43:46a
1:43:47security challenge but there's always
1:43:48going to be a political
1:43:50dimension that's informed by where
1:43:52people are doing the analysis from and
1:43:54it's it's i'm very excited to see where
1:43:57people take that
1:43:58um
1:43:59and and how this work contributes to it
1:44:02um but we are
1:44:04you know running low on time so maybe
1:44:06instead of getting more abstract my
1:44:08brains is already worrying we can we can
1:44:10track back to a couple um
1:44:12more concrete questions
1:44:14i mean one is you know one of the
1:44:16panelists noted in our panelists chat
1:44:19that there's so much agreement on
1:44:21different issues but i was wondering
1:44:22when when you were each reading each
1:44:23other's reports were there any insights
1:44:25or
1:44:26findings that you found particularly
1:44:28interesting that you hadn't hit upon in
1:44:31your own work um
1:44:35yes that's a is that is that a fair
1:44:37question
1:44:43i mean it's not something i disagree
1:44:45with but i was very struck by you know
1:44:48the the the deep
1:44:49digging that joanne and ryan did on the
1:44:52netscape bounty
1:44:54uh i think is something that will that
1:44:56there's a section of their paper that
1:44:58has is worth reading even aside from
1:45:01everything else that's worth reading in
1:45:02that paper it is
1:45:04you know how persistent is the idea that
1:45:06you can use these kinds of mechanisms to
1:45:09just shoe a pr problem under the rug
1:45:11well it was the from the first one
1:45:13onwards um
1:45:15so i was very struck by that and and
1:45:17really enjoyed reading that part in
1:45:18particular um
1:45:21cam i know you were about to jump in
1:45:22there as well no i was gonna say i think
1:45:24some of the first thing that that
1:45:27was very odd for for us when we started
1:45:29this research project is initially we
1:45:31thought it was going to be short and we
1:45:33were going to get away with a small
1:45:35paper which of course we massively
1:45:37failed at this and one of the first
1:45:38thing that was just very puzzling is
1:45:41how much feelings and disagreements
1:45:43people had on both bounties and they
1:45:45would just you know like our
1:45:46interviewees they would just like really
1:45:48disagree with each other and it would be
1:45:49different schools of bug bounties and
1:45:51such strong disagreements within the
1:45:53field and i think at first it was a bit
1:45:55head spinning for us being
1:45:57how is it that people can disagree on
1:45:59everything here so much including the
1:46:02history of bug bounties what is or is
1:46:04not about boundaries things on which you
1:46:05could see like you know you you you
1:46:07could conceive perhaps that there would
1:46:09be more agreements and um i felt for us
1:46:12it's at that time that we
1:46:13met juwan and ryan whose research was
1:46:15very grounding both in putting some you
1:46:19know some some semblance of like all
1:46:21right these are the things that are
1:46:23actually documented that people align on
1:46:25and perhaps on explaining some of the
1:46:28deeper dynamic that explained some of
1:46:31the passion and disagreements that we
1:46:34very quickly saw and i will admit that
1:46:36we that we took a hot second to to
1:46:39process and pack
1:46:42one thing i mean i learned so much from
1:46:44reading um matt your report as well as
1:46:46josh mcmillan your report but one thing
1:46:47that like
1:46:49wasn't front of mind for me when i was
1:46:51doing this work was thinking about like
1:46:52this alternative model of adversarial
1:46:54bounties
1:46:55like how useful it might be how fraud it
1:46:57might be how difficult it can be but
1:46:59really ultimately how useful
1:47:01um in our space of the infosec world
1:47:03adversarial boundaries like don't really
1:47:06they're not really there that's not
1:47:07where the market went if you talking
1:47:08about adversarial bounty it kind of
1:47:09looks more like the offensive market
1:47:11right people buying and selling exploit
1:47:13kids like that's the adversarial market
1:47:15which is beyond the scope of sort of
1:47:16what we look into for a variety of
1:47:18reasons but it made me wonder like um
1:47:21in a very serious way like did we
1:47:23institutionalize the wrong model here
1:47:25like did bug bounties evolve in a way
1:47:27through good intentions and bad
1:47:29intentions and happenstance and
1:47:31everything else that we detailed did we
1:47:32institutionalize the wrong model and so
1:47:36reading camille and josh's work and the
1:47:38report from ajl more generally it made
1:47:40me think like
1:47:42here we have a chance maybe to get it
1:47:43right and i hope we do and so that was
1:47:46something i took away that was not you
1:47:48know on my radar at all one of the many
1:47:49things but really resonated with me
1:47:51reading the report and resonated even
1:47:52more today hearing um it talked about in
1:47:55such a clear way
1:47:58so we're supposed to disagree now here
1:48:00we are just patting each other in the
1:48:01back again we failed we failed in
1:48:02running short reports and we failed and
1:48:04arguing oh no
1:48:09um
1:48:09[Music]
1:48:11okay so yeah so i'm i'm walking away
1:48:13from this thinking like i mean there's
1:48:15this one
1:48:16big level of challenges which is you
1:48:18know how do we even think of what the
1:48:20buckets and the classifications and
1:48:21these kind of epistemological challenge
1:48:23there's also this very pressing much
1:48:25more concrete need just to bring people
1:48:28you know a more diverse pool of people
1:48:30who can identify
1:48:32uh more
1:48:34you know wider range of of issues and
1:48:36also be heard right to amplify them
1:48:40and these are two two big challenges i'm
1:48:42also curious you know if having finished
1:48:44this work what are was there were there
1:48:46any things that you wished you could
1:48:48answer that you couldn't get at or
1:48:50things that you would like
1:48:52you know the students or other
1:48:53researchers who are watching this or
1:48:55reading a report to kind of
1:48:57pick up on and continue the thread
1:49:00forward to um you know for the next
1:49:03round of phase of research on this
1:49:04important
1:49:05topic
1:49:07um
1:49:08i have thoughts immediately on that um
1:49:10and
1:49:11the first thing i think of is the
1:49:14question of labor issues and our report
1:49:16you know focuses significantly on the
1:49:18working conditions and labor issues
1:49:19related to bugbani programs and bounty
1:49:21programs in general
1:49:23i don't think we've solved the problem
1:49:24and i don't know if there ever will be a
1:49:26solved problem here but
1:49:27i don't know if we have
1:49:30solved the problem of figuring out what
1:49:32the solution looks like i did mention
1:49:34the idea of creating guilds or groups of
1:49:38hackers aka unions who would advocate
1:49:40for baseline standards of treatment that
1:49:42seems like a really
1:49:43great way forward according that will be
1:49:45extremely hard given the international
1:49:48labor market we're dealing with whereas
1:49:50in in you know historically and
1:49:52typically unions are based on people who
1:49:54work at least in a country because you
1:49:57have certain rules are applied in your
1:49:59region and by the governments that have
1:50:01jurisdiction where you live
1:50:03and so with that said i think that how
1:50:06will the labor conditions improve in
1:50:08this market that is an open question and
1:50:10i would hope that um people begin
1:50:13working on that because we've looked at
1:50:14how hackers experienced this we've
1:50:16looked at
1:50:18what people's experience of this market
1:50:20is and we know that many dr benefit from
1:50:23it um benny loved this field they love
1:50:26working in this in in this
1:50:28field and doing this work and i would
1:50:30assume that it's going to be the same
1:50:31case for algebra algorithmic arms um but
1:50:34i hope that there is
1:50:36more work and better answers to the
1:50:39question of how do hackers then how do
1:50:41workers and infrastructure workers
1:50:43protect their rights because of the
1:50:45issues that we've highlighted
1:50:50there's
1:50:51two things i really wish
1:50:52i could know and hope to find out in the
1:50:54future the first is like we look so much
1:50:57on the side of the labor of the people
1:50:59who are finding and disclosing bugs i
1:51:00would love to know more about the people
1:51:02who have the most thankless job in the
1:51:04world which is triaging the incoming
1:51:05reports and bugs it's like seeing their
1:51:08perspective understanding how the market
1:51:10works from their view is something i
1:51:11would love to know more about and i hope
1:51:13to find out in the future
1:51:14the other thing that i would love to
1:51:15know more about if i could wave my magic
1:51:17wand is to get access to sort of the
1:51:19books for the bug bounty platforms like
1:51:22i am dying to know about more details
1:51:25about their financial model are they
1:51:26going to survive are they going to make
1:51:28it
1:51:29um these are big vc backed companies but
1:51:32they've been around now they're coming
1:51:33up on a decade
1:51:34and they've become important parts of
1:51:37the vulnerability disclosure pipeline
1:51:39for many many companies and in the back
1:51:41of my mind i have like a real serious
1:51:43worry about what happens if they go away
1:51:46because we built a lot of things on that
1:51:47foundation but we still don't really
1:51:48understand
1:51:50like the plumbing of those companies in
1:51:52a way that i would like to know so that
1:51:53would be if i could wave wave my wand go
1:51:55back to the plumbing slash digestive
1:51:56metaphors i want to know more about the
1:51:58financing on that side so those are my
1:52:00two my two things i'd like to know more
1:52:02about
1:52:03ryan if i could say like hey we've built
1:52:05a lot of things on this foundation but
1:52:07does anyone have any clear idea of the
1:52:09plumbing is the overall story of cyber
1:52:11security
1:52:16um i think there's so much uh that that
1:52:19we would love to know more about i'm
1:52:20hoping that our report sort of helps uh
1:52:24you know
1:52:25give give others research directions
1:52:27from what happened to adversarial
1:52:29bounties to or how do some of these
1:52:32concepts translate to other spaces i
1:52:35will add a small note of something that
1:52:37came up in our work which is doing this
1:52:39work with the perspective of um
1:52:42inclusion in mind and translating to
1:52:45spaces that are very uh more
1:52:47socio-technical and perhaps more aware
1:52:49of uh the language we use and how we
1:52:52talk about this and how words shape uh
1:52:55shape who participates and how those
1:52:57programs are seen we struggled a bit
1:53:00with the vocabulary that is widely used
1:53:02in infosec starting with the word bounty
1:53:05that uh it's you know that's that's
1:53:07something that came up for us and and i
1:53:09think there's also a little bit more to
1:53:10do here to reinvent not just how these
1:53:13programs work but you know the the
1:53:15really the words we use to talk about um
1:53:18some key concepts in infosec so i'll
1:53:21just add this small note
1:53:28and i am fully seated on the topic of
1:53:30bug bounties this
1:53:32i'm good no i'm kidding uh i think i
1:53:34would share ryan's interest in the
1:53:37financial plumbing of the
1:53:38platforms um
1:53:40particularly given what we've seen in
1:53:42sort of other
1:53:44loosely comparable um
1:53:47i'm gonna use the word crowdsourcing
1:53:50it's not quite right read the report to
1:53:51find out more
1:53:53but those kinds of approaches may not be
1:53:55as sort of scalable durable profitable
1:53:58as
1:53:59um
1:54:00i think some in silicon valley have
1:54:03led
1:54:04founders the public governments to
1:54:06believe
1:54:07and it comes with
1:54:09really really serious costs for the
1:54:12people who are
1:54:13working in these ecosystems and i would
1:54:16just like to express gratitude to have
1:54:18been able to
1:54:20work on this topic at the same time
1:54:22um as these wonderful folks and many
1:54:24others who are sort of pushing the boat
1:54:25out in this space it's been really
1:54:27really interesting and
1:54:30um
1:54:31can't wait to see uh how folks build
1:54:33upon this and and move forward
1:54:37uh yeah to add to that matt is an
1:54:38extremely humble person but it's because
1:54:41of matt that we know each other because
1:54:43he had these conversations with both of
1:54:45us
1:54:46through dave's society and connected us
1:54:47and so i just gotta give credit where
1:54:49it's due and matt thank you for
1:54:51connecting us and making this happen
1:54:53um despite never taking credit for what
1:54:55you do
1:54:57that's very flattering but there's a lot
1:54:59of other people at that society integral
1:55:01to that so it's it's it it's a it's a
1:55:05community right
1:55:07yeah
1:55:08one tiny correction as well i said i
1:55:10really said the federal trade commission
1:55:12and i realized
1:55:13about 20 minutes ago i met federal
1:55:15communications commission so i just
1:55:16wanted to correct that for the record
1:55:21nice the transcript will be appended
1:55:23suitably i don't know if there's going
1:55:25to be transfer but
1:55:26yeah well this is great i mean thanks
1:55:28everyone for uh inviting me to
1:55:30participate in this i learned so much
1:55:32from you all thanks to columbia for
1:55:34hosting this
1:55:36jason do you have some final thoughts
1:55:38for us
1:55:39yeah absolutely hope i just learned a
1:55:42ton out of this this was really
1:55:44fantastic these are topics that i kind
1:55:45of thought i knew something about and i
1:55:48guess i did but i just learned so much
1:55:50more so thank you very much
1:55:53um also um we've got a ton of other
1:55:55things coming up uh that we hope that
1:55:58you that have joined here uh will will
1:56:00like to
1:56:02visit also for example we have teamed up
1:56:05our uh salzman institute of war and
1:56:07peace studies has teamed up with
1:56:09columbia's school of the arts
1:56:11the digital storytelling labs for defrag
1:56:13which is the hacked
1:56:15film festival we've got an event on war
1:56:17games that is going to be coming out
1:56:19on the 24th of february that's going to
1:56:22be featuring we're pretty sure um
1:56:25director of sissa jen easterly one of
1:56:28the top
1:56:29cyber security officials in the country
1:56:32as well as and i think this is the first
1:56:33time
1:56:35for this we're actually going to have a
1:56:37general from norad we hope to have on
1:56:39the panel so both from the norad angle
1:56:41and from that we have a lot of other
1:56:44events coming up as part of our nigella
1:56:46rhoden digital futures forum so please
1:56:49keep an eye out for that and also the
1:56:51events coming up from the salsman
1:56:53institute of war and peace studies thank
1:56:56you very much have a great weekend
1:56:59but happy lunar new year for those that
1:57:01are celebrating

About this transcript

This page contains the full transcript of Debugging Bug Bounties in Cyberspace: From Vulnerability Discovery to Algorithmic Harms Redress by Columbia SIPA, generated from the public captions YouTube serves with the video. The transcript has 20,677 words across 3,358 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.