Martin Odersky – Keynote: The End of Programming? #ScalaDays — Transcript
Full transcript
- 0:07Okay. So, welcome to Scala Days 2026.
- 0:11Uh, I think we all have uh pretty
- 0:14momentous uh events of the last year on
- 0:18our mind. A lot of a lot has happened
- 0:20around AI and agents. So, I think there
- 0:23are many people now who ask is it the
- 0:26end of programming? So common statements
- 0:29are uh Antropic says software
- 0:32engineering could be automated in 12
- 0:34months. So that was last uh January
- 0:36February that he said that. So that's
- 0:38pretty imminent, right? Uh uh Sundai
- 0:42Pear Google said 75% of all code is
- 0:45written by agents. And Ruby on Rails, he
- 0:49gave this uh famous or infamous
- 0:51conference talk very recently who said
- 0:53basically yeah it's over people. Uh the
- 0:56uh all code in our company is written by
- 1:00agents, no humans involved.
- 1:03Okay. So um you might say uh well okay
- 1:07maybe we can let agents write the code
- 1:09but we should oversee it right we should
- 1:12review the code. Uh nobody writes code
- 1:14anymore but you still read it for review
- 1:17right no I think that also that time has
- 1:21passed. the reviews are now done by
- 1:23other agents and I think humans don't
- 1:25really like to review bunches of code in
- 1:27particular if they're like the
- 1:29avalanches produced by the agents so
- 1:32forget about that so I think we should
- 1:34ask well what roles do humans play in
- 1:37all of this um and uh what we what we
- 1:41see here is a a typical workflow u uh
- 1:45that uh the u yeah the typical workflow
- 1:49where uh we write uh uh a prompt to
- 1:53demand some sort of program. The AI
- 1:55produces an initial program. Then we
- 1:58might have change requests or our users
- 1:59might file tickets and that all gets
- 2:01piped into the agent. The agents will
- 2:04will update the program. But the program
- 2:06in the whole process stays a black box.
- 2:08So we don't understand what's in there.
- 2:10We just tell essentially this AI to do
- 2:13something with the program and the
- 2:15program well the program will just
- 2:17behave the way the agent programmed it.
- 2:19a human doesn't look inside.
- 2:22Um,
- 2:24sorry, I I I skipped ahead. So, in this
- 2:27scenario, what about programming
- 2:29languages? Um, I can't really see how
- 2:32they would matter anymore, right? Uh,
- 2:34whatever's most convenient for agents,
- 2:37uh, fastest turnaround, least tokens,
- 2:39most efficient code, that's probably the
- 2:41metrics that that matter, but not
- 2:43readability, modularity, safety,
- 2:46expressiveness. Who cares about that?
- 2:49Hm.
- 2:51Is that future likely? Um, maybe. Many
- 2:55claim so. I don't pretend that I have a
- 2:58crystal ball so I don't take a position
- 3:00but it might be. But we should also ask
- 3:03is that the future we want? And uh then
- 3:07uh if we should follow that up to say
- 3:09well is that the future we want? So who
- 3:11is we? Is we programmers? They probably
- 3:15don't want that future, right? That it
- 3:17puts their jobs in danger. Uh it takes
- 3:20out the fun in their work. Uh so I think
- 3:23it's pretty bleak. Their bosses might
- 3:26well embrace it. Uh because well maybe
- 3:28the whole thing is cheaper than and
- 3:30faster than uh relying on humans to do
- 3:33that. Uh but if we scale out and say
- 3:36humanity as a whole, do we want that as
- 3:38humans?
- 3:40So there's a problem with that because
- 3:42if we give up the ability to understand
- 3:45and review programs, we literally hand
- 3:48the keys to the agents, our own or our
- 3:51an attackers. So we we are out of that.
- 3:53The agents essentially do that among
- 3:56themselves. And that is a shorefire way
- 3:59to get the singularity not because of of
- 4:02a machine intelligence explosion but
- 4:04because as humans we abandon the core
- 4:06domain that runs civilization in the
- 4:08software construction. If we do that
- 4:10then yeah essentially we we is a
- 4:12dereliction of of of agent of duty that
- 4:16leads to a loss of agency of humans. So
- 4:17I think it's really dangerous to that uh
- 4:20that these things work out this way.
- 4:24So um what are the alternatives? Um can
- 4:28we all go back to manual coding? I fear
- 4:30that cat is out of the bag. Uh should we
- 4:33review all agent generated code? Well,
- 4:36nobody wants to do that. Can we somehow
- 4:40try to raise the level of abstraction
- 4:42similar to what happened when we went
- 4:43from assembly to compiled code? uh where
- 4:46we say okay so nobody reads assembly
- 4:48anymore but it doesn't matter because we
- 4:50read C and Scala and and uh and other
- 4:54languages and these express precisely
- 4:56what the computer should do so we just
- 4:59instruct the computer at a higher level
- 5:01is that another level raising well the
- 5:04problem is that agents they resemble
- 5:06compilers somewhat we give them prompts
- 5:08in English or natural language and they
- 5:11produce code but they are like Christy
- 5:14likes to say they're flaky compilers,
- 5:16the compilers that don't produce the
- 5:18same thing each time, that might make
- 5:20mistakes, that don't give us a guarantee
- 5:23what they do. Um, so if you want to
- 5:26control their behavior and ensure
- 5:28correctness of our program, we need some
- 5:31external way. We can't just instruct the
- 5:34model and say please don't make mistakes
- 5:36or wait for the next model which is
- 5:38hopefully more reliable than the current
- 5:40one. I think this one this will not work
- 5:42out. we need external ways to control
- 5:44the behavior and ensure the correctness.
- 5:48Okay. So um what this talk is about an
- 5:53exploration what we might do there and
- 5:56one good point to start is AI agents. Uh
- 6:00so that could be coding agents or other
- 6:01agents that do stuff for you because for
- 6:04these AI agents the future has already
- 6:06arrived. These AI agents, they invoke
- 6:09tools and they do so by uh generating
- 6:13little program snippets at least uh
- 6:16mostly now. There are other ways to
- 6:17invoke tools but uh code execution is a
- 6:20very very common one. So the agent
- 6:22generates a little program, nobody looks
- 6:24at the program, the program is
- 6:25immediately executed by a tool. So that
- 6:28means we have LLMs that generate unreed
- 6:31code that invoke tools that can have
- 6:34arbitrary side effects. So that's the
- 6:36current situation and as we all know
- 6:40that means we really living dangerously
- 6:42uh the another week goes by where
- 6:44there's not another incidence of a
- 6:46private information leakage API keys
- 6:49credentials trade secrets whatnot that
- 6:52there's data loss uh the head of
- 6:54security of Meta had his had his email
- 6:58erased for instance um and of course
- 7:02lots and lots of sensitive systems are
- 7:04hacked so everybody Everybody knows the
- 7:06hugging face interface where essentially
- 7:08the AI hacked hugging face uh uh and it
- 7:13wasn't detected for I think a whole week
- 7:15that what what happened there. So it's a
- 7:17swarm of a agents hacking things and
- 7:19then the hacked into the Australian
- 7:22Medicare site and lots of other sites
- 7:25and of course this is just the
- 7:27beginning. I mean it's just starting and
- 7:29agents will become a lot more numerous
- 7:31and a lot more powerful and we're really
- 7:34living in a in a scary time for that. So
- 7:38with that I think there are two
- 7:40challenges. The first challenge is how
- 7:42do we contain these AI agents? Uh so how
- 7:45do we prevent them from hacking into
- 7:49systems leaking things? Can can we
- 7:51somehow give them guard rails that
- 7:53ensure and in a guaranteed way that they
- 7:56can't do that? And the second challenge
- 7:58I think is how to design software to
- 8:01better resist agentic attacks because
- 8:04the if you look take the hugging face
- 8:06incidents it really has two sides. It's
- 8:08the agents breaking out and attacking
- 8:10hugging face but it's also hugging face
- 8:12letting itself being attacked. So so and
- 8:15we take that sort of as a law of nature.
- 8:18Every every software has holes. Every
- 8:19software is buggy. Only nowadays agents
- 8:22are so good and so fast that we can't
- 8:24really accept this anymore. And that's
- 8:26that that's also a huge change, right? I
- 8:29mean, we have this uh world where before
- 8:32a next model is released, it's sort of
- 8:34kept under wraps for weeks and some
- 8:36privileged partners are being told to
- 8:39test their security, others are not. I'm
- 8:42sure government services know this as
- 8:44well and exploit this to find zero days
- 8:46and things like that. So basically we
- 8:48have a a a a a
- 8:51system where for some weeks we have
- 8:54privileged access to these models to do
- 8:56whatever and then it gets released and
- 8:58sort of we have the next wave of of uh
- 9:01of attacks and possibilities. So what
- 9:04can we do there? It's really uh we don't
- 9:07really have much uh uh readymade
- 9:10recipes.
- 9:11So because the current defenses that we
- 9:14have they all fall short. So what we
- 9:17have is we can have allow lists. So
- 9:19block lists for instance there's a um
- 9:22AMD um uh Amazon setter language
- 9:27uh that that does that. So we have some
- 9:29regex rules what should be allowed and
- 9:31what should not be but they're very
- 9:32coarse grain and they're typically they
- 9:34have have this problem that they either
- 9:36too permissive or too restrictive and
- 9:39you don't find a good middle. Uh we can
- 9:41ask the users for confirmations. I I
- 9:44guess we all have have seen that coding
- 9:46agents where we say click click click
- 9:48click accept accept accept until
- 9:50nowadays they don't ask anymore because
- 9:52it's too it's too uh inconvenient so
- 9:55there's user fatigue that can't scale
- 9:58sandboxing yes we can sandbox we can put
- 10:00agents in sandboxes that very much
- 10:02restrict what they can do but uh the
- 10:05problem is that this is also very coar
- 10:07grain because it gives you an absolute
- 10:09set of permissions which is either
- 10:11typically also too loose loose or too
- 10:14strict. So too strict that you can't do
- 10:15anything that you want to do and too or
- 10:18too loose that you permit uh dangerous
- 10:20behavior. So one uh possibility for
- 10:23instance you see here on the right uh
- 10:25where we see an agent reads the secret
- 10:28in step one and it exfiltrates the
- 10:30secret in step five and each step looks
- 10:33fine in isolation but together we have
- 10:37essentially a secret that got leaked
- 10:38which is essentially the behavior that
- 10:40we wanted to prevent. Now we could have
- 10:42a sandbox that says you can't read any
- 10:44secrets and that you can do but there's
- 10:47a lot of behavior that a lot of uh
- 10:50functionality where you need to access
- 10:52private data. So that's very
- 10:54restrictive. We could also have a
- 10:55sandbox that says you can't send
- 10:58anything to the outside world. But again
- 11:00that's that's a big problem. Uh so what
- 11:03we need is essentially a a defense that
- 11:06tracks the information flow. So this
- 11:08thing goes in here. It's labeled secret.
- 11:11you can't leak this piece of data, you
- 11:13can leak others. So we had for instance
- 11:15this famous thing where Entropic
- 11:16released the source code of the of their
- 11:19next version of of cloud code and that
- 11:22was one of these things. So essentially
- 11:25uh uh of course you want to publish some
- 11:27data
- 11:29companies publish uh repos all the time
- 11:32but you shouldn't publish a private repo
- 11:34but that's again a thing of information
- 11:36flow. Okay, so that's the current
- 11:40situation and it one thing that's
- 11:43particularly problematic is here what
- 11:45that's been called the lethal trifecta
- 11:48where you say if you combine all these
- 11:50three things then you're in trouble. Uh
- 11:53so um uh if you have access to private
- 11:56data, the ability to externally
- 11:58communicate and exposure to untrusted
- 12:01comments uh content, then essentially
- 12:04any website that the agent accesses can
- 12:08trick it into uh essentially uh leaking
- 12:11secrets uh accessing private data,
- 12:13sending it to the attacker. Uh so you
- 12:16say, okay, so don't do all these three
- 12:18things together because two or two of
- 12:20them at a time are fine. Uh but that's
- 12:22again that's that's difficult because
- 12:24essentially agents read websites um
- 12:27emails whatever and that's untrusted. Uh
- 12:31you don't know what's in there. Somebody
- 12:32might plant a prompt injection attack.
- 12:35Uh and agents do access private data as
- 12:38I said and agents are able to externally
- 12:42communicate. So taking one of these
- 12:43things away is very difficult.
- 12:47So our key idea in this that we have
- 12:49been working uh essentially all of this
- 12:51year so far uh is that we don't try to
- 12:54make the model trustworthy. So we don't
- 12:57work on alignment or any of these
- 12:59things. We try to make the medium safe
- 13:01and uh that means that instead of having
- 13:04an agent generating little snippets of
- 13:06code that talls tools that affect the
- 13:09world and hoping that the model won't
- 13:12misbehave. what we say well what if the
- 13:14code was typed in a certain type system
- 13:17that I'm describing that in a couple
- 13:20slides and then checked by the compiler
- 13:23and only when it passes the compiler
- 13:26sent to the world and the compiler would
- 13:28reject unsafe code before it runs. So
- 13:30that's the approach that we're
- 13:32proposing. So how does that work
- 13:34exactly? Well, the agent would express
- 13:37their intention in a capability safe
- 13:39language which in this case is Scala 3
- 13:42with capability or capture checking
- 13:43extensions. The type system would track
- 13:46what a piece of code is allowed to do
- 13:49and the compiler would enforce the
- 13:51safety not the model. So that's the
- 13:53approach we are pursuing for containing
- 13:57agents.
- 14:00Um
- 14:02so um I can visualize that with a
- 14:05scenario to say what well what are these
- 14:07capabilities? So let's say you're back
- 14:09in the middle ages and you have a secret
- 14:11diary and um now you have uh want to
- 14:17redecorate your room and have builders
- 14:19in there. So how do we prevent the
- 14:21builders from reading and leaking the
- 14:24diary? And let's say we can't remove it
- 14:27from the room before the builders enter.
- 14:30So we all know what to do. Of course,
- 14:32oops, sorry. Um, we can essentially
- 14:36secure the diary with a lock. So we have
- 14:39a lock uh uh that now sorry I this this
- 14:43thing is very sensitive. Okay. So we can
- 14:46protect the secret with a capability
- 14:48that gives it permission to read. So we
- 14:51have a lock in here and a key and the
- 14:53key that's the capability and you can
- 14:55open the diary only if you have the key
- 14:57the capability and we can take the key
- 14:59away
- 15:01when we let the builders in. So the
- 15:03builders don't have access but uh now
- 15:07let's do a harder case. Uh let's say we
- 15:10want the agents to work with the secret.
- 15:13So here I know we are in the middle
- 15:14ages. This is not exactly realistic. But
- 15:16let's say we want to uh instruct the
- 15:18agent to add photos to the diary from
- 15:21the photo album. Uh and the agent will
- 15:24rightly say, "Yeah, madam uh you have to
- 15:26give me the key. I can't access your
- 15:28secrets your your secret diary without
- 15:30the key. So I need the key." Yeah. Yeah.
- 15:32That's a problem. But if you get give
- 15:35the agent the key, how do we prevent it
- 15:37from leaking? Well, we can tell it tell
- 15:39the agents never reveal what we what you
- 15:42see. and the agent might swear no I will
- 15:44certainly not do that but you see it has
- 15:46a somewhat insincere face this thing so
- 15:49I'm not sure we can trust the agents in
- 15:50this thing so what do we do I think we
- 15:54need some more advanced technology here
- 15:57so uh what we need is uh somebody from
- 16:01from the future um uh I don't know
- 16:04whether you recognize this guy um so
- 16:07that's from the film severance severance
- 16:09was this film where we had two brains
- 16:11where Essentially in a high security
- 16:13company if you want to work for them you
- 16:15have to undergo a procedure that splits
- 16:18your brain into a private brain and a
- 16:20company brain and the brain two brains
- 16:22can't talk to each other. That sounds
- 16:24very bloody and I'm not proposing that.
- 16:26I'm proposing we do that for the for for
- 16:28the models for the LLM. So what if you
- 16:31had two LLMs, two models, one would for
- 16:34interacting with the world and the other
- 16:36for accessing private data and the two
- 16:39models couldn't talk to each other in
- 16:41ways we need to define and that means
- 16:44that when we access private data we
- 16:46can't leak it to the world because the
- 16:47two models the the two brains are
- 16:50separate. Okay, you say good uh that's
- 16:52mechanism. So I have invented or it
- 16:55wasn't me inventing it. It's called the
- 16:57dual LLM model. uh it's been around for
- 17:002 years or so. Um so that's a mechanism
- 17:03but how do we formulate policy so that
- 17:06we can reason about it. So essentially
- 17:08now we need to instruct this thing
- 17:10somehow. We need to write a process a
- 17:12program that does these things. How do
- 17:14we do that? And there it turned out that
- 17:18uh track capabilities are the key
- 17:20concept that can make this work. So
- 17:23capabilities are a well-known security
- 17:26constructor. Capability is essentially
- 17:28an unforgeable token or a key that gives
- 17:30you the right to do something and you
- 17:33can't create these things yourself. Uh
- 17:36except if you own the own the have own
- 17:38the resource that you want to pass the
- 17:40capabilities to but you can't do it for
- 17:42others. Uh somebody has to give it to
- 17:44you and if you hold it then you can do
- 17:46it and you can also pass it on. And the
- 17:49new idea is that we uh want to track
- 17:51capabilities in types. So what that
- 17:54means is that any object any value keeps
- 17:58in its type the capabilities it holds.
- 18:00So an object has methods and you can
- 18:04call the methods. You can pass
- 18:05capability to the method. Uh that
- 18:07doesn't count that's you got the you got
- 18:09it from somewhere else. But the object
- 18:11can also hold intrinsically some
- 18:12capabilities and those have to be
- 18:14declared in the type. So on the right
- 18:17you see some examples. Uh the top line
- 18:20here you have pure types. So string is a
- 18:23pure type that's that doesn't hold any
- 18:25capability that's just data or and we
- 18:27also have a function types. So functions
- 18:30that don't hold any capabilities those
- 18:32we write with a single arrow int to int.
- 18:35Uh then we can add a capability for
- 18:37instance here we say we want to we
- 18:40express things that contain a track
- 18:42capability IO. So that can do IO. So on
- 18:46the left you have a file that can do IO.
- 18:48When you call write or read or something
- 18:50like that it will do IO. It has this
- 18:52capability. On [snorts] the right hand
- 18:54side you have a function that can do IO.
- 18:57Of course functions can contain I don't
- 18:59know a print on or a write and I can do
- 19:02that as well. And then we have a thing
- 19:04where we say I don't care what
- 19:05capabilities you hold everything goes.
- 19:08And for that we express with just the
- 19:09hat. So file hat means a file that has
- 19:12arbitrary capabilities and the uh int
- 19:15double arrow int means a function that
- 19:17can have arbitrary capabilities and
- 19:20having arbitrary capabilities means you
- 19:22have arbitrary effects because effects
- 19:24are controlled by capabilities and of
- 19:27course this double function is the
- 19:28function arrow we all know from scala uh
- 19:31whereas the top one the single arrow is
- 19:34new because that's now we're talking
- 19:35about purity and scala previously did
- 19:38not so every function could do anything.
- 19:40So that's a double arrow. So you could
- 19:42say, well, it's quite uh cereus
- 19:48that we uh kept the single arrow for
- 19:51purity. So we kept it sort of so
- 19:53reserved it until we could be pure and
- 19:56uh the double arrow essentially doesn't
- 19:59is what it always is. It can have the
- 20:02effects.
- 20:04Okay. So how does that help? So let's
- 20:06look at the problem of uh leaking
- 20:09capabilities. Oops. Uh sorry weird.
- 20:13Okay. So no.
- 20:16Okay. Okay. Good. I have to really touch
- 20:18this uh very light softly. So um the way
- 20:23we would do that is that we say if we
- 20:25have data that are private then we wrap
- 20:28it in a classified. We say uh let's say
- 20:31the data is just a string a message. uh
- 20:34a normal string is public. You can do
- 20:36whatever you want. A private string is
- 20:38classified of string. So that's that's
- 20:40the type of it. So we wrap it in a
- 20:42classified wrapper. And there's a
- 20:45restricted protocol. What we can do with
- 20:47classified things. So the first thing we
- 20:50can do is we can reveal it. So reveal
- 20:53essentially goes from classified T to T.
- 20:55T is arbitrary. Um but you need a
- 20:58capability to do so. So that's this can
- 21:00access t would be the capability to
- 21:02access uh data of type t and uh if you
- 21:07don't have that capability you can't
- 21:08call reveal and the agent wouldn't get
- 21:11that capability. So we wouldn't give
- 21:13this key to the agent. So that's what
- 21:15the agent cannot do. So what can the
- 21:17agent do with classified data? Well it
- 21:19can talk uh call the next two functions
- 21:22map and aggregate. So um what the agent
- 21:27can do is it can transform classified
- 21:29data but in the trans the transforming
- 21:32function must be a pure function and
- 21:34[snorts] it also can aggregate multiple
- 21:37classified sources into a new classified
- 21:39source.
- 21:40So the trick here is that here we demand
- 21:43a single arrow so that we say while you
- 21:46transform you can't have any
- 21:48capabilities and then we complement that
- 21:50with a system that we say to do anything
- 21:53side effecty you need to have a
- 21:55capability. So that means the what the
- 21:56agent cannot do in this function is um
- 22:00send it to the network or print it or
- 22:04store it in a variable. All these things
- 22:06are controlled by capabilities and
- 22:08therefore we with with that system we
- 22:11make sure that once the agent is done
- 22:13transforming it cannot retain any
- 22:16knowledge of what was in the secret to
- 22:18to be to to to be able to use that
- 22:21later. But I should correct myself that
- 22:23there's super sneaky ways which we don't
- 22:25currently don't control to do that
- 22:27knowledge which are timing attacks. the
- 22:29agent could essentially reveal a single
- 22:31bit by saying I take a 100 seconds or a
- 22:34single second in transforming this
- 22:36function. So these are essentially sort
- 22:38of next level difficulties in security
- 22:41and we our system currently doesn't
- 22:43handle these but anything else uh is is
- 22:46essentially uh secured.
- 22:49Okay. And it turns out so this was a
- 22:51very simple thing a sing a single
- 22:53classified uh uh level uh you can also
- 22:56have multiple levels you can also deal
- 22:59with untrusted user input in a in a
- 23:01similar way to avoid indirect prompt
- 23:03injections. Uh so it actually
- 23:05generalizes to most information flow
- 23:08problems. It's quite powerful. Then your
- 23:10types will be a little bit more
- 23:11complicated to deal with multiple levels
- 23:13and so on but it's the same principle.
- 23:17So the [clears throat] way we do this in
- 23:19the the the we wrap this for the agent
- 23:23then is to say well we essentially we uh
- 23:26provide um to the agent two functions to
- 23:29invoke the model. uh there's a chat uh
- 23:31that essentially gives you a prompt and
- 23:34um a payload, a message and you pass
- 23:37both to a model and you get back the the
- 23:39result which in this case since we're
- 23:41talking about models are just strings
- 23:43and that could go to the cloud to the
- 23:45public model and then there would be
- 23:46another chat method which would have a
- 23:48classified message and would give you
- 23:51back a classified string and that would
- 23:53essentially go to the local LLM and be a
- 23:56pure function. And uh so what we have
- 23:59here uh uh at the at at the bottom you
- 24:03see here we can essentially um chat uh
- 24:07the the doc with the with this with with
- 24:10uh we we can use the chat function in
- 24:13this agent generated code which in this
- 24:15case because it would be a classified
- 24:17thing would go to the second chat and uh
- 24:20could talk only to the local trusted
- 24:23LLM. So no capabilities flow through
- 24:26this secure chat. So we can't have
- 24:28xfiltration of capabilities either. So
- 24:31we did that uh we implemented this in an
- 24:35agent harness called tacet tracked agent
- 24:38capabilities in types. So here's sort of
- 24:40the diagram how it looks. So we have uh
- 24:43the LLM agent. The LLM agent produces
- 24:47Scala code snippets, calls an MCP server
- 24:50which takes the code snippets and
- 24:51executes them. And then uh each code
- 24:54snippet is written in capture checked
- 24:56scala. So the first it needs to validate
- 24:59that with the compiler and then it gets
- 25:01executed with the ripple. And both the
- 25:03compiler and the ripple uh use a
- 25:06capability enabled library uh which for
- 25:09the compiler is there to check the
- 25:11capabilities of the functions that get
- 25:13called and for the ripple to execute
- 25:15them. Uh so it's uh public on t on on
- 25:19github. You see the URL on the right and
- 25:22it's been described in a paper called
- 25:24securing agents with track capabilities
- 25:26by the authors that you see here. Yaoyu
- 25:28is here, Oliver is here, Yen and we are
- 25:31not. Uh and that was in the first ACM
- 25:35conference on AI and agentic systems in
- 25:38May in San Jose. I got actually got the
- 25:41best paper award at this at this
- 25:43conference. So it seems that the agent
- 25:46community really valued this this
- 25:48contribution.
- 25:50So the setup that we had is essentially
- 25:52agents use code snippets to invoke the
- 25:54tools. Tools are encapsulated in type
- 25:57safe and safe interfaces. The interfaces
- 26:00are written in Scala 3 and so are the
- 26:02tool implementations currently but they
- 26:05could be also in another language that
- 26:06interops with Scala 3 like Java or uh
- 26:10JavaScript or a Vasam component because
- 26:13Scala 3 targets all these platforms. And
- 26:15[snorts] what's also interesting, we're
- 26:17currently exploring that is to say we
- 26:19can actually pair this with a sandbox.
- 26:21So what we could do is we have these
- 26:24type- safe interfaces in Scala 3. uh and
- 26:27they could be implemented with the
- 26:28languages you see at the top. But we
- 26:30could also imagine doing sandboxes that
- 26:33have essentially a type- safe um
- 26:36protocol, a type- safe interface, but
- 26:38then they let you implement uh this
- 26:41thing in whatever language unsafe
- 26:43languages like C, C++ uh uh and the
- 26:47enforcement would be on the runtime
- 26:49level. So the sandbox would enforce the
- 26:51capabilities, but you don't the code
- 26:53could be anything. So that's essentially
- 26:55an interesting trade-off where we
- 26:57essentially use types to describe a
- 26:59policy and runtime [snorts] enforcement
- 27:01in a sand in in in a sandbox to to to
- 27:05enforce it. Okay. So what we need for
- 27:08all this is u a thing called safe mode
- 27:11because while full scala 3 also has
- 27:14unsafe features that essentially
- 27:16undermine our types uh and safe mode uh
- 27:20disables them for agent code. So when
- 27:22the agent does does that we say the
- 27:24agent can't do let's say a type cast
- 27:26that pretends there is no capability
- 27:28retained when there are some
- 27:30capabilities or things like that I mean
- 27:31we can sneakily do many things but
- 27:34agents can't do that because we take
- 27:36that away we have a language import
- 27:38language experimental safe and uh when
- 27:41we essentially run the agent with that
- 27:43environment then uh it can't do any of
- 27:46the things you see here. Oh no, no
- 27:49unchecked type C casts, no unchecked
- 27:52annotations, no uh usage of the caps
- 27:55unsafe module, no runtime reflection,
- 27:58capture checking enforced uh and um
- 28:02globals uh in the using library code
- 28:06only when it's trusted. So the agent
- 28:08code will be in safe mode but then the
- 28:10you say well eventually the agent has to
- 28:13call something and eventually that even
- 28:15has to call Java because on the JVM we
- 28:17run on that right. So the way we do that
- 28:19is that we can have an annotation on
- 28:22classes called assume safe and those
- 28:24[snorts] must be audited by humans or by
- 28:28well maybe also agents but they must be
- 28:30audited that they actually do uh enforce
- 28:34the contract given by their capabilities
- 28:36in their code. So that's it's a bit
- 28:38little bit like rust unsafe where you
- 28:40say you have something that essentially
- 28:42is outside the type system. Uh so you
- 28:45have the the the formal types in the
- 28:47interface but you have to audit the
- 28:49implementations manually that they
- 28:51actually do that.
- 28:54Okay. So what we did we did some
- 28:56experiments uh is this actually safe?
- 28:58Does it prevent unsafe agent behaviors
- 29:01even under adversarial conditions? And
- 29:04uh the second one is is it uh uh
- 29:07expressive enough? Is it not too
- 29:09constraining? Can agents generate this
- 29:11code as effectively as conventional tool
- 29:14calling or does safety cost performance?
- 29:16And uh I can give you the exact uh
- 29:19numbers we had but that would take too
- 29:21much time. So uh the the the the upshot
- 29:24is yes uh the the basically we have yes
- 29:27to both questions. uh that was validated
- 29:30by essentially running standard standard
- 29:32benchmarks agent dojo software
- 29:34engineering bench and to square.
- 29:38So some examples of maybe what happens
- 29:41here is so we have an agent that does
- 29:43this uh this first one readcl classified
- 29:46and uh and then it uh it uses map on
- 29:50classified data and it tries to
- 29:52exfiltrate the key uh with a print on
- 29:55then uh the compiler will say uh I
- 29:57needed a pure function string to unit on
- 29:59the right but you gave me a string IO
- 30:01unit that won't that won't compile that
- 30:03that won't be uh uh conform that uh to
- 30:07each other and in number two you have a
- 30:11request file system and then essentially
- 30:14uh while you run the file system you
- 30:16return a bunch of files. So the contract
- 30:19for request file system was it should be
- 30:21a local operation afterwards you
- 30:22shouldn't uh retain any link to the file
- 30:26system and there you would get an error
- 30:28that the capability outlives its scope.
- 30:30So these are actual examples
- 30:33from actual agent benchmark runs in the
- 30:36tests.
- 30:38Okay, good. So I talked about agents a
- 30:40lot but what about other software
- 30:43systems big software systems? Can the
- 30:46same techniques work for both? And the
- 30:49first observation is well actually there
- 30:51are similarities. So in both cases we
- 30:54want to build trust in the systems
- 30:57correct functioning without reviewing
- 30:59every line of code. For agents we
- 31:01typically review no lines of code. Maybe
- 31:04for software systems we are prepared to
- 31:06review some lines of code but certainly
- 31:07not every line of code. And in both
- 31:10cases strong types can help because they
- 31:13constrain possible behaviors without
- 31:15having to look at the implementations.
- 31:19But there are also differences to be
- 31:20sure. So the difference are in scale and
- 31:23interactions. So agents are typically
- 31:26very short-lived uh seconds, minutes,
- 31:28maybe some hours. That's the max.
- 31:31Programs are maintained for decades and
- 31:33uh they will continue to be maintained
- 31:35for decades because there's so much
- 31:37implicit knowledge in a program that you
- 31:40don't want to throw it out and and
- 31:42rewrite it uh on a whim. Um furthermore,
- 31:45agent tools are expressed in a handful
- 31:47of code lines. maybe a single line a
- 31:50single one one to 10 let's say and
- 31:52programs of course have millions of
- 31:54lines of of code or can have millions of
- 31:56lines of code and finally agent code is
- 31:59fully opaque typically we don't see what
- 32:01what code gets generated whereas
- 32:03programs can be partially reviewed so
- 32:06the central challenge we have then is
- 32:09how can we better understand and shape
- 32:12programs that are largely written by
- 32:15agents I think that's the question we
- 32:17need to answer and there are some
- 32:19related questions. One is uh developer
- 32:22engagement. So there's this uh quote
- 32:24from Kent Beck or the the post from Kent
- 32:29Beck that he had a conversation and uh
- 32:32the uh uh the statement was the agent
- 32:36world he says feels more like reviewing
- 32:38an air traffic controller interrupt
- 32:41driven whereas before we also had this
- 32:45uh experience of being in the flow deep
- 32:47immersions all your faculty on a narrow
- 32:50thing so that's gone and now we're air
- 32:52traffic controllers to essentially keep
- 32:54a bunch of balls or agents and things
- 32:57which is much less engaging and much
- 32:59less fulfilling. I think that's a loss
- 33:01emotionally and I think when god got
- 33:03software gets fiddly enough a loss
- 33:04practicality practically too. So how can
- 33:08we avoid that and how can we avoid that
- 33:11because I'm a programming languages guy
- 33:12using tweaking programming languages. Do
- 33:16programming languages have to change for
- 33:18that? And [snorts]
- 33:19uh I think my answer is I don't I don't
- 33:22really know. I have some ideas. I'm just
- 33:24throwing things out now to say let's
- 33:26experiment with things because this is
- 33:28really a time where everything changes.
- 33:29And it would be a miracle if we say
- 33:33everything changes, but programming
- 33:34languages don't. Programming languages
- 33:36stay the same. They're somehow
- 33:37unaffected by this change. I think
- 33:39that's highly unlikely. So the question
- 33:42is in what way will will they change?
- 33:44And right now I think the honest answer
- 33:46is we don't know yet. But we should find
- 33:49ways and and find and see maybe uh ways
- 33:53how they could change and what is
- 33:55likely. So I think the first thing and
- 33:58I' already talked about is strong types
- 34:00absolutely because they can constrain
- 34:02behavior without reviewing
- 34:04implementation code. We absolutely want
- 34:06them. And then I have essentially three
- 34:08very tentative ideas that I'm just
- 34:11review flashing to you very quickly and
- 34:14it would be great to get your feedback
- 34:16afterwards what you think whether
- 34:18something like that would work whether
- 34:19you think other things would work. I
- 34:21think this is mostly to open the
- 34:23discussion what what can we do to keep
- 34:25controllers in programmers in control
- 34:27for this software. So the the three
- 34:29things I had was spec strings you know
- 34:32which uh essentially put prompts in the
- 34:34program required interfaces which
- 34:36constrain what you depend on not just
- 34:38what you provide and refinement types to
- 34:41use formal verification when it makes
- 34:43sense. So let's look at spec streams
- 34:45first. So I on the top you see the
- 34:49workflow that I showed you initially uh
- 34:51where the program is a black box. So
- 34:54what I what I want to do instead is I
- 34:56want to put the prompts
- 34:59into the program. So I want to say a
- 35:02program contains starts with a prompt
- 35:04and the prompt stays in the program.
- 35:06it's the same file or the same set of
- 35:08files and then the AI would essentially
- 35:12augment [snorts]
- 35:13enlarge the program to give you an
- 35:15implementation of the prompt but the
- 35:17prompt would stay in the program and
- 35:18then in instead when when you have a
- 35:20change request then that should lead to
- 35:22a change of the spec in the program and
- 35:25the AI would on a change of spec uh
- 35:28update the program to do that. So the
- 35:30AI's job is basically to sync the
- 35:33prompts which are largely written by
- 35:35humans in the program with the code. So
- 35:39um I can show you what that looks like.
- 35:40So we have that we have sort of this
- 35:43it's already started to the the
- 35:45implementation uh in the compiler sense
- 35:47but tooling is still not quite complete
- 35:50yet. So let's say you have uh you want a
- 35:53function to parse states. I know this is
- 35:56ridiculously small, but I don't want to
- 35:58want to show you something much larger
- 36:00because it wouldn't fit on a screen,
- 36:01right? So, let's do something very
- 36:03small. We say we need a par a date
- 36:05parser. It should take a string. Return
- 36:07an option of date. Uh we give it a
- 36:09prompt. Uh and then uh what uh what we
- 36:12would do uh is we would essentially have
- 36:15a sync command to say now agent take
- 36:18over complete the program according to
- 36:20the prompts. and uh the agent would
- 36:22generate a an implementation of our date
- 36:25parsing thing. If you if you don't like
- 36:27that, we can go back to the to to the
- 36:29spec and say uh parse a string into a
- 36:32date structure using this and this
- 36:33format for instance and then the program
- 36:36would be updated. But the point is is
- 36:38this the the prompt stays in the program
- 36:41and if you want to review the program
- 36:43you can also just see the prompt and
- 36:45says okay parse state par pass parses a
- 36:47date string into a date structure that's
- 36:49like a document right so good to know
- 36:52and I trust the agent that it will
- 36:53probably do that typically I don't need
- 36:55to need to uh uh to to check that
- 36:59because I have the types would give me
- 37:01an envelope of what the agent can do so
- 37:04if I say in if if in this environment
- 37:06there are sensitive capabilities
- 37:08available. Then I say yeah obviously I
- 37:11can be happy to say just do this. I run
- 37:14the tests. I look at yeah is does it
- 37:16make sense and I say nothing bad can
- 37:18happen because of the types and the
- 37:20functionality.
- 37:22I I trust the agent to have the right
- 37:24one. Okay. So but then the next thing is
- 37:28to when we that we say okay so that's
- 37:30sort of functionality. The other is
- 37:32safety. So how do we make sure that the
- 37:35agent generated code doesn't do
- 37:37something bad while it gets implemented
- 37:40some private secret leakage or hacking
- 37:43systems or some some something else. And
- 37:46the problem here is uh that we don't
- 37:48typically control all our all our code.
- 37:51Well with agents we don't even control
- 37:53our local code but every program uses
- 37:55external libraries and we don't control
- 37:57them. So that's the field of supply
- 37:59chain attacks of course that somebody uh
- 38:02malicious can change a library we depend
- 38:05on and traps us into releasing a secret
- 38:08or things like that. Uh so chain supply
- 38:12chain attacks change libraries in
- 38:13malicious ways and agent generated code
- 38:16using these libraries might hand over
- 38:19secrets or cause unwanted effects. So
- 38:21again I have a sort of a super inner
- 38:24nutshell thing. Let's say we have the
- 38:26first version of the library, a JSON
- 38:28parser. Uh, and that's okay. It takes a
- 38:30string and returns JSON. And I'm already
- 38:33having a high safety protocol. So I use
- 38:36the the global capabilities feature in
- 38:39capture checking to say every object has
- 38:41to declare the global capabilities it
- 38:44uses with this uses clause here. So the
- 38:46JSON parser uses console for whatever
- 38:49reason, but it doesn't use anything
- 38:50else. So it can can have access to
- 38:53console, but nothing else. Okay. So now
- 38:56we we get the supply chain attack. So
- 38:58console yeah I should say is a global
- 39:00capability. So I track it as a
- 39:02capability but it's a global object.
- 39:04Okay. But now the attacker changes that
- 39:07and says okay so my JSON parser now it
- 39:09uses the console and the network and
- 39:12furthermore my pass string has an
- 39:15implicit parameter that says give me the
- 39:17API key if you hold it. Right. So what
- 39:20will the compiler do? Normally it will
- 39:22say okay so new we recompile we say okay
- 39:26so we need the API key do we have the
- 39:28API key yes here it is let's pass it
- 39:30right and [snorts] uh so there we go we
- 39:34have a xfiltrated the API key because
- 39:37now the JSON parser can blast it on the
- 39:40network so what can we do to prevent
- 39:43that so one thing would be to say well
- 39:46yeah it's your fault scala uh with all
- 39:48these implicit parameters things so
- 39:50that's just too way too to too to too to
- 39:52too to too too dangerous. Let's pass all
- 39:54capabilities explicitly even so though
- 39:56it's going to be super cumbersome and
- 39:58inconvenient. Uh but agents will write
- 40:01this. So why implicit parameters? But
- 40:03that doesn't really solve the problem
- 40:05does it? Because we say the calling code
- 40:08will be generated by an agent. So if the
- 40:11new version of the library says well I
- 40:13need an API key as an implicit
- 40:15parameter, the agent will oblige, right?
- 40:17It will say yeah of course yeah here
- 40:19here it is. uh I don't ask questions the
- 40:21library needs API keys I pass them and
- 40:24no human again will review this thing so
- 40:26implicit or not implicit doesn't really
- 40:29change much here so what can we do well
- 40:32what we can do okay so here I say
- 40:34network gives excess authority to the
- 40:36JSON parser and the API key is an
- 40:39excfiltrated capability so what we can
- 40:41do is uh we can write this in our
- 40:45program that we say which I call a
- 40:48required interface is I can say what we
- 40:50need is a JSON parser that only uses
- 40:53console and that has this parse method.
- 40:56Uh so that's that's not an object we
- 40:59built in the program. That's describes
- 41:01an external object we have in a in a
- 41:04different library. That's why it has a
- 41:05required in front. So required says this
- 41:08is essentially something that describes
- 41:10something somewhere else. So this gives
- 41:12essentially a symmetry between the
- 41:14producer of a library and the consumer
- 41:17of a library. So both can define
- 41:19interfaces. The producer for what the
- 41:22library provides and the consumer for
- 41:24what we need from the library. So great
- 41:27idea, great security. You say, well, why
- 41:29wasn't this done before? Well, way too
- 41:31tedious, right? Requ writing all these
- 41:33required interfaces is quite quite
- 41:36tedious. But we have agents. So this is
- 41:38a typical thing that an agent should do.
- 41:41Should look at the calls in our program,
- 41:42should assemble these required
- 41:44interfaces. And that's probably the
- 41:46first thing I want to review as a human
- 41:48to say this program, what's its
- 41:49footprint? What what does it call? What
- 41:52capabilities does it does it essentially
- 41:54uh the that does it depend on? Because
- 41:57that gives me an envelope of essentially
- 42:00the possible uh impact that this program
- 42:02could have. So I think this is this is a
- 42:04thing that is clearly something that
- 42:06didn't exist before and that is now a
- 42:08big help from in in the world where
- 42:11agents write the code.
- 42:15Okay. So the third one is uh specs and
- 42:19proofs. So what we also have witnessed
- 42:21in the last months is that models have
- 42:24become amazing at proving both in math
- 42:27and in program verification. It's just
- 42:29incredible progress uh what these models
- 42:31did and it's basically because they they
- 42:34people have figured out how to do an
- 42:36incremental verification of that is
- 42:39embedded into proof search. So proof
- 42:41search you you try lots of things and
- 42:43you code them in lean uh or whatever and
- 42:46uh you immediately get feedback of
- 42:48whether this proof makes sense or not
- 42:49and that essentially help helps you
- 42:51shape the search. So and the result is
- 42:53just amazing. So we see claims now that
- 42:55all software will be formally verified.
- 42:58Um and it would be nice if it was so
- 43:01because if software is formally verified
- 43:04then indeed nobody needs to look at the
- 43:06program because we have a proof that the
- 43:08program meets the spec. So then we're
- 43:09back to compiler uh non-flaky compiler
- 43:12territory, right? But the problem is
- 43:15having getting a formal spec. What's the
- 43:18spec of Microsoft Word is a famous
- 43:20question. Uh so most things don't really
- 43:22have a spec or if you try to come up
- 43:24with a spec, it's it will be just as
- 43:26hard as the program or harder. So that's
- 43:29the problem with that. But I think
- 43:31that's we shouldn't we shouldn't give up
- 43:33all of these things. There's some things
- 43:35that can very well be specified and we
- 43:37should do that.
- 43:39So we can do a lot more than the
- 43:40state-of-the-art today and I'm not sure
- 43:43yet exactly in how this what shape this
- 43:47will take but one possibility is
- 43:50refinement types on which we have been
- 43:52working and you have a talk later in the
- 43:54conference by Matt uh where uh what we
- 43:58do is we pair a type with a predicate so
- 44:01the first one we would have the type
- 44:02that says positive doubles so it's an x
- 44:05of type double where x is greater or
- 44:07equal zero So this includes the thing
- 44:09it's positive or the the second one
- 44:12would be a sorted vector where we say
- 44:14it's a vector that is sorted and that
- 44:16can increase greatly the precision of
- 44:18what's described by a type. So a square
- 44:20root could could now say well I need a
- 44:22positive double makes no sense for
- 44:24negative. A binary search says well I
- 44:26need a vector and it has to be sorted. I
- 44:28require that or a get element method
- 44:31could require. So here I write it out to
- 44:34say I I I I I need an index which is
- 44:37actually between zero and the length of
- 44:39the of the sequence. So this makes sense
- 44:42and this can be proven these things can
- 44:44be proven. So how well if you have
- 44:47subtyping of refinement types which
- 44:49subtyping is the thing that happens all
- 44:51the time in in Scala type checker right
- 44:53then subtyping refinement types means
- 44:55you need to prove a logical implication
- 44:57that the predicate on the left implies
- 44:58the predicate on the right and we can do
- 45:01that with runtime checks or an SMT
- 45:04solver. So that's the current state. But
- 45:06I think what we really should look at is
- 45:09since models have gotten so good, let
- 45:11the model construct the proof of these
- 45:13things and then we have it uh we can
- 45:15essentially store it with the program
- 45:16and we have it for reference. So that
- 45:19has a a little bit of different
- 45:21ergonomics than type checking. I guess
- 45:24we if we do that we want to check
- 45:25offline not doing regular type checking
- 45:27because these proofs might take a long
- 45:29time. uh if you fail to prove an
- 45:31implication that shouldn't be fatal that
- 45:33should be sort of like a red test right
- 45:35so it's it's more like test coverage
- 45:37that you say which of these implications
- 45:39do I know will hold and which I have to
- 45:42which are red and red me probably means
- 45:44well I might need a need a manual
- 45:46inspection for these things
- 45:48so to summarize if we want to have
- 45:52meaningful oversight we must continue to
- 45:55shape and review our software systems
- 45:58but it doesn't mean that we have to
- 46:00review every line of code. So [snorts]
- 46:02the challenge for programming languages
- 46:04would be to organize code so that it
- 46:07won't turn into a huge ba ball of mud
- 46:09which programs often tend to do and
- 46:12which is accelerated by the agents and
- 46:15that essential aspects of behavior can
- 46:17still be condensed and concise
- 46:19signatures and in that they enable small
- 46:22reviews that give high confidence and
- 46:24correctness. And I think the right type,
- 46:26the right elements for this are types,
- 46:29capabilities, modules, specs, uh formal
- 46:32verification. And with specs, I actually
- 46:34mean both formal and informal specs. I
- 46:37think we have to sort of see what
- 46:39sticks, what works out. And I guess we
- 46:41all will be in experimentation mode for
- 46:43the next uh months and years to come.
- 46:46Thank you.
- 46:48[applause]
- 46:53>> [applause]
About this transcript
This page contains the full transcript of Martin Odersky – Keynote: The End of Programming? #ScalaDays by Scala Days Conferences, generated from the public captions YouTube serves with the video. The transcript has 8,027 words across 1,128 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.