00:00:00.960 - 00:00:23.300

Welcome to the Remote Warfare Programme podcast. In this episode we have a recording of a panel from the Conceptualising Remote Warfare Conference which the Remote Warfare Programme held in collaboration with the University of Kent on the 28th of February and 1st of March. The conference pulled together a wide range of experts from the military, government,

00:00:23.640 - 00:00:42.700

academia and civil society to discuss the past, present and future of remote warfare as well as the implications of this approach. We couldn't have organised a conference without the support of the Conflict Analysis User Centre at the University of Kent and the British International Studies Association. If you like what you hear on this podcast you can hear more

00:00:42.700 - 00:01:06.940

panels in our upcoming episodes and you can read in more depth about the topics in our upcoming book released in early 2020. For now enjoy the podcast. Great, okay. So thank you for coming. This is a heavy lift. Friday afternoon, you know, last session and having to move. So I really appreciate you coming here. To me this

00:01:06.940 - 00:01:35.420

is really exciting because the last two days which have been really awesome, it's well done. We're kind of pivoting now from the past and present to the future and what does the future of remote warfare look like? What does the future of war look like? And I argue that we are at a really pivotal time in human history and we have not thought through a lot of the ramifications

00:01:35.420 - 00:02:11.960

of what is happening today. So since we haven't thought through that we have some great people to be thinking about that. So I'm going to take the liberty of going first and I hope we'll be kind of a broad scene setter and also hope to show the value of kind of a diverse set of perspectives. So here we go. Okay, great. And so, you know, I've really enjoyed this conference

00:02:11.960 - 00:02:35.880

a lot and sometimes I've just had to bite my tongue because some of my favorite topics have come up, you know, civilian casualties, civilian protection, drones, Yemen, you know, working with partners in general. And so, you know, kind of thinking about all the stuff that I've been doing in the last 20 years, if you know what I've done, you might go, what in the world is

00:02:35.880 - 00:03:02.660

this? So I just wanted to say just for a minute kind of how I got into this area. So I've done a lot of technical work for the U.S. military and working with other militaries on, you know, really data-based approaches to civilian protection and just being more effective. So that was really, you know, very operational work, working directly with militaries, often out in the field. Then I

00:03:02.660 - 00:03:25.500

was asked to join the State Department because basically they said, you know, we have a policy gap with respect to civilian protection, so can you come and kind of work in the policy area? And that kind of led to me going to Yemen. No, not Yemen, actually. I went to Saudi Arabia, working in that context. But as I was doing that work, there was a lot of discussion about

00:03:25.500 - 00:03:49.300

artificial intelligence and lethal autonomy. And so someone said, you know, it seems like with these new technologies in war, we should be concerned about civilians. So maybe you can just kind of join in those discussions. So I started going, and there was an interagency policy committee on lethal autonomy and AI, and they added me to the U.S. delegation to the U.N. for

00:03:49.300 - 00:04:16.060

lethal autonomous weapon systems and so forth. And I was really struck because there just wasn't like an operationally informed perspective. You had, I mean, there were good perspectives. You had a civil society perspective. You had a technologist perspective, a futurist perspective. But it's like it wasn't really a connecting of the dots from where we are here to, you know,

00:04:16.080 - 00:04:43.860

in 2040 we'll have, you know, drones with drones with little drones on top. And, you know, and what are the risks of this, right? So after the State Department and the new administration, where we don't need a State Department, so I went back to CNA, and basically CNA was like, this is a really important aspect of future war. So we want you to kind of lead a kind of

00:04:43.860 - 00:05:07.580

an interdisciplinary effort on this. So it's, you know, Center for Autonomy and Artificial Intelligence. So it's not just me. It's, so I sort of bring sort of the operational perspective and also, honestly, the civilian protection perspective, because that is a really important piece of kind of thinking about this technology more. But I'm also surrounded by really smart

00:05:07.580 - 00:05:29.580

people. I have one guy who's written a graduate level textbook on AI and keeps me honest in all the very little details about the technology. And I think, you know, we're better together, right? So that's kind of the perspective that we take in the organization. I do have a link there if you're curious, because we've been writing a lot of reports to try to delve into some of

00:05:29.580 - 00:05:54.620

these issues. Okay, so that's sort of background in a nutshell. So just a few slides. So first of all, you know, this is, in some respects, this is nothing new. And some people will say there's nothing new here. Militaries always look for a military edge from technology. On the other hand, AI is really fundamentally different. And, you know, some people call this the third

00:05:54.620 - 00:06:19.320

revolution, military warfare. But AI is going to change everything. And that includes the waging of war. And we also see this other, another thing changing, which is that research and development has always sort of been a military and government led practice. But now, and we see more and more, the commercial sector having a bigger role. And so you see a lot of changes kind of all at once.

00:06:19.320 - 00:06:46.320

And I want to take a minute just to talk about AI. And really just make one point, I could, I could spend an hour here, and that would probably put you all to sleep. But But one thing I want to make clear is that, first of all, there's a long history of AI. So really, 50s, 60s was really where it came in the play. I have this really funny quote here. So in 55, they said,

00:06:46.320 - 00:07:08.100

we're going to have this two month, 10 man study, we're basically just going to solve what AI can do. Okay, well, we're still working on that. And you see this a lot is, there's kind of a mismatch between where the technology is and where we think it is and what we think we can achieve. The biggest point I think I want to make here is that so you have machine learning,

00:07:08.700 - 00:07:35.760

and the slide here, and then it kind of continues, right? So it's lots and lots of stuff happening. We're in this machine learning era. What this what this means for us is that we do not have sentient artificial beings. What we have is basically, we have machines that do optimization over multi dimensional surfaces, to find, you know, optimal problems for very

00:07:35.760 - 00:08:02.160

narrow problems. So, so you know, when people think about, okay, you know, you're like, like a killer robot, a machine, we are not we are not nowhere close to having a capability that is sentient, that is making its own fundamental decisions. We have things like Google Maps, we have things like anti fraud software, we can do really specific things

00:08:02.160 - 00:08:26.600

and use big data and do really powerful things. But it's also fragile, it's very narrow. So, and, and this is the kind of technology that militaries are looking to use. So just a little bit. So, so the kind of the current situation in one slide. So what we see is we have government, and then we have civil society is a little oversimplific, but so forgive me for

00:08:26.600 - 00:08:54.360

that. But militaries, they are keen, they're really keen to apply AI in military solutions. And they, you know, people talk about AI race, and I'd argue it's somewhat self inflicted, because each country is listening to each other's rhetoric, and it's making them go faster. But I do think that AI is definitely a high priority for a lot of states. But they don't understand the

00:08:54.360 - 00:09:17.300

details, they don't understand the limitations, they don't understand the requirements of the technology. So they're not nearly as far along as you may think that they are. And US in particular, and I've been trying to nudge them in this area. They've identified some virtuous uses of AI, I'm going to talk about that a little bit, but nothing done today. Okay, civil society. So for

00:09:17.300 - 00:09:46.140

lethal autonomy, there's there's a big push for preemptive ban on lethal autonomy. The problem is that it's really based on this general AI idea, which doesn't exist. It's also based on kind of a trigger pull decision, narrow view on the use of force, that I think is missing a lot. So, so, so neither is really considering all the actual risks, or the opportunities for

00:09:46.140 - 00:10:10.220

civilian protection. So I've been trying to think a lot about how can we better do this? And how can we better account for the real risks? Okay, because, because I'm not saying that they're not real, not risks, there are significant risks. And by the way, Jennifer, in her presentation yesterday about metadata, is a great example of some real and scary risks of this technology

00:10:10.220 - 00:10:29.640

that is not being discussed. So next month, at the UN talking about lethal autonomy, they're not going to talk about that. But we should be talking about it because it's scary. And, and, and there are, so I've tried to identify a number of different areas of risk. And, again, we're really not talking about these things. And then I have this because

00:10:30.120 - 00:10:56.220

so much of the discussion is part of this is in the UN is talking about, well, who's going to pull the trigger. And my point is, that's just the tip of the iceberg, you have to think about how AI could be used in all of these different pieces. Okay. And it could be used in good ways. But it also could be used in very risky ways. And so we need to broaden this conversation much more than

00:10:56.220 - 00:11:24.300

what we have now. So this is just really a start to help us to think about having this broader discussion. The other thing I want to make clear is AI can be used for good. So one of the things that I did last year, I worked with the US military on a report on civilian casualties in Mosul and Raqqa. And it was ugly. And what happened in Mosul and Raqqa was not good.

00:11:25.320 - 00:11:52.860

So, so because I'm supposed to be working on AI, you know, the next thing I did was, okay, now, so after that analysis, I understand exactly how civilian casualties happened in Mosul and Raqqa. So the next question is, can AI help? You know, there's all this interest in the US to apply AI. So can we kind of use some of those resources and energy to have have a positive effect? So,

00:11:53.260 - 00:12:17.360

and so basically, these are based on the actual problems we saw in Mosul and Raqqa. And yes, you could actually design basically machine learning techniques to do each of these functions, which would help problems that we have now operationally. So, and this is just, you know, thinking about this for a little bit. So again, there's a lot more that can be done in this area.

00:12:17.760 - 00:12:48.720

So last slide. So, so again, research and development is out there, is very important. Civil society is increasing in influence. Militaries are developing strategies. They're thinking right now. And in fact, the US DUD AI strategy just came out two weeks ago. And it has in there it has, we will use AI to help protect civilians. Okay. They don't know how

00:12:48.720 - 00:13:14.980

to do that. And there is not an internal real constituent for that purpose. So it would be, it would be helpful for them to hear from external audiences, that these are good things to put resources into. So, so this last quote is just basically, we are, we're at a very interesting time, because there's lots of change, there's lots of resources being devoted.

00:13:17.260 - 00:13:51.160

And we have to be thinking about risks. And I have this here, because I think all of these need to be working together in concert, which right now, we're not there. So I've, I've gone over myself, so I'm going to stop. So you want to go? Okay. Thank you very much, Larry. I'm going towards that direction. And I'm going to talk about

00:13:51.960 - 00:14:21.580

how AI applications within the context of NATO peace operations. Thank you very much for having me today. In fact, AI, artificial intelligence within NATO, I think it's worth mentioning that AI is not that new within NATO, or the world of defense. The military has been testing applications for around 60 years. And NATO, for example, already used it in its

00:14:21.580 - 00:14:53.360

activities, such as in the development of, for example, interoperable semi-automated forces, or computer generated forces, used for simulations for exercise scenario. And these would require least human resources. But the technology is used so as to be able to operate in more complex environment. And one example of that is NATO's air power mission

00:14:53.360 - 00:15:21.200

training through distributed simulation. And that technology involves fair use and multiple air crew simulators, for example, as well as training centers and multiple command and control. So now, why am I working on narratives? I think it's a very important aspect of the policymaking world. I have been studying these as they are powerful tools in shaping

00:15:21.840 - 00:15:52.380

social life at all levels. And they can shape the interactions of the actors involved in the process, the decisions they are making, even the courses of their actions. And many narratives, in fact, define agenda setting. For example, whether AI is important to NATO. The narratives they are shaping would really define the agenda setting. And also, the actors shaping these

00:15:52.380 - 00:16:23.100

narratives, the places where they are produced. For example, in the case of NATO, these narratives are delivered, for example, at Harvard, a big security conference, such as Globesex conference, or the Munich conference. And all of these play important roles in the process. So, on the whole, three main factors, I think, have influenced the shaping of NATO AI narratives.

00:16:23.100 - 00:16:53.280

And these narratives are quite new, but they emerged out of three main contexts, I think. Threats, rivalry, slash competition, and the local AI landscape development. First, the alliance, its partners, and the countries it considers to be strategic in its operations have facing more and more cyber threats. And such a situation of uncertainty and instability

00:16:53.280 - 00:17:21.580

called out for significant and firm stand, both at the political and the operational level. For example, the case of Estonia, as you know, regarding cyber threats, attacks. And the case of Ukraine, which is constantly facing hybrid warfare. And second, these last two years, I think, have seen important development in how AI has been adopted by nation states.

00:17:22.000 - 00:17:50.760

And you would hear very often about NATO narratives singling out China and Russia all the time. And one thus can hear about a NATO that could stand up to the bullies of the international arena and say, no, we must do something. And finally, at the national level, when NATO member states and partners started to show interest in AI like the US,

00:17:51.960 - 00:18:22.120

some developed strategies, AI national strategies, and while others decided that they would integrate AI in their respective national defenses, for example, France. So those situations also played key roles. And obviously, those local situations are paired with the ongoing and rather fast development of AI in the industry world and the discussions

00:18:22.120 - 00:19:03.180

as well that have been triggered by policymakers, as I mentioned earlier, scholars and the media on it. And all of these are more and more requiring NATO to, from their speech discourse, adapt, innovate, and most importantly, maintain competitive advantages. So where does it... And in terms of the actors involved, who are the main shapers of these narratives within

00:19:03.180 - 00:19:31.200

NATO? First of all, there is NATO ACT, which is based in Norfolk. And it is what I consider to be the main backbone of AI applications in the alliance. It is, in fact, taking charge of leading NATO's transformation in terms of warfare development of military structure, forces, and capabilities and doctrine. There is also the NATO Science and Technology Organization,

00:19:31.620 - 00:20:00.460

STO, which is a subsidiary body conducting deep research, they organize events, and they implement educational activities as well. And there is also NATO Parliamentary Assembly Science and Technology Committee. And it is, in fact, separate from NATO, but among its aim is really to provide a greater transparency of NATO policies and foster a better understanding of the alliance's

00:20:00.460 - 00:20:28.160

objectives and missions among legislators and the citizens of the alliance. And finally, at the political level, we have individual actors such as the SG, the Secretary General, the Deputy SG, and I would also include former NATO key leaders in that category, for example, the former SG. But before discussing about these narratives, let me give you a brief

00:20:28.160 - 00:20:56.460

overview of the state of AI within the alliance. Just a few examples from my papers. For NATO ACT, a great majority of the work that is currently doing has to do a lot with strategic foresight testing, as well as setting up proof of concepts, especially for the latter. It's very important because they say that it's very difficult to get the member states on board with the idea of

00:20:56.460 - 00:21:32.940

integrating AI in the operations. And for example, during the most current exercise, the test is around 20 AI-related technologies. And one of the flagship initiatives of NATO is the so-called autonomy project in the field of cyber defence, where interoperability is also an important issue that has been raised. Now, I think it's very important, the work that STO is

00:21:32.940 - 00:22:05.000

doing, the Science Technology Organization, there have been around 16 initiatives they've been implementing since 2014, of which eight are active, six are under the planning phase, and two are awaiting the publication of the reports. And there is a strong focus on activities involving planning, management, procurement, also activities related to cost. For example,

00:22:05.120 - 00:22:35.960

there is a project on enterprise resource planning, ERP, and one related to the domain of digital employees for network management and control. And this explores, in fact, the possibility of using intelligent agents to perform tasks such as responding to customers' calls. STO, in fact, views the latter to be one of the areas where it sees first the wide-scale

00:22:35.960 - 00:23:08.420

adoption of artificial intelligence in NATO's military domain, as the products available from the commercial IT market are likely to align well with the needs of the NATO military. And it also has initiatives aiming at improving what I consider to be capacities for day-to-day operations, as well as on the ground. For example, they have projects on cloud-based

00:23:08.420 - 00:23:36.500

and data-centric security services and the use of deep machine learning for cyber defence. I just wanted to point out that there are so many applications of AI that are not discussed above, and NATO is working on these. For example, there are also initiatives on cognition, synthetic biology, as well as technologies for medical healthcare professionals.

00:23:38.420 - 00:24:07.560

And I think STO is doing the most important work, I think, because they are the most vocal about ethical issues related to AI, such as issues concerning trust, explainability, fairness, robustness, and accountability in machine learning systems. Human autonomy is key as well, and human in the loop for AI. Regarding the narratives, there have been

00:24:07.560 - 00:24:36.200

various narratives at play, but we don't have time, so I will only single out one, I think, which is very important. This actor-centric narrative, I think it's really important because it will have significant implications in terms of how AI is governed and how it will be applied within the alliance. And that narrative is that of inclusion and

00:24:36.200 - 00:25:09.620

adaptation, and they started to include both outside the traditional defence industry, such as start-up, and STO is doing the same, and I think it is a bold move, and one can see here a process towards a new mode of governance, if I could say. And why I'm saying that, it's because for now, and before that, science and technology within NATO uses a collaborative

00:25:09.620 - 00:25:35.900

business model, right? And the nations define their own resources, and they elect and choose these resources, and there is the in-house delivery business model where activities are conducted by the executive bodies, but that kind of framework, I think, opens up to future possibilities, and maybe we could talk about a new mode of governance, a model of governance

00:25:35.900 - 00:26:03.960

towards a democratic one within NATO, it may be possible, but that being said, we are not there yet. Do I still have time? 20 seconds. Okay. But what I have noticed is that there are less NATO-AI narratives on ethics among these actors I've been discussing, and trusting the topics we've been discussing here in the political narratives would be very

00:26:03.960 - 00:26:32.520

difficult, but one would hear much more narratives on deterrence, defence, and also enhancing capabilities in terms of information and situational awareness. That is where it's important, the role of NATO-PAI parliamentary assembly, because they are the ones who push the discussion, for example, towards the need to regulate development of unmanned

00:26:32.520 - 00:27:00.600

defence technologies and AI, the weapons, because they think that the role of the alliance is to protect civilians, right? And they are also the ones who are pushing, for example, for an increased policy of cryptographic technology, such as AI and big data, and also, I think it's interesting as well, they also push the need for an international treaty

00:27:00.600 - 00:27:41.500

on cyber security, and the role of NATO-PAI, which is outside the whole body of NATO, I think it's very important, but they do not have this capacity to shape, influence, and dominate narratives, and I think that we need more work, we need to do more work at this level. Thank you very much. All right, I used my own timer as well, so just to be on the safe side. All right, so I will talk

00:27:41.500 - 00:28:06.720

mainly about so-called future practices of remote warfare in the context of autonomous weapon systems, and investigating the significance that these so-called laws, depending on what definition you follow, will have on remote warfare, and I make two specific points. The first is something about distance and these emerging AI-driven technologies,

00:28:07.620 - 00:28:31.400

and to what extent, and I think I'm sitting in between both, so I will touch upon both general and narrow aspects of AI, because I think it's important to keep both of them in mind when we think about AI-driven technologies in the use-of-force context, and then I will talk about the issue of meaningful human control, and here I will make the argument that this has been emerging

00:28:31.400 - 00:28:57.060

as the potential focus of the debate in Geneva, so basically, many states say that it's inappropriate to delegate the use-of-force decision completely to machines, but then we come immediately to this idea of meaningful human control, and what this means, and what this delimits, you know, and different understandings of that, and I will talk about that in the context of historical

00:28:57.060 - 00:29:17.200

practices of remote warfare, and to what extent these actually imply meaningful human control, and I will argue that there have been lots of practices that have not been deliberated or verbalized about meaningful human control, what are the limits of that, and what kind of limits can we accept, that are now coming out through this debate about autonomous

00:29:17.200 - 00:29:39.460

weapons systems, interestingly. Okay, so let me start with the first one, the two types, two observations about distance I want to make, and here, really, I'm talking about the most negative vision we could think about when we think about an autonomous weapon system, so a system that has autonomy in its critical functions, which is most often talked about at the end of

00:29:39.460 - 00:30:01.920

Geneva, you know, so at the very end of the targeting process, so these are systems that would be able to engage in target selection, identification, and attack without requiring further human supervision, and I think these systems, if you think about them, they basically would remove human decision-making from the immediate decisions on the use-of-force, and

00:30:01.920 - 00:30:24.860

therefore just exacerbate some of the trends we've talked about in the last two days, so first of all, the trend of technological distance, you know, so that basically the one side is then completely, in any shape or form, even mentally, you know, detached from war fighting, in a sense, whereas, of course, on the other side, people will always be dying, and we always, and we see

00:30:24.860 - 00:30:48.300

like the negative repercussions that these asymmetrical effects can have already in the research on drones, and we've talked about this this morning, over the last two days, so the fact that it leads kind of to this feeling of helplessness on the other side, right, and to this feeling of kind of powerlessness in the face of a technologically very superior enemy, and I

00:30:48.300 - 00:31:12.860

think that this reality actually is also present at the debate in Geneva, because I found that many of the states who at the moment are supporting a pre-emptive ban on these kind of fully autonomous weapon systems, as they're called, are all states of the global south, right, and I think that's quite telling, given the history of interventions, and it's

00:31:12.860 - 00:31:32.940

also worth noting that I haven't really seen the same extent of kind of unified organization among the global south and also the society actors in other disarmament forums, so I think that itself is quite an interesting observation, has to do with this fact of tech distance, and I think there's also logical distances at the heart of the targeting process, if you think about these AI

00:31:32.940 - 00:31:53.420

enabled weapons, that it's becoming increasingly difficult to question data outputs, because there's basically no, I mean, even in my limited conversations that I had with AI specialists, there's even a limit to how far they can understand what machine learning algorithms actually do and how they reach their decisions, right, so there's always

00:31:53.420 - 00:32:10.200

this black box in the middle, and you can potentially correct the output if you know that the output is wrong, but you still don't really know anything about the decision-making process, and I think that's very problematic in terms of how these systems could be used across all different stages of the targeting cycle, because if you want to question the judgment of

00:32:10.200 - 00:32:32.200

that system or the output that they provide you with, how are you going to question that, because you have no way of knowing where it probably went wrong or where you could have intervened, so I think that sets definite limits to what we consider meaningful human control in any shape or form, because it's kind of a limit of

00:32:34.260 - 00:32:56.340

our human reasoning capacities in a sense, right. And I think there's also this argument around, which I find quite speculative at this point, and this is where I follow also what Larry has been saying, I think, is that at some point in the future, machines might be more capable than humans to

00:32:56.340 - 00:33:20.920

make these decisions about proportionality, about distinction, than we are as flawed human beings. This is very prominently put out there by an eye specialist called Arkin, who always talks about the fact that because these machines would not be subject to emotions of anger or revenge, of course also of empathy or other more positive

00:33:20.920 - 00:33:42.440

emotions we might want them to be capable of, but that basically means they would have this detached stance that you could take in making these assessments. But I think that is at the moment, I think it's more or less, he's always alone in being cited in representing that particular notion, and it seems to me quite a speculative argument at

00:33:43.300 - 00:34:07.780

present. And I think it also brings me to this issue that I think there's this desire in the tech community that we can find tech solutions for political problems. So if only we could program this kind of perfect algorithm that would help us solve these intricate social issues, then we would be fine. And I think that this kind of tech solutionism narrative is very dangerous

00:34:08.460 - 00:34:30.620

because it basically outsources the tricky decisions about whether this is something we normatively want to, again, a machine. Okay, well now I'm only at six minutes, or more than six minutes, so I will talk a bit about need for human control. So this is the idea to what extent do we think systems need to be under some kind of human

00:34:30.620 - 00:34:53.120

control to be appropriate, to be considered appropriate. And I think it's quite difficult to think about this because there's so many stages in the targeting cycle and so many touch points of human-machine interactions that we could think about when exercising human control. And although this has emerged as being quite a promising concept in Geneva,

00:34:54.540 - 00:35:13.920

the reason why this has been so promising is because it's very vague. So at the moment, states can more or less decide for themselves what they think would be meaningful human control. So I think you have a lot of states coming out saying, okay, we want to have the human always involved, but in what shape or form can be very different.

00:35:15.420 - 00:35:41.780

And I think actually that we've seen in the past already systems that have been deployed sometimes for years that have been in some shape or form compromising meaningful human control. And in the case of BNI-driven systems I'm most interested in, these are, for example, different systems of missile defense or cruise missiles. So where you have some of

00:35:41.780 - 00:36:05.540

these limited AI-driven capabilities already there, and they have been used, of course, for decades now and they are quite widely spread. And what I think these systems have led to are a certain type of non-deliberative or non-verbalized practice around what we think is appropriate in terms of human control. And this is kind of my analytical point I would like

00:36:05.540 - 00:36:30.680

to make. So I'm interested in how norms on the use of force change and how practices can help us study these changes. And so far in my field, the focus has always been on deliberative practices. So we're coming together in Geneva, discussing potential norms that can guide in the future our use of force. And at some point, maybe states accept, or maybe not, some kind of negotiated

00:36:30.680 - 00:36:52.620

outcome. And at that point, it becomes guiding for state behavior. So I think these deliberative practices are very important and there are many instances of international disarmament treaties where I've seen them kind of played out and become important. But I think at the same time, we have always been missing out on these non-deliberative practices. So the practices

00:36:52.620 - 00:37:11.340

that emerge from how weapons, new weapons especially, are used in the field. Of course, these weapons have to be IHL compliant and usually they should be tested in that regard under Article 36. Of course, the extent to which these tests are really what they should be is a different point. But I still think that incrementally, states have been using these

00:37:11.340 - 00:37:33.820

systems in particular ways. And these particular ways are what I think are these non-deliberative practices. And around this, we see crystallizing a certain understanding of what is appropriate when it comes to using these systems. And only now that we're actually talking deliberatively about issues such as meaningful human control, I think do these kind of practices that have

00:37:33.820 - 00:38:00.540

been around for decades actually come out and are kind of voiced. But I think that at the same time, these non-deliberative practices have already led to a certain norm. A norm in the understanding of a standard of appropriate behavior that has not been verbalized and is now basically competing with these deliberative efforts to create a new norm. And I think we see this very clearly also

00:38:00.540 - 00:38:26.140

in the way that some states want to limit discussions in Geneva to these more futuristic systems. And when the discussion comes to missile defense or drones, the issue always is that these are legacy systems or semi-autonomous systems and we shouldn't really talk about them. But I think it's vital to talk about them if you want to understand what are the limits of meaningful human control

00:38:26.140 - 00:38:54.740

and also what these systems have already led us to think are the limits and what is kind of appropriate in terms of human control. And I think that's a really crucial point that we're missing at the moment. So I think to conclude, I just want to, in this kind of whole process, what I think is also a very kind of dangerous idea that I see is that sometimes many states

00:38:54.740 - 00:39:18.080

have a tendency to want to portray these kind of systems of AI-driven development as inevitable. So there's a certain progress. We are on the trajectory. We have to use these systems and there's nothing we can do. At some point, we're just going to accept them. So we become more used to them. So there's this argument if we get used to more autonomous cars in our daily

00:39:18.080 - 00:39:38.400

lives, then eventually it will just be normal. So we will just use AI in our daily lives and that's going to be it. But I think we should also remember who has actually agency in this process and who can guide this process. These are exactly the people who at the moment are saying, this is inevitable, just kind of deal with it. Because I think technology is always ambivalent,

00:39:38.440 - 00:40:02.240

it's never deterministic. So just because it's there, it will be shaping our future. It doesn't do this on its own, but it does do this in a certain social context. And I want to end here with the Colin Rich dilemma. Colin Rich is a historian of technology and he talked about the common trade-off between knowing the impact of a technology

00:40:02.240 - 00:40:22.940

and the ease of influencing its social, political, or innovation trajectories. And he basically said, when change is easy, the need for it cannot be foreseen. Or when the need for change is apparent, changes become expensive, difficult, and time consuming. And this is kind of the dilemma. I think this describes the situation quite aptly,

00:40:22.940 - 00:40:46.140

because we're still at the initial stage of rolling out these technologies in different social fields. Not many systems that are in use in terms of military use of force have these AI driven, or have significant AI capacities. And this makes it at the moment maybe harder to assess what actually the problems will be, and where will they lead in the future. But at the same time,

00:40:46.160 - 00:41:04.880

I also think that's a piece of good news, because that means that we're not at this costly end of the development trajectory, but kind of still firmly in the middle. So we are at the stage where it's less difficult, potentially less time consuming, to actually make these changes and adjustments. And this is really why I think we should have these conversations very critically

00:41:04.880 - 00:41:28.400

about this now. Okay, great. Home stretch now. Good afternoon. Thanks for having me, and thanks to Abigail and Evanot's Remote Warfare Programme for hosting what I think everyone can agree has been a really productive two days, ranging from the theoretical contributions all the way to the policy

00:41:28.400 - 00:41:47.260

implications. For those of you who I haven't introduced myself to yet, my name is Robert Clark. I'm a postgraduate researcher at King's College London. My background is 12 years in the infantry, so I served in Iraq and Afghanistan, and more recently across East Africa in what the British Army called the STTTs, the short term training teams, or in sort of remote warfare

00:41:47.260 - 00:42:06.660

jargon, the security force assistance. My contribution today is going to be the implications of the rise of AI and robotics, in particular unmanned ground vehicles and remote weapon systems in the British Army, and how that can factor into remote warfare as a broader concept, which I think we've debated enough over the last two days' definitions, and I'm not going to go

00:42:06.660 - 00:42:26.340

there, but I'm up to debating that in the Q&A after. Okay, so the three immediate implications for the rise of AI and robotics. The first one is the continued increased investment across this technology. Secondly, it's the changing character of warfare that really drives this, and we're going to see this a lot more frequently in coming short to medium term

00:42:27.400 - 00:42:52.680

conflicts. And finally, there's a requirement for a much greater degree of transparency and the ethical and the legal frameworks which govern the technology and considerations further. Okay, so if you look at the increased investment from the UK last year, the 2018 Modernising Defence Programme, although it wasn't widely well received across various levels, in terms of

00:42:52.680 - 00:43:17.680

increased investment, it promised £160 million for the Defence Transformation Fund. That's specifically geared towards artificial intelligence within the Army and MOD, but hopefully by 2019, the spending review, that will gear up to potentially a maximum of £500 million. So it's quite a sizable sum for developing AI. If we compare that to the US, last year,

00:43:18.040 - 00:43:43.440

the Department of Defence initiated the joint AI centre, and that had a budget of £1.5 billion there or thereabouts. And then Assistant Secretary of Defence Shanahan described it as a departmental priority to organise their AI programmes, roughly 600 or so within the Pentagon. Now, what's interesting with these figures, the defence spending for AI between the UK and the US

00:43:43.440 - 00:44:09.560

is, once we look at the defence budgets of the UK and the US, last year, the UK defence budget was around £35-36 billion, and the US in pounds was around £525 billion. So it's about 15 times difference in budget, but when we consider the breakdown between the JIC budget and hopefully with the spending review, it's actually the UK spending just under five times the amount than

00:44:09.560 - 00:44:27.600

the US in terms of developing AI and robotics. I think that's quite an interesting telltale indicator of the spending. Okay, so why is it? Why is this increased spending? So it's the changing character of warfare that we've seen over the last few years. If we look at a quote from Chief of the General Staff, General Carlton Smith, last year, he stated that warfare is

00:44:27.600 - 00:44:47.580

increasingly moving into non-traditional spaces, and in particular artificial intelligence and autonomous technology. Now, these non-traditional spaces that he talks about are permeated by the characteristics that we term remote warfare, and in particular, recently, the developments with new AV technology. If we look at Syria and the use of the US Special Forces, the Switchblade,

00:44:47.580 - 00:45:15.020

which is the first offensive autonomous drone technology that's been used in theatre against anti-IS, anti-Daesh drone technology itself and missiles. The rise in new AV technology, this was touched on earlier by Peter and a few others, and it's really reinforced within the UK defence. Last year, wrong, this year actually, Gavin Williamson actually reconfigured the

00:45:15.020 - 00:45:34.440

operational criteria for the Iraq and Syria operational service model for personnel, specifically for UAV pilots and support personnel who weren't in the conflict area themselves, but were in mainly RAF Wallington in Lincolnshire and RAF Akita in Cyprus. So we've had that debate already about how far removed now people within conflict and their eligibility for

00:45:34.440 - 00:45:55.260

mental criteria, and Gavin Williamson stated that this reflects the changing character of warfare. A more stark quote from Major General Tickle, he's the Director of Capability at MOD. He stated last year that we will need autonomous machines to make us more effective and to make us more lethal. The lethality suggests a greater ability to neutralise targets and threats whilst the

00:45:55.260 - 00:46:17.280

effectiveness is a much broader range of military capability, and it's this duality between the lethality and the effectiveness that I seek to address. If we look at the effectiveness for a moment, it's really the last several years in conflict in Iraq and Afghanistan. What's really been burdened and overly pressured is the logistical and supply, the resupply and logistical

00:46:17.280 - 00:46:45.860

change in the army and wider military. If we consider the 2015 SDSR, the capability for creating two strike brigades, in addition to the global Britain approach that the British foreign policy is looking towards, then this logistical resupply burden will only increase in incoming conflict. So what's to be done? Okay, so last year the DSTL commissioned the AI Research and Development

00:46:45.860 - 00:47:06.880

Autonomous Last Mile Resupply System. Now this had three aims. First was to reduce the logistical burden I've just mentioned. Second was to provide new operational capabilities, so in particular new online delivery systems which were autonomous and autonomy in vehicles and weapon systems. And thirdly is to reduce the operational casualties that we've seen in recent conflicts.

00:47:07.820 - 00:47:31.280

So the bulk of the ALMRS programme was Autonomous Warrior 18, which was last year. It was conducted in Salisbury Plain by one armoured brigade. Now it really focused on the five remaining projects out of 50. Those aims enhance the situational awareness and citizen support, movement and manoeuvre, firepower, and crucially when we look at remote warfare, force protection and

00:47:31.280 - 00:47:57.340

sustainability. So one of the projects that really came to the fore of Autonomous Warrior was the FEMUS platform. Now it's got several different platforms as you can see in the pictures. The bottom picture is the transport platform. The top left is the Spanish and acquisition platform, the ELIX-XL. Now that can incorporate internal organic UAV autonomous

00:47:57.340 - 00:48:18.420

technology. And top right is the remote weapon system, so the ADA, which has a 50-cal heavy machine gun. And there's also the projector, which has a javelin anti-tank capability as well. When we look at the roles that these machines and the autonomy machines can have within current operations, we can really see it as a force multiplier if we consider within offensive

00:48:18.420 - 00:48:42.060

operations. So it has two real roles. One is the remote weapon system consideration in fire support. And the second role is the support role that I've alluded to earlier, transport and resupply. Now with the original platform at the bottom on the transport platform, in terms of like an infantry capability on the ground, that can actually resupply a platoon

00:48:42.060 - 00:49:01.800

200% of its operational capacity in terms of food, water, ammunition and medical supplies. So it relieves a huge burden on the resupply chain, quite a low structure. If we look at the remote weapon system consideration, now this can relieve a two to three man gun crew or an entire mechanised platform, which can provide a similar capability,

00:49:01.980 - 00:49:24.560

sort of either the Foxhound or the Mastiff. And also crucially for the casualty extraction, you can set the transport platform for pre-designated waypoints, set remotely, and this can bring back casualties two at a time. Now two casualties in a conventional sense would require at least an eight man litter team, and also considerations from platoon sergeants and

00:49:24.560 - 00:49:43.120

company sergeant majors, which have other roles on the battlefield as well. So that really frees up manpower, which is a huge help. Okay, in defensive operations, we can have it as area denial. So the remote weapon systems themselves, in addition to the defence and target acquisition platforms, can really sort

00:49:43.120 - 00:50:02.980

of hold and defend that key terrain in place. Again, further freeing up the manpower. Now, when we consider how this can be implemented into the UK's remote warfare approach, if we look at the existing capabilities, they're strengthened through especially the UGV technology and Themis. So for example, the security force assistance roles, if you consider Iraq,

00:50:02.980 - 00:50:23.140

Afghanistan, Syria and Somalia, they are as a true force multiplier. In these regions, in the training roles, roughly 50% or so of personnel are involved in force protection. So, i.e. securing the area and guarding the trainers and making sure it's a safe environment to operate in. So 50%, that's a lot of manpower. So straight away, you're relieving that burden

00:50:23.140 - 00:50:46.680

through the dispensing target acquisition platforms, the organic UGV drone technology within them to free up the force protection side of these missions. And again, in the more hot areas, especially like Somalia and Afghanistan, where the SFA roles are, the forward operating bases, you can relieve manpower for defences with the RWS systems as

00:50:46.680 - 00:51:11.620

well. And then when we consider the UAV deployments that I've spoke about in Syria and in Iraq, both offensive and defensive capability, the switchblade and the anti-drone capability that they can have there, these are already being operationalised in theatre as we speak against IS. So, with all that in mind, there are ethical and legal considerations to be had.

00:51:12.680 - 00:51:27.020

Now, I'm going to leave this more for, I'm sure there's many more people more qualified than me that can talk about the legal considerations. We've had some of the legal framework, but I think certainly a help straight away is refocusing the narrative to one of autonomy in weapons, not just autonomous weapons.

00:51:29.020 - 00:51:48.820

That'll allow the debate to progress. What I will say straight away is the UK government needs to engage more actively with the conventional asserts on conventional weapons. So far, there's been very little dialogue, if any at all. But really, by finding state-based definitions for things like remote weapon systems, offensive detergents, autonomy in weapons,

00:51:49.240 - 00:52:10.020

by finding these state-based definitions through the CCW, that's really the basis for establishing the legal framework to actually go forward with this technology. And in so doing, it'll actually attempt to redefine our ideas of normative acceptance and appropriateness, which is crucial. And yeah, by engaging the CCW further, the UK maintains its global role pioneering

00:52:10.020 - 00:52:43.340

the UGV technology and autonomy in weapons. Thank you. Okay. All right. So yeah, I would just like to echo Rob and say that I have also learned a hell of a lot in the last... Sorry, I'm just in my... Yeah, I've learned loads from people from across the spectrum in the last couple of days, so thank you very much. And thanks for the organizers as

00:52:43.340 - 00:53:08.920

well. So I'm delivering the paper. I'm obviously Pete and not Camilla, so Camilla is the lady up there. I will say in the Q&A, so I'm more... I come at this not from the techie AI side of things, so Camilla works for the APP on Jones. So any techie type questions, direct them to Camilla or the other panel members, please. Yeah, the oversight I can deal with or try to anyway.

00:53:10.980 - 00:53:36.500

So what I'm going to do is broadly structured in three parts. So current UK remote operations, I won't go into the details of it. I think we've probably all got an idea of what remote welfare might be or could be or might entail. But yeah, so I'll just... A very high level in that. Then get into the current UK parliamentary, what the debate is around oversight, where we stand

00:53:36.500 - 00:54:04.520

currently with oversight, what oversight looks like, what some potential impediments to parliamentary oversight might be. And then I'll end with just some, again, quite high level ideas about how we might try to develop or further develop oversight within the UK parliamentary system. So yeah, like in some ways, what remote warfare is and whether it's even useful to define

00:54:04.520 - 00:54:30.680

all these things under one umbrella is definitely up for debate. And there's some... Yeah, I can understand why you, in some instances, might chop and change them in and out, but they basically encompass a broad range of things, including cyber weapons, cyber defence, training, all of that stuff we've been talking about in the last couple of days. So this is kind of a tabular

00:54:30.680 - 00:54:57.740

demonstration of... So this is an indicative list of some of the parliamentary committees that would have some oversight over different aspects of remote warfare. It's by no means exhaustive, but I've probably tried to hit the big ones. So if you think we've missed any of the big ones, right, which potentially is possible, then do let us know. But yeah, so it's quite an impressive list

00:54:57.740 - 00:55:19.220

of committees when you think about it. You've got the Intelligence Security Committee, which is a joint committee, Defence, and then a subcommittee as well, the House of Commons, Foreign Affairs, International Relations, and so on and so forth. So some of the key committees in the UK parliamentary system represented there. And you can't... I mean, and there is oversight, right? Like there has,

00:55:19.580 - 00:55:42.220

in some cases, instances of quite impressive oversight and scrutiny, and sometimes it doesn't go as far as the parliamentarians involved might like. So there's been quite a lot of discussion already of the ISC investigation into the Rara Khan strike, for instance, where the ISC pushed back and said, here's some conclusions, here's some information, but we would have liked more.

00:55:43.560 - 00:56:08.900

And another instance would be the ISC's report from last summer into, like it's kind of third report into the rendition system. And yeah, so there is clearly some oversight. Some people might say there's too much, some people say there's not enough, but the puzzle isn't, is there oversight? The puzzle is, have we got enough oversight? Is it fit for purpose going forward? Similarly,

00:56:08.900 - 00:56:31.520

the APPG. So if you're not familiar with the UK parliamentary system, an APPG is an all-party parliamentary group, which is basically exactly what it says on the tin, group MPs from any party across the spectrum or independent MPs, as we seem to have an increasing number. Some people might... You might get text messages, like a whole bunch might leave in the next 10 minutes.

00:56:31.520 - 00:56:56.220

And yeah, so, and they do work together on, and you get like really interesting groups of MPs who seemingly have nothing else in common except an interest in a particular issue. And they can be chaired by Labour MPs, chaired by Tory MPs, so on and so forth. And again, the list there is indicative, but we've certainly gone to try to highlight some of the most important ones.

00:56:57.680 - 00:57:20.940

And then on the final table on the right-hand side, that is an indicative list of potential barriers to parliamentary oversight. That isn't to say that in some cases that there aren't legitimate reasons for those being in operation. So for instance, that clearly in some instances, it wouldn't be right for the government to comment on actions, right? Or certainly not

00:57:20.940 - 00:57:43.360

before they were taking place. There might be like... Yeah, like sometimes... So a lack of clarity over definitions, and we've kind of had a lot of discussion around that over the last couple of days. Like, what do we mean by, for instance, cyber weapons? Like, if anyone can tell me what we mean by cyber weapons, like, I would love to know. And maybe the UK government might

00:57:43.360 - 00:58:09.420

like to know, or not, right? Maybe it's convenient that they don't know. Maybe. But it could be down to stuff like failures to collate information. The most obvious one is, it's been alluded to a couple of times already, like the fact that the UK can honestly stand up and say that we've... What are we up to now? Over 1,900 airstrikes in the campaign, the most recent campaign, and apparently

00:58:09.420 - 00:58:35.320

there's one civilian being killed. Like, it just isn't credible. It doesn't stand up to any kind of scrutiny. Like, the US's own models, like, say it's about a civilian gets injured or killed about every 2.5% of the time, right? And we're way, way, way below that. I just... So that's an indicative barrier of... An indicative list of potential barriers to oversight. What does the

00:58:35.320 - 00:58:55.720

debate like? Again, we've had some of the discussion around this. Part of it is to do... Comes down to the Constitutional Convention around the use of force. I think sometimes the focus on that is overplayed. Clearly, it's part of the picture, but I don't think it's all of the picture. Like, kind of, some people say the convention doesn't exist. If you're interested in the debate,

00:58:56.580 - 00:59:14.300

regardless of where you stand, I would kind of point you in direction of this recent report by the Society of Conservative Lawyers. Like, we were going to have a quote in it, but basically the title says it. Even if you're in favour of the convention, I would encourage you to read it, because it's actually a really interesting report to read. It's actually very well reasoned. I don't

00:59:14.300 - 00:59:36.020

actually agree with the conclusions of it, but I learned a lot by reading it. There's the, kind of, changing nature of remote warfare, right? Like, whether... And it'll probably look different in 15 years' time. You could have had the same conference, like, maybe five years ago. It would have been much smaller, and probably every paper would have been on drones. Clearly, drones

00:59:36.020 - 01:00:00.140

is a big part of the picture still, but it's moved beyond that. There's debates about the assistance, and we've touched on that. There's a quote there from a Yemeni official talking about the UK's role in handing off intelligence, and it's used in drone strikes. So, yeah, so there's kind of a broad-ranging and continuously expanding debate around oversight,

01:00:00.140 - 01:00:25.660

and whether or not, you know, what would be legitimate oversight in these operations. This is... And so, with this one, again, we'd love to know if you think any of this could be moved around, or if we've missed any important parts of this in the next slide. This is kind of a pictorial demonstration of the current site, the current state of the parliamentary groups

01:00:25.660 - 01:00:44.420

involved in parliamentary oversight. And so, ranging from kind of groups that provide information, right, so the House of Commons Library to, you know, National Audit Office, those kind of things, like cross-party groups that are official part of the UK state, and any parliamentarian, regardless of where they sit in the House, can go and get information from

01:00:44.420 - 01:01:06.140

them, they publish reports, over to, like, direct role in policy formation. So, there clearly are lots of other things that sit below this, but we've had to kind of pick and choose other ones. I was thinking, when I was doing my final notes for this, it'd be interesting to know if anyone could think about how to incorporate the difference between the parties, and then the kind of the

01:01:06.140 - 01:01:27.320

backbench groups, like the 1922 committee, right, because sometimes they operate as one, and sometimes they can operate quite distinctly. So, if anyone's got any ideas how or where in this graphic, or maybe they wouldn't, like, let us know. So, yeah, so kind of moving up to the, the kind of privileged parliamentary committees, i.e. the ISC, in the top right-hand corner.

01:01:28.880 - 01:01:54.840

This is the activities that these groups engage with. So, again, starting up in the top left, you've got the kind of briefs from the House of Commons Library, etc., etc., and then down to, kind of, in the bottom right-hand corner, the influences on parliamentarians beyond parliament. So, their constituents, their, you know, the media, public opinion. It could be

01:01:54.840 - 01:02:16.620

their prior careers. People quite often sit on committees that they've, because they've got prior experience of whether they were a lawyer, or they were, had a certain role in industry, and so, and then going all the way up to, kind of, the, that's, so vote on military action, that's the constitutional convention. So, the most potentially active role, or not,

01:02:16.620 - 01:02:40.340

in stopping parliament, or giving acquiescence to. On the, I think, an interesting aspect of this, in some ways, this would be good if you could have it 3D, but obviously we can't, right? So, you could have, like, another axis where you would, kind of, go from proactive, so that's, kind of, like, the current PACAC, and the acronym of that body, oh, okay, yeah, the acronym of that

01:02:40.340 - 01:03:02.480

body escapes me at the moment, but they've got an inquiry into, kind of, the current state, and the future state of UK operations, to cut to reactive things, so, like, the response to the Red Car Strike, so you could have a third axis, axis wasn't on this, but that doesn't work in this instance. So, Rob, you covered a bit of this, so, basically, there's going to be a buttload of

01:03:02.880 - 01:03:26.580

money put into things that, kind of, fall under the barrier of warfare, everyone's, kind of, aware of it, Rob said it already, so, yeah, but you've made the point, yeah, done. So, this was just, I was trying to think about how oversight might develop, right? So, the technology is developing all the time, I don't always fully grasp exactly what that technology is yet,

01:03:27.040 - 01:03:47.320

and I think lots of parliamentarians are probably in the same position, but it is developing, and the way it's used is developing, and so, parliamentary scrutiny, we should at least be having the conversation about how that might develop, and so, rather than have, kind of, simplistic, like, these are some recommendations, I think, we thought it might be useful to try and

01:03:47.320 - 01:04:11.800

have this show as a process, so it's ongoing, right? So, the first instance is, so, make policy available to parliament, so, in some instances, it might be, there might be a need to create new policy, new guidelines, new definitions of things, and so, bring parliamentarians on board, right? There are expertise in parliamentary committees, in bodies like APBGs, within individual parliamentarians,

01:04:12.840 - 01:04:36.540

and I guess, I mean, it's not on there, because this is on, but, kind of, beyond that as well, bring the academy in, bring civil society in. You know, continue to define definitions of concepts, right? If we're not having, if it's not war, if remote warfare isn't war, then is it combat, right? Like, that seems, we need to define this stuff, and if parliament

01:04:36.540 - 01:04:59.760

doesn't do it, the courts will do it, right? So, you know, do you want to just leave yourself subject to the court? Yeah, and so, identifying gaps, right? So, there will be gaps that will develop, and we don't necessarily know what all of those gaps are, so, how, but in some cases, we do, right? So, and in some cases, there's just lacunae that's developed, so there's

01:04:59.760 - 01:05:18.960

no, for instance, minister assigned for the remit of looking after cyber weapons, and I think that's probably just because it's a relatively new area, and so no one's particularly been given it, but if a parliamentary committee wanted information of it, whether, like, public or in private, then it would be great for them to be able to go, right, that's the minister,

01:05:19.480 - 01:05:41.020

and then kind of, in another instance, codify and expand responsibility, and we've talked, like, the IFC is the obvious one to get more responsibility, but clearly, it would need much more resources putting into it, or you could have another committee. We've talked yesterday a little bit about the, sorry, I've gone over, 10 seconds, so, for instance, special

01:05:41.020 - 01:06:10.200

forces, right? There is no body assigned in parliament to oversee special forces, either publicly or in camera. It seems a huge gap, and I will leave it there. Thank you very much. Okay, I hope I can still add something, Arvo, between the last one and the two days. It's difficult, I think, but I'll give it a try. My research is about norms and autonomous

01:06:10.200 - 01:06:35.580

weapons, and I make a bit of a broader statement on the normative dimension of autonomous weapons, and AI, and I'm particularly interested in the questions what normative consequences the increasing role of AI in weapons systems has, and also what makes the use of AI-driven security technologies widespread and also acceptable, and I first talk a bit about the

01:06:35.580 - 01:07:02.080

background and what I think the research problems are before I talk about the effects of remote warfare, the conditions of possibility, and also what I think are kind of the future problems of remote warfare. To start with the background, I think the central problem of remote warfare will be how a technologically compromised human agency can influence how, when, and why

01:07:02.080 - 01:07:28.140

force is used, but currently the debate is mainly focused on law and how remote warfare and different practices of drone warfare, for example, such as targeted killing or target identification or the development of a so-called kill list, how they are important contestation of international law. So the political and also academic debate is at the moment very

01:07:28.140 - 01:07:51.940

much interested in considerations of how to define autonomy and also how to define the quite new concept of meaningful human control, and we have heard about this before, and we have also heard that the definition of both terms or concepts turns out to be extremely difficult and they are therefore not very useful at the moment, neither analytically nor

01:07:51.940 - 01:08:16.420

politically nor legally. So overall I think the political debate is too focused on trying to pin down what autonomous weapons are as the future of remote warfare, and I think the problem is that this directs the focus away from important political questions and it also means that we mainly talk about regulation but not enough about constitution, for example,

01:08:16.940 - 01:08:46.140

in the sense of what makes the emergence of AI in remote warfare unavoidable and also acceptable. And I think related to this is also the question to what extent the existence of specific technologies shape the future of remote warfare, how security policy is shaped by technologies and less by political reflections, for example. So I mean there

01:08:46.140 - 01:09:11.960

are important political questions, questions of political deliberation, for example, of accountability, of responsibility, of protecting human life, which means of all asking whether remote warfare is legal or not is not enough in this regard, I think. Now to talk a bit about the constitutive effects of remote warfare and AI, I think the first

01:09:11.960 - 01:09:36.640

step is to defocus from autonomy as a technical feature and to do actually more analysis on the human-machine interaction, quite broadly understood, and the question is here how is human control compromised by the electronic processing of data, for example, and by aspects such as the inexplicability of algorithms and we have also heard about this kind of

01:09:36.640 - 01:10:03.120

black box problem of algorithms. Also because remote warfare is not only about a weapon that kills but particularly also about an entire process of surveillance, of data collection, of target identification, of target tracking, so there's a whole range of elements which play into this arrangement of using force and this concept of meaningful human control

01:10:03.120 - 01:10:28.860

also concerns all these dimensions and not only how a weapon is operated, who presses the button, to put it like this. The ethical argument is that a certain extent of human control and human decision making needs to be maintained to make the use of force, the act of killing, acceptable but for me the question is also does it make the acceptance

01:10:28.860 - 01:10:54.360

of the idea of meaningful human control also the acceptance of technical solutions more likely and I think there's a risk that technology in a way defines and is constitutive for what meaningful human control means also in the political process. There are at the moment strong signs that AI will play a major role in the future of warfare and in the sense

01:10:54.360 - 01:11:22.160

of course it will also define what remote warfare will be in the future and I think this particular concerns research in learning algorithms and machine learning which suggests that the aim is a technical solution for a political normative problem and to just give an example DARPA, the US defence research project agency, just announced in September

01:11:22.160 - 01:11:43.780

of last year a multi-year investment of more than 2 billion US dollars in the so-called AI Next campaign and they will be researching what they call robustness and reliability of AI systems, enhancing the security of machine learning and pioneering the next generation of AI algorithms and applications so there will be a lot of research and research funding

01:11:44.560 - 01:12:11.160

invested into developing AI for use in weapon systems for example and I think this points to a further dehumanisation of remote warfare in the sense that the distance from human decision making is actually increasing. We've heard about the ambivalence of distance and in the case of remote warfare it can also mean that distance is decreased but I think

01:12:11.160 - 01:12:39.640

when AI is introduced it will certainly decrease when we think about human decision making and this challenges but can also shape our understanding of responsibility and accountability I think and it can also reinforce a constitutive effect on what acceptable use of force is. Now to talk a bit briefly about the conditions that make these developments possible, I think

01:12:39.640 - 01:13:03.820

we should focus more strongly on AI and on the different dimensions of AI and also how dual use applications for example travel between the civilian and the military research and development. So I think we should also take a closer look at domestic security, military research and also the cooperation with leading tech companies and we have heard

01:13:03.820 - 01:13:26.060

about Project Maven for example, it's a quite prominent example but overall I think there's a lack of awareness and also knowledge of the civilian military cooperation which I think can be crucial for military AI solutions and how they are used. Project Maven was a Google corporation and Google will still continue to cooperate with

01:13:26.060 - 01:13:49.960

the military but also companies such as Amazon or Microsoft cooperate with the US military or security agencies. So I think many of the problems in terms of accountability and responsibility are not only military problems but they are related to the way AI is handled also in the civilian life and I think this is important for how AI is or will become acceptable in

01:13:49.960 - 01:14:15.460

security and defence. And by that I mean for example the role of AI in quite different dimensions such as the future of work or medicine or care or policing or education. So I think we could do more analysis on the conditions that make a potentially problematic future of remote warfare possible and acceptable and I think these are legal, political but

01:14:15.460 - 01:14:40.900

also economic and cultural conditions. I think when the role of AI becomes more acceptable in these dimensions it will also have constitutive effects for what is acceptable in remote warfare and I think an example is how we take care of our data for example which can be used as training data for machine learning algorithms and data is one of the most valuable

01:14:40.900 - 01:15:08.160

resources so security and military agencies have also interest in this data and regulating AI in all dimensions is I think also primarily about regulating the collection and use of data and this also plays a role in the international dimension of security. For example the US are collecting vast amounts of data in Pakistan or Afghanistan and without access to this

01:15:08.160 - 01:15:34.100

data remote warfare will be probably quite different because it's really important for developing machine learning algorithms. Finally to talk a bit about the future of remote warfare I think we will certainly see a further intensification of smart intervention and smart technologies and AI will play a fundamental role in that and there's this problem of dual use of AI

01:15:34.100 - 01:16:01.520

and in my mind I think this means that research into weaponising AI will actually not stop and it will be also very difficult to regulate it by an international agreement for example so I think the CCW will not really influence this development regardless whether it manages to come to an agreement or not. In this sense I think there's a strong risk that human

01:16:01.520 - 01:16:25.120

control will be further lost in remote warfare not least since the concept of meaningful human control is rather utopian even in contemporary warfare and doesn't really correspond to practices in the very strict sense. So I think there will be also further depolitisation because legal regulations do not exist or are too vague and the political process doesn't really

01:16:25.120 - 01:16:49.960

interfere in major economic interests and also in technological trends which points again to the role of tech companies for example. So this doesn't mean that meaningful political control is impossible but I'm a bit pessimistic due to the wide acceptance of AI driven technologies assisting humans and how we are happy to give away our data

01:16:49.960 - 01:17:15.280

for example. So in sum I would expect a further broadening and deepening of AI and power security in a domestic but also in a warfare context and I think we will see more of kind of an enhanced surveillance using AI solutions and also maybe autonomous policing solutions in a domestic context and I think this is also important for the future

01:17:15.280 - 01:17:42.280

of remote warfare and how this future of remote warfare will be perceived as appropriate and acceptable which might not be considered as illiberal but rather kind of in line with liberal practices also with a neoliberal political economy for example. So I think the fundaments of remote warfare are really at home and not somewhere in a remote country.

01:17:43.000 - 01:17:43.640

Thank you.