Responsibility must remain in human hands
by Dr Michael Stehr, Advocate, Board Member of EuroDefense, Bonn, Germany
A cultural bias seems to dominate debates on “autonomous systems” with perceptions and expectations shaped by an independently acting “Terminator” or “Skynet”, a superior and amoral artificial intelligence (AI) system with nearly unlimited faculties.
The mundane reality
The reality is that there is a variety of unmanned systems, in the air, on land and at sea: tanks, flying drones, mine hunting sub systems, ships and submarines. Today UxV1 technologies are developed and designed to hit targets when remotely controlled by soldiers. They only can act independently during take-off and landing, automatically controlled cruising and the “lost-link-mode”. So far, no system is steered by a tactical computer although scientists are developing computers with advanced, AI driven capabilities. The technological outcome is not predictable.
In consideration of human responsibility, there is a need today, and even more in the future, to differentiate two categories of systems: automated systems and autonomous systems, as defined by the UK Ministry of Defence2:
“An automated or automatic system is one that, in response to inputs from one or more sensors, is programmed to logically follow a predefined set of rules in order to provide an outcome. Knowing the set of rules under which it is operating means that its output is predictable.”
“An autonomous system is capable of understanding higher-level intent and direction. From this understanding and its perception of its environment, such a system is able to take appropriate action to bring about a desired state. It is capable of deciding a course of action, from a number of alternatives, without depending on human oversight and control, although these may still be present. Although the overall activity of an autonomous unmanned aircraft will be predictable, individual actions may not be.”
Violation of international law?
International law is binding on belligerent parties and their personnel in international armed conflicts. One has to consider the possibility that using automated or autonomous systems could be in violation of international law. Three major rules from the “Protocol Additional to the Geneva Conventions of 12th August 1949 and relating to the Protection of Victims of International Armed Conflicts (Protocol 1)”3 can be taken as examples:
Rule 1: Article 35
1. In any armed conflict, the right of the Parties to the conflict to choose methods or means of warfare is not unlimited.
2. It is prohibited to employ weapons, projectiles and material and methods of warfare of a nature to cause superfluous injury or unnecessary suffering (…)
Prohibited weapons are e.g. poison gas, lasers to blind someone permanently, ammunition which is invisible to x-ray devices. Methods of warfare that exceed the purpose of incapacitating or killing enemy combatants are also prohibited. Consequently, the operation of remote controlled or automated or autonomous systems as such is no violation of this rule.
Rule 2: Articles 48, 51, 57
Article 48 Basic rule
In order to ensure respect for and protection of the civilian population and civilian objects, the Parties to the conflict shall at all times distinguish between the civilian population and combatants and between civilian objects and military objectives and accordingly shall direct their operations only against military objectives (…).
Article 51 Protection of the civilian population
(…) 2. The civilian population as such, as well as individual civilians, shall not be the object of attack (…)
(…) 4. Indiscriminate attacks are prohibited. Indiscriminate attacks are:
a) those which are not directed at a specific military objective;
b) those which employ a method or means of combat which cannot be directed at a specific military objective;
(…) 7. The presence or movements of the civilian population or individual civilians shall not be used to render certain points or areas immune from military operations, in particular in attempts to shield military objectives from attacks or to shield, favor or impede military operations. The Parties to the conflict shall not direct the movement of the civilian population or individual civilians in order to attempt to shield military objectives from attacks or to shield military operations.
Article 57 Precautions in attack
1. In the conduct of military operations, constant care shall be taken to spare the civilian population, civilians and civilian objects (…)
An attack aiming primarily at a legitimate military target is not an indiscriminate attack and does not violate these Articles if it causes no excessive collateral damage – the latter effectively avoided by precision. The use of technically advanced unmanned systems is therefore no different from a legal perspective. On the other hand, the belligerent party that places a radar in a village or town and uses its population as a human shield is clearly in violation of Article 51 para. 7.
Rule 3: Article 41 Safeguard of an enemy hors de combat
A person who is recognized or who, in the circumstances, should be recognized to be hors de combat shall not be made the object of attack. (…)
Ethical challenges to soldiers and politicians
Soldiers are accustomed to C4ISR systems with increasing degrees of automation. The next revolution in military affairs will be the ongoing replacement of manned systems and remotely controlled weapons by UxV. Currently, soldiers tend to observe rather than operate systems. UxV is however an opportunity to withdraw soldiers from the first line of fire and from dull, dirty or dangerous operations. Some soldiers – not all! – will tend to become operators or technicians rather than combatants.
What about the psychology of soldiers fighting ennemy UxV while risking their lives? It may be easier to destroy machines than to kill humans – but there’s also the knowledge of risking lives against mere objects.
Given the theoretical possibility that a future unmanned system could decide on the start, conduct and end of an operation without any human participation, the question arises as to whether the military should retreat from observing and controlling that process. The answer is a very definite No! There is not only the question of the ultimate responsibility of the military for all its operations but also the fact that the very act of applying lethal force is a core element of the soldier’s job. Furthermore, even the most advanced AI system lacks a conscience and the link to existential issues. Humans are vulnerable and finite – and the knowledge about this is part of human intelligence. Decisions made by AI are very different from human decisions – to say the least.
Last but not least, UxV pose an inescapable question to political and military leaders. Can you take responsibility for your soldiers risking their lives fighting enemy UxV on the battlefield without using UxV themselves?
Information about Dr Stehr’s book on unmanned systems and cyber operations (10/2020): https://bit.ly/3tZ3zJL
Dr Michael Stehr
is a German lawyer with experience as staff member of the scientific branch of the administration of the German Parliament. He was born in Hannover in 1966. Dr Stehr has been the editor of “Law of the sea and international law” of the German periodical MARINE FORUM since 2000, dealing with piracy and seaborne terrorism, International Law and CSDP. He is a Board Member of EuroDefense, Germany (https://www.eurodefense.de) and Vice-President of the German Naval Association (https://deutscher-marinebund.de).
1 UxV stands for any one of the four categories of unmanned vehicles: ground, air, surface or undersea. Humans operate these vehicles from a remote location.
2 UK Ministry of Defence, Joint Doctrine Publication 0-30.2 Unmanned Aircraft Systems, P. 9-20, https://bit.ly/32sBWg3
3 https://bit.ly/3eiguzz
