NIMBUS – User Interface for Bundeswehr Multi‑Domain Operations with Unmanned Systems

The goal of NIMBUS is to determine how a future-proof, cross-domain interface must be designed so that one or more operators can monitor and control multiple unmanned systems from different domains – air, land, sea, and potentially space – in joint operations. The central question is how information superiority and high levels of automation can be leveraged without operators being overwhelmed by data overload, complex operating logic, or non-transparent automation decisions.

The first step was to understand which unmanned systems the Bundeswehr currently uses, in which units and domains they are employed, in what types of which mission they occur, and how they are currently operated. From this, the task emerged to capture the actual operating reality – including work environments, role models, interfaces, and levels of automation – from the perspective of UxV operators. On this basis, concrete user requirements for a multi-UxV interface were to be derived that support both the operation of individual systems and the simultaneous control of multiple platforms in multi-domain operations.

In a second step, it was necessary to determine how roles and levels of automation can be meaningfully modeled so that responsibilities remain clear and Meaningful Human Control is maintained. NIMBUS goes beyond classic categories such as “human-in-the-loop” by linking human roles, system affordances, as well as legal and safety-related requirements.

In parallel, it was to be demonstrated how mixed-reality technologies can contribute to the control of unmanned systems. For this, a realistic reference scenario was necessary that describes the joint use of unmanned systems by the Air Force and Navy using a clearly defined, security‑policy relevant example: the protection of critical underwater infrastructure in the Baltic Sea.

NIMBUS follows the human-centered design process according to DIN EN ISO 9241‑210. The 2024 base phase focused on understanding and describing the context of use, identifying user requirements, and documenting the state of science and technology. The 2025 option phase built on this and developed design solutions in the form of a mixed‑reality interaction concept with a demonstrator.

Context of Use

At the beginning, a comprehensive description of the context of use was created. For this purpose, all unmanned systems currently used in the Bundeswehr were systematically recorded: 

• Air: Micro-UAVs (MIKADO, Black Hornet), tactical and operational systems (ALADIN, LUNA, KZO, Heron 1, Heron TP)
• Land: tEODor, RABE, PackBot 
• Sea: Seehund, Seefuchs, SeaCat, REMUS 100, Deep Trekker Revolution 
• Space: Research demonstrators such as the POLARIS systems (MIRA, NOVA)

For all systems, key data such as range, operating altitude or depth, purpose, user service branch, weight, and armament were collected and assigned to the respective domains.

Online Survey

In addition, an online survey was conducted among twenty UxV operators from the Army and Air Force. It captured UxV expertise, experience with multi-domain operations, collaboration with other domains, typical missions, as well as the usability and level of automation of current ground control stations.

The results show that collaboration between the land and air domains is already common practice in many types of missions, while cooperation with the maritime, cyber, and space domains occurs much less frequently. At the same time, current ground control stations tend to be viewed neutrally, with a desire for greater – but transparent – automation, particularly in the area of navigation.

Usage Requirements

On this data basis, usage requirements were derived that cover aspects such as adaptability, automation, usability, trust, displays, warning systems, and control. For example, they specify:

• Operators should be able to customize controls and displays to meet individual needs.
• Navigation tasks should be largely automated, while control over the payload and its effects remains with the human.
• Critical system conditions must be clearly identifiable through both auditory and visual cues. 

In addition, requirements for the visibility of typical interface elements were defined – such as situation map, UxV position, route, camera image, mission information, engine values, or communication history –, differentiated into “always visible” and “situation-dependent”.

State of the Art

The state of the art in science was summarized based on numerous Fraunhofer FKIE projects, including INSIDE, VOPZ UAS, AutoAuge, RoboMine, GeMUx, UGViXRC, and PAA, as well as initiatives on FCAS, loitering munition, automated convoy driving, and reconnaissance for fire brigades and civilian users. In addition, relevant NATO and EU activities such as STANAG 4817, INTERACT, and CAMELOT were analyzed.

The state of the art in technology included commercially available systems for controlling unmanned systems (including SitaWare, HAT.tec SCALE, Kongsberg MDCS, MESE), open-source solutions such as QGroundControl and QGC-Gov, as well as fleet management software for UAVs and UGVs. The analysis showed that while there are powerful individual solutions, there are hardly any cross-domain multi‑UxV interfaces with clear HMI standardization that can be used consistently.

Roles and Automation Model

As part of the option phase 2025, role models and automation concepts were developed. For UAV, USV/UUV, and UGV missions, roles such as Mission Commander, Air Vehicle Operator, Vehicle Operator, Payload Operator, Intelligence Analyst, and Safety Director were analyzed and harmonized. This resulted in a cross-domain role model that maps both classic command roles and specific functions for unmanned systems.

In parallel, the handling of automation and autonomy was examined in greater depth, especially with regard to Meaningful Human Control. Building on the international debate on autonomous weapon systems and the OODA cycle (Observe, Orient, Decide, Act), not only abstract levels of autonomy but also concrete requirements for user roles, system behavior, and user interfaces were defined.

To this end, the TISA framework (Trust, Involvement, Situation Awareness) was developed as part of the NIMBUS project. It enables practical operationalization of Meaningful Human Control for the design and evaluation of human‑UxV interfaces.

Reference Scenario: Baltic Sea

A key element of the option phase was the development of a realistic scenario: the protection of the submarine communications cable C‑Lion1 in the Baltic Sea. Against the backdrop of real sabotage acts on pipelines and cables and a tense security situation, a scenario was modeled in which the Air Force deploys a MALE UAV (Heron) equipped with maritime reconnaissance sensors, and the Navy utilizes several USVs and an ROV. Objective: to identify suspicious vessels above the cable, inspect the area, and detect any possible tampering. In doing so, legal frameworks of maritime law, the structure of the German Exclusive Economic Zone (EEZ), and the role of Bundeswehr bases on the Baltic Sea were taken into account.

Mixed-Reality Interaction Concept

From a technical standpoint, a mixed‑reality interaction concept was created with two components:

• 2D situation workstation on a 27‑inch touchscreen monitor – providing a cross‑domain situational picture, mission tasks, and fleet management functions. 
• Mixed‑reality mode on a Meta Quest 3 – enables the immersive teleoperation of an underwater ROV, while the situational picture remains visible displayed as a floating 2D panel in form of augmented virtuality. 

The information architecture is based on concepts of situation awareness, the OpenBridge design system for maritime HMI, and an atomic design approach for component‑based implementation in Unity.

Results of the base phase 2024

As a result of the base phase, NIMBUS provides:

• a detailed description of the usage context of all relevant unmanned systems of the Bundeswehr,
• an empirical analysis of multi‑domain cooperation from the operators’ perspective,
• a structured requirements catalog for a multi‑UxV interface, and
• a systematic compilation of the state of science and technology. 

The study clearly shows where cross‑domain cooperation already takes place today, where gaps exist, and which HMI aspects are particularly critical for acceptance, trust, and performance.

Results of the option phase 2025

The 2025 option phase adds a cross-domain role and automation concept in which Meaningful Human Control is concretely operationalized for interface design. The Baltic Sea scenario and the mixed‑reality demonstrator provide a clear demonstration of how an integrated situation workstation and an immersive teleoperation view can work together – to deploy unmanned systems in complex maritime situations without losing overall situation awareness.

Outlook

On this basis, several further developments are possible: 

• User Studies: The demonstrator can be used to conduct systematic evaluations with operators from the Air Force, Navy, and Army – focusing on usability, workload, situation awareness, and confidence in automation. 
• System Integration: The demonstrator can be connected to real unmanned systems and simulations to test the NIMBUS concepts in training, exercises, and, in the future, in operational use. 
• NATO Integration: As an “HMI modul”, NIMBUS offers the opportunity to contribute results to NATO activities on Meaningful Human Control, STANAG 4817, and interoperable multi-domain control stations. 

In the long term, NIMBUS can thus make a key contribution to a standardized, user‑centered user interface for unmanned systems in the Bundeswehr.

Project duration
2024 to 2025

Client
Federal Office of Bundeswehr Equipment, Information Technology and In-Service Support (BAAINBw)

Project partners
Bundeswehr Technical Center for Aircraft and Aeronautical Equipment (WTD 61)

Deutsches Institut für Normung (DIN). (2020). DIN EN ISO 9241‑210:2020‑03: Ergonomics of human‑system interaction – Part 210: Human‑centred design for interactive systems. Berlin: Beuth.

Drury, J. L., Scholtz, J., & Yanco, H. A. (2003). Awareness in human-robot interactions. In SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics (pp. 912–918). IEEE.

Endsley, M. R. (1988). Situation awareness global assessment technique (SAGAT). In Proceedings of the IEEE 1988 National Aerospace and Electronics Conference (pp. 789–795). IEEE.

Kwik, J. (2022). A practicable operationalisation of meaningful human control. Laws, 11(3), 43.

Lasquetty, J. G., Pérez Carrasco, F., Maciás, M., Piwiński, J., Kłoda, R., Goodchild, C., et al. (2018). CAMELOT Architecture and Data Model. TEKEVER ASDS, PIAP, BAES, EXODUS, UPVLC, NAVAL.

Loper, M., & Sitterle, V. (2023). Evolving LVC to include evaluation of human-AI teaming dynamics. In Proceedings of the 2023 Winter Simulation Conference (pp. 2506–2517).

McLean, T., Hoke, J., Vogl, T., & Schnell, T. (2013). LVCA: An integrated architecture of live, virtual, constructive and automated elements for UAS experimentation and training. In AUVSI Unmanned Systems North America 2013 Conference Proceedings.

Müller, W., Segor, F., Mühlenberg, D., Driessen, B., De Geus, V., Papp, Z., Svenmarck, P., Kalis, A., & Amditis, A. (2023). Interoperability of armed forces unmanned systems: the INTERACT project. In Open Architecture/Open Business Model Net-Centric Systems and Defense Transformation 2023 (Proc. SPIE Vol. 12544, pp. 205–216).

NATO Science and Technology Organization (STO), Human Factors and Medicine Panel (HFM). (2025). Human Systems Integration for Meaningful Human Control over AI-based Systems. STO‑TR‑HFM‑330 (pre‑release).

NATO Standardization Office (NSO). (2023). APP‑6E: NATO Joint Military Symbology. STANAG 2019 (Edition E, Version 1).

Prinz, L. M., Bhattacharya, S., & Witte, T. (2022). Virtualisierte Operationszentrale für Unbemannte Systeme (VOPZ UAS). Project report E/E610/LG042/KF079, Fraunhofer FKIE.

Prinz, L. M., & Mathew, T. (2024). Support lines and grids for depth ordering in indoor augmented reality using optical see-through head-mounted displays. In Proceedings of the 2024 ACM Symposium on Spatial User Interaction.

Prinz, L. M., Mathew, T., & Klöckner, J. (2024). NIMBUS – Nutzer-Interface für Multidomainoperationen der Bundeswehr mit Unbemannten Systemen. Base phase 2024 – Usage contexts and current-state assessment. Interim report E/E610/QG006/KF079, Fraunhofer FKIE.

Prinz, L. M., Mathew, T., & Klöckner, J. (2025). NIMBUS – Nutzer-Interface für Multidomainoperationen der Bundeswehr mit Unbemannten Systemen. Option 2025 – Mixed reality interaction concept. Final report E/E610/QG006/KF079, Fraunhofer FKIE.

Prinz, L. M., & Mathew, T. (2026, March). Operationalizing meaningful human control for human-robot interface design. In Companion Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (pp. 361–366).

Rule, A., & Forlizzi, J. (2012). Designing interfaces for multi-user, multi-robot systems. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction (pp. 97–104).

Schoenherr, J. R. (2025). Meaningful human control of autonomous weapons systems: Translating functional affordances to inform ethical assessment and design. In M. E. Erendor (Ed.), Cyber Security in the Age of Artificial Intelligence and Autonomous Weapons (pp. 173–199). CRC Press.

Sibley, C., Coyne, J. T., & Morrison, J. (2015). Research considerations for managing future unmanned systems. In AAAI Spring Symposia.

UxV is a placeholder term and is generally interpreted as “Unmanned X Vehicle”.

UAV: Unmanned Aerial Vehicle

UGV: Unmanned Ground Vehicle

USV: Unmanned Surface Vehicle  

UUV: Unmanned Underwater Vehicle