Researcher profile

Tobias Fischer

Tobias Fischer contributes to research discovery and scholarly infrastructure.

ResearcherAffiliation not importedOpen to collaborate

Trust snapshot

Quick read

Trust 21 - EmergingVerification L1Unclaimed author
23works
0followers
14topics
4close collaborators

Actions

Decide how to stay connected

Follow researcher0

Identity and collaboration

How to connect with this researcher

Claiming links this public author record to a researcher profile and unlocks direct collaboration workflows.

Log in to claim

Direct collaboration

Open a focused conversation when the fit is right

Claim this author entity first to unlock direct invitations.

Research graph

See the researcher in context

Open full explorer

Inspect adjacent work, topics, institutions and collaborators without jumping out to a separate graph page.

Building this graph slice

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Published work

23 published item(s)

preprint2026arXiv

Ensemble-Based Event Camera Place Recognition Under Varying Illumination

Compared to conventional cameras, event cameras provide a high dynamic range and low latency, offering greater robustness to rapid motion and challenging lighting conditions. Although the potential of event cameras for visual place recognition (VPR) has been established, developing robust VPR frameworks under severe illumination changes remains an open research problem. In this paper, we introduce an ensemble-based approach to event camera place recognition that combines sequence-matched results from multiple event-to-frame reconstructions, VPR feature extractors, and temporal resolutions. Unlike previous event-based ensemble methods, which only utilise temporal resolution, our broader fusion strategy delivers significantly improved robustness under varied lighting conditions (e.g., afternoon, sunset, night), achieving a 57% relative improvement in Recall@1 across day-night transitions. We evaluate our approach on two long-term driving datasets (with 8 km per traverse) without metric subsampling, thereby preserving natural variations in speed and stop duration that influence event density. We also conduct a comprehensive analysis of key design choices, including binning strategies, polarity handling, reconstruction methods, and feature extractors, to identify the most critical components for robust performance. Additionally, we propose a modification to the standard sequence matching framework that enhances performance at longer sequence lengths. To facilitate future research, we will release our codebase and benchmarking framework.

preprint2026arXiv

ESO Expanding Horizon White Paper: Revealing the properties of matter at supranuclear densities with gravitational waves

Understanding dense matter under extreme conditions is one of the most fundamental puzzles in modern physics. Complex interactions give rise to emergent, collective phenomena. While nuclear experiments and Earth - based colliders provide valuable insights, much of the quantum chromodynamics phase diagram at high density and low temperature remains accessible only through astrophysical observations of neutron stars, neutron star mergers, and stellar collapse. Astronomical observations thus offer a direct window to the physics on subatomic scales with gravitational waves presenting an especially clean channel. Next-generation gravitational - wave observatories, such as the Einstein Telescope, would serve as unparalleled instruments to transform our understanding of neutron star matter. They will enable the detection of up to tens of thousands of binary neutron star and neutron star - black hole mergers per year, a dramatic increase over the few events accessible with current detectors. They will provide an unprecedented precision in probing cold, dense matter during the binary inspiral, exceeding by at least an order of magnitude what current facilities can achieve. Moreover, these observatories will allow us to explore uncharted regimes of dense matter at finite temperatures produced in a subset of neutron star mergers, areas that remain entirely inaccessible to current instruments. Together with multimessenger observations, these measurements will significantly deepen our knowledge of dense nuclear matter.

preprint2026arXiv

Human-in-the-Loop Segmentation of Multi-species Coral Imagery

Marine surveys by robotic underwater and surface vehicles result in substantial quantities of coral reef imagery, however labeling these images is expensive and time-consuming for domain experts. Point label propagation is a technique that uses existing images labeled with sparse points to create augmented ground truth data, which can be used to train a semantic segmentation model. In this work, we show that recent advances in large foundation models facilitate the creation of augmented ground truth masks using only features extracted by the denoised version of the DINOv2 foundation model and K-Nearest Neighbors (KNN), without any pre-training. For images with extremely sparse labels, we use human-in-the-loop principles to enhance annotation efficiency: if there are 5 point labels per image, our method outperforms the prior state-of-the-art by 19.7% for mIoU. When human-in-the-loop labeling is not available, using the denoised DINOv2 features with a KNN still improves on the prior state-of-the-art by 5.8% for mIoU (5 grid points). On the semantic segmentation task, we outperform the prior state-of-the-art by 13.5% for mIoU when only 5 point labels are used for point label propagation. Additionally, we perform a comprehensive study into the number and placement of point labels, and make several recommendations for improving the efficiency of labeling images with points.

preprint2026arXiv

Robust Scene Coordinate Regression via Geometrically-Consistent Global Descriptors

Recent learning-based visual localization methods use global descriptors to disambiguate visually similar places, but existing approaches often derive these descriptors from geometric cues alone (e.g., covisibility graphs), limiting their discriminative power and reducing robustness in the presence of noisy geometric constraints. We propose an aggregator module that learns global descriptors consistent with both geometrical structure and visual similarity, ensuring that images are close in descriptor space only when they are visually similar and spatially connected. This corrects erroneous associations caused by unreliable overlap scores. Using a batch-mining strategy based solely on the overlap scores and a modified contrastive loss, our method trains without manual place labels and generalizes across diverse environments. Experiments on challenging benchmarks show substantial localization gains in large-scale environments while preserving computational and memory efficiency. Code is available at https://github.com/sontung/robust_scr.

preprint2026arXiv

Ultrasensitive Real-Time Detection of SARS-CoV-2 Proteins with Arrays of Biofunctionalized Graphene Field-Effect Transistors

With the growing interest in graphene field-effect transistors (GFETs) for biosensing applications, there is a strong demand for strategies enabling flexible and multiplexed biofunctionalization, as well as highly parallel, real-time electronic readout integrated with microfluidic control. Here we present a methodology that addresses these challenges by enabling real-time, parallel monitoring of multiple GFETs integrated on a single microfabricated chip within an automated electronic and microfluidic platform. We demonstrate the capabilities of this approach through ultrasensitive detection of the SARS-CoV-2 spike (S) and nucleocapsid (N) proteins. GFET chips are functionalized via van der Waals assembly using 1 nm-thick molecular two-dimensional (2D) materials - carbon nanomembranes - which enable multiplexed biofunctionalization. The chips are integrated into a custom-developed microelectronic and microfluidic system that allows parallel, real-time, and automated measurements of 15 GFETs. We present in situ biofunctionalization of the GFETs with antibodies, followed by highly specific detection of the S- and N-proteins with limits of detection down to 10 aM and a dynamic range spanning four orders of magnitude. Owing to its versatility, the presented methodology is readily adaptable for sensing a wide range of biological and chemical targets.

preprint2026arXiv

Why Domain Matters: A Preliminary Study of Domain Effects in Underwater Object Detection

Domain shift, where deviations between training and deployment data distributions degrade model performance, is a key challenge in underwater environments. Existing benchmarks testing performance for underwater domain shift simulate variability through synthetic style transfer. This fails to capture intrinsic scene factors such as visibility, illumination, scene composition, or acquisition factors, limiting analysis of real-world effects. We propose a labeling framework that defines underwater domains using measurable image, scene, and acquisition characteristics. Unlike prior benchmarks, it captures physically meaningful factors, enabling semantically consistent image grouping and supporting domain-specific evaluation of detection performance including failure analysis. We validate this on public datasets, showing systematic variations across domain factors and revealing hidden failure modes.

preprint2022arXiv

3D template-based $Fermi$-LAT constraints on the diffuse supernova axion-like particle background

Axion-like particles (ALPs) may be abundantly produced in core-collapse (CC) supernovae (SNe), hence the cumulative signal from all past SN events can create a diffuse flux peaked at energies of about 25~MeV. We improve upon the modeling of the ALPs flux by including a set of CC SN models with different progenitor masses, as well as the effects of failed CC SNe -- which yield the formation of black holes instead of explosions. Relying on the coupling strength of ALPs to photons and the related Primakoff process, the diffuse SN ALP flux is converted into gamma rays while traversing the magnetic field of the Milky Way. The spatial morphology of this signal is expected to follow the shape of the Galactic magnetic field lines. We make use of this via a template-based analysis that utilizes 12 years of $Fermi$-LAT data in the energy range from 50 MeV to 500 GeV. In our benchmark case of the realization of astrophysical and cosmological parameters, we find an upper limit of $g_{aγ} \lesssim 3.76\times10^{-11}\;\mathrm{GeV}^{-1}$ at 95$\%$ confidence level for $m_a \ll 10^{-11}$ eV, while we find that systematic deviations from this benchmark scenario induce an uncertainty as large as about a factor of two. Our result slightly improves the CAST bound, while still being a factor of six (baseline scenario) weaker than the SN1987A gamma-ray burst limit.

preprint2022arXiv

Hyperdimensional Feature Fusion for Out-Of-Distribution Detection

We introduce powerful ideas from Hyperdimensional Computing into the challenging field of Out-of-Distribution (OOD) detection. In contrast to most existing work that performs OOD detection based on only a single layer of a neural network, we use similarity-preserving semi-orthogonal projection matrices to project the feature maps from multiple layers into a common vector space. By repeatedly applying the bundling operation $\oplus$, we create expressive class-specific descriptor vectors for all in-distribution classes. At test time, a simple and efficient cosine similarity calculation between descriptor vectors consistently identifies OOD samples with better performance than the current state-of-the-art. We show that the hyperdimensional fusion of multiple network layers is critical to achieve best general performance.

preprint2022arXiv

Point Label Aware Superpixels for Multi-species Segmentation of Underwater Imagery

Monitoring coral reefs using underwater vehicles increases the range of marine surveys and availability of historical ecological data by collecting significant quantities of images. Analysis of this imagery can be automated using a model trained to perform semantic segmentation, however it is too costly and time-consuming to densely label images for training supervised models. In this letter, we leverage photo-quadrat imagery labeled by ecologists with sparse point labels. We propose a point label aware method for propagating labels within superpixel regions to obtain augmented ground truth for training a semantic segmentation model. Our point label aware superpixel method utilizes the sparse point labels, and clusters pixels using learned features to accurately generate single-species segments in cluttered, complex coral images. Our method outperforms prior methods on the UCSD Mosaics dataset by 3.62% for pixel accuracy and 8.35% for mean IoU for the label propagation task, while reducing computation time reported by previous approaches by 76%. We train a DeepLabv3+ architecture and outperform state-of-the-art for semantic segmentation by 2.91% for pixel accuracy and 9.65% for mean IoU on the UCSD Mosaics dataset and by 4.19% for pixel accuracy and 14.32% mean IoU for the Eilat dataset.

preprint2022arXiv

Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments

Spiking neural networks (SNNs) offer both compelling potential advantages, including energy efficiency and low latencies and challenges including the non-differentiable nature of event spikes. Much of the initial research in this area has converted deep neural networks to equivalent SNNs, but this conversion approach potentially negates some of the advantages of SNN-based approaches developed from scratch. One promising area for high-performance SNNs is template matching and image recognition. This research introduces the first high-performance SNN for the Visual Place Recognition (VPR) task: given a query image, the SNN has to find the closest match out of a list of reference images. At the core of this new system is a novel assignment scheme that implements a form of ambiguity-informed salience, by up-weighting single-place-encoding neurons and down-weighting "ambiguous" neurons that respond to multiple different reference places. In a range of experiments on the challenging Nordland, Oxford RobotCar, SPEDTest, Synthia, and St Lucia datasets, we show that our SNN achieves comparable VPR performance to state-of-the-art and classical techniques, and degrades gracefully in performance with an increasing number of reference places. Our results provide a significant milestone towards SNNs that can provide robust, energy-efficient, and low latency robot localization.

preprint2021arXiv

Core-collapse supernova simulations and the formation of neutron stars, hybrid stars, and black holes

We investigate observable signatures of a first-order quantum chromodynamics (QCD) phase transition in the context of core collapse supernovae. To this end, we conduct axially symmetric numerical relativity simulations with multi-energy neutrino transport, using a hadron-quark hybrid equation of state (EOS). We consider four non-rotating progenitor models, whose masses range from $9.6$ to $70$ M$_\odot$. We find that the two less massive progenitor stars (9.6 and 11.2 M$_\odot$) show a successful explosion, which is driven by the neutrino heating. They do not undergo the QCD phase transition and leave behind a neutron star (NS). As for the more massive progenitor stars (50 and 70 M$_\odot$), the proto-neutron star (PNS) core enters the phase transition region and experiences the second collapse. Because of a sudden stiffening of the EOS entering to the pure quark matter regime, a strong shock wave is formed and blows off the PNS envelope in the 50 M$_\odot$ model. Consequently, the remnant becomes a quark core surrounded by hadronic matters, leading to the formation of the hybrid star. However for the 70 M$_\odot$ model, the shock wave cannot overcome the continuous mass accretion and it readily becomes a black hole. We find that the neutrino and gravitational wave (GW) signals from supernova explosions driven by the hadron-quark phase transition are detectable for the present generation of neutrino and GW detectors. Furthermore, the analysis of the GW detector response reveals unique kHz signatures, which will allow us to distinguish this class of supernova explosions from failed and neutrino-driven explosions.

preprint2021arXiv

Intelligent Reference Curation for Visual Place Recognition via Bayesian Selective Fusion

A key challenge in visual place recognition (VPR) is recognizing places despite drastic visual appearance changes due to factors such as time of day, season, weather or lighting conditions. Numerous approaches based on deep-learnt image descriptors, sequence matching, domain translation, and probabilistic localization have had success in addressing this challenge, but most rely on the availability of carefully curated representative reference images of the possible places. In this paper, we propose a novel approach, dubbed Bayesian Selective Fusion, for actively selecting and fusing informative reference images to determine the best place match for a given query image. The selective element of our approach avoids the counterproductive fusion of every reference image and enables the dynamic selection of informative reference images in environments with changing visual conditions (such as indoors with flickering lights, outdoors during sunshowers or over the day-night cycle). The probabilistic element of our approach provides a means of fusing multiple reference images that accounts for their varying uncertainty via a novel training-free likelihood function for VPR. On difficult query images from two benchmark datasets, we demonstrate that our approach matches and exceeds the performance of several alternative fusion approaches along with state-of-the-art techniques that are provided with prior (unfair) knowledge of the best reference images. Our approach is well suited for long-term robot autonomy where dynamic visual environments are commonplace since it is training-free, descriptor-agnostic, and complements existing techniques such as sequence matching.

preprint2021arXiv

Muonization of supernova matter

The present article investigates the impact of muons on core-collapse supernovae, with particular focus on the early muon neutrino emission. While the presence of muons is well understood in the context of neutron stars, until the recent study by Bollig et al. [Phys. Rev. Lett. 119, 242702 (2017)] the role of muons in core-collapse supernovae had been neglected--electrons and neutrinos were the only leptons considered. In their study, Bollig et al. disentangled the muon and tau neutrinos and antineutrinos and included a variety of muonic weak reactions, all of which the present paper follows closely. Only then does it becomes possible to quantify the appearance of muons shortly before stellar core bounce and how the post-bounce prompt neutrino emission is modified

preprint2021arXiv

Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition

Visual Place Recognition is a challenging task for robotics and autonomous systems, which must deal with the twin problems of appearance and viewpoint change in an always changing world. This paper introduces Patch-NetVLAD, which provides a novel formulation for combining the advantages of both local and global descriptor methods by deriving patch-level features from NetVLAD residuals. Unlike the fixed spatial neighborhood regime of existing local keypoint features, our method enables aggregation and matching of deep-learned local features defined over the feature-space grid. We further introduce a multi-scale fusion of patch features that have complementary scales (i.e. patch sizes) via an integral feature space and show that the fused features are highly invariant to both condition (season, structure, and illumination) and viewpoint (translation and rotation) changes. Patch-NetVLAD outperforms both global and local feature descriptor-based methods with comparable compute, achieving state-of-the-art visual place recognition results on a range of challenging real-world datasets, including winning the Facebook Mapillary Visual Place Recognition Challenge at ECCV2020. It is also adaptable to user requirements, with a speed-optimised version operating over an order of magnitude faster than the state-of-the-art. By combining superior performance with improved computational efficiency in a configurable framework, Patch-NetVLAD is well suited to enhance both stand-alone place recognition capabilities and the overall performance of SLAM systems.

preprint2020arXiv

Charged-Current Muonic Reactions in Core-Collapse Supernovae

The steady advance in core-collapse supernova simulations requires a more precise description of neutrino processes in hot and dense matter. In this work, we study the rates of charged-current (CC) weak processes with (anti)muons in supernova matter. At the relativistic mean field level, we derive results for the rates of CC neutrino-nucleon reactions, taking into account full kinematics, weak magnetism and pseudoscalar terms, and $q^2$-dependent nucleon form factors in the hadronic current. In addition to muonic semileptonic processes we also consider purely leptonic processes. In particular, we show that inverse muon decay can dominate the opacities for low energy $ν_μ$ and $\barν_e$ at densities $\gtrsim 10^{13}~\rm{g~ cm^{-3}}$.

preprint2020arXiv

Constraining the onset density of the hadron-quark phase transition with gravitational-wave observations

We study the possible occurrence of the hadron-quark phase transition (PT) during the merging of neutron star binaries by hydrodynamical simulations employing a set of temperature dependent hybrid equations of state (EoSs). Following previous work we describe an unambiguous and measurable signature of deconfined quark matter in the gravitational-wave (GW) signal of neutron star binary mergers including equal-mass and unequal-mass systems of different total binary mass. The softening of the EoS by the PT at higher densities, i.e. after merging, leads to a characteristic increase of the dominant postmerger GW frequency f_peak relative to the tidal deformability Lambda inferred during the premerger inspiral phase. Hence, measuring such an increase of the postmerger frequency provides evidence for the presence of a strong PT. If the postmerger frequency and the tidal deformability are compatible with results from purely baryonic EoS models yielding very tight relations between f_peak and Lambda, a strong PT can be excluded up to a certain density. We find tight correlations of f_peak and Lambda with the maximum density during the early postmerger remnant evolution. These GW observables thus inform about the density regime which is probed by the remnant and its GW emission. Exploiting such relations we devise a directly applicable, concrete procedure to constrain the onset density of the QCD PT from future GW measurements. We point out two interesting scenarios: if no indications for a PT are inferred from a GW detection, our procedure yields a lower limit on the onset density of the hadron quark PT. On the contrary, if a merger event reveals evidence for the occurrence of deconfined quark matter, the inferred GW parameters set an upper limit on the PT onset density. (abridged)

preprint2020arXiv

Core-collapse Supernova Explosions Driven by the Hadron-quark Phase Transition as a Rare $r$-process Site

Supernova explosions of massive stars are one of the primary sites for the production of the elements in the universe. Up to now, stars with zero-age main-sequence masses in the range of 35--50~$M_\odot$ had mostly been representing the failed supernova explosion branch. In contrast, it has been demonstrated recently that the appearance of exotic phases of hot and dense matter, associated with a sufficiently strong phase transition from nuclear matter to the quark-gluon plasma at high baryon density, can trigger supernova explosions of such massive supergiant. Here, we present the first results obtained from an extensive nucleosynthesis analysis for material being ejected from the surface of the newly born proto-neutron star of such supernova explosions. These ejecta contain an early neutron-rich component and a late-time high-entropy neutrino-driven wind. The nucleosynthesis robustly overcomes the production of nuclei associated with the second $r$-process peak, at nuclear mass number $A\simeq 130$, and proceeds beyond the formation of the third peak ($A\simeq 195$) to the actinides. These yields may account for metal-poor star observations concerning $r$-process elements such as strontium and europium in the Galaxy at low metalicity, while the actinide yields suggests that this source may be a candidate contributing to the abundances of radioactive $^{244}$Pu measured in deep-sea sediments on Earth.

preprint2020arXiv

Event-based visual place recognition with ensembles of temporal windows

Event cameras are bio-inspired sensors capable of providing a continuous stream of events with low latency and high dynamic range. As a single event only carries limited information about the brightness change at a particular pixel, events are commonly accumulated into spatio-temporal windows for further processing. However, the optimal window length varies depending on the scene, camera motion, the task being performed, and other factors. In this research, we develop a novel ensemble-based scheme for combining temporal windows of varying lengths that are processed in parallel. For applications where the increased computational requirements of this approach are not practical, we also introduce a new "approximate" ensemble scheme that achieves significant computational efficiencies without unduly compromising the original performance gains provided by the ensemble approach. We demonstrate our ensemble scheme on the visual place recognition (VPR) task, introducing a new Brisbane-Event-VPR dataset with annotated recordings captured using a DAVIS346 color event camera. We show that our proposed ensemble scheme significantly outperforms all the single-window baselines and conventional model-based ensembles, irrespective of the image reconstruction and feature extraction methods used in the VPR pipeline, and evaluate which ensemble combination technique performs best. These results demonstrate the significant benefits of ensemble schemes for event camera processing in the VPR domain and may have relevance to other related processes, including feature tracking, visual-inertial odometry, and steering prediction in driving.

preprint2020arXiv

Improved axion emissivity from a supernova via nucleon-nucleon bremsstrahlung

The most efficient axion production mechanism in a supernova (SN) core is the nucleon-nucleon bremsstrahlung. This process has been often modeled at the level of the vacuum one-pion exchange (OPE) approximation. Starting from this naive recipe, we revise the calculation including systematically different effects, namely a non-vanishing mass for the exchanged pion, the contribution from the two-pions exchange, effective in-medium nucleon masses and multiple nucleon scatterings. Moreover, we allow for an arbitrary degree of nucleon degeneracy. A self consistent treatment of the axion emission rate including all these effects is currently missing. The aim of this work is to provide such an analysis. Furthermore, we demonstrate that the OPE potential with all the previous corrections gives rise to similar results as the on-shell T-matrix, and is therefore well justified for our and similar studies. We find that the axion emissivity is reduced by over an order of magnitude with respect to the basic OPE calculation, after all these effects are accounted for. The implications for the axion mass bound and the impact for the next generation experimental axion searches is also discussed.

preprint2020arXiv

Neutrino signal from proto-neutron star evolution: Effects of opacities from charged-current-neutrino interactions and inverse neutron decay

We investigate the impact of charged current neutrino processes on the formation and evolution of neutrino spectra during the deleptonization of proto-neutron stars. To this end we develop the full kinematics of these reaction rates consistent with the nuclear equation of state, including weak magnetism contributions. This allows us to systematically study the impact of inelastic contributions and weak magnetism on the $ν_e$ and $\barν_e$ luminosities and average energies. Furthermore, we explore the role of the inverse neutron decay, also known as the direct Urca process, on the emitted spectra of $\barν_e$. This process is commonly considered in the cooling scenario of cold neutron stars but has so far been neglected in the evolution of hot proto-neutron stars. We find that the inverse neutron decay becomes the dominating opacity source for low-energy $\barν_e$. Accurate three-flavor Boltzmann neutrino transport enables us to relate the magnitude of neutrino fluxes and spectra to details of the treatment of weak processes. This allows us to quantify the corresponding impact on the conditions relevant for the nucleosynthesis in the neutrino-driven wind, which is ejected from the proto-neutron star surface during the deleptonization phase.

preprint2020arXiv

Time of Flight and Supernova Progenitor Effects on the Neutrino Halo

We argue that the neutrino halo, a population of neutrinos that have undergone direction-changing scattering in the stellar envelope of a core-collapse supernova (CCSNe), is sensitive to neutrino emission history through time of flight. We show that the constant time approximation, commonly used in calculating the neutrino halo, does not capture the spatiotemporal evolution of the halo neutrino population and that correcting for time of flight can produce conditions which may trigger fast neutrino flavor conversion. We also find that there exists a window of time early in all CCSNe where the neutrino halo population is sufficiently small that it may be negligible. This suggests that collective neutrino oscillation calculations which neglect the Halo may be well founded at sufficiently early times.

preprint2020arXiv

Track to Reconstruct and Reconstruct to Track

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstruction-based tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.

preprint2017arXiv

DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self

This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative exploration and manipulation of its environment, where the initiative can originate from both the human and the robot. The framework, based on a biologically-grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor learning algorithms, as well as planning abilities and an autobiographical memory. The architecture as a whole drives the robot behavior to solve the symbol grounding problem, acquire language capabilities, execute goal-oriented behavior, and express a verbal narrative of its own experience in the world. We validate our approach in human-robot interaction experiments with the iCub humanoid robot, showing that the proposed cognitive architecture can be applied in real time within a realistic scenario and that it can be used with naive users.