Knowledge Graph Fusion for Cross-Modal Semantic Communication

This paper proposes a knowledge graph-enhanced multi-source fusion (KG-MSF) scheme, a novel cross-modal semantic communication system to robustly fuse visual and textual data for tasks such as visual question answering (VQA) over wireless channels. The proposed KG-MSF scheme integrates knowledge graph reasoning into a multi-stage fusion and encoding pipeline, utilizing bidirectional cross attention between modalities and structured semantic triplets to enhance semantic preservation and resilience to channel impairments. Specifically, image objects and question tokens are first aligned via cross-modal attention, then enriched with shallow and deep semantic triplets extracted through knowledge graphs, which are subsequently fused and transmitted using joint source-channel coding. Extensive simulation results are provided to demonstrate that the proposed KG-MSF scheme significantly outperforms the competing ones under both AWGN and Rayleigh fading channels, indicating KG-MSF’s superior semantic robustness and efficient cross-modal reasoning in wireless environments.