Multi-LLM Consensus Framework for Evaluating Banking-Sector NIDS Dataset Coverage of MITRE ATT&CK Techniques

Abstract

The systemic criticality of global banking networks has ren-dered them high-priority targets for advanced persistent threats, neces-sitating Network Intrusion Detection Systems (NIDS) whose operational effectiveness must extend beyond statistical accuracy. However, a signif-icant validation gap persists between experimental NIDS performance and real-world effectiveness: NIDS models that achieve high accuracy on standard benchmarks often fail in operational banking environments because generic datasets lack sector-specific patterns, such as SWIFT and ATM-related intrusions, that characterize real financial threats. To address this, the paper investigates a sector-aware evaluation method-ology that systematically assesses how well existing NIDS benchmark datasets cover the attack behaviors most relevant to banking infrastruc-ture. The methodology maps documented adversary behaviors from the MITRE ATT&CK knowledge base to NIDS benchmarks while enforcing the realistic sensor limitations defined by NIST SP 800-94. Leveraging a multi-LLM consensus engine with four state-of-the-art models, we evalu-ated 210 banking-specific adversary techniques to derive a baseline of 68 network-observable behaviors for systematic coverage analysis. Results across five benchmark datasets demonstrate that UNSW-NB15 achieves the highest utility with an 82.2% weighted coverage score (though only 18.4% reflects direct, technique-level evidence), while CIC-DDoS2019 re-veals an 89.9% blind spot for core banking behaviors. These findings es-tablish a reproducible foundation for sector-aware NIDS evaluation and highlight the urgent need for banking-native datasets.