Driver vigilance estimation plays a critical role in preventing fatigue-related traffic incidents. Current multimodal methods leveraging EEG and EOG signals often suffer from high computational costs due to the reliance on self-attention mechanisms like Transformers. To address these challenges, we propose a novel framework, Selective Attention Sequential Model (SAMM), which integrates a dynamic channel attention mechanism and the Mamba sequence modeling approach. By replacing traditional Transformer modules with Mamba's selective state spaces, our model achieves linear-time complexity while effectively capturing both local and global features.The SAMM framework fuses EEG and EOG signals using early fusion and employs a deep channel attention mechanism to enhance localized feature extraction. Mamba further complements this by efficiently modeling global dependencies in multimodal data, thus reducing computational costs while maintaining high accuracy. Extensive experiments on public datasets, SEED-VIG and SADT, demonstrate that SAMM achieves state-of-the-art performance with a significant reduction in inference time.