Journal / Conference
Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, 2019)
[PDF link: link]
[Code link: link]
Keywords
Action Recognition, Deep Reinforcement Learning, Key Frame
Abstract
Deep learning based methods have achieved remarkable progress in action recognition. Existing works mainly focus on designing novel deep architectures to achieve video representations learning for action recognition. Most methods treat sampled frames equally and average all the frame-level predictions at the testing stage. However, within a video, discriminative actions may occur sparsely in a few frames and most other frames are irrelevant to the ground truth and may even lead to a wrong prediction. As a result, we think that the strategy of selecting relevant frames would be a further important key to enhance the existing deep learning based action recognition. In this paper, we propose an attention-aware sampling method for action recognition, which aims to discard the irrelevant and misleading frames and preserve the most discriminative frames. We formulate the process of mining key frames from videos as a Markov decision process and train the attention agent through deep reinforcement learning without extra labels. The agent takes features and predictions from the baseline model as input and generates importance scores for all frames. Moreover, our approach is extensible, which can be applied to different existing deep learning based action recognition models. We achieve very competitive action recognition performance on two widely used action recognition datasets.
Method/Framework
Process of training attention-aware sampling (AS) agent via deep reinforcement learning (spatial stream). The AS agent receives state and takes action to select part of the video. The feedback reward is computed based on the difference between the original video-level prediction and the new prediction. At inference stage, the probabilities are viewed as importance scores for each frame and are used to pick the most discriminative frames.
Experiments
We evaluate our approach on two challenging action recognition datasets: UCF101 and HMDB51. Extensive experimental results show that the proposed method is effective and achieves the competitive results.
Highlight
- We discover a novel problem for action recognition, which is that irrelevant frames should be differentiated and discarded during the testing stage.
- To address this issue, we propose an attention-aware sampling method to select discriminative frames in videos, where the agent is trained by DRL.
- We conduct experiments on two widely used benchmark datasets to demonstrate the effectiveness of our method and achieve competitive results.
Citation
@InProceedings{AS_2019_AAAI,
author = {Dong, Wenkai and Zhang, Zhaoxiang and Tan, Tieniu},
title = {Attention-aware Sampling via Deep Reinforcement Learning for Action Recognition},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2019}
}
