It is, when you have to take your eyes totally off the screen and focus somewhere else to do whatever else it is you need to do.
When the action is still rolling, you're compromising whatever's happening on-screen and potentially making sacrifices to look away and fiddle with the controller. I'd rather have a mini-HUD on-screen when this is going on so I can keep the in-game action in my periphery and then quickly back out of whatever action I'm taking if something important happens on the screen.
That can easily be the difference in making a kill or getting killed, making a save, or scoring a goal.