Should I be concerned about the oil pressure light flickering when I brake hard or idle?

Yes, the oil is the life line to the engine, without proper pressure or circulation the engine will die, no different than us humans. The reason why the light goes off is that the engine is running at higher RPM pushing more pressure to the sensor therefore it turn off the light. When the engine running at low RPM the oil pressure is low therefore it not enough pressure to turn off the switch. 1st. it cheap, replace the oil pressure switch and make sure the oil dip stick at proper level. If replacing oil pressure switch does not rectified the issue, the oil pump might be going bad and not pumping proper pressure. Replacing oil pump is a major job. remember, on any vehicle, if it run proper there should be no lights on and that is the reason why they put the lights there for a reason. Check and see you have a small oil leaks.