Does my battery need to be replaced?

I charge my doorbell and security cams with batteries. I usually connect it right to a any micro-USB charger. NORMALLY, during charging both the red and yellow (I think) lights turn on. When the battery is fully charged, everything is green. And I've been doing it this way for a couple years with several batteries throughout my 3 devices. But now, one of my batteries, when plugged in to be charged.. isn't lighting up at all. I've checked the cable, and another battery does appear to get charged. So I think the cable is good. Is this simply a case of one of my batteries just dying on me? And I need to buy a new one? thanks!