Is it OK to charge my lights with a mains USB adapter?

Yep. Any reputable branded 5 volt charger will work fine e.g. Iphone charger. In fact, it’ll probably charge your light heaps faster than your trusty old PC.