Is it OK to charge my lights with a mains USB adapter?

Yep. Any reputable branded 5 volt charger will work fine e.g. an Iphone charger. In fact, it’ll probably charge your light faster than a computer USB!