You should clean your floggers, or any leather item, once it is dirty. It may seem like an obvious answer, however it is also the most true.
If your leather items get dirty every time they are used, then they should be cleaned every time. If they don't get dirty during their use, then they don't need to be cleaned right away.
As odd as it may seem, floggers and leather toys that are used, but don't get dirty, can often be in better condition than their unused counterparts. Unused leather toys often get put into rubber bins to keep them safe from moisture while being store in a garage or closet space. While this seems ideal to protect the leather, it can actually cause them to dry out if they aren't conditioned semi-regularly (every few months depending on your climate) or grow mold if they are too well conditioned or stored in a moist environment. A breathable suitcase or cloth bag is best for storing them.
Toys that get used on skin regularly actually pick up body oils from the people using them, which can help condition the leather; it is skin after-all. Should they get covered in sweat, dirt, other bodily fluids, or anything else on them that makes you think they are dirty, go ahead and clean them with Saddle Soap or Glycerin Soap (whichever you prefer). Always make sure to condition them afterwards to prevent them from drying out.
Because climates, storing conditions, how often they're used, and a variety of other variables play a role in when your toys should be clean, it really does come down to cleaning them when they are dirty and conditioning them when they feel dry.