Get some good leather motorcycle gloves with protection for the palms and knuckles. Some even have Kevlar in the palms; most have thick leather or even Kevlar over the knuckles. They should also have a gauntlet: that big thing that comes up your arms a few inches past your wrist and seals with Velcro. You should tuck your jacket into the gauntlet while you ride. This gives you some overlap in case of asphalt sliding. (The same goes for zipping the pants and jacket together.) In addition to Velcro at the jacket end of the glove, there should be a wrist strap that wraps around and further helps hold the glove in place.
Don't get those little gloves that end right at your wrist bones. They may be OK for Tiger Woods, but they won't give you enough protection in a crash.
Lots of companies make gloves, and lots of shops sell them. That's why there are no specific recommendations here. Make sure they fit, though. They should be a little difficult to remove even when the velcro straps aren't closed. And they definitely should stay on once you close the retainers. If you can pull them off when they're locked in place, they're too big. The leather should stretch a little after you wear them a while.
Don't make the common mistake of getting gloves that are too big. They should be snug but comfortable, just like a helmet. If the fingers are too long, they'll hook on the levers and otherwise get in the way. If they're too big in the palms, they'll bunch up between the bars and your hands. This can get quite uncomfortable.
Gloves are for protection. Make sure they fit, or take or send them back.
More information on glove fit at Sport Rider Magazine
Gloves in the rain
There's no such thing as a waterproof glove; they have too many seams. It's best to just use your normal gloves and add a layer over the top. The cheapest are latex or nitrile gloves, or regular old rubber gloves from a grocery or hardware store. You can also go to Aerostich and get their Triple Digit Rain Covers. Hippo Hands fit on the bike and are another solution.
For cold rainy days, try neoprene kayak paddler's gloves under your armored gloves. Your hands don't stay completely dry, but warm and comfortable. And they're "pre-curved" to wrap around the handle bar grips. Look at REI or NRS.
It doesn't hurt anything to get your gloves wet. You can just air-dry them, or use a fan (not a clothes dryer). Wet gloves shouldn't be really uncomfortable unless you're on a long ride in very cold weather.
Should you decide to, say, leave your gloves out in the humid garage for a couple months, you may not like the way they smell. We suggest stuffing something fragrant inside them for a while, such as a dryer sheet, a solid air freshener, or a little hotel coffee pouch. It's preferable that you do this while you're not riding your bike.