Your down gear could be soaking up water just because it is dirty. First, try cleaning it with Nikwax Down Wash.Direct, and if it is fairly new, or recently re-waterproofed, the Durable Water Repellency (DWR) will be revitalized just from cleaning.
eBay