Well, if you have space in the "container" (and I think that means the shelf where mugs of pens can go, not a single mug?), then I'm pretty sure Dana K. White would say keeping them is fine! Especially if they bring up nice memories of when/where you bought them when you use them. :-)