Saltwater Shrimp messing with feather duster?

I have a Feather duster ( invertebrate ) in the sand in my reef tank. Recently the shrimp in the tank have been messing with it * cleaner shrimp, peppermints * They will scare the featherduster into its tube and then pick at it in there.....what do i do?? The poor featherduster comes back out and it's all natted and they won't leave it alone.