why won't they leave the island*

They leave via a boat that shows up when you StreetPass someone. Sometimes they won't leave which is either a visual glitch or downloading a pass that was stored for a while in NetPass. That's what I've noticed from my experience