My violin end button is crooked, can I fix it?

I just recently got my violin and, granted it was used, but I noticed the end button is crooked in the hole. I took it to the nearest place I could think of but they didn't have any luthiers to look at it. They did suspect that it was probably dropped under tension. It isn't broken though that I can see. If I were to take the strings off, could I simply push the button back in or do I absolutely need to have a luthier look at it?

Also, is it doing serious damage to my instrument as it is (it hasn't affected the sound that I can tell, though I'm no professional?