Charlotte, l love the stone and I think it is smart to keep it. I also really love the wall paper. I think if you painted the stone white and places some of those white animal bust above the fire place it would look really nice. I also think you could build a simple built in shelf on the left side of the fireplace. That would take away from the weird indent between the brick chimney and the fireplace.
Here are some images I found
painted stone
wall paper fireplace 2
wallpaper fireplace
animal bust