The Palace of Mysteries is not Level 5, just to be clear. You will not be able to destroy the brick walls for a little while. If you look closely on your map, you have actually seen the Level 5 entrance somewhere on the other side of the river, but have not yet been able to access it.