You may need to use a footstool in between the walkable area of the fortress and the ledge. Also make sure you're using the right type of ladder.