There are a surprising number of plants you can grow in the the garden, even when you are occasionally visited by deer. The deer in my garden tend to leave the following plants alone. There are a number of plants they will nibble on occasionally but not really harm, such as the summer spireas, golden chain tree, ornamental cherry trees, camellias, red maples, japanese maples, smokebush, mock orange, and several more.
I have found deer tend to not prefer plants with strong scented foliage or ones with rough foliage. Herbs do well. With plenty of natives in the area, they only cruise through my back yard like a trip through the salad bar, picking up a few things that look tasty. In winter when they are hungry, they are a lot less picky.
For a full sun garden:
For a part sun garden: