Mandarin dragonet [aka mandarin goby] is a fish that requires a large tank for it's feeding habits. They will not commonly accept prepared or processed foods, so they eat the copepods and amphipods within an aquarium to survive. Nano reefs are simply not large enough to provide a replenishing supply of food for mandarins, and they should be avoided.
Other common fish that simply require a much larger tank than any nano reef can provide include; tangs, surgeon fish, butterfly fish, lionfish, puffer fish, angel fish, and dwarf angels. Please keep in mind that while some people may keep these fish in their nano reef, that doesn't make it right. Some people will also keep these fish in their nano reefs for a short period of time, and move them to one of their larger tanks. This should be avoided.
Carnivorous star fish should not be kept, as they can easily corner a fish or shrimp in such a small tank and eat it. Nudibranches and sea slugs are also something to avoid, as certain types can be toxic, and have special needs. Turbo snails are alright in larger nano reefs, but older turbo snails can sometimes become very large and knock over corals. If you already have turbo snails, I would not be very concerned about them, but if you are just starting your tank, go for astrea snails.
Gonipora, aka flowerpot coral, will not survive easily in captivity, especially not in a nano reef. Their exact needs are not entirely known at this time, but whatever it is that is missing within our tanks is enough to kill them in a matter of months.
Corals with stinging sweeper tentacles such as galaxia and frogspawn corals should be avoided in smaller tanks. If they are kept near other corals, their sweeping tentacles will come out and night and sting or kill the surrounding corals. These will do ok in larger nano reefs, as long as they are somewhat secluded to a certain area.