As a nano reef hobbyist, you must realize that a nano reef does have certain limitations as to what can be healthily kept within it. There are certain corals, invertebrates, and fish that should not be kept in a nano reef, due to their instincts, size, habits, or quick growth. The following are some more common items that hobbyists should not keep in their nano reef. Please keep in mind that this does not include all things that should be avoided. Even though some of these animals might be able to live in a nano reef environments, it is our responsibility as hobbyists to give them the best environment possible for their well being.
Mandarin dragonet [aka mandarin goby] is a fish that requires a large tank for it's feeding habits. They will not commonly accept prepared or processed foods, so they eat the copepods and amphipods within an aquarium to survive. Nano reefs are simply not large enough to provide a replenishing supply of food for mandarins, and they should be avoided.
Other common fish that simply require a much larger tank than any nano reef can provide include; tangs, surgeon fish, butterfly fish, lionfish, puffer fish, angel fish, and dwarf angels. Please keep in mind that while some people may keep these fish in their nano reef, that doesn't make it right. Some people will also keep these fish in their nano reefs for a short period of time, and move them to one of their larger tanks. This should be avoided.
Carnivorous star fish should not be kept, as they can easily corner a fish or shrimp in such a small tank and eat it. Nudibranches and sea slugs are also something to avoid, as certain types can be toxic, and have special needs. Turbo snails are alright in larger nano reefs, but older turbo snails can sometimes become very large and knock over corals. If you already have turbo snails, I would not be very concerned about them, but if you are just starting your tank, go for astrea snails.
Gonipora, aka flowerpot coral, will not survive easily in captivity, especially not in a nano reef. Their exact needs are not entirely known at this time, but whatever it is that is missing within our tanks is enough to kill them in a matter of months.
Corals with stinging sweeper tentacles such as galaxia and frogspawn corals should be avoided in smaller tanks. If they are kept near other corals, their sweeping tentacles will come out and night and sting or kill the surrounding corals. These will do ok in larger nano reefs, as long as they are somewhat secluded to a certain area.