I’d say that the wood lovers are the most dangerous for 2 reasons. There is a phenomenon called wood lovers paralysis that occurs with large doses of those species that are found growing on wood. It appears to be most associated with Psilocybe Azurescens, but has been reports with Psilocybe Cyanescens and Psilocybe Ovoideocystidiata as well. There are also extremely poisonous look alikes. Gallerina Marginata and other members of the Gallerina genus can be pretty hard to distinguish from many members of the Psilocybe genus. Most psilocybin containing mushrooms will stain blue when bruised even slightly. This blueing should be significant and does not occur with Gallerina mushrooms, however, you do need yo positively ID each individual mushroom as they share the same habitat and distribution and flush at the same time. I have personally observed them fruiting in the middle of a patch of Psilocybe Ovoideocystidiata. The potential for mistakes is a lot higher than people realize. Essentially all poisonings associated with psychedelic mushrooms have been a case of misidentification. Among those that I know of being commonly used recreationally there are none that are thought of as poisonous on their own. However, the habitats in which they occur can really lend themselves to contamination. Especially with Psilocybe Ovoideocystidiata. They grow in some pretty disgusting environments, filled with goose poop and standing muck. Furthermore, they also share environments with copperheads, which I have personally encountered and almost stepped on. So these are mushrooms not without their own perils. All of which should be taken in as necessary parts of an environment that are working together and should demand respect.