my best guess as I have had something similar and what I found was some small caterpillars or some kind of larvael worm/ or fly maggot. You got to get out there at night with a flashlight and see whats up after dark. I should have saved the worm but I have plenty of odd flies in and out around the greenhouse. While lizards dont eat the bloom or the bud, there isnt any rules against them taking a bit of it if it contains a meal ya know.
I you want to stop the bugs you got to think like a bug