Go over short fence at the Bomb Flower spot. Face away from the fence and angle toward the right, then perform a backflip to land on the ledge. You don't need to move the bomb flower to perform the backflip, just stand next to it before flipping.
If you are relying on the Goron Brace/Gauntlets to enter Dodongo's Cavern, you can speed things up a little. When you toss the bomb flower over the short fence, a cutscene plays that prevents you from moving. If you backflip before the cutscene plays, you can drop down during the cutscene, saving some time. You can also get the Heart Piece above the cavern this way.