"When it rains, do you run from doorway to doorway, trying to stay dry, getting wet all the while? Or do you just accept the fact that it's raining and walk with dignity?"
Some powders are more susceptible to ambient heat in the chamber or environment than others. But in the end, excess heat is just one more variable introduced into accuracy that can cause change. It is no different than a slight variation in overall length / bullet seating depth, a slightly thicker neck on a case, etc. Accuracy is all about consistency. Minimizing variables in cartridge construction / reloading, shooting technique (< especially that one) and environment all translate on paper.
I'd agree on the consistency factor. My groups tend to drop 1-2 inches with more lateral movement as the barrel gets hot. For me it starts about mid-way through the 2nd 20 round mag, at a fire rate of maybe 3 rounds per minute interspersed with a 5 minute walk down the range and back.
Like Sarge said this is about the time you don't want to touch the barrel with bare skin.