My first question is, Why did you put it back in the factory stock? And, yes, it can make that much difference. Then, like somebody said, back to basics. Is everything tight? Personally, I would put it back in the McMillan stock, be sure everything is tightened properly (bedding screws and scope rings and bases) and shoot it again. Not another two dozen rounds. Pick the load that shot best and put three to five more down the tube. That should tell you if you've gained anything. If not, I would probably be trying a different powder.
That's my plan. I put the wood stock on because it's pretty, wanted less of a tactical look. At this point I don't care how pretty it is, I just need it to shoot straight. I know it likes Varget, but case density is too low, and I think that was part of the problem in the hills (8500 ft). H1000 shoots well, but it's too much the other way, a compressed load with a regular BT, and with the TSX all copper bullets, too much powder. IMR 4831 is what Barnes said is most accurate in testing, case density is fine, running 73 grs max, so close to 80%, and I'm still cranking 3100 fps or so at muzzle. I need a big punch out over 500 yards for the elk with this 168gr bullet, so I want to stay on the hot side.
I've been reloading for more than ten years and this is the first time I've heard of annealing. So no, I pitch my cases when they crack. I don't sort by weight, never had to. I've always had loads shooting ragged holes at 100, under 2" at 200. If I need to do that stuff to get good results, I better hang it up. And I'm loading for hunting, I don't compete anymore - used to shoot silhouettes in Colorado - iron sights and off hand, if it fell over, I dun good. All I need to do is hit a pie plate sized target out to at least 500 yards. If I miss, I just send another one with a better hold.
Thanks for the helps, fellas.