I'm new to shooting higher powered calibers, so the recoil is taking some getting used to but its a nice challenge during lockdown

I've started with some Hornady 'Lites' 125gn which are grouping nicely from the bipods, but I also ran a couple of 180gn Federal blue rounds through it and noticed that they pulled considerably right (like.. 5inches at 100m). Still grouped ok, but I was surprised by the horizontal shift. Is that normal for a different load to shift that much on a bigger caliber? or is it just me not doing my bit? Need to get some ~150gn rounds and try them too..
I'm also keen to get what is a very heavy trigger lightened and I don't think its an easy adjustment - can anyone recommend a good gunsmith in the Geelong area?