Maybe your occasional fliers are just due to dented/damaged pellet skirts...
Does the crown look good (smooth, concentric, no burrs?)
There are a whole load of factors that affect springer accuracy, so don't feel bad, it really is a learning curve.
For instance: when you load a pellet, do you seat it in the bore the same way every time? Same depth? Believe me, it makes a big difference. I use fat ballpoint "click" pens with plasticky "noses" to press my pellets in, the plastic doesn't mar the pellet skirts, and by pressing in until the pen is stopped by the bore it seats the pellets to the same depth reliably...
Lots of little tricks. Keep asking questions, maybe you won't have to spend as much as you think

Oh, and get as many kinds of decent pellets as you can to find what works well in your gun!