How do I make certain enemy formations rarer than others in the same formation set?

Simple indeed this is. When a battle starts, the game will randomly pick a formation to battle. So if you want to have more chance of a certain formation, simply add more of them to the list. If you want less chance (rare) of a certian formation, then just make multiple other formations. Like adding two of all the formations except the ones you want rare. You get the idea.