In this paper, we aim to design a general framework for the Pursuit-Evasion (PE) game of multiple UAVs. Based on the distance between the pursuers and the evader, the whole pursuit-evasion process is decoupled into a seeking stage and a herding stage. A formation control method is proposed to make sure that the pursuers can find and drive the non-cooperative evaders along a desired trajectory towards the designated target while maintaining a preset formation. In the meantime, an adaptive potential function is designed to achieve collision avoidance to both obstacles and other agents in the formation. The mian contribution is that it combines formation control and collision avoidance in the problem of indirect herding, which is rarely addressed before. Also, the convergence and effectiveness of the designed controller for trajectory tracking and formation maintenance is proved and verified in the paper. Simulation results in two and three dimensional space show that the proposed framework achieves the goal of coordinated herding and collision avoidance in a pursuit-evasion scenario under uncertainties and external disturbances.