this guy from PCgamers tryed Arma III in the E3, he says its much more smooth than ArmaII, altrough i have to see it first to judge.
Also, you should have said UT 3 engine, not RO2. And i think the best one would be Cryengine 3, its free as long as you dont charge for yoru product, easy to mod, modern,etc.
The cons I see is maps can only be 2x2 km max, But knowing most maps in FH2, i think they are tiny enought.