I realize that not many people actually make maps in Max but I'm hoping that this question can be answered anyway.
How do you UV map a terrain without having it look too obviously like repeating squares of the same texture?
You have to look for "tilable" texture. these are generally squared textured of a motif drawn in a way that it can repeat itself indefinitely without being too noticeable.
I don't know if it helps but just try googling tileable texture to see if it what you're looking for
Is there a recommended size limit or ratio. Like if the map is 256 X 256 how big of an area could it do before the resolution get too ugly?
I was also wondering if there was a tesselate command for the mapping.
For the resolution question, i've never done map in max so i'd have a hard time telling the limit for good looking texture once imported in freedom force.
If by tesselate you mean choosing how many time the texture actually appear...hum, i don't remember, hav'nt open max in weeks. i just do a planar mapping on the ground and scale the guizmo (the plane representing the orientation of the projected mapping)
You might try looking at some of the blank terrains from FF to get an idea as to what works.
You can make say three variant tiles that all match up with each other on the edges, and map them to the terrain in a kind of random fashion. This works really well for artificial environments like paving, or space-station metal floors, but can also work for natural environments like rock piles.