A little copper sulfide in the root system will eliminate that tree hurting the fence issue.If the future forecast calls for trouble, leave enough room to maintain without getting on their property. You'll be glad you did for the classic argument, the tree issues. Put the fence on the line, and you can't cut the tree down on the other side that's jacking your fence up. Also, they can't stack or lean crap up against your fence accelerating rot or rust. If you have room, make it a two row mower pass, be it push or ride. Make sure survey markers are easy to find for the future. Put a yard ornament on top or something. This will be most important if they sell and you have to deal with new neighbor ownership. My dad had the fence on his property in a standard residential neighborhood house. The neighbors put a privacy fence up right against his without communication. It was just there in a day. He told the neighbor that they built the fence on his property, and they argued that the first fence was the property line. He had the surveyor come out and it cost the neighbors a shiny dime to move that fence. If the neighbors put their house up for sale, flag the boundary through their process to save trouble. The amount of room you leave is contingent on the size of your property, and how much you're willing to give up if smaller.