Just noted that the original poster mentioned having both remote Internet players and LAN players join the server. Are you using the LAN address for the LAN players, and only using the external IP address for the remote players? That should give the best results.
Some routers can use the external IP address for all players, but this requires a bit more work for the router to do, and could (in theory) slow down some weak routers. I think the feature is called "Filter Internet NAT redirection".
If you really want to jump into deep voodoo, you could start up WireShark (
www.wireshark.org) on the server and configure it to capture all traffic to TCP port 2010, watching for DUP ACK, RST, or a lot of out of order packets. That could indicate further network setup is warranted. Note that Wireshark can eat a lot of CPU power, so have a strong server for this.