On both the client and the server go to http://dslreports.com/speedtest or https://secure.dslreports.com/speedtest and test your speeds. They should be the same. If one is slower than the other, try to find the connection issue and fix it.
Other solutions include re downloading Minecraft as it may have gotten corrupted somehow with the mods. I would try both client and server versions.
Good luck.
So, I am currently making my server on minecraft and now when I join I get a message that says "Internal Exception: java.io.IOException: Connection reset by peer". I can successfully join the server, but I get this message every 1 second-10 minutes. Everyone else does not have this error. So, how can I fix this? Thank you, in advance!