If someone has your IP address, and you are not protected by a software or hardware firewall, it is conceivable he could:
1. Break into your computer, via any of several cracking programs;
2. Infect your computer with a worm, trojan, virus, etc.;
3. Attack your computer in a way that makes your use of the Internet impossible;
4. Engage in illicit activities online, and "forge" his IP address to be your IP address, thus making it look like you were the person who did the illicit activities.
There's more, but that should give you the drift.
If you have a router between your computer and the modem, there's nothing to worry about. If you have a decent software firewall, such as ZoneAlarm or even Windows firewall, there's nothing to worry about.
If you don't have a router but turn your computer off at night, there's nothing to worry about; you'll get a new IP address the next morning, anyway.
UPDATE:
Having a firewall should protect you from anything he might try to do to your computer. It is exceedingly difficult to get around even a software firewall.
Your best protection is always packet and port filtering, and your best bet along those lines is a Linksys router with the latest firmware.