Like Quad said, check your form. Rows won't work too well if you don't do them just right. Don't use momentum to bring the weight up, that defeats the purpose. Try to feel the muscles in your back contracting as you bring the weight up to your torso.
There is also a thread on Lyle's site that talks about rows, pulldowns, etc and how to do them properly. I'll link to it in a minute.
You can also try a larger variety of exercises for your back. Are you deadlifting yet? I've found that deadlifting and specifically doing rack deadlifts from about knee level in a rack are great for the back.
Another thing to watch is to make sure you are involving the muscles in your back properly. From
Lyle's back thread:
<div></div><div id="QUOTEHEAD">QUOTE</div><div id="QUOTE">
2. A lot of people have trouble targetting/training/hitting the back. Part of it is that they can't see it, part of it is poor technique. A lot of trainees feel a ton of arm involvement and almost no back involvement with back exercises. This is very often the case if they have tight shoudlers from too much bench pressing. A key cue is to think of the arms as hooks or ropes and focus on what's going on with the elbows.
During rowing, you should think about pushing the elbows behind you and getting the shoulders blades together. This has the conseuence of making you stick your chest out. During the eccentric, the shoulders should move forwards a bit and they must come back during the concentric. Someone with tight pecs and front delts (again, too much benching) will tend to have problems with this and should stretch their pecs/front delts prior to every set of back.
During pulldowns, the elbows should be pushed down, the focus should be on letting the shoulders come up (and scapula elevate/rotate outwards) during the eccentric and push them down during the concentric.
</div>