Greetings, thedon.
Milk fat, in its unadulterated form (raw, uncooked but certified free of contaminants) is a very good fat to take. Interestingly, natural sources of CLA comes from beef, milk fat and fat in cheese. Furthermore, your fat soluble vitamins (A, D and K and perhaps, E?) in milk are dissolved in the fat. Skim the fat off and you lose these vitamins.
What makes milk fat bad would be the heat damage it undergoes via pasteurization or worse, UHT (ultra heat treatment). The homogenization of milk prevents the fat from appearing as a seperate layer on top the milk, but it also means that the fat is now an alteration of its natural, original state. Read somewhere that this itself makes milk fat toxic...
If you are able to get your milk raw but certified free of contaminants (one or two states in the US has this...) then by all means indulge yourself. If you are like the rest of us, who are restricted by nanny-state laws, then your milk-fat would be damaged by such processes mentioned above, and you'd be better off avoiding milk fat as you would a hydrogenated fat or a heat damage fat.
Godspeed, and happy HSTing.