The problem with diesel is microscopic particulate matter (aka "PM"). CA and NY have tighter health standards than most of Europe.
The short answer is, extremely small particles of anything are more of a health-threat than small amounts of CO, HC, etc. I forget the sizes, but we're talking tens of microns and smaller.
The problem is particles of that size are mostly man-made; they are too small for you lungs natural systems to clean out, like dust and pollen is, and too large to go into the bloodstream, like gasses are, where they are bound and/or excreted. So they tend to linger for very long periods of time in your lung tissue, cause irritation and cancers.
Call me a traitor, but I think CARB does a good job on the whole, and I'm damn glad they are able to force manufacturers into changing. We'd still be driving 4000 lb. carbureted drum-brake cars, choking to death at 55 if it was left up to Detroit. Detroit can't "self regulate" no more than your kids can "self regulate" Coke and candy intake; luckily for you they don't have lobbiests ("but Pa!").
PS: The only reason large trucks are exempt from a lot of these regulations is politics. The stuff could be phased in slow enough to not burden independent owner/operators (all engines get replaced eventually).
The truck engine manufacturers whine about being unable to meet standards, but that's a crock of shit, what are they living in the 19th century? They can buy a computer or two for design like everyone else.