If we don't need color, then we can consider it noise. If there is an edge (a step change in pixel value) in hue that is hard to detect in a grayscale image, or if we need to identify objects of known hue (orange fruit in front of green leaves), then color information could be useful.
For many applications of image processing, color information doesn't help us identify important edges or other features.
