Different sounds may have a low or high frequency, and the side you use or the pressure you exert may be better for hearing those particular frequencies.
Examples of low frequency sounds are Korotkoff sounds when taking a blood pressure, bowel sounds, and heart tones/murmurs during cardiac auscultation.
An example of high frequency sound is breath sounds when auscultation the lungs.
It depends on the stethoscope, but typically the bell (usually smaller, commonly mistakenly called "the pediatric side") is better for low frequency sounds and the diaphragm (usually the larger side) is better for high frequency sounds. A lot of expensive stethoscopes like Littman have tuneable diaphragm that can change from high or low frequency by the pressure you exert when applying the diaphragm to the patient (low pressure is low frequency, high pressure is high frequency).
AHA recommends using the bell (for low frequency), when taking a blood pressure, however, few actually do it this way.
Does it really make a difference? In my opinion, no.
I personally don't notice a difference either. Some people have told me they do after I tell 'em about it, but I think it's kinda a placebo "Oh, wow, it's way louder when I auscultate the blood pressure with the bell!"