You gotta prime them :)

  - Parlez-vous anglais?
  - No
  - Je ne parle pas bien le français, mais ... [followed by
    English with French-style word mangling and descriptive
    gestures and sounds]

Once they know you suck at French, they will spe-ak slo-o-o-w-e-r and cut you a break in general.