A material is described as brittle if it easy to fracture. Fracture involves the breaking of chemical bonds, and so most crystalline materials (e.g. metals) are brittle. The only way the material can respond to a stress is through stretching of chemical bonds, which then break.
Some polymers are much less brittle because they can respond to an applied stress in a different way, by sliding the chains past one another. This involves no breaking of bonds, and so the material will not fracture. The large CL atoms (amongst other things) prevent this sliding and so the material becomes more brittle.