With your current diagram, even if M=0, the beam still rotates counter clockwise. Are you sure your value of 3.5kg is correct?
Also, am I correct in saying that the 30 degrees written down was for the purposes of finding 3.5, and the question is: if you can move the pulley to any position, what is the maximum value of M possible while still being in equilibrium? If not, I don't fully understand what you are asking. But your current free-body diagram is rotating, and you'd need a clockwise moment (-ve M in your diagram) to correct this.