It takes two to Tango
There has to be mutual consent from both sides - If you both know each other and you are both over 18, you both have mutual respect and you enjoy the actions of that person and go along with the actions of that person without giving any indication you want them to stop - I would say that is not sexual assault.
Your post is ambiguous - Is this someone a stranger? Even 'starts touching you in a sexual manner' is open to interpretation. Is this a boyfriend or girlfriend? Has there been previous consent? If you 'go along with it' are you giving consent (are you returning the contact?) If you are both sexually immature it may be that you are both in the fumbling embarrassed phase of learning to grow up; growing up and learning about sexuality there will be many mistakes. If you don't like something make sure you do something differently next time or verbalise and tell someone to stop.
If you don't know what is pleasurable or ok for you, then how is that 'someone else' expected to know what you like or dislike unless you tell them? You do need to know your own boundaries. If you are so concerned and don't want contact then don't put yourself in such a position until you know what you really want or you have discussed those concerns first with that 'someone' If you don't want anyone to touch you at all say so right at the outset so there are no misconstrued actions or interpretations.
Look at it from the perspective of that 'someone' - what would you be thinking exploring someone else's sexual or personal space? There is a lot more to defining consent than a verbal 'yes' or 'no' If you are genuinely frightened for your life or are in total fear to say anything then not saying anything can still be interpreted as not giving consent.