Its all about the context.
Nosotros hablamos con ellos ayer-- We talked to them yesterday.
Nosotros hablamos con ellos a veces -- We talk to them sometimes.
With -er verbs its quite obvious which one. For other conjunctions you need to know what the context is. If someone is talking about what they did yesterday, and half of what they are saying is in the imperfect, you'll know they are using the preterite. If that makes any sense?
If not when im back in an hour or so I'l try explain it better.