My understanding of the system is:
Unless you're earning £5,044 (as an employee) or £5,075 (self employed) then you'll not be counted as having completed a "qualifying year" towards your state pension. You will need to have accumulated 30 qualifying years before you retire in order to be eligible for a state pension.
So yes, you'll miss out on 3/4 years worth of contributions. But the fact you only need to total 30 years of contributions, should give you plenty of time to catch up - assuming you work until normal retirement age.