It would need to be damn close across the rest of the UK for the scottish vote to decide it considering we make up less than 10% of the UK's population. You also need to consider how the vote goes in both Wales and Northern Ireland as I'm pretty sure both are more pro-remain than England especially the latter which I'm pretty sure is the most pro-EU part of the UK though it is the smallest in population terms.
The pro-independence lobby in scotland (which is more than just the SNP) is somewhat torn over this issue as while a UK-wide leave vote but a remain vote in Scotland does raise constitutional questions, it doesn't create a good case for an independent Scotland as the SNP's reliance on the oil industry to bolster the economy has backfired now. It makes a referendum harder to win if you basically have a choice between independent Scotland inside EU or United Kingdom outside of the EU.
If the situation the OP described arose then yes, England would have ground to ask union-based questions though I feel love for the union is much stronger in England than Scotland so I don't feel it would have too much support. The government would also dismiss it because they got the result they wanted but I doubt it would do anything to heal the rifts within the Conservative party.
Regardless of all this speculation, I think that this referendum is going to decided by turnout more than anything else. Lots of people just don't care enough and that's why turnout at european elections is always bad in the UK.