I'm not a 100% sure but this is what I would do:
You have: y = 6a^2x - abx - 12b^2x
First thing I would do is take x out:
y = x (6a^2 - ab - 12b^2)
Then you can take two approaches, either factorize it regularily, by taking (a - b) (a - b) and respectfully changing the signs and the number
or you could use the formula:
(-b (plus/minus)(square root of b^2 - 4ac)) / 2a = x
In this case you have the variables: a = 6, b = -1 and c = -12, as seen in your equation above.
If you plug in the values you will get:
x = 1 (plus/minus) (root of: (-1)^2 - 4*6*(-12)) / 2*6
x = 1 (plus/minus) (root of: 1 + 288) / 12
x = 1 (plus/minus) (root of: 289) / 12
you can find the root of 289, which is 17 and put it in:
x = 1 (plus/minus) 17 / 12
From this you will get two values:
x = (1 + 17)/ 12 and x = (1 - 17)/ 12
Thus you will get two values for x, being x = 3/2 and x = -4/3
Thus;
y = x (a -3/2b) (a + 4/3b)
or, more neatly:
y = x (2a -3b) (3a + 4b)
If you want to check it would give:
y = x ((2a)(3a)+(2a)(4b)+(-3b)(3a)+(-3b)(4b))
y = x (6a^2 + 8ab - 9ab -12b^2)
y = x (6a^2 - ab -12b^2)
And
y = 6a^2x - abx -12b^2x
I hope this helped.