If the angular speed was slower than the minimum, then the resultant force on the particle (the one on top) is
>mrω2, right. That means it has an acceleration towards the centre that's bigger than
rω2. That means it will actually fall towards the hole. Friction will be trying to prevent that (relative motion), but it will fail as it can't get any bigger.
Something that's often helpful is to consider extreme values, in Mechanics, because our physical intuition can confirm for us whether our equations are right. Example: let's imagine that
ω is ridiculously small and then ridiculously large. If it's small what happens? Clearly the particle rapidly slides towards the hole if there isn't a significant rotation. If it's large, i.e. fast, then it will move outwards, which could also be seen as moving to a larger 'orbit'.
I just remembered actually doing this experiment in real life. We took a fine wire, attached small masses to each end, threaded the wire through a long glass tube, and then you can hold it up and whirl it around
Here's a simple diagram:
http://www.batesville.k12.in.us/physics/phynet/mechanics/circular%20motion/labs/cf_and_speed.htmSo like in this question, one mass sits at the bottom and the other one rotates in a (hopefully) horizontal circle. There is no friction here (apart from air) so it's not exactly the same, but it's something. You see that the faster you force it to rotate, the larger the circles the rotating mass makes.
"Problems involving non-horizontal forces" is a title suggesting that they are
not dealing with non-horizontal circular motion; it will probably just be a matter of resolving forces in the direction towards the centre of the circle (radial). Nothing particular there (will be on inclined planes and so on).
"non-uniform speed" - this can be a bit more demanding. What does *not* change is the formula for centripetal acceleration, and that you have to find out how the existing forces cause that acceleration. What does change is that, with changing speed, there is also acceleration in the transverse direction. There is no change in the Physics here, it's just more complicated (for example, if you use energy conservation you have to notice that the kinetic energy of the particle rotating is changing).
It's difficult to answer the 'do I need to do more reading'. I can see from your attempts at answering that you know how to construct the diagrams and equations. The more difficult non-constant-speed case is just the same principles applied to a more difficult situation. More reading/watching vids about circular motion would be a good idea to help you to see how it works in different ways and get you to feel more comfortable with it (but - is the problem about how circular motion works, or about how friction works?)