Thinking mathematically, assuming the boat maintains a constant velocity B m/s and does not need to slow down or use time to change direction for the return journey, when the driftwood floats at a constant velocity D m/s, with a tide T m/s, where the distance between the house and the start position of the driftwood is S and the distance between the start position and finish position of the driftwood is F:
The return time is given by
t = (B-T)/(S+F)
EDIT: this also assumes that you notice it is driftwood the moment you get there and turn back immediately.