One opinion in the Midrash imagines a vignette involving Yosef and his brothers on their return from Canaan following the burial of their father Yaakov. The Tora associates the death of the family’s father with a sense of increasing dread on the part of the brothers, but suggests that this was brought about simply by Yaakov’s absence, which in their minds would embolden Yosef to finally vent his anger against them:
Beraishit 50:15
And when Yosef’s brethren saw that their father was dead, they said: “It may be that Yosef will hate us, and will fully requite us all the evil which we did unto him.