Remember that chart? At the bottom, you may have noticed that the form for the first person, past tense for each modal was included. Here's the entire past tense conjugation for wollen:
ich wollte, du wolltest, er wollte, wir wollten, ihr wolltet, sie wollten, Sie wollten.
Once you know the first person past tense form for each modal, you know how to conjugate the whole tense, because all seven verbs take the same endings.
Future tense gets a little clunky at the end of the sentence, but here's the basics. Remember werden and how it works? Everything else gets sent to the end of the clause. Here's an example:
"We will have to stay for three hours."
In German, here's how that would look: "Wir werden auf drei Stunden bleiben müssen."
So the action verb becomes the next-to-last word, followed by the modal infinitive (bleiben means "to stay").