Once you're familiar with present tense French subjunctive, this will come very easy. The indicative mood analogue of past tense subjunctive is the passé composée, just as the normal present tense subjunctive is an analogue of normal present tense. Indeed, past tense subjunctive is structured almost exactly the same as the passé composée, with either avoir or être in a present tense conjugation as helping verbs, and then with the addition of the past participle. The only difference is that avoir or être are conjugated into the subjunctive present tense. Easy!
Whether you use avoir or être as the helping verb is determined in just the same way as with the normal past tense. As a general rule, the auxiliary verbs, or “Mr and Mrs Vandertramp”, take être as a helping verb, as well as any reflexive verb. Everything else takes avoir as a helping verb.
Both avoir and être are highly irregular in the subjunctive:
être: je sois, tu sois, il/elle/on soit, nous soyons, vous soyez, ils/elles soient
avoir: j'aie, tu aies, il/elle/on ait, nous ayons, vous ayez, ils/elles aient
Please keep in mind that rules of agreement still apply to the past subjunctive. Those constructions that use être as a helping verb must still agree in gender and number with the subject. Other special cases of agreement in the past tense must also be met.
Once you've got this half of the conjugation done, it's time to move on to the past participle. This is easy, because it's exactly the same as with passé composée. In general, you acquire the past participle of a verb depending on its ending. -er verbs have the -er ending removed and replaced with -é, -re have their endings replaced with -u, and -ir verbs are in turn replaced with -it. There are numerous exceptions, however, so it might be a good idea to refresh on those.