Mix skim milk and pudding. Stir in sweetened condensed milk. Put in the frig to set at least 15 minutes.
Line the bottom and sides of a trifle bowl or 9 x 13 pan with vanilla wafers.
Add a layer of bananas on the bottom.
Fold whipped topping into pudding. Spread half of the pudding mixture on top of bananas.
Add a second vanilla wafers, bananas, and the rest of the pudding. Garnish the top with vanilla wafer crumbs. Chill for at least 1 hour before serving.