Prepare the white cake according to the package directions. Bake until fully cooked and allow it to cool completely.
Cut the cooled cake into small cubes and set aside.
In a medium bowl, whisk together the sweetened condensed milk, key lime juice, and lime zest until smooth and slightly thickened.
In another bowl, whisk the instant vanilla pudding mix with cold milk for about 2 minutes until thickened. Let sit for 5 minutes.
In a large mixing bowl, beat the softened cream cheese and powdered sugar until smooth and creamy with no lumps remaining.
Add the prepared pudding mixture into the cream cheese mixture and mix until fully combined.
In a chilled bowl, beat the heavy whipping cream and vanilla extract until soft peaks form.
Gently fold the whipped cream into the cream cheese pudding mixture using a spatula until light and fluffy.
Place a layer of cake cubes into the bottom of a large trifle bowl or serving dish.
Spoon part of the key lime filling over the cake layer.
Spread a layer of the creamy pudding mixture over the lime filling.
Sprinkle crushed graham crackers evenly across the cream layer.
Repeat the layers with the remaining ingredients until the bowl is full.
Cover and refrigerate the trifle for at least 4 hours or overnight for best results.
Before serving, garnish with whipped cream, lime slices, lime zest, and extra graham cracker crumbs.
Serve chilled.