Line an 8x8-inch baking pan with parchment paper. Leave extra parchment paper hanging over the sides for easy removal later. Press the parchment gently into the corners of the pan and set aside.
Add the white chocolate chips, sweetened condensed milk, and butter to a medium saucepan. Place the saucepan over low heat.
Stir the mixture slowly and continuously as the chocolate melts. Continue stirring until the mixture becomes smooth, creamy, and fully melted with no lumps remaining.
Remove the saucepan from the heat. Stir in the key lime juice, lime zest, vanilla extract, and green food coloring if using. Mix until fully combined and smooth.
Pour the fudge mixture into the prepared baking pan. Spread evenly using a spatula, making sure the mixture reaches all corners.
Sprinkle graham cracker crumbs evenly across the top of the fudge while the surface is still soft.
Place the pan into the refrigerator and chill for at least 3 hours or until the fudge is fully firm.
Lift the chilled fudge out of the pan using the parchment paper. Transfer to a cutting board and slice into small squares using a sharp knife.
Serve chilled for the best creamy texture. Store leftovers in an airtight container in the refrigerator.