Combine the sugar, butter, and milk in the 4-quart pan. Boil until the thermometer reaches 234° F/112° C, also called soft ball stage in candy making. Remove from heat.
Add the peanut butter and stir until melted.
Add the remaining ingredients and stir until well-blended, but don’t overdo it. Do not scrape the sides or the fudge will be gritty.
Pour into the prepared 9x13 pans. Allow to set in the refrigerator overnight.
Cut into 1-inch squares and store in a the refrigerator in a tightly sealed container.