Preheat the oven to 325°F (163°C). Grease and flour a 9×5-inch loaf pan.
In a large bowl, cream the softened butter and granulated sugar together using an electric mixer on medium speed until light and fluffy, about 4-5 minutes.
Add the eggs one at a time, beating well after each addition.
In a separate bowl, whisk together the all-purpose flour, baking powder, and salt.
Add the dry ingredients to the butter mixture in three additions, alternating with the whole milk, beginning and ending with the dry ingredients. Mix each addition just until incorporated.
Stir in the fresh lemon zest, lemon juice, and vanilla extract until evenly combined.
Pour the batter into the prepared loaf pan and smooth the top with a spatula.
Bake in the preheated oven for 55 to 65 minutes, or until a toothpick inserted into the center comes out clean.
Remove the cake from the oven and let it cool in the pan for 15 minutes.
Turn the cake out onto a wire rack and allow it to cool completely before slicing and serving.