Lee was born at Stratford Hall Plantation, in Westmoreland County, Virginia, the fourth child of Henry Lee and Anne Hill (née Carter) Lee. Henry “Lighthorse Harry” Lee was a Revolutionary War, major-general, cavalry hero. Henry was injured in surpressing a riot in Baltimore and died from the wounds. He was raised by his widowed mother and attended private schools.
Lee went to West Point in 1825.
While there, he became corps adjutant, the chief post of honor for a cadet. He graduated in 1829, ranked 2 out of 46 in his class. When he graduated West Point, he had not only attained the top academic record but was the first cadet to graduate the Academy without a single demerit.
He was commisioned in the Engineers as a 2nd lieutenant and assigned to the Corps of Engineers. Lee served for 17 months at Fort Pulaski on Cockspur Island, Georgia.