Who was the first president to invite an African-American man to the White House?

Theodore Roosevelt, who invited Booker T. Washington, was the first president to invite an African-American man to the White House.