The phrase passing the buck probably came into use in American poker games during the late nineteenth century.
In 1872, Mark Twain wrote that players would pass an object, a buck, to remind them of who was to deal.
It is also possible that buck is an old word for bet.