There is no exact definition of who the Founding Fathers were. For the most part, I think people consider the Founding Fathers as those of the Revolutionary era who signed the Declaration of Independence, Articles of Confederation, and Constitution. But it could encompass other builders of the early republic, the leading military commanders of the Revolutionary War, and even the regular soldiers who fought for independence. But there isn’t an official group that gets the designation. The term has really only been popularly used in the past 100 years, thanks largely to Warren G. Harding, as I’ve written about before.
No matter which group is defined as the Founding Fathers, there were many others besides John Adams who didn’t own slaves.