The Freedmen's Bureau encountered many struggles while trying to set up schools for a numerous amount of reasons. There were many cases where whites would oppose the location of the colored schools and would protest against it. For example, reported on September 22nd, 1865 in Washington D.C., the Freedmen Bureau saw it convenient to establish a school at the Freedmen’s hospital on the corner of 14th & M st. However, before it could open Judge Wylie and his wife protested in person against it.(6) They did not want the opening of colored schools in their neighborhood. Not only did they encounter protests from whites but many of the Freedom schools were burned down. On March 11th, 1866 , two white men burned down a colored church at Centreville, Queen Anne's County, Md. because a school for colored children was held in it.(7) There are numerous reports stating that schools and churches have been burned or stoned in order to prevent the freedmen from becoming educated. Many of the colored teachers also faced hard times and there were cases were they were even killed. A freedmen stated, "The Radicals build school houses, the Conservatives burn them."(8) Through all the major accomplishments of the Freedmen's Bureau, establishing good will between the whites and the freed slaves were not one of them.
Education, declared a Mississippi freedman, was "the next best thing to liberty."
Despite the many hardships that the Freedmen’s Bureau faced while setting up colored schools, their most notable achievements were in education and health care. W.E.B. Du Bois had written, “The greatest success of the Freedmen's Bureau lay in the planting of the free school among Negroes, and the idea of free elementary education among all classes in the South” . The Bureau spent more than 5 million dollars trying to set up schools in the south. By 1869, more than 3,000 freedmen schools were set up in the South and more than 150,000 students were enrolled.(9) The average attendance rate throughout the South was between 79 and 82 percent. (10) Stated in the Sub-assistant commissioner's monthly report in January 1868, some people thought it would be more helpful if the bureau could set up one free school in ever county that would be available to people of all color.(11) The Bureau did such a good job setting up schools that the very successful Savannah Educational Society wanted to be under the immediate care of the Freedmen Bureau when the society could no longer increase their contributions.(12) At the end of the Bureau's time, General Howard’s wish was to transfer the education system to a permanent government division. The success of the school establishment in the South had much to do with the freedmen's desire to be educated. J.W. Alvord, who was an ispector for the bureau, stated that freedmen, "have the natural thirst for knowledge." (13)