Built in 1856, College Hall was the first building in America erected for instruction in scientific agriculture. It is believed to have been the first building constructed on campus, with six other buildings following during that year. These buildings were constructed on the site where Beaumont Tower now stands. The hall was built as the school's principal classroom, laboratory and office building. It was the center for all classroom work until 1870.

Few public buildings have been so poorly constructed and plagued by defects that may have been the work of careless or dishonest contractors. College Hall's foundation rested on plank footings, and one corner of the basement wall enclosed a large stump. In 1918, students and alumni, wishing to preserve College Hall as a landmark, sought to rebuild the structure as a student union. After workers found the walls to be hollow, bricks soft and a generally unstable foundation, such plans were dropped.

One August day in 1918, as the National Anthem was being played by the band at a nearby war trainees retreat, the walls of College Hall began to collapse. Members of the football team were used to finish tearing down the building.