The fort was never fully armed, and until 1861 it was usually occupied only by army caretakers. Following South Carolina's secession from the Union, fears that the Federal government would send troops to occupy North Carolina's forts prompted local secessionist militia troops to seize Fort Caswell on 9 Jan. 1861. But North Carolina had not seceded, and so the fort was returned, only to be retaken on 16 April, after the Civil War began.
Now armed, the fort became one of the main Confederate defenses of the Cape Fear River. The Confederates strengthened it by constructing a massive earthwork defense around the fort and on top of the main work to accommodate 29 heavy guns. However, the fort was never directly attacked; its guns occasionally fired on Union warships in the ocean blockading the Cape Fear entrance. After Union forces captured Fort Fisher on the opposite side of the river on 15 Jan. 1865, retreating Confederate forces detonated gunpowder magazines in Fort Caswell on the morning of 17 January, literally blowing away the southeast face and shattering the west face. After the war, the fort reservation was returned to caretaker status.
In 1885 Secretary of War William C. Endicott's Fortifications Board selected Fort Caswell to receive modern defenses. The work continued from 1894 to 1904, significantly altering the original fortification. The army used the reservation until after World War I, when, in 1923, it was closed by the War Department and listed for sale as surplus property. In 1925 it was purchased by Florida developers. With the onset of World War II, the U.S. Navy bought the reservation in 1941. Sold to the Baptist State Convention in 1949, the reservation has been used as the North Carolina Baptist Assembly.