Skidaway Island was originally a hunting and ceremonial ground of the Timucua Indians. European settlement of the island was successful between 1754 and 1771, and included John Milledge (the father of John Milledge), who established a plantation on the northern end. He named it "Modena," presumably after the Italian city for its famed production of silk. Silk was an early industry of the European settlers of coastal Georgia. The Modena Plantation grew corn, cotton, oranges and mulberry trees, and kept sheep, cattle, hogs and horses. It survived as a plantation into the 1840s, until John Milledge III sold it in 1843. The name "Modena" is still used to refer to the northern part of the island.

After the American Civil War (1861–1865), many of the plantations on the island were unable to continue without slave labor, and their owners gradually sold them to wealthy northerners, who mostly held them on speculation. Much of the island reverted to wilderness and was used mainly for hunting, trapping, fishing and lumber for several decades. The Modena Plantation changed hands sixteen more times after the war. In 1927 the property became a private hunting preserve for Ralph H. Isham, best known for purchasing a collection of James Boswell's private papers from Baron Talbot of Malahide and passing them to Yale University.

Mr. Isham sold the property in 1934 to Robert C. Roebling (great-grandson of John Augustus Roebling). The Roebling family (Robert, Dorothy, and five children) established a farm for Black Angus cattle while living on their 176-foot (54 m) schooner, the Black Douglas, which provided all of the power needed by the farm until it was sold in 1941 to the United States Fish and Wildlife Service. The cattle farm was very successful for the next twenty years, until it came to an end in 1954 due to post-WWII economic pressures. Many of the buildings from the farm are still standing and in use by SkIO, most notably the seven-sided barn with its compass rose mosaic at the center.

Another successful European settler of the mid-18th century was Henry Yonge, who had a plantation located just southeast of Modena, called Orangedale. In 1877 it came into the ownership of the Benedictine Order. The Benedictines tried to establish a monastery and Catholic school for the newly freed former slaves on the island, but were unsuccessful. They abandoned their efforts in 1889. The "Priest tract," as the property had come to be called, became one of many acquired by a partnership of Thomas Bourke Floyd, James Boog Floyd, and A. Goden Guerard Jr. Thomas Floyd eventually assumed ownership of the entire middle portion of the island, until economic hardship forced a transfer to C&S Bank in 1924. The Union Camp Corporation purchased it from the bank in 1941.