Riverside Park School was originally a boxy wooden-frame school house built in 1891 with multiple additions (due to population increase) until it was considered a fire hazard and unstable. Duval County voters passed a one million dollar bond in 1915 to build more than a dozen new brick school houses. This one was first known as Public School Number Four but later was renamed Annie Lytle Elementary School, after its former long-time teacher and principal.
Construction was started in 1917, Designed by architect Rutledge Holmes, built by Florida Engineering and Construction Company, and Completed in 1918 . It cost over $250,000 and originally overlooked Riverside Park, before the construction of I-95 & I-10 isolated it in the 1950's.
In 1960 the last student passed through those front doors and the school was closed to the public. It was used for office space and storage after that before being condemned in 1971 and abandoned. It may have also been rented out to a catholic school for a year or two in the early 70s or late 60s but is not confirmed.