Union Church was the first building to be erected in Petersburg. It opened in October 1879. In the mornings it was used by the Presbyterians and in the afternoons by the Primitive Methodists. The Primitive Methodist Church withdrew from the town and a Wesleyan Church opened in August 1880. The Baptists became the sole occupants of the Union Church until their own church was built in 1883.  The Church of Englands first service, in September 1884, was in the Union Chapel then they purchased the building in December 1884. It became Church of St Peter until it was replaced by the present St Peter's Church in Kitchener Street in 1888. The Union Chapel became the Salvation Army Barracks in the 1890s. Afterwards it was used as a school, residence and store room, before being demolished in the 1960s.