Polly's design of his house was heavily influenced by his military experiences. "The Fort,” as it was to become known, was designed with two floors. Like a fort, the first floor featured one door for entry and then entry to other spaces was thru portals only large enough for one man to fit thru. Additionally, each space had gun ports to defend against attacks. Lastly, there was a "palisades" space running the length of the front house equipped with gun ports for the defense of the house. The second floor had been designed for living quarters and offered a front veranda and middle house breezeway. The second floor was approached by stairways, front and back, leading from the ground to the top.
The Fort had been sited on a southwest to northeast axis along the Privilege Creek waters. Limestone from the nearby creek was quarried and used for construction. Once again, Polly's experience at fortified stone garrisons made his decision to use the native stone an easy one. After, stone footings were installed; limestone walls 24" thick were constructed two stories tall. Native cedar timbers were harvested and used for floor and roof construction. The roof was finished with rough sawn cedar shingles and a large fireplace was installed at one end of the house. Lastly, Polly constructed a high stone fence all the way around his ranch. Often, during Indian raids, neighbors sought refuge at the house and therefore became known as “The Fort.”