This paper argues that a strong focus on residential segregation limits the understanding of the role of the built environment. The city is used as more than just a place of residence; urban life is far from restricted to where we live. The potential for interplay that develops as people share public space is argued to be just as important for integration processes as the residential mix. In addition, this article examines shortcomings related to the definition of residential segregation because of limitations within the scientific analysis of urban space: the evident difficulties in delimiting relevant geographical units and delimiting relevant social groups. The study is based on empirical analysis of Sodertalje. Sodertalje topped international news when its mayor informed the US Congress that the city had managed to receive more refugees from the war in Iraq than the US and Canada combined. However, to what extent are these new immigrants given access to Swedish society through everyday practices? The results highlight how segregation in public space—including impaired accessibility to a range of resources such as places of work and contact with other people—is a very strong feature of excluded areas and is strongly disadvantageous for newcomers. These results challenge some of the beliefs in the current public debate as well as some of the principles used by Swedish authorities to ameliorate segregation.