Gelfand, Serconek, Retakh and Wilson defined algebras they call A(G) associated to layered graphs G. These algebras arise in the context of factoring polynomials over non-commutative rings. Under certain conditions on G, A(G) is quadratic, and the question of whether A(G) is Koszul arises. Cassidy, Phan and Shelton answer this in case G arises as (an appropriate modification of) the face graph of a regular cell complex, X. We show that the answer to this question is a topological invariant (i.e. that it doesn't depend on the cell complex structure). In particular if X is a regular cell complex of dimension d, A(G) will be Koszul if and only if the homology of X is concentrated in the top dimension, and if any singularities of X are not detectable through local homology below the top dimension.