PLANTED CLIQUE DETECTION BELOW THE NOISE FLOOR USING LOW-RANK SPARSE PCA
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
PLANTED CLIQUE DETECTION BELOW THE NOISE FLOOR USING LOW-RANK SPARSE
PCA
Alexis B. Cook1 and Benjamin A. Miller2
1
Department of Applied Mathematics, Brown University, Providence, RI 02906
alexis cook@brown.edu
2
MIT Lincoln Laboratory, Lexington, MA 02420
bamiller@ll.mit.edu
ABSTRACT We study approaches to solving the planted clique problem that
are based on an analysis of the spectral properties of the graph’s
Detection of clusters and communities in graphs is useful in a wide modularity matrix. Nadakuditi proved in [8] the presence of a phase
range of applications. In this paper we investigate the problem of transition between a regime in which the principal component of
detecting a clique embedded in a random graph. Recent results have the modularity matrix can clearly identify the location of the clique
demonstrated a sharp detectability threshold for a simple algorithm vertices and a regime in which it cannot. Through understanding the
based on principal component analysis (PCA). Sparse PCA of the breakdown point of this simple spectral method for clique detection
graph’s modularity matrix can successfully discover clique locations in high dimensional settings, we hope to motivate new algorithms to
where PCA-based detection methods fail. In this paper, we demon- detect cliques smaller than previously thought possible.
strate that applying sparse PCA to low-rank approximations of the Sparse principal component analysis (SPCA) has recently been
modularity matrix is a viable solution to the planted clique problem shown to enable detection of dense subgraphs that cannot be detected
that enables detection of small planted cliques in graphs where run- in the first principal component [7, 9]. The semidefinite program-
ning the standard semidefinite program for sparse PCA is not possi- ming formulation known as DSPCA has also been proposed as an
ble. approximation that nearly achieves the information-theoretic bound
Index Terms— planted clique detection, graph analysis, com- in [4], and as a complexity-theoretic bound for planted clique detec-
munity detection, semidefinite programming, sparse principal com- tion [10]. However, DSPCA has great computational burden. This is
ponent analysis because several eigendecompositions are performed over the course
of the procedure. For subgraphs that are relatively close to the de-
tection threshold, however, the subgraph still manifests itself signif-
1. INTRODUCTION icantly in the largest eigenvectors. This suggests that these cliques
would be detectable even if only a low-rank subspace of the orig-
Real-world graphs are naturally inhomogeneous and exhibit nonuni- inal matrix were considered. In this paper, we investigate the use
form edge densities within local substructures. In this setting, it is of DSPCA with a low-dimensional approximation
often possible to break graphs into communities, or sets of vertices √ of the modularity
matrix. The running time of DSPCA is O(n4 log n/), so reduc-
with a high number of within-group connections and fewer links be- ing the dimensionality of the problem can potentially improve the
tween communities. The problem of community detection in net- running time by multiple orders of magnitude.
works has become increasingly prevalent in recent years and has im- The remainder of this paper is organized as follows. Section 2
portant applications to fields such as computer science, biology, and outlines the problem model and defines notation. Section 3 briefly
sociology [1–3]. While community detection often considers par- discusses recent relevant research in this area. In Section 4, we
titioning a graph into multiple communities, a variant of the prob- demonstrate analytically—using an approximation—that a large
lem considers detection of a small subgraph with higher connectiv- fraction of the magnitude of a vector indicating the planted clique
ity than the remainder of the graph [4], a special case of which is the will manifest itself in the eigenvectors of the graph’s modularity
planted clique problem [5–7]. matrix associated with the largest eigenvalues. Section 5 provides
A clique is a set of vertices such that every two vertices are con- empirical results demonstrating improved performance using a small
nected by an edge. In the planted clique problem, one is given a number of eigenvectors (rather than only one), and in Section 6 we
graph containing a hidden clique, where each possible edge outside summarize and outline possible future work.
the clique occurs independently with some probability p. Detecting
the location of this maximum-density embedding in a random back-
2. DEFINITIONS AND NOTATION
ground graph is a useful proxy for a variety of applications—such
as computer network security or social network analysis—in which
In the standard (k, p, n) planted clique problem, one is given an
a subgraph with anomalous connectivity is to be detected.
Erdős-Rényi graph G(n, p) with an embedded clique of size k, and
This work is sponsored by the Assistant Secretary of Defense for Re-
the objective is to determine the hidden locations of the clique ver-
search & Engineering under Air Force Contract FA8721-05-C-0002. Opin- tices. Formally, given the graph G = (V, E), where V is the set of
ions, interpretations, conclusions and recommendations are those of the au- vertices and E is the set of edges (connections), with |V | = n, the
thors and are not necessarily endorsed by the United States Government. desired outcome is the subset of vertices V ∗ ⊂ V , |V ∗ | = k that
978-1-4673-6997-8/15/$31.00 ©2015 IEEE 3726 ICASSP 2015belong to the clique. where k · k0 denotes the L0 quasi-norm. We will apply a convex
In our procedure, instead of working directly with the edge set relaxation and use the lifting procedure for semidefinite program-
E, we will analyze the adjacency matrix corresponding to our model. ming (SDP), following the method of [12], to yield a semidefinite
The adjacency matrix A of an undirected, unweighted graph is a program:
useful representation of the graph’s topology. Each row and column X̂ = arg max tr(BX) − ρ1T |X|1
is associated with a vertex in V . After applying an arbitrary ordering X∈Sn (3)
of the vertices with integers from 1 to n, we will denote the ith vertex subject to tr(X) = 1,
as vi . Then Aij is 1 if vi and vj are connected by an edge and is 0
where tr(·) denotes the matrix trace, and Sn is the set of positive
otherwise. The model for the adjacency matrix of the planted clique
semidefinite matrices in Rn×n . Here, ρ > 0 is a tunable parameter
problem is:
8 that controls the sparsity of the solution. After applying DSPCA to a
∗ graph observation, we take the principal eigenvector of X̂ and assign
Aij = 1 with prob p if vi ∈ / V ∗ or vj ∈/ V∗ the vertices associated with the k largest entries in absolute value as
∗ belonging to the clique. We use the DSPCA toolbox provided by the
/ V ∗.
>
:0 with prob 1 − p if v ∈
i / V or vj ∈
authors of [12] for the results in this paper.
Consistent with the methodology developed by Newman [1], we Fig. 1 compares the results returned by PCA and DSPCA for a
will study the modularity matrix B of our observed graph, defined sample test case. We generated an Erdős-Rényi graph G(500, 0.2)
as: and embedded a clique of size 10. The PCA-based technique fails
Bij = Aij − p, here, as the entries in the principal component corresponding to the
clique vertices (marked by magenta dots) are not consistently high
where we have subtracted the expected value of the adjacency matrix
in absolute value. However, the vector returned by DSPCA, or the
for the Erdős-Rényi model that does not contain the planted clique.
principal eigenvector of solution to the SDP relaxation described in
(3), accurately identifies the clique vertices. Here, ρ = 0.6 was
3. RECENT METHODS AND RESULTS used. For this sparsity level, the DSPCA solution contains a very
clear signal with high values at the clique vertices and is almost zero
It has been shown previously that thresholding the principal eigen- at background vertices.
vector of B can yield the locations of the clique vertices. The unit-
normalized principal eigenvector u1 of the modularity matrix asso- 0.2 0.4
ciated with an Erdős-Rényi √
graph without an embedded clique is
Component Value
Component Value
0.3
asymptotically distributed as nu1 ∼ N (0, I) [11]. Using this fact, 0.1
Nadakuditi establishes an algorithm for detecting vertices V̂ ∗ in the 0
0.2
planted clique [8]: 0.1
−0.1
n √ “ α ”o 0
V̂ ∗ = vi : | nui1 | > FN −1
(0,1) 1 − , (1)
2 −0.2
0 100 200 300 400 500
−0.1
0 100 200 300 400 500
where ui1 denotes the ith entry in u1 and the false-alarm probability Vector Component Index Vector Component Index
of identifying a non-clique vertex as part of the clique is α. (a) (b)
PCA-based clique detection has been shown to have a well-
defined breakdown point in high dimensions. Nadakuditi elucidates
Fig. 1. Example PCA result (a) and DSPCA result (b).
this breakdown point in the following theorem.
Theorem 3.1 (Nadakuditi [8]) Consider a (k, p, n) planted clique DSPCA is an iterative algorithm that needs to compute a full
problem where the clique vertices are identified using (1) for√a sig- eigendecomposition of a perturbation of the modularity matrix at
nificance level α. Then, for fixed p, as k, n → ∞ such that k/ n → each step. Thus, for large graphs, the algorithm can run quite slowly.
β ∈ (0, ∞) we have However, while the clique does not stand out in the principal com-
( q ponent, it may still be well-represented within a low-dimensional
p
a.s. 1 if β > βcrit. := 1−p subspace corresponding to the eigenvectors with the largest eigen-
P(clique discovered) −−→ values. In this paper, we will show that a low-rank approximation
α otherwise.
of the modularity matrix can replace its full-rank counterpart in the
DSPCA algorithm, to result in a faster method that still breaks the
This theorem implies that, for sufficiently large graphs, the al- detection threshold go the PCA-based algorithm.
gorithm for PCA-based clique detection
q described in (1) performs
np
poorly when the clique size k ≤ 1−p
. Fortunately, there is an- 4. PERFORMANCE PREDICTION: APPROXIMATION
other spectral technique that has proven capable of getting past this AND LOWER BOUND
detection threshold.
We can approximate the modularity matrix B as the sum of a
We will use an approach relying on sparse PCA to attempt to find
real, symmetric matrix X and a rank-1 matrix P = θuuT , where
a vector that is “close to” the principal component, but that contains
Xi,j + p ∼ Bernoulli(p),
√ θ = k(1 − p), and u is a vector whose
exactly k nonzero entries, in the hope that the entries will correspond
to the clique vertices. Formally, we would like to solve: ith entry ui is 1/ k if i ∈ V ∗ and is 0 otherwise. Note X is diago-
T
nalizable, and we can write X = UX DUX , where D is a diagonal
x̂ = arg max xT Bx matrix of eigenvalues, λ1 ≥ . . . ≥ λn , of X. The spectrum of
kxk2 =1 (2) X + P is equal to the spectrum of D + θũũT , where ũ = UXT
u, and
subject to kxk0 = k, there is a unique mapping between their eigenvectors via UX . Thus,
3727as in [13], we will analyze this matrix rather than the modularity Differentiating with respect to σ, we have:
matrix to determine the concentration of u among the eigenvectors. Z 1 „Z 1
−c2 dx
„ ««
Computing the principal m-dimensional eigenspace, 1 ≤ m ≤ n, ∂J dx
= 2 1 + 2θ − 1.
can be recast as maximizing the trace for a symmetric orthonormal ∂σ a (λ(x) + σ) a λ(x) + σ
projection of the original matrix, i.e.,
“ “ ” ” With the constraint that g 2 (x) integrates to 1, we can find the critical
U ∗ = arg max tr U T D + θũũT U , point at Z 1
U ∈Um dx 1
=− . (5)
λ(x) + σ θ
where Um is the set of n × m matrices with orthonormal columns. a
Let: » – Differentiating with respect to c yields the same equation. We can
Im−1 0m−1 use (5) to solve for σ at a given θ, then solve for c by normaliz-
Vm = , ing the integral to 1. As n → ∞, the distribution
0n−(m−1)×m−1 v p of eigenvalues
will converge to a semicircle with radius R = 4np(1 − p) [15].
where v ∈ Rn−(m−1) . We will use Vm ∈ Um to get a lower bound Therefore, we can change variables into a space where λ is the de-
for the representation of ũ in the principal eigenspace, since the first pendent variable and x is the cumulative density function of λ. We
m − 1 columns disregard the impact of ũ. We know that: achieve this through the following substitution, where γ ranges from
“ “ ” ” “ ” 0 to π:
tr U ∗T D + θũũT U ∗ ≥ max tr Vm T
(D + θũũT )Vm . λ(γ) = R cos(γ)
v
We can rewrite the right-hand side as: 1
x = f (γ) = (γ − sin(2γ)).
!2 π
m−1
X n n Substituting this into (5) gives us:
´ X X
λi + θũ2i + 2
`
max vi−m+1 λi +θ ũi vi−m+1 . (4)
v 2 π sin2 γdγ
Z
i=1 i=m i=m 1
=− .
√ π f −1 (a) R cos(γ) + σ θ
We will approximate ũ as a vector where each entry is ±1/ n with
n
equal probability. Through introducing a new vector g ∈ R , where Solving this integral analytically is complicated and beyond the
gi = ũi vi−m+1 for i ≥ m and gi = 0 for i < m, we note that the scope of this paper, but it expresses a relationship between R, a,
quantity in (4) is maximized at the following value of g: and θ that can be computed numerically. Also, since the radius R
0 n 12 of the semicircle determines the detection bound using the PCA
n technique, we can use this to characterize the norm of the projection
1 2
X X1
arg max gi λi + θ @ gi A . of ũ onto Vm independent of R by considering θ as a proportion
g n n
i=m i=m of the detection threshold. After solving for c, we can return to the
vector formulation:
As n → ∞, we can recast g as a continuous function, where we will
n n
maximize: X g /n i
X 1/n 1
«2 = ≈− .
Z 1 c λ i +σ θ
„Z 1
g 2 (x)λ(x)dx + θ g(x)dx , i=m i=m
a a
R1 The square of the L2 norm of the projection of the signal vector ũ
where m/n → a as n → ∞. Let h(x) = − x g(t)dt, so onto the column space of Vm is given by:
0
h (x) = g(x). Adding a Lagrange multiplier σ to ensure that g is 0 n 12
m−1
unit length, we want to maximize: T 2
X 2 X
kVm ũk2 = ũi + @ gi A
Z 1 „Z 1 «2 „Z 1
(6)
«
i=1 i=m
g 2 (x)λ(x)dx + θ g(x)dx + σ g 2 (x)dx − 1 .
a a a m − 1 “ nc ”2
= + .
n θ
Rearranging terms gives us an objective function J, where:
T
Z 1 Then, using (6) we can plot kVm ũk, a lower bound of the pro-
∗
J= L(x, h(x), g(x))dx − σ, jection of ũ onto U , as shown in Fig. 2. The percentages (rela-
a tive to the threshold in Theorem 3.1) correspond to cliques of size
k = 10, 8, 6, and 4 embedded into a graph where N = 5000 and
L(x, h(x), g(x)) = g 2 (x)λ(x) − θg(x)h(a) + σg 2 (x).
p = 0.01. kVm T
ũk is a lower bound for kU ∗T ũk, the projection into
Then by the Euler–Lagrange Theorem [14], J is maximized where: the top m eigenvectors. Even significantly below the PCA detection
d ∂L threshold, a large portion of the signal vector’s magnitude lies in the
= 2 [g(x) (λ(x) + σ)]0 = 0, space spanned by a relatively small percentage of eigenvectors.
dx ∂g
Fig. 3 compares the prediction to empirical performance. The
meaning g(x) (λ(x) + σ) must be a constant c. This yields a for- black dashed curve is predicted based on the red curve in Fig. 2, the
mula: blue curve uses Vm for a random instantiation,
√ the green curve uses
c U ∗ when ũ has entries that are ±1/ n, and the red curve uses U ∗
g(x) = ,
λ(x) + σ when ũ each entry is drawn from a standard normal and the vector is
which can be substituted into the objective function, yielding: unit-normalized. The approximation where ũ has equal magnitude
Z 1 „Z 1 «2 ! in each component is always greater than the lower bound, and we
2 dx dx see similar performance when using a ũ whose entries are drawn
J =c +θ − σ.
a λ(x) + σ a λ(x) + σ from a normal distribution.
37281
Predicted Lower Bound
0.8 Lower Bound
0.8 Uniform
Random
0.6
u||2
0.6
u||
||VTm~
||UT~
0.4
0.4
41% over threshold
0.2 13% over threshold 0.2
16% under threshold
44% under threshold
0 0
0 0.1 0.2 0.3 0.4 1 4 16 64 256 1024
Proportion of Eigenvectors Used Num. Eigenvectors
T
Fig. 2. L2 norm of Vm ũ for different clique sizes. Fig. 3. Norm of ũ when projected into the principal eigenspace, for
n = 5000, p = 0.01, and θ = 6. The plot includes the predicted
lower bound, the average lower bound, the average using true eigen-
5. SIMULATION RESULTS vectors for a uniform ũ, and the average using a random ũ from the
unit hypersphere. Error bars indicate the maximum and minimum
When an approximation of the modularity matrix is used, we obtain over 10 random instantiations.
Bm , a rank-m approximation of B, by keeping the first m terms of
its eigendecomposition: 8−Vertex Clique 10−Vertex Clique
1 1
Probability of Detection
Probability of Detection
m
0.8 0.8
X
Bm = λi ui uTi ,
i=1 0.6 0.6
where λi is the ith largest eigenvalue of B, and ui is the corre- 0.4 First Eigenvector 0.4 First Eigenvector
m=5 m=5
sponding eigenvector. We will evaluate how the performance of the 0.2 m=10 0.2 m=10
DSPCA algorithm varies as a function of m through generating re- m=15 m=15
m=500 m=500
ceiver operating characteristic (ROC) curves that use the entries of 0
0 0.2 0.4 0.6 0.8 1
0
0 0.2 0.4 0.6 0.8 1
the vector returned by DSPCA as test statistics. Probability of False Alarm Probability of False Alarm
The results in this section demonstrate the outcomes of 100-trial (a) (b)
Monte Carlo simulations. In each simulation, an Erdős-Rényi graph
G(500, 0.2) is generated, and a clique of size k = 8 or k = 10 is
embedded on a subset of its vertices. Fig. 4 demonstrates that, even Fig. 4. DSPCA detection performance for several values of m, for
within the regime below the threshold elucidated in Theorem 3.1 (in G(500, 0.2) and k = 8 (a) or k = 10 (b).
this case approximately 11.18), most of the power is in the upper
eigenvectors. The DSPCA solution to the planted clique problem
represents a significant improvement over PCA methods, even in the Other future work involves studying the relationship between the
case of small m. parameter ρ in DSPCA and the associated false-alarm probability
of the algorithm. This will involve analyzing the noise characteris-
tics as more eigenvectors are added. Combined with some more ex-
6. CONCLUSIONS AND FUTURE WORK tensive Monte Carlo simulations over the various parameters of the
model, this could reveal a method to directly compute a detectability
In this paper we have demonstrated that applying DSPCA to a low- bound based on the background parameters, foreground parameters,
rank approximation of the graph’s modularity matrix is a viable al- and number of eigenvectors used.
gorithm to solve the planted clique problem. When the size of a
planted clique is reduced below the threshold where it is detectable
via PCA, it may still be well-represented in relatively few principal 7. REFERENCES
eigenvectors, which enables higher detection performance in a low-
dimensional space. One future direction involves determining the [1] M. E. J. Newman, “Finding community structure in networks
detection threshold for SPCA algorithms. We are currently working using the eigenvectors of matrices,” Phys. Rev. E, vol. 74, no.
on quantifying the breakdown point for the DSPCA approach, us- 3, 2006.
ing either the modularity matrix or its rank-m approximation, to the
[2] M. E. J. Newman and M. Girvan, “Finding and evaluating
planted clique problem. Since SPCA is NP-hard and thus compu-
community structure in networks,” Phys. Rev. E, vol. 69, no. 2,
tationally intractable, all existing algorithmic approaches motivated
2004.
by SPCA are relaxations of the initial formulation described in (2).
Alternative SPCA algorithms could prove to have higher detection [3] Jianhua Ruan and Weixiong Zhang, “An efficient spectral al-
power than DSPCA. gorithm for network community discovery and its applications
3729to biological and social networks,” in Proc. IEEE Int. Conf. [10] Quentin Berthet and Philippe Rigollet, “Complexity theoretic
Data Mining, 2007, pp. 643–648. lower bounds for sparse principal component detection,” in
[4] Ery Arias-Castro and Nicolas Verzelen, “Community detection Conf. Learning Theory, Shai Shalev-Shwartz and Ingo Stein-
in random networks,” Preprint: arXiv.org:1302.7099, 2013. wart, Eds., vol. 30 of JMLR W&CP, pp. 1046–1066. 2013.
[5] Noga Alon, Michael Krivelevich, and Benny Sudakov, “Find- [11] Florent Benaych-Georges, “Eigenvectors of Wigner
ing a large hidden clique in a random graph,” in Proc. ACM- matrices: Universality of global fluctuations,” preprint:
SIAM Symp. Discrete Algorithms, 1998, pp. 594–598. http://arxiv.org/abs/1104.1219, 2011.
[6] Yash Deshpande p and Andrea Montanari, “Finding hidden [12] Alexandre D’Aspremont, Laurent El Ghaoui, Michael I. Jor-
cliques of size N/e in nearly linear time,” preprint: dan, and Gert R. G. Lanckriet, “A direct formulation for sparse
http://www.stanford.edu/∼montanar/RESEARCH/FILEPAP/ PCA using semidefinite programming,” SIAM Review, vol. 49,
clique.pdf, 2013. no. 3, pp. 434–448, 2007.
[7] Benjamin A. Miller, Nadya T. Bliss, Patrick J. Wolfe, and [13] Florent Benaych-Georges and Raj Rao Nadakuditi, “The
Michelle S. Beard, “Detection theory for graphs,” Lincoln eigenvalues and eigenvectors of finite, low rank perturbations
Laboratory J., vol. 20, no. 1, 2013. of large random matrices,” Advances in Math., vol. 227, no. 1,
pp. 494–521, May 2011.
[8] Raj Rao Nadakuditi, “On hard limits of eigen-analysis based
planted clique detection,” in Proc. IEEE Statistical Signal Pro- [14] I. M. Gelfand and S. V. Fomin, Calculus of Variations, Dover,
cess. Workshop, 2012, pp. 129–132. 1963.
[9] Navraj Singh, Benjamin A. Miller, Nadya T. Bliss, and [15] Zhidong Bai and Jack W. Sliverstein, Spectral Analysis of
Patrick J. Wolfe, “Anomalous subgraph detection via sparse Large Dimensional Random Matrices, Springer, second edi-
principal component analysis,” in Proc. IEEE Statistical Sig- tion, 2010.
nal Process. Workshop, 2011, pp. 485–488.
3730You can also read
