Validating Ajax Applications Using a Delay-Based Mutation Technique
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Validating Ajax Applications Using a Delay-Based Mutation
Technique
∗ ∗
Yuta Maezawa, Kazuki Nishiura, Shinichi Honiden Hironori Washizaki
The University of Tokyo Waseda University
Tokyo, Japan Tokyo, Japan
{maezawa, k-nishiura, honiden}@nii.ac.jp washizaki@waseda.jp
ABSTRACT side and asynchronously retrieve data necessary to dynam-
The challenge of validating Asynchronous JavaScript and ically update Web page content without page transitions,
XML (Ajax) applications lies in actual errors exposed in a and this is crucial to attracting Web users [3]. Moreover,
user environment. Several studies have proposed effective the number of users exceeded 2.4 billion as of June 2012 [4],
and efficient testing techniques to identify executable faults. making Ajax applications essential platforms for daily life.
However, the applications might have faults that are not ex- The challenge to validating Ajax applications arises from
ecuted during testing, but might cause actual errors in a user their event-driven, asynchronous, and dynamic features. De-
environment. Although we have investigated static methods spite concerted efforts by developers, these features mean
for finding “potential faults” that seem to cause actual er- that not all possible behaviors of running applications can
rors if executed, developers need to confirm whether or not be predicted, and these difficult-to-predict behaviors may
the potential faults are actually executable. Herein, we pro- be error-prone. To find faults in Ajax applications, a state-
pose a mutation-based testing method implemented in a tool based approach may be more effective than navigation-model-
called JSPreventer. Even if the potential faults are not easily based, code-coverage, or black-box ones [5]. Several state-of-
executable in a given environment, our method mutates the the-art studies on state-based testing [6, 7, 8, 9] have effec-
applications until they are executable using two delay-based tively and efficiently found executable faults using the state
mutation operators to manipulate the timing of the appli- space of Ajax applications. However, the applications might
cations handling interactions. Thus, JSPreventer provides have faults that are not executed during testing if the faults
executable evidences of the not-easily-executable faults for cause actual errors only when specific conditions are met,
developers, if it reveals actual errors by testing the mutated such as unexpected user operations, Web browser behaviors,
applications. We applied our method to real-world applica- and network delays. Since it would be unrealistic to test the
tions and found actual errors that developers could debug applications under all possible conditions, these faults are
to improve their reliability. Therefore, JSPreventer can help not easily detectable even if developers use the state-of-the-
developers validate reliable real-world Ajax applications. art testing techniques. Consequently, users might encounter
erroneous behaviors in the applications when a user envi-
ronment meets the specific conditions.
Categories and Subject Descriptors In our previous studies [10, 11], we have investigated static
D.2.5 [Software Engineering]: [Testing and Debugging] methods for identifying Ajax application behaviors that seem
to cause errors when specific conditions are met (e.g., un-
Keywords able to login using a correct username and password because
of improper use of asynchronous communications). Figure
Ajax; JavaScript; Web application testing; Software valida-
1 overviews our tool, called JSPreventer, that is based on
tion
our previously proposed tool, called JSVerifier, and these
features are highlighted below.
1. INTRODUCTION
Implementation We assume that Ajax design patterns [12],
Asynchronous JavaScript and XML (Ajax) [1] make Web
which have been collected from observations in many
applications responsive [2]. Ajax applications improve us-
real-world Ajax applications, describe of the properties
ability because they can handle user events on the client
of the application behaviors expected by developers.
∗Shinichi Honiden and Hironori Washizaki are also affiliated Hence, the implemented Ajax design pattern informa-
to National Institute of Informatics, Japan. tion (IADP info) can be leveraged to verify whether
Permission to make digital or hard copies of all or part of this work for an Ajax application is running as expected.
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear Extractor We assume that interactions with Ajax applica-
this notice and the full citation on the first page. Copyrights for components tions can trigger changes in the application states. In-
of this work owned by others than ACM must be honored. Abstracting with teractions correspond to how user events, asynchronous
credit is permitted. To copy otherwise, or republish, to post on servers or to server responses, and timeouts are handled. Because
redistribute to lists, requires prior specific permission and/or a fee. Request the source code contains these events and event han-
permissions from Permissions@acm.org. dlers, our extraction method [10] statically analyzes
ASE’14 , September 15 - 19 2014, Vasteras, Sweden
Copyright 2014 ACM 978-1-4503-3013-8/14/09 ...$15.00. the relationships among interactions via a call graph
http://dx.doi.org/10.1145/2642937.2642996 and constructs a finite state machine.
Ajax Design Patterns † Implemented Ajax Design Pattern Information This is because our mutation operators change non-
Implementation functional aspects of Ajax applications, but the ap-
plications can handle specific server responses after a
Previous IADP Info† Test Data/Oracles Proposed
Work Extension given delay time has elapsed. We expect that test-
JSVerifier JSPreventer ing the mutated source code with the given test data
Formulator
and oracles will reveal actual errors. Thus, JSPreven-
Interaction ter provides developers with executable evidences of
Invariants … …
Finite State Machines
…… ……
… … ……
not-easily-executable faults during testing, i.e., subtle
Revealer
Verifier
… …
network delays are required for revealing actual errors.
Mutated Source Code
Extractor Debugging Outputs from JSVerifier and JSPreventer (i.e.,
Faulty Interaction Sequences an extracted finite state machine, identified faulty in-
Actual Errors
teraction sequences, and revealed actual errors) are
…
… …
…… ……
…
… …
…
… used to debug the original source code until JSVerifier
… … …
… … …… Debugging
… … …
…
…
does not report any potential faults. JSPreventer can
Faulty Source Code Debugged Source Code then be used for validating reliable Ajax applications.
Figure 1: Overview of JSPreventer Compared with our previous studies [10, 11], the novelty
of this study lies in revealing actual errors due to poten-
Formulator and Verifier Using the IADP info as a guide, tial faults in Ajax applications. Since Ajax design patterns
our verification method [11] instantiates interaction in- aim to improve the usability of Ajax applications, code vi-
variants, and the applications handle their interactions olations against the design patterns do not always lead to
in invariant occurrence and order. Whether the ex- actual errors being debugged. Additionally, the faulty inter-
tracted finite state machine satisfies the interaction action sequences that are identified are counterexamples in
invariants is automatically verified using the NuSMV the extracted finite state machine. Counterexamples in an
model checker1 . Finally, JSVerifier obtains faulty in- abstract model can be spurious; therefore, it is important to
teraction sequences from counterexamples of the verifi- reanalyze them in an actual system [15]. Consequently, we
cation results, e.g., {onload, init(), onclick, callback(), argue that the novelty of our study is that it helps developers
... }, and reports the presence of “potential faults” that validate Ajax applications.
seem to cause actual errors if executed. Our contributions are:
Although we claim that faulty interaction sequences are • A mutation-based testing method that reveals actual
associated with potential faults reported by JSVerifier, the errors due to potential faults in Ajax applications.
following two research questions remain unanswered.
RQ1 Can JSPreventer reveal actual errors caused by po- • Definitions of synchronous and asynchronous delay-
tential faults in Ajax applications? This question can based mutation operators.
be answered by testing whether actual errors are due
to potential faults. However, the existing testing tech- • An implementation of our method in JSPreventer.
niques cannot reveal such errors because the potential
faults might not be easily executable. • Case study experiments on real-world Ajax applica-
tions demonstrating that JSPreventer can reveal ac-
RQ2 Can JSPreventer reveal these errors in a feasible amount tual errors due to potential faults.
of time? Although testing all possible scenarios in ev-
ery environment should reveal all errors, such a method This paper is organized as follows. Section 2 uses an exam-
would be unrealistic. ple to illustrate the challenge of Ajax application validation.
Section 3 describes our mutation-based testing method, while
This paper presents a mutation-based testing method to
Section 4 evaluates our method. Section 5 covers related
answer the above questions.
work, and Section 6 is the conclusion.
Revealer JSPreventer gets the faulty interaction sequences
as input from JSVerifier and tries to find executable ev-
idences of the potential faults. However, JSPreventer
2. MOTIVATING EXAMPLE
may not easily execute Ajax applications on the faulty To illustrate the aforementioned challenge, we implemented
interaction sequences because a specified environment a simple Ajax application called QAsite2 . QAsite is a typical
does not meet specific conditions to reveal actual er- Q&A website where users can ask and answer each other’s
rors due to the potential faults. Therefore, we assume questions, such as in Experts Exchange3 and Quora4 . In
that an unexpected network latency [13] may make implementing QAsite, we referred to the User Action, On-
the potential faults executable. To emulate an un- Demand JavaScript, and Direct Login patterns described
expected network latency, we define synchronous and in the Ajax design patterns [12].
asynchronous delay-based mutation operators (Section Figure 2 briefly depicts the source code, while Figure 3
3). Although a program mutation technique is basi- shows screenshots of QAsite, which runs as follows.
cally used for injecting artificial faults [14], we lever- 2
Running examples and analysis results are available from
age the technique to allow JSPreventer to make po- mzw.jp/yuta/research/ex/QAsite
tential faults executable in the specified environment. 3
www.experts-exchange.com
1 4
nusmv.fbk.eu www.quora.com
1 31 function t r y L o g i n ( r e s ) {
2 32 if ( isSuccess ( res ) ) {
3 < s c r i p t t y p e =”t e x t / j a v a s c r i p t ” 33 // C r e a t e l o g o u t a n c h o r
4 s r c =”j s / p r o t o t y p e . j s ”> 34 // D i s a b l e l o g i n form
5 < s c r i p t t y p e =”t e x t / j a v a s c r i p t ”>
18 p a r a m e t e r s : ” t a s k=g e t s e e d ” , 48 . . .
19 onSuccess : handleHttpGetSeed }) ; 49 QAsite . . .
20 } 50 Q u e s t i o n . . .
21 } 51 Answers . . .
22 function h a n d l e H t t p G e t S e e d ( r e s ) { 52 Good!
23 /∗∗ S e t r e s p o n s e t o ” s e e d ” v a r i a b l e ∗∗/ } 53 ...
24 ... 54
25 function v a l i d a t e L o g i n ( ) { 55 ...
26 new Ajax . R e q ues t (LOGIN PREFIX , 56 L o g i n / out . . .
27 method : ”GET” , 57
28 p a r a m e t e r s : ” t a s k=c h e c k l o g i n & . . . ” , 58
29 onComplete : t r y L o g i n } ) ; 59
30 } 60 . . . < / body>
Figure 2: Source code of our motivating example: QAsite
Figure 3: Screenshots of QAsite
Table 1: Actual errors in the QAsite
FIS# Ajax design pattern Brief explanation of erroneous behavior Actual error
1 User Action QAsite might not prevent multiple calls to validateLogin Unnecessary
when a user unexpectedly double-clicks on the login button. server sessions
2 On-Demand QAsite can handle onGood before successful login Invalid user
JavaScript due to a dead link to the mask image file. operations
3 Direct Login QAsite can send requests for login attempts without seed data. Invalid login
Then, a user cannot log in with his or her username and password. attempts
Faulty interaction sequence #3
Executed interaction sequence #3
init
onload
Nameless_55
onfocus
getSeed
loggedIn
(F-1)&& !hasSeed]
!hasSeed
hasSeed]
Without seed data
Branch_136
(E) Immediate resp.
onSuccess
onmouse
onm
dow
down handleHttpGetSeed
(F-2) QAsite sends account info. validateLogin onmousedown
onm
onmous
Figure 4: Finite state machine extracted from QAsite
(i) Page load: Upon a page load, QAsite registers user therefore, JSVerifier can find all the interactions in a static
event handlers for the login form and the “Good!” but- manner. For example, in Figure 2 (line 8), JSVerifier finds
ton (lines 7-12). The User Action pattern notes such that QAsite changes its state when handling the onfocus
implementations so that Ajax applications can handle event. To construct a finite state machine based on the in-
user events anytime. Additionally, the QAsite masks teractions, JSVerifier leverages a call graph that represents
the answers (line 54) to prohibit guest users from view- function caller-callee relationships because the interactions
ing them. This implementation is similar to the On- are function callbacks in response to events. Thus, JSVerifier
Demand JavaScript pattern, in which QAsite should outputs an interaction-based finite state machine containing
handle the click on the “Good!” button after users application behaviors that are independent of the testing en-
login. vironment. Figure 4 shows a finite state machine extracted
from QAsite.
(ii) Login: We implemented the login form according to Additionally, JSVerifier runs the NuSMV model checker
the Direct Login pattern. When a user sets the cursor to verify the correctness of the extracted finite state machine
on the input forms (lines 8-9), QAsite determines that (Verifier in Figure 1). However, the model checker cannot
the user intends to log in, and QAsite asynchronously determine whether or not the behaviors of Ajax applica-
retrieves seed data from the server (lines 14-23). When tions are ones developers expect. Fortunately, Ajax design
a user clicks on the login button (line 10), QAsite sends patterns are based on observations in many real-world Ajax
the username and a hash value with the password and applications. Therefore, we assume that the design patterns
seed data for secure communications to validate the present the properties of the application behaviors expected
account on the server side (lines 25-30). After receiving by developers. Hence, we define invariants to be verified
the validation result (line 29), QAsite creates a logout from the behavior properties relevant to the interactions.
widget and disables the login form if the information For example, the Direct Login pattern indicates that QA-
is valid (lines 32-35). site should receive the seed data before sending the account
(iii) Answer view: While it is creating the logout widget, information.
QAsite removes the mask (lines 32-35). The user can JSVerifier also uses the property pattern mappings5 , which
then view the answers and click the “Good!” button provide template formulas for finite-state verification, to ex-
(lines 11 and 39-40). When the user logs out, QAsite press invariants in verification formulas. As template param-
disables the logout widget, enables the login form, and eters, JSVerifier requires developers of functions or events
masks the answers again (lines 42-46). where they implement Ajax design patterns (IADP info)
so that it instantiates verification formulas (Formulator in
2.1 Results of Our Previous Studies Figure 1). As an example of the Direct Login pattern in
Although QAsite seems to run as expected, JSVerifier pro- QAsite, developers input the handleHttpGetSeed and val-
posed in our previous studies [10, 11] reports the presence idateLogin functions (lines 22 and 25 in Figure 2) as the
of three potential faults, which are listed in Table 1. Herein, IADP info. From a counterexample of the verification re-
we explain how JSVerifier works. sults, JSVerifier then identifies faulty interaction sequence
JSVerifier statically extracts a finite state machine from #3 regardless of whether or not it is executable in a testing
the source code of Ajax applications (Extractor in Figure environment (visualized by the bold solid arrow in Figure
1). In our static extraction method, we focus on interactions 4). Thus, JSVerifier reports the presence of potential faults
with Ajax applications, such as mouse clicks and server re- in Ajax applications that seems to cause actual errors if ex-
sponses, as triggers that can change the application states. ecuted.
The interactions are implemented at event handlers for user
5
events or asynchronous communications in the source code; patterns.projects.cis.ksu.eduOriginal code Revealing errors due to potential faults 1
(Step 1-a) 2 ...
Faulty Interaction Executed Interaction 3 < s c r i p t s r c =”h a n d l e C l i c k . j s ” />
Sequences (FISs) Executing on FISs Sequences (EISs)
Test data Yes Executable Figure 6: Loading a JavaScript file with improper
FISs = EISs faults (EFs) timing
(Step 2) No
Applying mutation operators Mutated code 3.1 Executing Faulty Interaction Sequences
(Step 1-b) Because executing Ajax applications results in actual er-
Executing on FISs EISs
rors if there are any, JSPreventer first attempts to execute
Delay-dependent the applications in accordance with the identified faulty in-
Yes potential faults
FISs = EISs (DdPFs) teraction sequences (Step 1-a in Figure 5). Although JSPre-
No venter uses the Selenium WebDriver6 in order to make its
*-dependent execution as automated as possible, developers must input
Yes Branch point No potential faults
found? (*dPFs) test data that cannot be inferred by any tools, such as a
Test oracle (Step 3) correct username and password, in order to test a successful
Testing mutated code Actual errors login scenario. JSPreventer executes applications with test
data and obtains the executed interaction sequences from
End the execution results. If the executed interaction sequences
and faulty interaction sequences are identical, JSPreventer
Figure 5: JSPreventer workflow determines that potential faults in the applications are exe-
cutable faults against the implemented Ajax design pat-
2.2 The Challenge terns. As previously mentioned, executable faults can be
The potential faults reported by JSVerifier can be exposed revealed through state-of-the-art methods.
under a specific set of conditions but have little chance of be- If the potential faults are not executed, JSPreventer fo-
ing found if the set is unknown. A problem with such faults cuses on the differences between the faulty interaction se-
is that developers have trouble detecting them during testing quences and the executed interaction sequences. Differences
because they cause errors only when complicated conditions may occur because Ajax applications cannot handle certain
are met [16]. Therefore, our challenge in this study is to interactions in faulty interaction sequences. The solid and
reveal actual errors due to faults in developers’ testing envi- dashed arrows in Figure 4 respectively represent faulty inter-
ronments before users encounter erroneous behaviors in the action sequence #3 and its executed interaction sequence.
applications. At the meshed circle in the figure, WebDriver emulates the
As an example of faulty interaction sequence #3, if QA- onfocus user action, but QAsite actually handles the onSuc-
site is validated with browsers in a reliable network (e.g., cess event because of an immediate server response. How-
with Firefox and Chrome browsers in a local host environ- ever, the ability to manipulate the timing of the applications
ment), it can immediately receive a response to obtain the handling the interactions allows Ajax applications to run as
seed data [(E) in Figure 4] and cannot run along with faulty faulty interaction sequences. Thus, a mutation technique
interaction sequence #3. However, an unexpected network can be used to manipulate the timing.
latency that occurs when the seed data is being obtained en- 3.2 Applying Mutation Operators
ables QAsite to send account information without the seed
data [(F-1) and (F-2) in Figure 4], which prevents the user By mutating the source code of the Ajax applications,
from logging in with his or her correct username and pass- JSPreventer enables them to run on faulty interaction se-
word. Despite concerted efforts by developers, identifying quences in a given environment (Step 2 in Figure 5). We
unexpected conditions, such as specific communication de- assume that an unexpected network latency might allow
lays in the unpredictable context of a running application, potential faults to be executable and define the following
is still extremely difficult. Consequently, JSPreventer is de- delay-based mutation operators.
signed to systematically test whether Ajax applications run
as expected on faulty interaction sequences that might not Synchronous delay: A Web browser parses the HTML
be easily executable in the given environments. code from top to bottom while loading a Web page.
When the browser finds a script element whose src
attribute has the location of an external JavaScript
3. OUR APPROACH file, it begins loading the JavaScript file. Because the
Figure 5 depicts the three-step process that JSPreven- JavaScript code can dynamically manipulate the Web
ter uses for validating Ajax applications. JSPreventer ini- page content, the browser blocks render the remain-
tially executes Ajax applications on the faulty interaction ing HTML code until the JavaScript file is completely
sequences identified by JSVerifier (Step 1-a). If the poten- loaded. Therefore, script elements can be implemented
tial faults associated with these sequences are not easily exe- at the bottom of the HTML code in order to render all
cutable, JSPreventer mutates the source code until the Ajax page elements as fast as possible. This is a well-known
applications do execute the potential faults (Steps 2 and 1- way to increase the perceived performance [17].
b). Then, developers test for the unexpected behavior due to
Loading a JavaScript file at the improper time may
the potential faults by using the mutated applications (Step
cause an actual error. Figure 6 shows brief code frag-
3). Additionally, JSPreventer classifies potential faults into
ments to explain our synchronous delay mutation op-
executable, delay-dependent potential, and *-dependent po-
6
tential faults, which are defined in Section 3.2. docs.seleniumhq.org/projects/webdriver1 // O r i g i n a l c o d e Our mutation operators do not change the functionalities of
2 $ . g e t ( ” m y s c r i p t . php ” , // URL Ajax applications because the applications can still handle
3 { name : ”Yuta ” } , // query s t r i n g
4 function ( d a t a ) { // callback
a specific interaction after the given delay has elapsed.
5 a l e r t ( ” Response i s ” + data ) ; To determine where to apply our mutation operators, JSPre-
6 }) ; venter searches for a branch point between the faulty inter-
7 action sequences and executed interaction sequences, such as
8 // Mutated c o d e
9 new D e l a y e d R e q u e s t ( 3 0 0 0 ) . a p p l y F u n c t i o n ( the meshed circle in Figure 4. The branch point may pre-
10 $ . get , // t a r g e t f u n c vent Ajax applications from running on the faulty interac-
11 ” m y s c r i p t . php ” , // URL tion sequences. The mutation operators are then applied to
12 { name : ”Yuta ” } , // q u e r y s t r i n g
13 function ( d a t a ) { // c a l l b a c k an interaction in the executed interaction sequences at this
14 a l e r t ( ” Response i s ” + d a t a ) ; point, e.g., the onSuccess event handler. Even if the server
15 }) ; immediately responds, QAsite does not handle the server
response within the delay period. By iterating Steps 1-b
Figure 7: Example of our wrapper function
and 2 in Figure 5 until the executed interaction sequences
equal the faulty interaction sequences, JSPreventer artifi-
erator. In this application, developers implement the cially sets up conditions in which potential faults can be ex-
handleClick function in the handleClick.js JavaScript posed. Finally, it classifies the potential faults that become
file and set the function as the onclick event handler executable by applying our mutation operators as delay-
of the button element. Following common practice, dependent potential faults. Thus, JSPreventer provides
the script element is then implemented at the bottom developers with executable evidences of the delay-dependent
of the HTML code. However, an unexpected network potential faults which are not-easily-executable faults be-
latency may delay loading of the JavaScript file. The cause they are not executed without subtle network delays.
application cannot respond to the onclick event oc- However, not all applications can be mutated to produce
currence because this scenario allows a user to click a identical executed interaction sequences and faulty interac-
button even though the browser has yet to register the tion sequences. Faulty interaction sequence #2 in Table
event handler. 1 means that QAsite can call the onGood function without
To emulate a JavaScript file loading delay due to an receiving the server response for login. Although JSPre-
unexpected network latency, JSPreventer inserts an ar- venter injects the same artificial delay as in the case of
tificial script element whose src attribute is located in faulty interaction sequence #3, the “Good!” button can-
our server-side program. This program then sends an not be clicked because the mask works as expected. This
HTTP response with a given delay time, causing a means that JSPreventer cannot make the potential faults
synchronous delay. This HTTP response consists of executable. JSPreventer classifies these potential faults as
header and body sections in which are set “content- *-dependent potential faults, which might cause actual
type: application/javascript” and an empty string, re- errors for reasons other than delayed injection.
spectively. In this manner, JSPreventer can manip-
ulate the timing of handling interactions relevant to 3.3 Testing Mutated Code
page loading, such as an onload event. Because developers implement Ajax design patterns ac-
cording to their intentions, we assume that they can also
Asynchronous delay: Although the most significant fea- implement test oracles to verify whether the patterns are
ture of Ajax applications is the asynchronous commu- correctly implemented in Ajax applications or not. With
nications between the client and server, an unexpected such test oracles, JSPreventer can determine whether the
network latency can significantly impact this feature. mutated code has passed the test (Step 3 in Figure 5). If
Unlike a synchronous delay, an unexpected network the code fails a test, developers can determine the actual
latency does not inhibit rendering of Web page con- errors from the test results.
tent or handling of user interactions. Consequently, JSPreventer outputs actual errors in addition to a finite
it is extremely difficult to consider all possible states state machine and faulty interaction sequences from JSVer-
where Ajax applications may have to handle a delayed ifier. We assume that these outputs will help developers
asynchronous server response. debug the original source code because they allow the com-
To emulate a handling delay with an asynchronous plex behaviors of Ajax applications to be understood, the
server response, JSPreventer rewrites the correspond- faulty behaviors of a finite state machine to be identified,
ing JavaScript code fragment by using our wrap func- and unexpected behaviors due to potential faults to be ob-
tion, DelayedRequest, as shown in Figure 7. JSPre- served. Thus, developers can use JSPreventer to prevent
venter finds a target function for an asynchronous server actual errors due to potential faults in Ajax applications.
request in the original code and parses its arguments,
such as URL, query string, and callback function. In- 3.4 Use Scenario of JSPreventer
stead of the original code, JSPreventer inserts an in- Figure 8 illustrates a JSPreventer use scenario based on
stantiation statement of the DelayedRequest with a the results of our motivating example, assuming that JSPre-
given delay time, e.g., 3000 msec. The DelayedRequest venter is applicable in the context of iterative and incre-
sends the same asynchronous server request as the orig- mental development [18]. In fact, developers often select
inal one, but does not invoke the callback function even a simplified and rapid iteration development life cycle [19].
if a response is immediately received. After the given The scenario in JSPreventer involves five steps: (i) the im-
delay time has elapsed, the DelayedRequest invokes plemented Ajax design pattern information (IADP info) is
the callback function. initially inputted to JSVerifier; (ii) JSVerifier suggests faulty(i) Input IADP Info (ii) Notice potential faults Faulty Interaction Sequence
…
(iii) Input test data and oracles
(v) Confirm correctness and reliability
(iv) Observe and debug actual errors
Figure 8: JSPreventer use scenario
interaction sequences in the applications against the IADP optical accessory supplier website was found at the top
info and reports the presence of potential faults. (iii) Devel- of the search results.
opers provide test data and oracles to JSPreventer; (iv) to
reveal unexpected runs, JSPreventer mutates and tests the UCDChina and ESA Additionally, we searched for “on-
applications. In this step, if the potential faults become to blur=checkInput” and “onsubmit=validate”, which are
be executable with subtle network delays, we assume that representative of the Live Form pattern. The Live
the executable evidences and the revealed actual errors can Form pattern suggests that Ajax applications should
be of help in validating and debugging the applications; (v) check form data before making a submission. From
through iterative development, highly reliable Ajax applica- the search results, we found Chinese and British com-
tions can be released. panies’ portal websites.
4. EVALUATION 4.2 Experimental Setup
We conducted case studies and evaluated the usefulness of Our case study experiments were conducted on a 64-bit
JSPreventer by answering the following research questions. Mac OS X 10.9.2 machine with an Intel Core i5 (2.3GHz)
and 16 GB of memory. First, we saved complete Web pages
RQ1 Can JSPreventer reveal actual errors caused by po- of the subjects by using Mozilla Firefox and obtained HTML,
tential faults in Ajax applications? CSS, and JavaScript source code files. Then, we leveraged
RQ2 Can JSPreventer reveal the errors in a feasible amount Code Beautifier10 to count lines of these codes, as listed in
of time? Table 2 (HT M L, CSS, and JS). The 1K-10K lines of code
range represents medium-largish Ajax applications. Note
4.1 Subject Ajax Applications that only client-side source code could be obtained; there-
Table 2 shows the Ajax applications which we used in fore, we implemented mock server-side scripts in PHP.
our case studies. To find real-world Ajax applications in Next, we used JSVerifier to extract the finite state ma-
which developers had implemented Ajax design patterns, we chine from each subject. Table 2 lists the extraction time
searched by using keywords relevant to Ajax design patterns (Te ) and the numbers of states and transitions of the finite
with NerdyData7 , which provides a search engine for the state machine (#states and #trans.).
source code on websites. Table 3 lists the interaction invariants used in our case
2020m We searched for “login controller.js” where James studies. The interaction invariants to be satisfied were de-
Dam8 implemented the Direct Login pattern9 . The termined for each subject (Table 4). In the source code of
7
all the subjects, we found implementations of asynchronous
nerdydata.com communications and page load event handlers. Therefore,
8
Internet Archive: goo.gl/S47AVV
9 10
Ajax Login System Demo: goo.gl/yCcxtN ctrlq.org/beautifierTable 2: Subject Ajax applications
Subject URL HTML CSS JS #states #trans. Te (sec) Tv (sec)
2020m http://2020m.com 188 420 2468 15 57 9.434 0.344
UCDChina http://ucdchina.com 1978 898 880 32 508 14.166 0.624
ESA http://www.easyservicedapartments.com 4751 964 3523 34 602 17.705 0.657
Table 3: Interaction invariants derived from Ajax design patterns
# Expected application behavior Ajax design pattern Pattern category
1 Handling user events during asynchronous communications XMLHttpRequest Call Fundamental Technology
2 Registering user event handlers at page load User Action Fundamental Technology
3 Validating form data before submission Live Form Functionality and Usability
4 Retrieving seed data before login attempt Direct Login Functionality and Usability
5 Disabling login form after successful login Direct Login Functionality and Usability
we determined that Invariants #1 (XMLHttpRequest Call) verse Lookup12 might allow interceptors to reverse-engineer
and #2 (User Action) should be satisfied in all the subjects. the hashed value, obtain the user password, and log in 2020m
Additionally, considering the search keywords, we also deter- illegally. Thus, we could infer a vulnerability to intrusion.
mined that Invariants #4 and #5 (Direct Login) in 2020m Additionally, JSPreventer revealed that UCDChina sent
and Invariant #3 (Live Form) in UCDChina and ESA should an empty text on its search form, despite developers having
be satisfied. JSVerifier then verified all the invariants for incorporated a goSearch function to prevent it. A cause of
each subject and reported the verification time (Tv in Table this erroneous behavior was that UCDChina could display
2) and the verification results (Resultv in Table 4). the search form before loading the code fragment containing
If the verification result was incorrect, JSPreventer exe- the goSearch function. Since the search form is the source
cuted the classification workflow shown in Figure 5. We set where users enter the search query, developers in this case
the delay time to be 3 seconds. Additionally, we provided all had intended to improve the search form. Although they had
the necessary test data and oracles, such as the correct user- expected only valid search queries would be sent from the
name and password, for successful logins in 2020m. Finally, search form, the error indicates that UCDChina did not val-
JSPreventer classified the potential faults that caused incor- idate the search query at all, and therefore, it possibly sent
rect verification results into executable, delay-dependent po- every search query. If so, there would be an additional pos-
tential, or *-dependent potential faults. The Resultc column sibility that a server would receive malicious search queries
in Table 4 corresponds to these classification results. JSPre- (e.g., an SQL code fragment) that might lead to an SQL
venter also reported the execution, mutation, and testing injection attack.
times (Tx , Tm , and Tt ). JSPreventer could find an undefined addthis_close func-
Additionally, we conducted comparative experiments with tion call in ESA by using the synchronous delay operator
Crawljax11 , which is a state-of-the-art tool for finding faults upon loading an addthis_widget.js file. The undefined
in Ajax applications [6, 8, 20, 9]. The default setting of function calls can be fatal errors, which might cause ESA to
Crawljax could not find any delay-dependent potential faults crash.
in the subjects. Although Crawljax ran until it completed its exploration
of the state space, it did not reveal the actual errors found by
4.3 Results and Discussions JSPreventer. These results suggest that JSPreventer reveals
Revealing actual errors due to delay-dependent actual errors due to delay-dependent potential faults.
potential faults (RQ1): JSPreventer could classify po- Feasible analysis time (RQ2): For each test subject,
tential faults reported by JSVerifier into executable, delay- JSPreventer revealed the actual errors within one minute.
dependent potential, and *-dependent potential faults. As Most of the Te , Tx , and Tt values were required to initial-
indicated in Table 5, it found actual errors in the subjects. ize parsers and to launch the test browser. Although Tv in-
We reviewed the source code of the subjects and confirmed creased linearly with the size of the extracted finite state ma-
that the actual errors could be exposed. chine, the extracted finite state machines were small enough
In 2020m, users could not log in even with their correct to be verified with the NuSMV model checker. Tm was much
username and password because 2020m sent the login re- shorter than the other amounts. These results indicate that
quest with the initial value at the declaration statement JSPreventer is practical.
of the seed variable. For secure password hashing, 2020m False positives in *-dependent potential faults: In
used the MD5 algorithm to a value obtained by adding our case studies, three potential faults could not be executed
the seed variable to the user password. Although 2020m using JSPreventer and the delay-based mutation operators.
should use the seed variable generated at runtime, it hashed As for the potential faults at Invariants #2 and #5 in 2020m,
the value with the initial value “0”; hence, 2020m sent the we observed that Mozilla Firefox prevented 2020m from run-
hashed value using the raw user password. The MD5 Re- ning on the identified faulty interaction sequences by making
11 12
crawljax.com search.cpan.org/~blwood/Digest-MD5-Reverse-1.3Table 4: Classification results of our case studies
Subject Invariant# (Ajax design pattern) Resultv Tx (sec) Tm (sec) Tt (sec) Resultc +
2020m 1 (XMLHttpRequest Call) Correct - - - -
2 (User Action) False positive 9.643 0.039 11.868 *dPF
4 (Direct Login) Potential fault 8.147 0.116 8.175 DdPF
5 (Direct Login) False positive 8.553 - - *dPF
UCDChina 1 (XMLHttpRequest Call) Correct - - - -
2 (User Action) Potential fault 11.677 0.183 11.816 DdPF
3 (Live Form) Potential fault 9.692 - - EF
ESA 1 (XMLHttpRequest Call) Correct - - - -
2 (User Action) Potential fault 10.520 0.133 10.335 DdPF
3 (Live Form) False positive 15.307 0.208 14.965 *dPF
+ EF, DdPF, and *dPF in the column of Resultc represent executable, delay-dependent potential, and *-dependent
potential faults, respectively.
Table 5: Actual errors due to delay-dependent potential faults
Subject Brief explanation Inferred vulnerability
2020m Login failed with correct username and password Intrusion
UCDChina Content search with an empty query SQL injection
ESA Undefined function call Application crash
a proper rendering block and page transition, respectively. 4.4 Threats to Validity
Since the extracted finite state machine did not contain the Internal validity threats: We considered two exter-
Web browser behavior, we determined that these potential nal factors that might affect results in our case study ex-
faults were false positives due to spurious counterexamples. periments. The results from 2020m, UCDChina, and ESA
Additionally, JSPreventer could not make the potential fault demonstrate the usefulness of JSPreventer because these
at Invariant #3 in ESA executable. In the source code of real-world applications were obtained via a public search en-
ESA, we found that a conditional branch was implemented gine provided by NerdyData. However, their actual server-
to prevent ESA from executing the potential fault. Analyz- side scripts were not available; therefore, we implemented
ing data-dependent impossible behavior is currently beyond mock server-side scripts in order to run these real-world ap-
the scope of our research; hence, this result was a false posi- plications on our machine. Because the mock server-side
tive due to our methods. However, this problem can be miti- scripts may be a threat to internal validity, we intend to
gated by using execution results to refine the extracted finite conduct additional case studies using real-world open-source
state machine, i.e., by conducting further dynamic analy- Ajax applications.
ses together with JSPreventer. A remaining issue is that Additionally, we provided the necessary information to
it is unknown whether Ajax applications actually execute run the test subjects (e.g., IADP Info, test data, and or-
*-dependent potential faults. Therefore, we are interested acles). Although the ability of developers to input correct
in defining additional effective mutation operators to make information into JSPreventer may affect the internal valid-
*-dependent potential faults executable. ity, the results of our case studies show that JSPreventer
Applying to generic Ajax applications: We designed can reveal actual errors due to potential faults. It should
JSPreventer for developers who use Ajax design patterns to be noted that the test scenarios involved typical tests, such
build applications. In actual Ajax application development, as verifying user login success with the correct username
developers have their own design patterns. If there are not and password. In the future, we intend to use JSPreventer
given IADP info for the own design patterns, JSPreventer in actual Ajax application development projects to evalu-
cannot determine code locations where our mutation opera- ate whether developers can input correct information into
tors should be applied. To test Ajax applications made with JSPreventer. We are also interested in combining a search-
their own design patterns, developers need to specify the based testing technique [21] with JSPreventer in order to
code locations of synchronous or asynchronous communica- generate test data automatically.
tions which they suspect as the cause of errors when specific External validity threats: Regarding the generality
network delays are present. JSPreventer then mutates the of our approach, JSPreventer can only reveal actual errors
code fragments at the specified code locations and tries to due to delay-dependent potential faults. However, potential
reveal the errors. faults may depend on other reasons, such as unexpected user
Debugging potential faults: We assume that the ac- operations and Web browser behaviors. Therefore, we are
tual errors revealed by JSPreventer can help developers de- going to collect *-dependent potential faults by conducting
bug potential faults. However, this debugging task depends additional case studies with JSPreventer and define effective
on the skills and experience of the developers. In the future, mutation operators for them.
we plan to establish a method to support debugging.Although 2020m, UCDChina, and ESA are practical Ajax Zheng et al. also conducted a rules-based static analy-
applications, it would be interesting to determine the scala- sis to detect data races due to asynchronous calls (i.e., the
bility of JSPreventer by obtaining experimental results using sort of interactions dealt with in this paper) in Ajax applica-
many real-world Ajax applications. tions [13]. Although they focused on global variables, which
are shared by asynchronous calls, JSPreventer can identify
faulty asynchronous calls without data races by leveraging
5. RELATED WORK the Ajax design patterns as oracles. Additionally, Zheng et
In this section, we describe the state-of-the-art studies on al. were not interested in testing unexpected behaviors in
the analysis and testing of Ajax applications. the applications if a detected data race occurred. Although
State-based analysis and testing: Marchetto et al. they suggested two methods to fix data races, we argue that
introduced the concept of state-based analysis and testing the actual errors found by JSPreventer can help developers
to Ajax applications [22]. Their tool, ReAjax[23], extracts debug their applications and confirm the correctness of the
finite state machines from the execution results of Ajax ap- codes.
plications. The finite state machines consist of Document Guarnieri et al. introduced a pure static taint analysis
Object Model (DOM)13 instances and the effects of callback for JavaScript code in order to identify security vulnera-
executions as states and transitions. However, ReAjax re- bilities such as cross-site scripting and SQL injection [28].
quires developers to manually execute Ajax applications to Wei et al. pointed out the dynamic feature of JavaScript;
sufficiently trace the execution results. JavaScript code can dynamically get additional code from
Towards automated execution, Mesbah et al. implemented the server at runtime. They proposed a blended taint anal-
Crawljax [24], which can automatically emulate user ac- ysis of the JavaScript code that can be collected by executing
tions, i.e., crawling Ajax applications. Crawljax has been test cases [29]. Although these analyses can output precise
used to detect DOM-related faults [6], to test cross browser control flows of only the JavaScript code, JSPreventer ana-
compatibility [8], and to determine whether CSS rules are lyzes stateful behaviors in the HTML, CSS, and JavaScript
unnecessary or not [20]. Additionally, Choudhary et al. code of Ajax applications.
leveraged Crawljax to improve the accuracy of identifying Mutation analysis and testing: Mutation analysis is
cross browser issues [9]. Although Crawljax is very useful a fault-based technique to assess the adequacy of test cases
for finding executable faults, such crawling approaches can- [14, 30]. It seeds artificial faults into the program under test
not reveal potential faults because they may not be easily and then measures whether the seeded artificial faults can
executable in the given environments. be detected in the test cases. This technique in its original
Arzti et al. presented Artemis [7] as a way to improve form might be applicable to Ajax applications [31, 32]. In
code coverage by using the feedback-directed technique [25]. contrast, JSPreventer leverages a deformed mutation analy-
Artemis analyzes historical test execution data and gener- sis technique, wherein seeded artificial delays do not act as
ates test cases to explore the state space of Ajax applica- faults to be detected but expose existing potential faults in
tions. The authors noted that Artemis may help Crawljax Ajax applications. Although JSPreventer cannot be used to
determine what user actions should be emulated. However, assess the adequacy of test cases, it can reveal actual errors
Crawljax and Artemis cannot be combined in a way that due to potential faults.
would determine potential faults in Ajax applications.
Hong et al. have a similar concern of finding concurrency
errors due to the improper timing of Ajax applications han- 6. CONCLUSION
dling the interactions [26]. Their tool called WAVE is an ex-
We proposed a support tool, called JSPreventer, for vali-
tended implementation of the WebKit14 browser framework
dating Ajax applications by finding executable evidences of
for managing event invocations on the JavaScript engine.
potential faults that may not be easily executed in the given
However, the HTML rendering engine can also affect event
development and test environments. We implemented delay-
invocations, such as the rendering block, as we mentioned in
based mutation operators to artificially make the poten-
Section 3.2. Since JSPreventer reveals errors due to poten-
tial faults executable and reveal actual errors due to delay-
tial faults of Ajax applications on standard Web browsers
dependent potential faults. Our experimental results show
that users actually use, it can more precisely suggest erro-
that JSPreventer reveals actual errors due to delay-dependent
neous behavior of the applications to developers.
potential faults in real-world Ajax applications in a feasible
Static approach: As for static approaches, Guha et
amount of time. By debugging the revealed errors, devel-
al. proposed a static method to prevent Ajax applications
opers can develop and validate highly reliable Ajax applica-
from handling invalid server requests [27]. Their frame-
tions.
work constructs a request graph through control flow anal-
Our future research will proceed in three separate direc-
ysis, which represents how Ajax applications handle asyn-
tions. One direction is to develop automated debugging.
chronous server requests with an invariant order. The re-
Besides establishing a method that effectively uses solutions
quest graph can be used as an oracle to validate runtime
in Ajax design patterns, we plan to combine JSPreventer
server requests. However, if developers cannot correctly im-
with mechanisms for determining actual errors in existing
plement Ajax applications, the request graph cannot reject
debugger tools, such as Firebug15 . The second direction is
invalid server requests relevant to potential faults. Thus,
to implement more mutation operators to reveal actual er-
developers debugging the potential faults using JSPreventer
rors due to *-dependent potential faults. The third direction
can construct a more proper request graph from the de-
is to expand the range of case studies on large-scale Ajax ap-
bugged applications.
plications in the real world.
13
www.w3.org/DOM
14 15
www.webkit.org getfirebug.com7. REFERENCES [17] Steve Souders. High Performance Web Sites. O’Reilly
[1] Jesse James Garrett. Ajax: A New Approach to Web Media, Inc., 2007.
Applications. www.adaptivepath.com/ideas/ [18] C. Larman and V.R. Basili. Iterative and Incremental
ajax-new-approach-web-applications. Developments: A Brief History. Computer,
[2] Brent Stearn. XULRunner: A New Approach for 36(6):47–56, 2003.
Developing Rich Internet Applications. IEEE Internet [19] Anthony T. Holdener, III. Ajax: The Definitive Guide.
Computing, 11(3):67–73, 2007. O’Reilly Media, Inc., 2008.
[3] Jakob Nielsen and Hoa Loranger. Prioritizing Web [20] Ali Mesbah and Shabnam Mirshokraie. Automated
Usability. New Riders Press, Berkeley, CA, 2006. Analysis of CSS Rules to Support Style Maintenance.
[4] Internet World Stats. World Internet Usage Statistics In Proc. Int’l Conf. on Software Engineering (ICSE),
News and World Population Stats. pages 408–418, May 2012.
www.internetworldstats.com/stats.htm. [21] Nadia Alshahwan and Mark Harman. Automated Web
[5] Alessandro Marchetto, Filippo Ricca, and Paolo Application Testing Using Search Based Software
Tonella. A case study-based comparison of web testing Engineering. In Proc. Int’l Conf. on Automated
techniques applied to AJAX web applications. Int’l Software Engineering (ASE), pages 3–12, November
Journal on Software Tools for Technology Transfer 2011.
(STTT), 10(6):477–492, 2008. [22] Alessandro Marchetto, Paolo Tonella, and Filippo
[6] Ali Mesbah and Arie van Deursen. Invariant-based Ricca. State-based Testing of Ajax Web Applications.
Automatic Testing of AJAX User Interfaces. In Proc. In Proc. Int’l Conf. on Software Testing, Verification
Int’l Conf. on Software Engineering (ICSE), pages and Validation (ICST), pages 121–130, April 2008.
210–220, May 2009. [23] Alessandro Marchetto, Paolo Tonella, and Filippo
[7] Shay Artzi, Julian Dolby, Simon Holm Jensen, Anders Ricca. ReAjax: a reverse engineering tool for Ajax
Moller, and Frank Tip. A Framework for Automated Web applications. IET Software, 6(1):33–49, 2012.
Testing of JavaScript Web Applications. In Proc. Int’l [24] Ali Mesbah, Engin Bozdag, and Arie van Deursen.
Conf. on Software Engineering (ICSE), pages Crawling AJAX by Inferring User Interface State
571–580, May 2011. Changes. In Proc. Int’l Conf. on Web Engineering
[8] Ali Mesbah and Mukul R. Prasad. Automated (ICWE), pages 122–134, July 2008.
Cross-Browser Compatibility Testing. In Proc. Int’l [25] Carlos Pacheco, Shuvendu K. Lahiri, Michael D.
Conf. on Software Engineering (ICSE), pages Ernst, and Thomas Ball. Feedback-Directed Random
561–570, May 2011. Test Generation. In Proc. Int’l Conf. on Software
[9] Shauvik Roy Choudhary, Mukul R. Prasad, and Engineering (ICSE), pages 75–84, May 2007.
Alessandro Orso. X-PERT: Accurate Identification of [26] Shin Hong, Yongbae Park, and Moonzoo Kim.
Cross-Browser Issues in Web Applications. In Proc. Detecting Concurrency Errors in Client-side
Int’l Conf. on Software Engineering (ICSE), pages JavaScript Web Applications. In Proc. Int’l Conf. on
702–711, May 2013. Software Software Testing, Verification and Validation
[10] Yuta Maezawa, Hironori Washizaki, and Shinichi (ICST), pages 61–70, March 2014.
Honiden. Extracting Interaction-based Stateful [27] Arjun Guha, Shriram Krishnamurthi, and Trevor Jim.
Behavior in Rich Internet Applications. In Proc. Using Static Analysis for Ajax Intrusion Detection. In
European Conf. on Software Maintenance and Proc. Int’l World Wide Web Conf. (WWW), pages
Reengineering (CSMR), pages 423–428, March 2012. 561–570, April 2009.
[11] Yuta Maezawa, Hironori Washizaki, Yoshinori Tanabe, [28] Salvatore Guarnieri, Macro Pistoia, Omer Tripp,
and Shinichi Honiden. Automated Verification of Julian Dolby, Stephen Teilhet, and Ryan Berg. Saving
Pattern-based Interaction Invariants in Ajax the World Wide Web from Vulnerable JavaScript. In
Applications. In Proc. Int’l Conf. on Automated Proc. Int’l Symp. on Software Testing and Analysis
Software Engineering (ASE), pages 158–168, (ISSTA), pages 177–187, July 2011.
November 2013. [29] Shiyi Wei and Barbara Ryder. Practical Blended Taint
[12] Michael Mahemoff. Ajax Design Patterns. O’Reilly Analysis for JavaScript. In Proc. Int’l Symp. on
Media, Inc., 2006. Software Testing and Analysis (ISSTA), pages
[13] Yunhui Zheng, Tao Bao, and Xiangyu Zhang. 336–346, July 2013.
Statically Locating Web Application Bugs Caused by [30] Yue Jia and Mark Harman. An Analysis and Survey of
Asynchronous Calls. In Proc. Int’l World Wide Web the Development of Mutation Testing. IEEE Trans.
Conf. (WWW), pages 805–814, April 2011. on Software Engineering (TSE), 37(5):649–678,
[14] R.A. DeMillo, R.J. Lipton, and F.G. Sayward. Hints September 2011.
on Test Data Selection: Help for the Practicing [31] S. Mirshokraie, A. Mesbah, and K. Pattabiraman.
Programmer. Computer, 11(4):34 –41, April 1978. Efficient JavaScript Mutation Testing. In Proc. Int’l
[15] Cong Tian and Zhenhua Duan. Detecting Spurious Conf. on Software Testing, Verification and Validation
Counterexamples Efficiently in Abstract Model (ICST), pages 74–83, March 2013.
Checking. In Proc. Int’l Conf. on Software [32] Kazuki Nishiura, Yuta Maezawa, Hironori Washizaki,
Engineering (ICSE), pages 202–211, May 2013. and Shinichi Honiden. Mutation Analysis for
[16] Michael Grottke and Kishor S. Trivedi. Fighting Bugs: JavaScript Web Applications Testing. In Proc. Int’l
Remove, Retry, Replicate, and Rejuvenate. Computer, Conf. on Software Engineering and Knowledge
40(2):107–109, 2007. Engineering (SEKE), pages 159–165, June 2013.You can also read
