Testing Applications for the Real World - Can the crowd deliver "better, faster, cheaper" testing for a software-driven world?
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Testing Applications for the Real World … Can the crowd deliver “better, faster, cheaper” testing for a software-driven world? November . 2012
Testing Your Applications for the Real World... Table of Contents 1. Background & Introduction 3 2. The Changing World of Business, IT and Applications 4 3. The Market Response: New Ways to Perform …Enter the crowd 5 4. Solutions Defined: How Does Crowdtesting Work? 7 5. Features & Benefits: When and How Is Crowdtesting Useful? 8 6. Applicability: When Is Crowdtesting Appropriate (and when is it not!) 11 7. Current Users: Who Is Crowdtesting Today? 12 8. Challenges in Using The Crowd 14 9. The Future of Innovation From The Crowd 16 10. About PASS and massolution 17 Figures 1. IT and Business Are Changing 3 2. Enterprise Crowdsourcing for Applications Testing 6 3. The passbrains Process for Crowdtesting 7 4. Sample Test Results from passbrains 9 5. Crowdtesting ROI for Market Facing Apps 11 6. The Crowd Comes From Many Directions 13 www.massolutions.com
Testing Your Applications for the Real World...
Background & Introduction
“Global firms need to deliver continuous access, any time and anywhere, often to thousands
of employees and sometimes millions of customers, over different types of devices, and are
struggling with the challenges…” 1
“ … the ability to effectively test the quality of software applications has never been more
critical to an organization’s reputation and operations.” 2
-- ComputerWeekly.com
When it comes to software applications testing, statements such as these speak directly to how crowd-
sourcing can deliver value and provide a new model for getting access to talent and resources where
companies use the scale, expertise, and diversity of the public domain to complete business-critical tasks.
Our goal is to ask and help answer questions about using the crowd for software testing as a critical part of
a company’s overall application deployment, with an emphasis on challenges facing the buyer, whether you
are a small, high-growth company or a Global 1000 enterprise.
The market is in its early stages, which calls for more clarity and education on everyone’s part, so that both
buyers and sellers can determine if and how using the crowd fits into their future applications testing. Thus,
we’ve included multiple views from buyers, analysts, and providers to assess whether crowdsourcing can
replace some current testing methods or add capabilities previously unavailable through either in-house or
traditional outsourcing approaches.
Written with the help of applications developers, software testers, and executives responsible for application
quality, PASS Technologies AG has sponsored this white paper through its passbrains division in collaboration
with massolution, a research and advisory firm, specializing in the design and implementation of enterprise
crowdsourcing and crowdfunding models. We hope you will join us for the conversation about Crowdsourced
Software Testing, or in short “crowdtesting” at www.crowdsourcing.org.
1 ‘Few Firms Test Security of Mobile Apps….” ComputerWeekly.com, Sep 2012
2 “Weighing Up The Options for Software Testing,” ComputerWeekly.com, Aug 2012.
www.massolution.com 3
Testing Your Applications for the Real World...
The Changing World of Business, IT and Applications
Today’s world is changing faster than ever. Trends in business and information technology are inextricably
linked for Global 1000 companies, governments, and consumers alike. One data point: the number of smart
phone applications is approaching 1 Million, while the number of mobile devices has surpassed the number
of desktop and laptop computers combined.3 Product life cycles have shrunk from years, to months, to
weeks in some cases, with all that implies for multiple and frequent software releases as companies struggle
to just to keep up with consumers. IT decisions and control have migrated from centralized to distributed
systems, and users are increasingly in charge (see Figure 1).
Figure 1 . IT and Business Are Changing … It’s All About the User
Old World New World
• Corporate-driven IT • Consumer-driven IT
• Centralized systems and apps • Distributed computing (The Cloud)
• Command-and-Control • Open source
• HW and SW “included” & OEM-driven • More HW & SW than ever … hundreds of thousands
• Sourcing, like IT, stays close to home of smart phone apps
• Sourcing, like IT, also becomes more open & distributed
In many cases, the physical world has moved online, from content to commerce, and applications are the life-
blood of many businesses, especially those that aspire to reach the mass markets through the latest media.
Companies are doing everything they can to refine, differentiate and maximize the end-customer experience,
which is as likely today to occur at the end of an application versus a phone call or in-person. Applications,
like other tools, must operate “to spec” (functionality), be secure yet easy to use, and work properly across
other systems (hardware, software, networks). Moreover, companies must be able to do all this at a reason-
3 “Infographic: Smartphone App Numbers,” WebPro News, Jun 2011.
www.massolution.com 4
Testing Your Applications for the Real World... able cost. As a portion of applications spending, testing comprises 20% to 30% of total costs, and “functional testing” typically makes up 80% of testing spend.4 Accordingly, F1000 companies have increased their use of third parties to get the job done. Globally companies invest more than $50 billion per year on applications testing and quality assurance,5 while research firms such as IDC and Forrester indicate the market will more spend nearly $20B of that figure on external providers, with an increase in outsourcing of 15% annually over the next several years.6 Drivers behind decisions to outsource applications testing include lower costs, better speed/time to market, and lack of in-house resources, tools and automation to ensure software works in the real world. The Market Response: New Ways to Perform …Enter the crowd With all the innovations to date - more tools, more automation, more outsourcing, the Cloud, etc. – many of the age-old business problems of effective (quality) and efficient (cost) software testing still exist: cost and time over runs, insufficient domain knowledge, tester attrition, and fragmented testing processes. As markets move faster than ever, there is now more testing, more often, and the quality of apps suffers as organizations try to keep up. For example, one global payment processing provider found that, while its software engine could process millions of individual payments without a glitch, customers who wanted to make simultaneous and sequential payments from the same account to multiple payees experienced a software failure, which is to say a “serious customer service issue.” Crowdtesting aims to raise the bar for achievable levels of quality, flexibility, speed and cost for testing software. The crowd comes in various shapes and sizes, but, as Figure 2 shows, crowdtesting for large enter- prises refers to managed services delivered over public networks through highly-skilled, uniquely qualified professionals who test consumer-facing applications, from corporate websites, to transaction processing engines, to mobile consumer life-style apps. Contrary to the populist hype around “the crowd”, getting access to your own group of dedicated testers involves a professional relationship with the testing company who can help execute on commercial testing needs. 4 “Independent Testing Services -‘It’s Time to Innovate’,” Everest Research Institute, May 2011. 5 “Top six trends to drive market for software testing in 2012,” TechJournal, citing Pierre Audoin Consultants, Nov 2011. 6 Ibid. www.massolution.com 5
Testing Your Applications for the Real World...
Figure 2 . Enterprise Crowdsourcing Model for Software Testing
Organized
• Led by certified program managers
• Single point contact
• Managed services approach offers ‘crowd management’
CLIENT LEAD
Professionalized
• Testers are screened and proven before deployment
• Tester rating system enables skill-level matching
• Many are full-time employees in lead IT shops
SENIOR QA SUPERVISORS
Standardized
• Same processes and quality on every project
• Reports generated in usable/actionable formats
• Enterprise users can re-use favorite testers 10-150 TESTERS
Online apps, broadly defined:
Company Website Mobile Payments Gaming & Entertainment Lifestyle & Social Media
Issues of privacy, security, and quality (addressed below) are all legitimate concerns for this new form of
software testing, but even without the crowd such problems exist. For example, more than half the software
commonly used by businesses fails to meet acceptable levels of security, according to a study of almost 3,000
applications.7 And the problem only worsens for mobile applications. The real question is ‘How can crowd-
testing actually help with these issues?’
Today, crowdtesting is becoming a viable testing option, able to operate effectively in enterprise environments,
where 47% of QA professionals indicate they know what crowdtesting is and how it works, and more than
one-third believe using the crowd is a trend that will change the landscape of enterprise application testing,
7 “Crowd open security testing to all developer, says Veracode,” re-printed on Cowdsourcing.org, Jan 2011.
www.massolution.com 6Testing Your Applications for the Real World...
according to a study by Forte.8 Global market
research firm IDC notes that “traditional “I think as an industry, we need to make crowd-
outsourcers will face increasing competition testing part of the normal testing process. At the
with evolving cloud {and} crowdsourcing same time you need to be clear on your objectives,
models for testing applications,” and that and use the crowd in a such a way that it is both
some “enterprises now prefer a crowd- simple and detailed in terms of bug tracking, so that
sourced marketplace … to get access to companies can integrate it into their overall testing
global talent and diverse set of skills.”9 process. Some people have the impression that
quality is an issue with crowdtesting, but I think they
have not spent enough time looking at the crowd.
How does Crowdtesting work? In my opinion, if you have the right provider, it
makes sense to invest in crowdtesting.”
For software that is consumer-oriented
and multi-platform, companies of all sizes Bodo Vahldieck, Senior Manager
and sectors are now exploring how to test Quality Management Localization Services
their applications using the crowd. As one Autodesk – a global leader in 3D design,
industry analyst and blogger notes, “… the engineering and entertainment software
crowd makes it possible to validate that appli-
cations perform to a given standard across
locations” {because} “each individual tester
figure 3 . Passbrains Process for Applications Testing
DELIVERY,
PREPARATION INITIATION EXECUTION EVALUATION ACCEPTANCE
& WRAP UP
Client requirements Test Cases Design Test Tasks Test Case Reports Test Report
analysis Execution Evaluation Delivery to
Test Configurations
customer
Test plan design Definition Test Case Test Result Metrics
Reporting Customer’s
Test resources Test Tasks Test Results Report
Project
determination Assignments Progress Design
Acceptance
• No. of Testers Monitoring &
passbrains PM- Testers’
• Skills Support Internal Project
System Set-up classification
• Configurations Review
Results refinement
Project Briefing
Proposal to client Monitoring Testers’ Feedback
for Testers Testers’
(fixed price and Test Tasks- remuneration
duration) Testers’ NDA
Reassignments /
Acceptance
Contract with client Recruitment of
Testers’ Task additional Testers
passbrains Project Acceptance (only if necessary)
Announcement
Testers recruitment
8 Efficacy & Applicability of Crowdtesting, Forte Consulting Group, Mar 2012.
9 “Top six trends to drive market for software testing in 2012,” TechJournal, citing Pierre Audoin Consultants, Nov 2011.
www.massolution.com 7Testing Your Applications for the Real World...
knows how the application must behave in his/her locale.”10 Sourcing software testing through the crowd
offers access to an on-demand, distributed and scalable workforce delivering on a pay-for-performance
basis. Figure 3 shows the passbrains process for an enterprise crowdtesting environment.
Large corporate users should expect “the same or better” in terms of the testing robustness, quality, and
detailed processes suitable for enterprise testing. It’s true that not all crowd testers are the same but
companies can identify and solve critical issues by using enterprise-capable crowdtesters. Accordingly,
the right enterprise crowdtesting solutions should offer …
• testing-as-a-service for mobile, web and enterprise apps
• “on-demand” test projects with selected, qualified test experts, with the opportunity to select and re-use
the best testers, based on customer experience
• guaranteed coverage of all major devices, computers, operating systems and browser versions
• exploratory, scripted and automated testing for functionality, compatibility, security, load/performance,
localization and end-customer usability
• the ability to test across all product development
phases, from early prototype and pre-release beta
testing to continuous post-release testing
• professional project management and quality assurance
When (and how) is crowdtesting useful?
The features and benefits of professional crowdtesting services include:
• Industrialized Testing
(also referred to as “real world testing”) “Testing is expensive and takes time and resources.
Crowdtesting certain apps is not only more cost
The irony of the evolution of software devel- effective but does things that companies cannot
opment is that no longer are corporate/ reproduce in a lab environment, which helps find
industrial software labs able to mimic and more bugs as well as issues that no one even thinks
test for all the possible permutations in the about internally. I think that while there are a few
end use of an application. In part, they lack pioneers out there now, more and more companies
the numbers (5 to 10 dedicated testers versus will decide to try crowdtesting because they will find
a crowd of 150 - 200), while in other cases that it generates real added value to their testing.”
they cannot replicate the literally thousands
of IT configurations that applications will face Vladimir Vlasic, Software Testing Manager
across multiple devices, operating systems,
Mobiliar, leading Swiss pensions and insurance
versions, and configurations.
provider
For example, in one passbrains testing
program for a Windows-based software
10 “Crowdtesting – Applicability & Benefits,” HCL ERS Blogs, Kunal Banerjee, Feb 2012.
www.massolution.com 8Testing Your Applications for the Real World...
product, there were multiple software versions, processors (32 and 64 bit) and 9 languages, including Arabic,
Chinese, English, and Russian, resulting in more than 60 different configurations and over 1,000 permutations
in just the desktop environment alone. This “ability to test under real-world conditions” is the #1 reason IT
shops turn to the crowd, as “crowdtesters normally find many more defects in an application compared to
traditional outsourcing.”11
Figure 4 . Sample Test Result from passbrains
600
500
400
# of Bugs Found
300
200
100
0
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7
Total Active Testers Total Tickets
11 Ibid.
www.massolution.com 9Testing Your Applications for the Real World...
• Speed to market
There are two types of speed that matter – the speed of getting something started and the speed of getting
something done! In an Information Week article, General Motors revealed how it has recently moved away
from traditional forms of outsourcing for applications development and testing, “so its teams can be more
nimble and creative.”12 The CIO wanted more “business-changing ideas from IT” and he wanted them faster.
He noted with traditional forms of outsourcing, however, that “When the business says ‘go’ then that means
we start working on a contract, we don’t start working on a project.” For companies that want speed and
flexibility, crowdtesting may offer a solution.
Regarding the rate of completion, enterprise buyers cited “quick-turnaround time” as the number two reason
why they look to crowdtesting.13 Typically, the completion rate is significantly faster, often in proportion
to the greater number of allocated test specialists. Whereas a typical enterprise applications test cycle
might require 2 to 3 weeks, a crowdtest solution might take only 2 to 3 days. Moreover, typically, crowdtest
resources are always “on call” so that the question “When can you start testing?” typically results in the
answer companies most want to hear … “now”. This is especially critical when testing and release cycles
once measured in months are now more likely to occur in weeks or even days.
• “Glocalization”
Crowdtesting is not the next evolution of outsourcing, so much as it is a specific strategy to address the
emerging challenges of applications in today’s world, namely how to track and test new applications across
thousands of configurations at market speed. In fact, where outsourcing led to offshoring, and the migration
of applications testing further away from the client, crowdtesting actually reverses this trend, by enabling
companies to localize testing around the ultimate end user, but on a global basis.
Crowdtest platform providers can source qualified testers for specific markets since commerce may be
global but buyers and their opinions, habits and values are often very local. Such coverage should include all
the possible virtual environments in which the applications are expected to work (operating system, device,
network, etc.), as well as all physical environments and demographic attributes – language, age, gender,
location, etc.
• Cost Savings
In passbrains’ experience, test execution through a crowdsourced platform typically costs 50% to 70% less than
traditional in-house solutions. According to analysts at Everest, “crowd labor is {about} 60% cheaper than other
models.”14 For those areas where the crowd makes sense – e.g., mobile, web and desktop software built for
B2C applications – the cost savings must also make sense. At the same, “buyer beware” applies to crowdtesting
just as it does to any other product or service, and the adage ‘you get what you pay for’ applies as well. Value
and price should align.
12 “General Motors Will Slash Outsourcing in IT Overhaul,” InformationWeek, Jul 2012.
13 Independent Testing Services -‘It’s Time to Innovate’, ...
14 “Every Crowd Has A Silver Lining,” Everest Group Research, 2011.
www.massolution.com 10Testing Your Applications for the Real World...
One key factor in lower costs through crowdtesting is the variable, “pay-as-you-go” model, something tradi-
tional forms of outsourcing have always aspired to but failed to achieve as they end up building the same
infrastructure they sought to help others avoid. Another key factor is that companies using crowdtesting
typically pay for valid bugs reported and other deliverables, as defined by the customer. Based on research
between passbrains and massolution, we find that the typical cost of an accepted and validated bug is 3 to 4
times higher in-house than with crowdtesting.15
Moreover, key applications that drive sales and customer interactions cannot afford bugs, which can not only
affect short-term sales but also damage a company’s brand and reputation as well. In one case, a global
e-retailer used crowdtesting to discover bugs that their in-house team did not find and which would have
cost the company $150,000 had the bugs gone undetected.16 Figure 5 below outlines the high-level features
and benefits of a crowdtesting approach versus either in-house or traditional outsourcing models
Figure 5: Crowdtesting ROI for Market-Facing Apps
TRADITIONAL
IN-HOUSE CROWDSOURCING
OUTSOURCING
COST “X” Savings of 20%+ Savings of 50%+
Focus on all apps with Large % of ERP Focus on external apps
SCOPE limited resources with ‘flexible’ resources with unlimited resources
SPEED Months Weeks Days
PRO: unlimited innovation
PRO: fewer IP issues PRO: services focus
QUALITY CON: potential security
CON: fewer brains CON: flexibility a myth
concerns
For market-facing applications aimed at customers or employees,
VALUE
crowdtesting can deliver more bugs, faster and at lower cost.
When is crowdtesting appropriate (and when is it not!)
So how does crowdtesting fit into an overall enterprise sourcing strategy and when is it appropriate to use
the crowd? There are several key variables companies should consider when thinking about where and how
best to use crowdtesting for their software applications.
15 Based on actual customer experiences at passbrains.
16 Ibid.
www.massolution.com 11Testing Your Applications for the Real World... Typically, crowdtesting works best when … • the Internet is ‘operating environment’ and connection mechanism • the application server is publically hosted, able to accept global requests • there is a need and capacity to handle large volumes of simultaneous users • the client computing device is open, with minimal requirements – PC, mobile • there is some level of application stability, pre- or post-release17 There are certain things that enterprises can only test through the crowd – e.g., multiple languages or “dirty configurations” that reflect actual users’ IT set up. Crowdtesting is also appropriate for multiple testing methods – e.g., waterfall, agile – and lends itself well to blackbox testing, obviously, for applications outside the firewall. However, the crowd is not a panacea. Companies cannot count on the crowd to test all the exceptions and adverse conditions driven from the web- or application-server side, especially if conditions there change before or after testing. Nonetheless, companies do use private crowds for white-box testing. For example, working with enterprises that manage sensitive customer records and data, passbrains has “anonymized” data sets or used dummy data in the past. In fact, inside the larger trend of crowdsourcing and applications testing, traditional outsourcing providers are attempting to develop dedicated, private crowds to increase the utilization rates of their own testing staff. In short, there is a time and place for companies to apply crowdtesting as a critical part of their overall testing strategy, based upon the factors above. Who is crowdtesting today? Crowdtesting is on a trajectory similar to most new markets for IT and professional services: providers of crowdtesting services are out ahead of their buyers in many cases, while many companies still seek to answer the question, ‘How can the crowd help me with my applications testing?’ Today roughly 80% of enterprise crowdtesting occurs in North American and European markets, primarily among large multi-national companies with multiple and multi-dimensional markets where the crowd can help. Leading industries mimic those who have lead the way for other types of sourcing (traditional, outsourcing, offshoring) and include financial services, telecom, publishing, health and pharmaceuticals, high-tech, retail, and travel. Again, the crowd works best wherever companies must push application-based services to the end consumer. Interestingly for smaller, high-growth companies, crowdtesting also offers a head start and accelerated testing capability, especially for companies with short product and service life cycles, which require faster time to market and more frequent application releases or updates – e.g., any iPhone banking app! Every business is, in some sense, becoming a technology company, which is why crowdtesting is so much more useful and accepted today than even a few years ago. As for the digital-age enterprises, who represent the current and next-generation stock board leaders, companies such as Amazon, eBay, Google, Groupon, Microsoft, and others are all testing their applications with the crowd. 17 “Crowdtesting – Applicability & Benefits,” HCL ERS Blogs, Kunal Banerjee, Feb 2012. www.massolution.com 12
Testing Your Applications for the Real World...
Massolution’s CsFactor™ enterprise crowdsourcing framework for identifying crowdsourcing providers and
tasks identifies over a hundred enterprise-grade providers and 65 enterprise applications of crowdsourcing,
including applications testing. Figure 6 depicts today’s “supplier market” for crowdtesting services, based
upon the various attributes and differentiators among current providers, which are broadly defined and
range in their focus and competence in areas such as testing coverage, depth of expertise, volume of testers,
geographic coverage and enterprise focus. It is important to remember that crowdsourcing is not as much a
type of supplier as it is a testing approach.
Figure 6: The Crowd Comes From Many Directions
HIGH
Managed Services
for Crowdsourcing
Legacy IT Providers
Improved Time & Outsourcers
TARGET OPPORTUNITIES
to Market
Lower Cost “The Crowd” At Large /
Independent Contractors
Higher % of
Critical Bugs Large Enterprises
Found
LOW HIGH
PERCEIVED RISKS
Protection of IP Security of Concerns over
Customer Data quality control
www.massolution.com 13Testing Your Applications for the Real World... Challenges in using the crowd As with all forms of sourcing, there can be issues around the relative comfort buyers have with alternative ways to work, and there are barriers and concerns around various methods of applications testing. For traditional outsourcers, companies have both internal and external worries, including:18 • lack of in-house testing, which makes it hard for outside providers to step in • an internal inability to compare costs to shop other options • a lack of test automation tools and expertise on the part of the provider • provider’s industry knowledge of the business and related applications Companies assessing crowdtesting may have some of these same concerns, while others are not as relevant (e.g., deep vertical or functional expertise may be less relevant for B2C apps). There are some concerns that are more particular to crowdtesting, such as the novelty of the market, issues around quality of processes and workers, and worries over data privacy, IP protection and security. Novelty - ‘It’s a new market, has the service proven itself?’ Crowdsourcing is a new model for getting work done, and crowdtesting applications is a part of this trend. As this paper shows, numerous buyers and sellers of crowdtesting are already active in the market. History tells us there will be a surge in users (whether it makes sense for all of them or not!), as the lemming effect takes hold. Later, as the market matures, industries and business will take a more deliberate but confident eye towards using the crowd. As one industry analyst notes, “The question is not whether or you should use crowdtesting, but only where and when.”19 Quality - ‘How do I know what I’m getting? Am I getting the right crowd?’ This is perhaps the most important and valid concern when considering where and how to use crowdtesting, especially given that not all crowds perform the same. Companies interested in using the crowd should focus on the quality of the provider’s platform, processes and systems … all of the things that will enable them to build and manage a chosen crowd of individual testers. Then, they should focus on the quality of those individuals – certifications, real-life experience, and the ability to complete tasks to high standards (i.e., find bugs that your in-house team could not). Key questions to assess the quality of a potential crowdtesting provider include: 1. About the testing platform and program management • How long / involved are your testing engagements (partner versus vendor)? • How do you break down component activities for crowd testers? 18 Based in part on Independent Testing Services -‘It’s Time to Innovate’, ... 19 “Crowdtesting – Applicability & Benefits,” HCL ERS Blogs, Kunal Banerjee, Feb 2012. www.massolution.com 14
Testing Your Applications for the Real World...
• How do you capture and organize accumulated knowledge across projects?
• How do you document your processes and how mature are they?
• What are your escalation procedures?
• What does a typical testing report look like – bugs found, severity, cost?
• How much time and money have you invested in your testing management system and platform (“years”
and “millions” puts you at the ‘enterprise level’)?
2. About the testers
• Where, how (and how quickly) do you source your testers?
• What security and background procedures do you follow?
• What is the profile of your testers – education, location, languages?
• How many of your testers operate at what level – e.g. % junior versus senior?
• How do you recognize and reward your testers?
• Where else are your testers employed (many will have other jobs)?
Given that apps testing is already a very prescribed and procedural activity, providers of crowdtesting
services should apply the same rigor and discipline to managing crowdtesters. For example, at passbrains,
in-house program managers source, interview, background check, and ‘beta test’ individual testers, all before
customers ever see or use them through their platform. Passbrains also engages their testers through
internal forums to send out and seek advice, so that not only are they tapping into unique testing skills for
specific environments and configurations, but they are also collectively learning how to go about testing in
the most efficient and effective way possible.
Another aspect of quality control is ‘testing fatigue’. Just as a handful of internal testers repeat the same
processes, fail to see new things or simply lose focus and interest, the same can be said of the crowd, though
it takes longer and is easier to avoid. In the case of passbrains, the company structures and layers their
testing resources to introduce new testers after 36 – 64 hours (a typical period when most crowdtesting
occurs and easy-to-find bugs emerge), and then calibrates testers’ compensation to incent and reward those
who stay with the testing process.
Security – ‘How do I protect my company’s IP and/or customer data?’
Companies using the crowd should demand the infrastructure that provides the best possible security – e.g.,
passbrains replicates “same as” systems and intranet-based testing environments as needed, depending on
the nature of the testing, including devoting dedicated servers specific to clients. However, given the nature
of crowdtesting, companies are often less worried about their IP and the guts of their apps (which testers
cannot see), than about the issue of privacy and customer data.
For example, if companies working with passbrains have privacy concerns, the company can anonymize the
data used in tested applications. Security, especially mobile security, is a huge issue whether companies
are using in-house or external resources. The gaping hole that was the entrée to many companies IP and
www.massolution.com 15Testing Your Applications for the Real World...
internal systems was once “the web” (or the website, to be more accurate). It has now switched to the
mobile world. Computer Weekly states that, “Organisations are struggling to manage the challenges of the
mobile era, with less than a third formally testing their mobile applications,”20 and “a mere 18% of 1,500
CIOs, IT directors and QA professionals {saying} their organisations were focusing on security,” according to
TechTarget.21
The Future of Innovation From The Crowd
The environment and circumstances for crowd testing will continue to evolve, but the real key is ‘flexible
innovation’. With hundreds of thousands of applications, changing platforms and devices, and endless
releases, companies are really in a constant state of beta testing their software in real-time with customers.
That’s a brave new world, and very different than the past.
According to PASS CEO and inventor of the passbrains platform, Dieter Speidel, “It’s really about harnessing
change, about innovation, which means you have to look beyond the crowd … it’s not so much a model but
a mindset.” This means that, while outsourced crowdtesting is certainly one way to get things done, as we’ve
discussed in this paper, it’s also a path to new ideas for how to use the crowd.
For example, your company might …
• engage a provider for a fully outsourced crowdtesting program
• use the crowdsourced pool of experts for consulting your employees in their day-to-day tasks and
challenges
• use crowd platforms to identify and recruit specific talent
• partner with a provider to build your own crowd, managed by someone else
• license a crowdtesting platform to build and manage your own crowd
• use the crowd to assess your in-house testing tools and capabilities
• engage yet other 3rd parties to help navigate the crowd – sourcing strategies, vendor management, and
governance for crowd-based solution
These are all possibilities depending on your company’s unique circumstances and goals. Flexibility is the
aspect of innovation that matters most … in how companies can identify, build and manage the crowd …
in how they can set up and run projects compared to traditional methods by tapping into vast resources,
quickly … and realizing that the crowd is not just about ‘doing the job’ but also about creating ideas for how
they can do the job better.
20 “Automate testing for agile quality,” ComputerWeekly.com, Jun 2012.
21 “Building software security testing skills for managers,” TechTarget, Feb 2012.
www.massolution.com 16Testing Your Applications for the Real World... Enter the crowd. When it comes to innovation, the truth is that if companies are doing it the same as they did a year ago, they are probably not as current as they think. Crowdtesting is a way to maintain and reinforce what you value – quality, reliability, usability. Today companies can blend traditional and crowdtesting approaches to maximize their ROI, especially for businesses with public-facing applications, which are difficult to test in a lab. Again, the question is not whether or you should use crowdtesting, but rather where and when. Maybe making your applications better means letting them outside the four walls of your business in new ways you have not tried before? ABOUT PASS and MASSOLUTION About PASS Group Building on long-term success in quality engineering and process excellence, the PASS Group is a Swiss- headquartered global IT and professional services provider with over two decades of experience in providing applications development and precision testing services. We help our customers, from small, high-growth start-up’s to Global 1000 companies, embrace “the crowd” through our passbrains platform and crowdsourced testing services as a way to close the speed versus quality gap in in a digital world where applications require more support, and testing, than ever before. For more information, please visit www.passbrains.com and www.pass.ch About massolution Massolution is a unique research and advisory firm specializing in the crowdsourcing and crowdfunding industry. Massolution works with leading organizations to deliver crowdsourcing and crowdfunding business models that access an on-demand, scalable workforce to deliver improved business performance and to drive product and service innovation and enhanced levels of customer engagement. Our team has experience working in large enterprise environments designing, implementing, and managing crowdsourcing initiatives. Massolution also operates the industry website Crowdsourcing.org. More information at www.massolution.com or www.crowdsourcing.org or via contact@crowdsourcing.org. www.massolution.com 17
You can also read
