APPLIED CASE STUDIES FOR CHILDREN'S DATA GOVERNANCE - An in-depth compilation of bridging principles to best practices and fostering positive ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
APPLIED CASE STUDIES FOR
CHILDREN’S DATA GOVERNANCE
An in-depth compilation of bridging principles to best practices
and fostering positive online/offline experiences for children
By: Stephanie Nguyen, Principal Research Scientist
IBM
IBM
NOTICE AND DISCLAIMER OF LIABILITY CONCERNING
THE USE OF IEEE SA CASE STUDY DOCUMENTS
This IEEE Standards Association (“IEEE SA”) publication (“Work”) is not a consensus standard document. Specifically,
this document is NOT AN IEEE STANDARD. Information contained in this Work has been created by, or obtained
from, sources believed to be reliable, and reviewed by participants that produced this Work. IEEE expressly
disclaims all warranties (express, implied, and statutory) related to this Work, including, but not limited to, the
warranties of: merchantability; fitness for a particular purpose; non-infringement; quality, accuracy, effectiveness,
currency, or completeness of the Work or content within the Work. In addition, IEEE disclaims any and all conditions
relating to: results; and workmanlike effort. This case study document is supplied “AS IS” and “WITH ALL FAULTS.”
Although the participants who have created this Work believe that the information and guidance given in this Work
serve as an enhancement to users, all persons must rely upon their own skill and judgment when making use of it.
IN NO EVENT SHALL IEEE BE LIABLE FOR ANY ERRORS OR OMISSIONS OR DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO: PROCUREMENT OF SUBSTITUTE
GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS WORK, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE AND REGARDLESS OF WHETHER SUCH DAMAGE WAS FORESEEABLE.
Further, information contained in this Work may be protected by intellectual property rights held by third parties
or organizations, and the use of this information may require the user to negotiate with any such rights holders in
order to legally acquire the rights to do so, and such rights holders may refuse to grant such rights. Attention is also
called to the possibility that implementation of any or all of this Work may require use of subject matter covered by
patent rights. By publication of this Work, no position is taken by the IEEE with respect to the existence or validity
of any patent rights in connection therewith. The IEEE is not responsible for identifying patent rights for which
a license may be required, or for conducting inquiries into the legal validity or scope of patent claims. Users are
expressly advised that determination of the validity of any patent rights, and the risk of infringement of such rights,
is entirely their own responsibility. No commitment to grant licenses under patent rights on a reasonable or non-
discriminatory basis has been sought or received from any rights holder.
This Work is published with the understanding that IEEE is supplying information through this Work, not attempting
to render engineering or other professional services. If such services are required, the assistance of an appropriate
professional should be sought. IEEE is not responsible for the statements and opinions advanced in this Work.
The following cases were submitted by the respective contributors, and do not constitute content developed by IEEE.
The organizations and products identified are for information, and inclusion in this report does not represent an
endorsement by IEEE or IEEE SA.
The Institute of Electrical and Electronics Engineers, Inc.
3 Park Avenue, New York, NY 10016-5997, USA
Copyright © 2018 by The Institute of Electrical and Electronics Engineers, Inc.
All rights reserved. Published . Printed in the United States of America.
IEEE is a registered trademark in the U.S. Patent & Trademark Office, owned by The Institute of Electrical and Electronics Engineers, Incorporated.
Trademarks included in this document are the property of their respective trademark owners. In referencing any trademarks, IEEE is providing information for
the convenience of users of this document and does not constitute an endorsement by the IEEE of these companies, products or services.
No part of this publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the prior written permission of the publisher.
Copyright © 2021 IEEE. All rights reserved. 2
TABLE OF CONTENTS
» PART 1: EXECUTIVE SUMMARY 5
» PART 2: BACKGROUND 6
» PART 3: RESEARCH OVERVIEW 7
» PART 4: CASE STUDY DATA PRINCIPLES 10
» PART 5: KEY LEARNINGS - FROM DATA POLICY TO PRACTICE 11
» POINT 1: POLICY AND LEGAL DOCUMENTATION 12
1.1 - Establish clear and robust consent language to ensure users are effectively informed
about and consent to data collection, use, and sharing 12
1.2 - Enable strong tenets supporting data minimization and data agency 12
1.3 - Integrate established guidelines to build upon best practices 13
» POINT 2: PRODUCT DESIGN, ENGINEERING & USER EXPERIENCE 12
2.1 - Build for those most at-risk communities in mind 14
2.2 - Strike a balance between parent control and child empowerment. 16
2.3 - Build ongoing or just-in-time feedback mechanisms to flag potential issues 18
2.4 - Implement a threat modeling and risk based approach 19
2.5 - Employ data minimization by default 20
» POINT 3: PROCESS AND DATA GOVERNANCE 20
3.1 - Integrate human review as opposed to rely on or strive for full automation 22
3.2 - Employ a participatory design approach to product creation and iteration 23
3.3 - Take a holistic approach to technology integration in a child’s life and all around well-being 24
3.4 - Increase transparency and accountability through open mechanisms
to continually improve the technology 26
» PART 6: CONCLUSION 27
» CASE STUDIES
Case study 1 - Parenting Phonecasts by Dost Education 28
Case study 2 - My Gem Inside by Turkcell 37
Case study 3 - Whiz Kids By Turkcell 43
Case study 4 - IBM Digital–Nation Africa 50
Case study 5 - Funzi 56
Case study 6 - LEGO Life 63
Case study 7 - PopJam by SuperAwesome 70
Case study 8 - neoGuard by Neopenda 75
Case study 9 - OWAS Platform by Privately 81
Case study 10 - Kennisnet 89
Case study 11 - AwesomeAds by SuperAwesome 97
Copyright © 2021 IEEE. All rights reserved. 3
ACKNOWLEDGEMENTS
We want to give many thanks to the 11 case study participants and teams (AwesomeAds, Digital Nation Africa, Dost
Education, Funzi, Kennisnet, LEGO Life, My Gem Inside, Neopenda, Privately (Privately’s OWAS Platform), PopJam, and
Whiz Kids) who spent several months writing, editing, and compiling examples included in this report. Thank you to the
IEEE Standards Association team and advisory members including Greg Adamson, John Havens, Ali Hessami, Aurelie
Jacquet, Konstantinos Karachalios, Nicholas Napp, Moira Patterson, Alpesh Shah, and Yu Yuan for providing early concept
strategy and advice to this activity. Finally, thank you to the other IEEE team members who also helped in the production,
marketing, and distribution of this work including Tanya Steinhauser and Donna Ceruto and designer, Debra Humphries.
The following cases were submitted by the respective contributors, and do not constitute content developed by IEEE.
The organizations and products identified are for information, and inclusion in this report does not represent an
endorsement by IEEE or IEEE SA.
Copyright © 2021 IEEE. All rights reserved. 4
PART 1: EXECUTIVE SUMMARY
This case study gallery is an in-depth compilation of bridging principles to best practices and fostering positive
online/offline experiences for children. We curated 11 unique examples of how data and privacy principles have
been designed in practice and incorporated into children’s technologies across Finland, India, the Netherlands,
Switzerland, Turkey, Uganda, United States, and the United Kingdom. The cases also represent a diverse set of
children’s related technology: learning platforms, online advertisements, social media platforms, machine learning
algorithms, newborn medical devices, training for job opportunities, and special education programs for children
with autism. For each case study and children’s technology, we outlined the following: mission and goals, key
problems, core users, process, data principles, core challenges, impact and progress, and information about the
team. The output of this work is an aggregated set of principles spanning categories across (1) policy and legal
documentation, (2) product design, engineering and user experience, and (3) process and data governance. The
principles derived from these case studies can lay the foundation for future standards related to children’s data
privacy and governance.
Data privacy and security are complex and highly regulated areas of law, particularly as related to minors. Both the
underlying technology and the regulatory regime are rapidly evolving, and applicable laws and regulations also vary
at the local, state, national and regional levels. This document recognizes but does not intend to restate or replace all
such laws and regulations, and users are responsible for referring to and observing all applicable legal and regulatory
requirements, including those related to data privacy and security for minors.
Overview of key learnings:
• 1.1 - Establish clear and robust consent language to ensure users (children and parents) are effectively
informed about and consent to data collection, use, and sharing
• 1.2 - Enable strong tenets supporting data minimization and data agency
• 1.3 - Integrate established guidelines to build upon best practices
• 2.1 - Build with those most at-risk communities in mind
• 2.2 - Strike a balance between parent control and child empowerment
• 2.3 - Build ongoing or just-in-time feedback mechanisms to flag potential issues
• 2.4 - Implement a threat modeling and risk based approach
• 2.5 - Employ data minimization by default
• 3.1 - Integrate human review as opposed to relying on or striving for full automation
• 3.2 - Employ a participatory design approach to product creation and iteration
• 3.3 - Take a holistic approach to technology integration in a child’s life and all around well-being
• 3.4 - Increase transparency and accountability through open mechanisms to continually
improve the technology
Copyright © 2021 IEEE. All rights reserved. 5
PART 2: BACKGROUND
With COVID-19 in 2020, millions of students across the world were increasingly reliant on digital platforms and
resources with social distance learning. In fact, nearly “93% of households with school-age children report some form
of distance learning during COVID,” reported1 the US Census Bureau’s Household Pulse Survey. This global pandemic
increasingly amplified already existing data-related inefficiencies with technology, especially as it relates to vulnerable
populations like children and youth. However, it is important to note that issues like data leaks, security vulnerabilities,
or overcollection of information through advertising have been long discussed before 2020. As data collection has
become ubiquitous across our lives, the likelihood of data privacy leaks as well as potential harms from barriers to
outcomes and opportunities increases, especially in the context of children. These risks are compounded by the
potential overcollection of data and the frequent lack of responsible and/or effective data management processes.
From education-related platforms, online gaming communities, and social networking sites, some negative
consequences of data sharing that children may face include “reputational damage, blackmailing, stalking or identity
theft,” researchers from the London School of Economics noted in their 2018 report,2 since children are perceived as
“more vulnerable” than adults due to their digital skills or awareness of information sharing to various audiences online.
These issues have paved the way for more policy-related interventions with these technology platforms and services.
For example, in September 2019, the United States Federal Trade Commission (FTC) issued3 a $170 million fine to
Google and YouTube for violating the Children’s Online Privacy Protection Act (COPPA). Specifically, they highlighted
how the video platform had been collecting personal information from children without their parents’ consent. As part
of the settlement, Google and YouTube would need to “implement and maintain a system that permits channel owners
to identify their child-directed content” so that they can better comply with COPPA, the FTC press release noted. In
October 2020, Ireland’s Data Protection Commissioner (DPC) launched two inquiries that highlighted how Instagram
“had made email addresses and phone numbers of users under 18 public,” the Telegraph reported.4
To date, questions around how to implement “privacy by design” through consent, data sharing, and daily use of
these applications and systems directly highlight how challenging it is to translate policy into a product or interface
in practice. For these reasons and more, the IEEE Standards Association launched the IEEE Standards Association’s
Case Study Gallery on Children’s Experiences project in 2020.
1
Schooling During the COVID-19 Pandemic.
2
Children’s data and privacy online.
3
Google and YouTube Will Pay Record $170 Million for Alleged Violations of Children’s Privacy Law.
4
Instagram under investigation for exposing millions of children’s contact details.
Copyright © 2021 IEEE. All rights reserved. 6
PART 3: RESEARCH OVERVIEW
In line with the IEEE Standards Association’s Digital Inclusion, Identity, Trust, and Agency program (DIITA), this case
study activity involves the collection, research, and writing of in-depth case studies. This includes live examples of
how privacy principles have been designed in practice and embedded in a children’s related product or service.
Mission: The mission of this work on a high level is to foster actionable and insightful information that can help
improve the interactions between technology, policy, and society in children’s lives. The community we convene
of case study contributors and the work we produce will ideally aim to set the foundations of insights for future
standards. The case study compilation highlights how teams around the world are translating privacy principles
and policies into practice and everyday use. The work also illustrates community-driven, human-centered
approaches to vulnerable populations like children and youth.
Selection criteria: Early in 2020, our team solicited nominations with clear criteria:
1. Cases must exhibit direct work with a children’s product or service. Is the team actively working on or have
worked on building or designing privacy related concepts into a children’s digital product or service?
2. Cases must be able to articulate examples of data governance through privacy, fairness, and respect, specifically
in the context of youth and children. Do teams have an example of how they contributed to better data
collection or management, privacy, security, governance, transparency, equity and/or fairness in some way?
3. Case studies must be able to be published and globally accessible by different audiences. Note: Teams will be
able to of course keep information proprietary as needed, but we would need to be able to show some level
of process detail to better examine the case study.
Diversity of cases: Based on the selection criteria and preliminary interviews with candidates, we were able to
gather 11 robust and diverse examples with representation from around the globe. We have submissions from
Finland, India, the Netherlands, Switzerland, Turkey, Uganda, United States, and the United Kingdom. The cases
also represent a diverse set of children’s related technology: learning platforms, online advertisements, social
media platforms, machine learning algorithms, newborn medical devices, training for job opportunities, and special
education programs for children with autism.
Case study framework: All 11 candidates explored important aspects of the same case study framework to outline
key principles that could be used to improve digital safeguards for children through the build and design of these
technologies. The framework for the case studies used the following structure:
• Mission and goals: Explain the goal of the technology. What do you hope to achieve?
• Key problems: What is the problem your team is solving? Why is this problem important to solve now?
• Core users: What audience(s) does the problem outlined above impact the most?
• Process: How does the platform, service, or application work? What are the key components of the
technology? How does it work on a high level?
• Data principles: What are some key data principles this work most clearly addresses? Authors focused on
principles paralleling IEEE’s Ethically Aligned Design principles.5
• Core challenges: What are two or three of the biggest challenges that your team has faced in order to make
this product or service successful?
• Impact and progress: What impact do you hope your product has on the people you intend to service? If you
already have documented progress, what impact has your product made on the communities you intend to serve?
• About the team: What team and/or organization is affiliated with this work?
Copyright © 2021 IEEE. All rights reserved. 7
On a high level, here are the case studies that were submitted. At the time of this publication, these were examples
of companies providing children’s technology products or services, identified based on the selection criteria and
preliminary interviews described above. This information is given for the convenience of users of this document and
does not constitute an endorsement by the IEEE of these companies, products or services.
Category Product / Service6 Summary of Technology
Phonecasts to support families with early child
Dost Education
development and education
A mobile application to support cognitive, emotional,
Educational My Gem Inside
and behavioral development of children with autism
A mobile application to deliver children an equal
WhizKids
opportunity for tech education
An online learning and innovation platform that
Digital - Nation
empowers individuals with in-demand skills for a range
Africa
of job opportunities
Skills Building
A mobile learning service focused on workforce
Funzi employability, career growth, entrepreneurship, and
personal development
A social digital play experience that inspires children to
LEGO® Life
build and share their creations
Social
A creative community for kids age 7-12, providing a
PopJam
moderated social network for children
A wearable vital signs monitor for neonates in low-
neoGuard™
resource settings
Health &
A privacy preserving Online Wellbeing and Safety
Well-being
(OWAS) artificial intelligence software that can be
OWAS Platform
integrated into apps and games to protect and assist
children in real time
A service concept to provide students with insight on
Data
Kennisnet who is accessing their learning data, when, for what
Management
purpose, and for what period
A digital advertising platform built to meet the needs
Privacy-
AwesomeAds ® of the children’s market: for compliance with COPPA,
Preserving
GDPR and other data privacy laws
5
Ethically Aligned Design
6
Equivalent companies, products and services may also be considered, if they can be shown to represent the same or substantially similar results.
Copyright © 2021 IEEE. All rights reserved. 8
How and why we selected these data-related principles: Each team picked two to four principles from the IEEE’s
Ethically Aligned Design principles7 which prioritizes human well-being for autonomous and intelligent systems.
The general principles include: human rights, well-being, data agency, effectiveness, transparency, accountability,
awareness of misuse, and competence. The principles selected were deliberately broad, allowing teams to choose
how to interpret and apply them to their work. They act as a strategic starting point (as opposed to a finite list of
principles) for these case studies because this was a framework that could help “guide and inform dialogue and
debate around the non-technical implementations of these technologies, in particular related to ethical aspects,”
explains the IEEE report.8 We emphasized that if teams had alternative principles they would like to use, they could
insert them into the case studies.
There were a few notes of dissonance with these principles we want to mention. They were written for “autonomous
and intelligent systems,” however, in the context of children and data-collecting systems, these technologies are
never fully autonomous. They require careful, constant integration with children to aid their ongoing and often
unpredictable learning and exploration. These experiences also require the attention and participation of multiple
stakeholders like parents, teachers, and caregivers.
Overall, the Ethically Aligned Design principles are still useful for this activity as a starting and discussion point.
The framing, priorities, and meanings of the original principles highlighted trends that help us uncover what
foundational standards can be drawn to build and design technology for children.
7
See footnote 5.
8
See footnote 5.
Copyright © 2021 IEEE. All rights reserved. 9
PART 4: CASE STUDY DATA PRINCIPLES
We aggregated and sorted all of the principles that the teams outlined. What we largely realized upon doing
this aggregation was that data agency, well-being, and awareness of misuse seemed to be more relevant to the
case studies than words related to accountability and transparency. “Data agency” was the most implemented
and relevant principle to the case studies, highlighting more opportunity to segment out the use of this term for
potential children’s related data standards. Competence was used zero times. We speculate that the principles
that were mentioned less often were due to the nature of the children’s case study and the data-related focus
seemed to position competence as less relevant.
Count Principle Definition
Technology creators shall empower individuals with the ability to
access and securely share their data, to maintain people’s capacity
8 Data agency to have control over their identity. We note that ‘agency’ is
complicated in the kids’ ecosystem, it needs to balance the agency
of two users: children and their parents, where applicable.
Technology creators shall adopt increased human well-being as a
7 Well-being
primary success criterion for development.
Awareness Technology creators shall guard against all potential misuses and
4
of misuse risks of the technology in operation.
Technology creators and operators shall provide evidence of the
3 Effectiveness
effectiveness and fitness for the purpose of the technology.
Technology that respects, promotes, and protects internationally
3 Human rights
recognized human rights.
The basis of a particular technology decision should always be
2 Transparency
discoverable.
Technology shall be created and operated to provide an
1 Accountability
unambiguous rationale for all decisions made.
Technology creators shall specify and operators shall adhere to
0 Competence
the knowledge and skill required for safe and effective operation.
Copyright © 2021 IEEE. All rights reserved. 10PART 5: KEY LEARNINGS—
FROM DATA POLICY TO PRACTICE
In this section, we take a look at all of the case studies and aggregate foundational elements in practice that can help
inform future standards in children’s products and services. The categories below can be articulated across:
1. Policy and legal documentation
Through the privacy policies and terms of service, these legal documents are positioned to disclose how
the company or organization is treating the end user’s data through collection, use, and sharing to other
third parties. These dense and complicated legal documents are likely not read by end consumers. However,
the reason why this is an important element to highlight is that it reflects the company’s culture and legal
commitments they are making to the parents, teachers, children, and other stakeholders who will be using
their products and services. A vague policy or a policy that does not include key tenets to protect the user can
be used as an indication of a company’s commitment (or lack thereof) to the people they intend to serve.
2. Product design, engineering and user experience
The product design, engineering and user experience elements featured in this section include the ways that a
technology looks, works, and can be used by children and other key stakeholders. This piece is crucial because
it is where the intangible product vision and principles translate into meaning and experiences directly with
people. So often, there is a gap in translating policies to product design and development. This section aims to
highlight ways that the 11 case study participants have integrated high level tenets like “data minimization” or
“clear user consent” or “duty of care” into the user
3. Process and data governance
The process of building technology and the ways that data is governed may not be obvious to an end user
or stakeholder since they only see the final product and its features. This section highlights whose voice
impacts the technology and how. It can include participatory design models and ways to incorporate checks
and balances into the lifecycle of a product. It is important to highlight these elements because they impact
how a product works, what the company or organization prioritizes, and most importantly, what positive and
negative impacts may occur toward a child, family member, or other stakeholder.
Copyright © 2021 IEEE. All rights reserved. 11POINT 1: POLICY AND LEGAL DOCUMENTATION
1.1 - Establish clear and robust consent language to ensure users are effectively
informed about and consent to data collection, use, and sharing
Example 1: My Gem Inside
“Through our Privacy Policy, we show what data is being gathered
and why along with what the rights of the users of the application
are. If we need other information outside what is mentioned in the
privacy policy, we ask for the consent of parents, legal guardians, or
other individuals legally responsible for the child before collecting
user information. For instance, we use emotional artificial intelligence
technology to teach emotions. Before adding the module that
processes facial images of the children, we inform and ask for consent
to process this data by issuing a pop-up. This pop-up informs parents,
legal guardians, or other individuals legally responsible for the child
that this technology is being used and asks for their permission to
move forward.”
Example 2: Kennisnet
“This dashboard shows the students their recent permission requests,
who has access to their data, and a log that shows when data was Figure 1: Kennisnet’s main dashboard shows who
accessed and by whom for what reason. The section ‘Who has access can access student data and for what reason.
to my data’ can be sorted by category or by service.”
1.2 - Enable strong tenets supporting data minimization and data agency
Example 1: Whiz Kids
“This is most clearly shown through the privacy policy
of the application that explains which personal data
is being processed for which purposes and its legal
grounds, and informs users of their rights related to
the protection of personal data as data subjects. Users
can review the privacy policy as it is freely accessible
within the application and use data subjects’ rights to
voice their questions and concerns. Whiz Kids gathers
only the necessary personal data, and some technical
information, of the device on which the child accesses
the system. This way, children may use the application
without losing progress; and parents can observe their
children’s development. Finally, the application does
Figure 2: Whiz Kids mobile application
not transfer personal data to third party receivers for
and online learning platform.
marketing or advertising purposes.”
Copyright © 2021 IEEE. All rights reserved. 12Example 2: Dost
“We require explicit consent from the user through a two-step registration before we enroll them on the phone
call program. Community workers or teachers build awareness about the program and walk them through how
they can sign up. If the user is willing to join, they need to dial Dost’s central phone number (place a “missed call”)
as a signal of explicit consent to register for the program. They immediately receive an automated welcome call
from Dost where we reconfirm their consent by giving them an option to opt-out if they do not want to participate
anymore. Users receive frequent options to unsubscribe throughout the duration of the program. Even if we have a
user’s phone number, we do not start a user on the program without this registration process.”
1.3 - Integrate established guidelines to build upon best practices
Example 1: AwesomeAds
“We set best practices to ensure any advertising seen by our audience complies with established content
guidelines from the Children’s Advertising Review Unit (CARU)9 in the US, the Code of Non-broadcast Advertising,
Sales Promotion and Direct Marketing (CAP Code)10 in the UK, and other equivalent applicable regulations.”
Example 2: Whiz Kids
“One example of how we uphold human rights in the application is that our team internally follows the
recommendations of the Convention on the Rights of the Child (CRC),11 the most widely ratified treaty.12
CRC helps guide our thinking to protect and uphold the rights and best interests of the children all around
the world, especially in terms of education and non-discrimination.”
Example 3: My Gem Inside
“We use UNICEF’s Worlds of Influence Report13 to define well-being as being in good mental health, having good
physical health, basic skills for interpersonal and career opportunities, and good quality relationships.”
9
Children’s Advertising Review Unit (CARU).
10
Non-broadcast Code - ASA | CAP.
11
Convention on the Rights of the Child.
12
Convention on the Rights of the Child text.
13
Innocenti Report Card 16 Worlds of Influence.
Copyright © 2021 IEEE. All rights reserved. 13POINT 2: PRODUCT DESIGN, ENGINEERING,
AND USER EXPERIENCE
2.1 - Build with those most at-risk communities in mind
Example 1: My Gem Inside
“Autism is a developmental difference where new skills can be taught with education; and this helps children to
continue their lives independently. To maximize the beneficial impact of education, children with autism should be
given special education without delay or without barriers to access such as cost, location, or socioeconomic status.
My Gem Inside includes more than 90 education programs developed by families and experts that support the
child’s cognitive, behavioral and emotional development.”
Figure 3: Navigating between games in the child section of My Gem Inside.
Example 2: Digital Nation Africa
“Africa is the youngest continent in the world and is home to one of the fastest growing economies.14 It is expected
that over 230 million jobs in Africa will require digital skills by 2030 and the demand will outgrow any other market
worldwide.15 [...] We define education as a fundamental human right. With the platform, we wanted to make world
class education free and accessible for every African individual. Keeping in mind that Africa has 650M mobile users, we
designed the platform to be mobile-friendly and later launched an Android app so people may learn on-the-go.”16
14
The future of entrepreneurship in Africa.
15
Digital Skills in Sub-Saharan Africa Spotlight on Ghana.
16
Africa’s mobile youth drive change.
Copyright © 2021 IEEE. All rights reserved. 14Example 3: Funzi
“The target audience of Funzi is career and entrepreneurship-seeking young adults ages 18 to 35 in urban and peri-
urban areas in emerging markets such as South Africa and Indonesia. We also allow children under 18 to use the app
related to skills training. Together with our partners, we were able to implement more human rights principles to
our work. For example, we worked with Zimba Women in Uganda to provide entrepreneurship training to women
in the International Trade Centre (ICT). The challenge was finding new and innovative tools for hubs to train women
entrepreneurs in Uganda and Africa, specifically women in ICT. So, to create stronger well-being through our
platform, we worked on migrant integration in Finland. In 2015, at the outset of the refugee crisis hitting Europe,
Funzi set up a program for migrant integration with several global partners. We created a refugee journey map to
understand the target audience, challenges, and needs.”
Figure 4. Funzi’s Migrant Integration in Finland and Entrepreneurship training to women in Uganda.
Copyright © 2021 IEEE. All rights reserved. 152.2 - Strike a balance between parent control and child empowerment
Example 1: OWAS Platform
“Privately is providing a fresh new approach to ensure safety and well-being of young children online. Historically,
online safety of children has been synonymous with parental control, filtering, and moderation solutions like those
from Netnanny and Qustodio. Most parental control solutions are primarily simplistic filtering-based solutions. This
means that parents can block websites and apps but may not be able to understand any risks like bullying or sexual
conversations that happen in the context of social networks where encrypted communication takes place. [...] The
OWAS technology makes it possible for children to be part of their own online safekeeping and well-being rather
than being excluded from this process as is the case for most of the solutions that focus on ‘parental control.’ A
study from LSE has argued for much more nuanced mediation by parents compared to current parental control
regimes. In our implementation of the ‘wup’ app for example, parents are also involved in the safekeeping of
children but not through surveillance. The parent and child promises from within the app are listed above.”
Figure 5 (left) and Figure 6 (right): Through the OWAS platform, here are
parent and child promise screens in the ‘wup’ app from ProJuventute.
Example 2: LEGO Life
“Since June 2020 every child under 16 is required to go through a full verifiable parental consent mechanism to
access the full experience. This has taken considerable investment, but offers an extremely robust consent solution
that actively engages parents and guardians in their child’s experience.”
Example 3: My Gem Inside
“We highlight a separate module that enables parents and teachers to configure the settings and follow the child’s
progress on the app. Both working with relevant stakeholders (like child psychologists and teachers) and allowing
parents and teachers access to the platforms helps to ensure effectiveness of our product for our community.”
Copyright © 2021 IEEE. All rights reserved. 16Example 4: Dost
“We use a user-centered approach to identify parent needs and design our content to be inclusive and empathetic
to the practical realities of parenting. How might we help a busy parent blend play and responsive caregiving into
their daily routine to turn regular moments into learning moments for the child? Our goal is to keep the ideas simple,
tactical and actionable. For example: Managing excessive screen time for kids through a simple storytelling activity
that builds vocabulary and communication in children.”
Example 5. PopJam
“The US Federal Trade Commission requires parental notice should a kid wish to enable push notifications,
so we use our “Kid Web Services” consent management platform to manage this process between kids and their
parents or caregivers.”
Figure 7: Parental consent through the PopJam platform.
Copyright © 2021 IEEE. All rights reserved. 172.3 - Build ongoing or just-in-time feedback
mechanisms to flag potential issues
Example 1: Neopenda
“Neopenda was created with a mission to innovate needs-based medical technologies for where they are most
vital. Improving the health and well-being is core to our mission, and is evidenced by our first product—neoGuard.
The vital signs monitoring device helps flag moments of distress for newborns in low-resource environments.
[...] By designing a solution specifically with and for users in low-resource environments, we are able to meet the
constraints witnessed by 85% of the world’s population. With neoGuard, we aim to enable clinicians to provide more
timely and appropriate treatment to patients, ultimately reducing patient morbidity and mortality rates.”
Figure 8: neoGuard patient
monitoring system functionality.
Example 2: LEGO Life
“We have integrated a Safety non-player character (NPC) into the app to deliver just-in-time safety notification
and to act as a guide throughout the entire experience. This is “Captain Safety” and he advises children on
everything that’s important, from what’s appropriate content to write in messages to why they shouldn’t share
personal information with strangers, always encouraging kids to take safety seriously. [...] Firstly, 100% of User
Generated Content (UGC) that is uploaded to the platform is pre-moderated through a process of AI and manual
review. Every user who has a piece of content rejected is delivered an educative and informative response
as every interaction with the user is considered as a learning loop.
Figure 9: Left: A photo of LEGO’s
safety pledge we share with
all users. Right: A just-in-time
“Captain Safety” notification
aimed at ensuring platform users
are careful about what they post
on the social application.
Copyright © 2021 IEEE. All rights reserved. 18Example 3: PopJam
“Some of the ways we limit personal data is by using real-time full name detection and human moderation
of all usernames chosen by children. This ensures we do not collect any personal data whilst keeping kids
anonymous and safe.”
Example 4: OWAS Platform
“In the ‘wup’ app by Pro Juventute (Switzerland’s child helpline), the ‘wup’ app features an Intelligent Keyboard
that picks up hate and toxicity in outgoing text messages and image analysis that provides immediate feedback to
the child in case any nudity is detected on a clicked or downloaded image. The app also detects bullying and sexual
conversations in incoming conversations on social media. Appropriate support is built into the app for each of the
use cases.”
2.4 - Implement a threat modeling and risk based approach
Example 1: OWAS Platform
“The whole purpose of the device monitoring done by the OWAS SDK is to determine the state of well-being of the
child—whether it is the exposure to bullying, sexting, excessive screen time, or late night usage and sleep impact.
All these measurements seek to serves children with automatic suggestions and guidance which help them to self-
regulate and use technology mindfully.”
Figures 10, 11, 12 (from left to right): The interventions and rich content of the Own It
app powered by the OWAS Platform.
Copyright © 2021 IEEE. All rights reserved. 19Example 2: Neopenda
“In terms of awareness of misuse, as an ISO13485-certified medical device company (i.e. has a quality management
system compliant to international standards), taking a risk-based approach to every decision is essential for
our activities. This includes an evaluation of human factors and applying usability engineering principles in the
development of our products. Understanding foreseeable misuse has been built into our design and development
process, and is deeply embedded in our risk management file for neoGuard. For example, neoGuard’s original
designs included the sensors placed in a baby hat. As we sought feedback from our users, our users highlighted that
it would be too difficult for them to clean the corners of a baby hat for reuse, and therefore could result in cross-
infection. We modified the design to be a headband that is easily cleaned and does not contain any sharp corners.”
Example 3: LEGO Life
“Within the App, we have avoided private chat rooms to mitigate the risk of harmful contact and carefully
curated the emoticons to mitigate harmful conduct. [...] In LEGO Life’s initial iteration, children were only able to
communicate through emoticons, a method that was considered to significantly reduce incidence of cyberbullying,
as the emoticons were chosen to represent positive sentiments. However, the singular use of emoticons significantly
reduces children’s ability to construct and communicate how they feel and limits the ranges of sentiments that can
be expressed. Over time we began to phase out the singular use of emoticons, and instead do complement this with
text. The principle of start safe and go from there remains the approach we take to technology.”
2.5 - Employ data minimization by default
Example 1: PopJam
“In PopJam, we have a creation tool which allows kids to create art
and share appropriate pictures providing they do not include their
faces or any other personal details. We identify any handwriting in the
creations, moderate any text objects added to the canvas, and ensure
that photo metadata is discarded prior to the image leaving the device.
[...] This ensures we do not collect any personal data whilst keeping kids
anonymous and safe. Traditionally, large amounts of data are captured
for analytics. To avoid this, we never directly send data to our third party
analytics system, which allows us to remove all personal data like IP
addresses and device IDs before it is sent to our analytics tools.”
Figure 13: PopJam app replacing
personal information as shared
content with emojis.
Copyright © 2021 IEEE. All rights reserved. 20Example 2: AwesomeAds
“AwesomeAds does not allow behavioral or interest-based advertising directed at children, retargeting or attribution
tracking of children. Our technology is designed with this at front of mind, and any new functionality we add is
carefully considered alongside this principle to ensure we are providing maximum benefits to our advertisers
without compromising the privacy of our users. An example of this is the campaign reporting we provide, which
gives visibility to aggregated insights into a campaign’s performance, excluding any data that could be used to track
or retarget a specific individual. [...] AwesomeAds does not allow links from kid-safe ads or content to websites and
apps that are not appropriate for children. Our system ensures only approved creatives can deliver impressions. [...]
All ad requests are categorized in multiple ways based on the associated contextual metadata, to provide granular
matching for the highly targeted campaigns.”
Example 3: Digital Nation Africa
“During the registration process, users are explicitly asked for their consent on the following: [1] Whether they
would like to subscribe to email updates for the platform. [2] Whether they would like to use their contact data
to keep them informed of related products, services, and offerings. Users can opt out of both if they do not wish
to do either. Additionally, we do not share personal information of our users without consent and legal approvals
with any external party.”
Example 4: LEGO Life
“In addition all our kids’ gaming experiences, including LEGO Life, are free from links to shop functions, third-party
websites, advertising, and other inappropriate or adult content. We never sell children’s personal data – or parents’
– to any third-party vendors or other partners. [...] LEGO Life has though, from its inception, consistently prioritized
children’s privacy. Examples of this are the random username generator that acts as a gamified solution designed
to prevent disclosure of children’s real names. It offers children the opportunity to select from a range of creative
names, making privacy feel fun and engaging.”
Example 5: Dost Education
“We do not externally share any user-level personally identifiable or demographic information, user-level call
engagement or survey response data (both touch tone responses and audio voicemail recording). All data used for
analysis for product improvements and impact reports are anonymized and aggregated.”
Example 6: OWAS Platform
Both of the apps that the OWAS Platform is integrated in (BBC’s Own It App and the ‘wup’ app by Pro Juventute) use
it to enact “‘Privacy Preserving Implementation,’ meaning that the child’s data never leaves the phone” by default.
Copyright © 2021 IEEE. All rights reserved. 21POINT 3: PROCESS AND DATA GOVERNANCE
3.1 - Integrate human review as opposed to rely on or strive for full automation
Example 1: AwesomeAds
“In the human review process, every ad creative and landing page is checked by experts to provide quick feedback to
the advertiser. [...] AwesomeAds does not allow links from kid-safe ads or content to websites and apps that are not
appropriate for children. Our system ensures only approved creatives can deliver impressions.”
Example 2: PopJam
“We also make sure all usernames generated or submitted by kids are checked for personal information by both
automated moderation and human moderators. [...] Any content delivered to children should be moderated and any
algorithms should be human-augmented. We use a combination of human and AI moderation to identify inappropriate
content. We abide by the Kidtech Standard.”
Figure 14: PopJam has a number of fun or creative features like filters, stickers, and art tools.
Example 3: LEGO Life
“Firstly, 100% of User Generated Content (UGC) that is uploaded to the platform is pre-moderated through a process of
AI and manual review. Every user who has a piece of content rejected is delivered an educative and informative response
as every interaction with the user is considered as a learning loop.”
Copyright © 2021 IEEE. All rights reserved. 223.2 - Employ a participatory design approach to product creation and iteration
Example 1: My Gem Inside
“My Gem Inside is an internationally certified educational game developed under the supervision of child psychologists
and teachers to provide an accessible education platform for children with autism. The application provides early and
continuous special education so that these children can have real education, not just in special education schools with
their peers with autism.”
Example 2: LEGO Life
“LEGO Life partnered with UNICEF (the first toy company to do so) in the development and implementation of our Digital
Child Safety Policy and on the co-creation and integration of the Child Online Safety Assessment tool (COSA) into our
business. The LEGO Group fully embraces safe-by-design, as recognized by the Australian eSafety Commissioner.
Example 3: Digital Nation Africa
“We frequently use design thinking to understand our target audience and their pain points.”
Figure 15: The learning paths based on the most in-demand job roles that are offered by the
IBM Digital – Nation Africa platform.
Copyright © 2021 IEEE. All rights reserved. 23Example 4: Dost Education
“Through a series of interviews and focus group discussions with 150 people, in urban and peri-urban low income
areas of Mumbai and Delhi over the span of two months, we learned a lot about these low-income parents.”
Figure 16 and Figure 17: Frontline workers working directly with community users of the Dost Education platform.
3.3 - Take a holistic approach to technology integration in a child’s life
and all around well-being
Example 1: Dost Education
“Within a span of two weeks, we spoke
with more than 300 families to understand
their needs during the pandemic. They
are under immense stress because of
loss of livelihood, fear for their physical
well-being and face increased instances
of domestic abuse at home. [...] Children
benefiting from the program are
measured by [1] Observations and early
childhood development assessments
done by our researchers on a sample of
users. [2] Percentage of parents reporting
the positive impact of the program on
their children through a combination of
automated surveys and phone surveys.”
Figure 18: Dost Education’s focus group session in Nizamuddin, Delhi, India
Copyright © 2021 IEEE. All rights reserved. 24Example 2: Funzi
“With Funzi, learners are able to customize their learning paths by choosing courses from livelihood and well-being-
related themes. Our goal is to increase employment, improve lives, and stimulate the growth of local communities
and economies. [...] Funzi also focuses on courses like COVID-19: Adapt & Thrive course in South Africa, Food safety
for migrants with S Group and Hanken & SSE Executive Education: Business lead “ microMBA” for migrants.”
Figure 19: Courses on Funzi are divided into handfuls of learning cards, crafted to fit mobile screens. Users are
able to learn on funzi.mobi while using little data.
Example 3: LEGO Life
“The LEGO Life app enables children to develop, through a digital learning-through-play experience, the 21st century
skills they need in an increasingly digitized world: creativity, collaboration, and critical thinking. Indeed, given that it
requires children to build, create, iterate, and problem solve offline in order to share online, it brings together the
best of technology and the best of hands-on, minds-on play to deliver a fluid and holistic play experience.”
Example 4: PopJam
“We design our services with the best interests of children in mind, and we aim to encourage positive, healthy
behaviors and balanced lifestyles. PopJam has closing hours where kids cannot socialize on the app between 11 p.m.
and 6 a.m. This is to help stop kids developing bad habits around phone usage at night. Our community management
teams and in-app messaging features all promote kindness and online safety. They help teach kids how to behave
online so they develop the skills needed to cope with adult social networks when they are older.”
Example 5: My Gem Inside
“One measure of effectiveness is that in focus groups with children with autism we observed that they increased the
duration of focus on educational content from a few minutes to approximately 45-50 minutes with the help of the
My Gem Inside application.”
Copyright © 2021 IEEE. All rights reserved. 253.4 - Increase transparency and accountability through open mechanisms
to continually improve the technology
Example 1: Kennisnet
“The EDM service designed here contains the history of approved
and denied access requests. so that there is a complete overview
of who may have access to this data about you including type of
data collected and the various information that relates to it such as
the time period that the data was collected. The student, parent,
or caregiver can revoke decisions in the EDM service at any time.
[...] The EDM service shows the current status of data access by
third parties and the history of data access by third parties. This
information is created by making use of the UMA (user managed
access) framework. Whenever a party needs access to the data there
is a check with the authorization server if consent is given by the user
as resource owner.”
Example 2: Whiz Kids
“To help ensure transparency, the evaluation criteria for student
scoring is also transparent, so users can understand how and why
they received specific scores and decision paths. Secondly, we use
open source code which makes the code publicly accessible. The
open source code means that it is possible to see that any person will Figure 20: The main dashboard showing
reach the same result of actions when they manually perform the who accesses student data, for what
same operations on the platform. This is important because the users purpose, and for which period.
will be able to understand what powers decisions behind the AI based
decision mechanism of Whiz Kids.”
Copyright © 2021 IEEE. All rights reserved. 26PART 6: CONCLUSION
It is difficult to find successful, specific and in-progress examples of putting thoughtful policies into practice. These
11 case studies highlight how companies have thoughtfully and meticulously built, designed, and ongoingly govern
technologies that stem from high level policies and principles like “privacy by design” or “stronger transparency.”
This research analysis allowed us to better highlight the types of data-protecting features that are critical to children
in different contexts—from education to social platforms.
• 1.1 - Establish clear and robust consent language to ensure users (children and parents) are effectively
informed about and consent to data collection, use, and sharing
• 1.2 - Enable strong tenets supporting data minimization and data agency
• 1.3 - Integrate established guidelines to build upon best practices
• 2.1 - Build with those most at-risk communities in mind
• 2.2 - Strike a balance between parent control and child empowerment
• 2.3 - Build ongoing or just-in-time feedback mechanisms to flag potential issues
• 2.4 - Implement a threat modeling and risk based approach
• 2.5 - Employ data minimization by default
• 3.1 - Integrate human review as opposed to relying on or striving for full automation
• 3.2 - Employ a participatory design approach to product creation and iteration
• 3.3 - Take a holistic approach to technology integration in a child’s life and all around well-being
• 3.4 - Increase transparency and accountability through open mechanisms to continually
improve the technology
The products were very diverse, ranging from a children’s social media platform for LEGO creations to hardware
devices for newborn babies in Uganda. Despite that, these companies had strong, overlapping tenets that could
help us draw principles from three categories: (1) policy and legal documentation, (2) product design, engineering
and user experience, and (3) process and data governance. These three categories can be used as an important
scaffolding for the creation of future child-related data and privacy principles to ensure that the approach
incorporates a broader system of changes within a company’s culture, product and legal systems.
This inaugural “Case Study Gallery” report focused on children’s technologies and serves as a solid example
and foundation for potential future processes that can spur IEEE standards creation. For example, future case
studies can narrow and focus on how to operationalize tenets like “data minimization,” “data portability,” or “data
consent” in practice. In addition, the analysis and case studies are positioned to share across policymaking, industry
practitioners, academic research and advocacy spaces. We hope this work is helpful to you and would be open to
hearing your feedback and insights on how this work and research is helpful to your related initiatives. Thank you for
your time and attention to this work.
Copyright © 2021 IEEE. All rights reserved. 27Parenting Phonecasts
by Dost Education
By: Sindhuja Jeyabal, Co-founder and CTO of Dost Education
Dost Education supports low income Indian
families with early child development and
education through phonecasts
28SECTION 1. MISSION AND GOALS
Dost’s mission
Dost’s mission is to get every child between the ages of 3 and 6 school-ready.
We want to put parents at the center of their child’s development and give them is to get every
the tools necessary to create an environment where their child can learn, develop, child between the
and thrive.
ages of 3 and 6
To do this, we use a combination of technology, research-based curriculum, and school-ready.
behavioral science to reach parents with busy schedules. More specifically, Dost
sends short one-minute daily podcasts over phone (“phonecasts”) to provide
families with strategically crafted information. The phonecasts are meant to nudge
and motivate parents to adopt healthy routines to support their child’s early
education and development while solving their everyday parenting problems.
For example, mothers in Hindi-speaking regions make rotis (round breads) daily,
typically priding themselves on the uniform size of each roti. They are also looking
to keep their children engaged and away from the hot stove while they cook.
Dost’s phonecast gives mothers an idea of making rotis of different sizes and
keeping the child occupied with a story that goes with it. This activity covers
an important pre-numeracy concept of “big” versus “small” objects that
children should typically develop by age 4 or 5.
Figure 1: A parent dialing Dost’s
phone number to register for the
phone call program
Copyright © 2021 Dost Education. All rights reserved. 29You can also read
