How to design and evaluate a Regulatory Experiment? - A Guide for Public Administrations - Öko ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
How to design and evaluate a Regulatory Experiment? A Guide for Public Administrations
Authors and contact details:
Prof. Dr. Dierk Bauknecht (d.bauknecht@oeko.de),
Dirk Arne Heyen, Dr. Peter Gailhofer
Oeko-Institut e.V.
Freiburg, Darmstadt, Berlin
Prof. Dr. Kilian Bizer (bizer@wiwi.uni-goettingen.de),
Dr. Daniel Feser
Chair of Economic Policy and SME Research
Faculty of Business and Economics
Georg-August-University Goettingen
Prof. Dr. Martin Führ (martin.fuehr@h-da.de),
Simon Winkler-Portmann Layout:
Society for Institutional Analysis (sofia) Bertram Sturm,
at the University of Applied Sciences Darmstadt. www.bertramsturm.de
Prof. Dr. Kilian Bizer (bizer@wiwi.uni-goettingen.de), Cover photo:
Thore Sören Bischoff, M.A., Dr. Till Proeger istockphoto / wildpixel
Institute for Small Business Economics at the
Georg‐August‐University Goettingen (ifh Göttingen) March 2021
Project background & acknowledgment:
These guidelines have been developed in the context of the University of Göttingen (ifh Göttingen), 3) the Society for insti-
two-year research project entitled “Regulatory experiments for tutional analysis (sofia) at the Darmstadt University of Applied
the reflexive and adaptive governance of innovation”, funded Sciences, and 4) the Öko-Institut – Institute for Applied Ecolo-
by the German Federal Ministry for Education and Research un- gy, Berlin/Freiburg.
der grant no. 16ITA213.
More information can be found on the project website: https://
The project has been conducted by four partners: 1) the Chair reragi.wordpress.com/reragi-englisch/ A list of published scien-
of Economic Policy and SME Research at the University of Göt- tific papers in this context can be found at https://reragi.word-
tingen, 2) the Institute for Small Business Economics at the press.com/2019/04/09/veroeffentlichungen/ and in the Annex.
V RI L D
6 R Q G H U I R U V F K X Q J V J U X S S H
, Q V W L W X W L R Q H Q D Q D O \ V H
How to design and evaluate a Regulatory Experiment?
2
A Guide for Public Administrations
Table of contents
Executive Summary 4
1 Regulatory experiments: enhancing field knowledge for
better regulation towards Sustainable Development 6
2 Types of regulatory experiments and their respective benefits 7
2.1 Regulatory Sandboxes: exemptions from existing legal rules 7
2.2 Regulatory Innovation Trials (RITs): experimenting with new regulatory options 8
3 Key issues to consider for regulatory experiments 10
3.1 Learning & evaluation throughout the experiment 10
3.2 Stakeholder involvement 11
3.3 Legal issues 11
4 Design, implementation and evaluation of regulatory experiments 14
4.1 Phase 1: Clarification: Goal of the experiment and determination of type 14
4.2 Phase 2: Preparation of the experiment 16
4.3 Phase 3: Implementation 22
4.4 Phase 4: Evaluation and upscaling 23
Annex: List of project publications and case studies 24
How to design and evaluate a Regulatory Experiment?
3
A Guide for Public Administrations
Executive Summary
Experimenting with regulations? What may sound strange legal situation. In turn, RITs also concern technical artifacts,
at first can be a fruitful way to support learning processes on infrastructures or social practices as part of the real-world
technical, social or regulatory innovations and better assess setting in which they take place. However, the distinction
their impacts ex-ante. is still useful since the focus and learning goals of the two
types of regulatory experiments differ as described in the
text box (see Chapter 2 for details).
What are regulatory experiments?
Broadly speaking, regulatory experiments can be defined What are such experiments good for?
as a means to deliberately deviate from the current reg-
ulatory framework to try out new or different rules in a As the Council of the European Union (EU) concluded in
real-world setting. General characteristics are the key role 2020, regulatory experiments “provide the opportunity for
of public regulations, the involvement of government actors advancing regulation through proactive regulatory learning,
(whether local or national), and the generation of learning enabling regulators to gain better regulatory knowledge
processes. and to find the best means to regulate innovations based on
real-world evidence, especially at a very early stage, which
When speaking of “regulation” and “regulatory experiments”, can be particularly important in the face of high uncertainty
we mean the whole range of public policy instruments, and disruptive challenges, as well as when preparing new
procedures and organizational structures. policies” (Council document 13026/20).
We differentiate between two key types of regulatory exper- What the Council of the EU concluded mainly regarding Reg-
iments (see text box, and Chapter 2 for details). While in type ulatory Sandboxes is even more true for RITs. Being standard
1 – “Regulatory Sandboxes” – regulation is mainly the frame- in the case of technical innovations in R&D processes, test-
work of socio-technical experiments, in type 2 – “Regulatory ing innovative regulatory options in a small but real-world
Innovation Trials” – regulation itself is the main object of ex- setting before being “rolled out” can help to better design
perimentation and learning. effective and efficient regulation from the outset. Address-
ees of regulation but also society as a whole can benefit
Î “Regulatory Sandboxes” aim to allow testing techni- from a less costly but still goal-effective policy design.
cal, social or organizational innovations by (initially)
temporary exemptions from existing legal rules (e.g. Regulatory experiments seem particularly useful for ad-
through experimentation clauses). Examples range from dressing the major sustainability challenges that our world
exemption clauses for testing autonomous driving and currently faces. Any kind of governance dealing with these
delivery to regulatory sandboxes for testing innovations challenges and potential solutions must cope with profound
in the energy and fintech sectors. problem interdependencies and uncertainties about the ef-
fects of interventions. This calls for a “culture of analysis” in
Î “Regulatory Innovation Trials” (RITs) aim to test new political decision-making, thoroughly scrutinizing the ben-
regulatory options and learn about their impact before efits and burdens related to a regulation. The knowledge
introducing them on a permanent basis and eventually gained in experiments is supposed to complement other
nationwide or elsewhere. Examples range from basic in- methods for ex-ante policy-impact assessments such as
come experiments with a limited number of participants modeling.
to testing new traffic rules in a geographically limited area.
At the same time, regulatory experiments should not be
The distinction between the two types is ideal-typical: Like used without a serious intention to learn, i.e. neither to justi-
RITs, Regulatory Sandboxes can also entail experimental ad- fy policy choices already taken nor to delay policy decisions
aptations of subordinate regulations and represent a new for tactical reasons.
How to design and evaluate a Regulatory Experiment?
4
A Guide for Public Administrations
Aim & background of these guidelines The regulatory experiment may be located in different con-
texts and different phases of the development of new regu-
The following guidelines aim to support officials in pub- lation. For example, the experiment may be carried out at an
lic administration (whether in the EU, national ministries, early stage to test new regulatory options, or it may be part
agencies, or local administration) in preparing, implement- of a more formal impact assessment, when certain options
ing and/or evaluating real-world regulatory experiments. have already been pre-selected in the policy process. Experi-
The overall aim is to foster regulatory learning and the de- ments may be stand-alone or part of a larger program. These
velopment of a better regulation around societal efforts to- guidelines are written to be useful for these different kinds
wards the UN Sustainable Development Goals (SDGs). of experiments.
Public officials who are both experienced and inexperienced The guidelines are based on insights from a research proj-
in regulatory experiments should benefit from the recom- ect on regulatory experiments that included an extensive
mendations and best practices presented. The guidelines are literature review as well as a detailed analysis of 27 cases of
written in a generic manner, primarily addressing the main ac- regulatory experiments (see Annex) from all over the world
tor in charge who drives and oversees the process – although related to Sustainable Development (see text box on page 2
different actors, different administrative units or organiza- for further information on the project).
tions might be responsible for different tasks and steps.
Key messages & recommendations
Regulatory experiments can help policymakers to design more effective and efficient regulation by
gaining field knowledge of the benefits, costs and side effects. Such real-world experiments com-
plement other methods for ex-ante policy-impact assessments.
While “Regulatory Sandboxes” primarily allow testing socio-technical innovations by granting ex-
emptions from certain legal requirements (see Chapter 2.1), RITs allow testing regulatory innova-
tions before being rolled out (see Chapter 2.2).
Given that experiments are about learning, a “culture of analysis” with an open-minded attitude, a
proper setup of the experiment, continuous monitoring and stringent evaluation are key to maxi-
mize knowledge gains from the experiment (see Chapters 3.1 and 4).
Broad stakeholder involvement – including all actors potentially affected by the socio-technical or
regulatory innovation – is another key factor for success (see Chapters 3.2 and 4.2.3).
Regulatory experiments entail several legal issues that need attention, from the overall legal frame-
work conditions to the legal base of the specific experiment (see Chapter 3.3).
While there is no standard procedure for regulatory experiments, the ideal-type phase approach in
these guidelines (see Chapter 4) can help to properly design, implement and evaluate such exper-
iments.
How to design and evaluate a Regulatory Experiment?
5
A Guide for Public Administrations
1 Regulatory experiments: enhancing field knowledge for
better regulation towards Sustainable Development
Designing effective and efficient public policies and reg- As part of responsive governance that aims to adequately
ulations from scratch or adapting them to new develop- cope with new developments, regulatory experiments can
ments has always been a challenge. Unintended side ef- be an effective approach to deal with the aforementioned
fects may emerge, the willingness of actors to contribute to challenges. As the Council of the EU concluded in 2020, “flex-
regulatory goals may be low, and the complexity of envis- ibility and experimentation can be important elements for
aged interaction might have been underestimated. Today’s an agile, innovation-friendly, future-proof, evidence-based
intertwined world with many multi-level and multi-actor and resilient regulatory framework” (Council document
interdependencies, major societal challenges – as captured 13026/20, p. 3). For public administration officials respon-
notably by the United Nation´s Sustainable Development sible for policy design, in particular regulatory experiments
Goals (SDGs) – and trade-offs between goals make policy “provide the opportunity for advancing regulation through
design a highly demanding challenge. proactive regulatory learning, enabling regulators to gain
better regulatory knowledge and to find the best means to
Typical ex-ante policy impact assessments usually rely on regulate innovations based on real-world evidence, espe-
models with many assumptions under uncertainty, and past cially at a very early stage, which can be particularly import-
experiences. However, the real effects of a changed regula- ant in the face of high uncertainty and disruptive challenges,
tory framework often differ because – inter alia – reactions as well as when preparing new policies” (ibid., p. 5).
of individual or corporate actors to the new regulation do
not follow the (often linear) expectations. Technical, social In the following, we clarify the term “regulatory experiment”
and organizational innovations that might serve as solutions and differentiate between two ideal types, presenting use
to sustainability problems face regulatory challenges like case and real-world examples for each of them (Chapter 2).
legal barriers and a wide range of uncertainties. This might Afterwards, key issues around experiment design and imple-
be the case simply because the legal status quo could not mentation challenges are discussed (Chapter 3). Although
anticipate current developments. Digital technologies have not intended as a deterministic step-by-step playbook, the
increased the gap between business innovations and regu- main chapter (4) presents concrete recommendations along
latory timeframes. different phases of an experiment. The recommendations
are again illustrated with examples from real cases.
photo: istockphoto / Tero Vesalainen
How to design and evaluate a Regulatory Experiment?
6
A Guide for Public Administrations
2 Types of regulatory experiments and their respective
benefits
In general, regulatory experiments can be broadly defined text for experimentation, enable where appropriate in a
as a means to deliberately deviate from the current reg- real-world environment the testing of innovative technolo-
ulatory framework to try out new or different rules in a gies, products, services or approaches […] for a limited time
real-world setting. The main characteristics of such exper- and in a limited part of a sector or area under regulatory su-
iments are: pervision ensuring that appropriate safeguards are in place”
(Council document 13026/20, p. 4).
the key role of public policies and regulations;
the involvement of government actors (whether local, Specifically, regulatory sandboxes can be based on:
national or supranational); and
the generation of learning processes. exemptions from prohibitions;
exemptions from or adaptations to prescriptive rules
When speaking of “regulation” and “regulatory experiments”, such as specific approval or documentation require-
in the following we mean not only prescriptive law in terms ments, technical standards, or traffic law rules;
of “command and control” approaches or the narrowly-de- adaptations to public tax or fee provisions;
fined regulation of networks and monopolies (e.g. energy compensation of costs that would occur under the cur-
grids), but rather the whole range of institutional arrange- rent regulatory framework.
ments of public policy instruments, procedures and orga-
nizational structures. Such exemptions or adaptations are usually based on pub-
lic ordinances, which themselves are based on either a law
We differentiate between two key ideal types of regulatory specifically dealing with regulatory sandboxes, or an “ex-
experiments. In the case of “Regulatory Sandboxes”, regula- perimentation clause”/“flexibility clause” in – for example
tion is mainly the framework of socio-technical experiments – mobility law (see text box). Such clauses explicitly autho-
that may need exemptions from legal rules. In the case of rize a government to deviate from the existing law by a pre-
“Regulatory Innovation Trials”, regulation itself is the main defined degree.
object of experimentation and learning. The two forms are
further elaborated in the following. Example of a general experimentation clause:
Based on a general experimentation clause in the Ger-
man Public Transport Act (Personenbeförderungsgesetz,
2.1 Regulatory Sandboxes: exemptions from ex- PBefG), exemptions have been approved, for example,
isting legal rules for testing autonomous driving and delivery as well as
new forms and business models of car-/ride-sharing. The
Regulatory sandboxes aim to allow testing specific techni- clause states: “For the purpose of practically testing new
cal, social or organizational innovations for which the cur- types or means of transport, the authorizing authority
rent legal framework poses significant challenges. This if may, upon application in individual cases, approve devia-
often the case with radically new technologies and business tions from provisions of this Act or from provisions issued
models, like in the context of digitization and automation. on the basis of this Act for a period not exceeding four
years, provided that public transport interests are not op-
The sandboxes are characterized by (initially) temporary posed thereto” (§ 2 Abs. 7 PBefG, own translation).
exemptions from the existing legal framework. The ex-
emptions or adaptations remove direct legal barriers (part The regulatory sandbox concept originates from financial
of the innovation is not allowed) or economic barriers, i.e. sector regulation in the UK, where it has also been adopt-
the experiment is not economically viable under the current ed in other domains like the energy sector. Today, there are
regulatory framework. regulatory sandboxes in many sectors and countries, e.g. the
financial sector in Denmark and Singapore, the energy sec-
The Council of the EU defines regulatory sandboxes “as tor in Germany and Norway (see text box), and the mobility
concrete frameworks which, by providing a structured con- sector – especially around autonomous driving and delivery
How to design and evaluate a Regulatory Experiment?
7
A Guide for Public Administrations– in several European countries but also California, for ex- menting with new regulations, one might discover that they
ample. do not work as intended, e.g. an emissions trading regime
might not sufficiently incentivize companies to reach emis-
Examples of specific legal frameworks for Reg- sion reduction goals, or implementation costs may be high-
ulatory Sandboxes: er than excepted. This represents valuable “regulatory learn-
In 2015, the Dutch Ministry of Economic Affairs and Cli- ing” and can help to avoid potential costs and downsides of
mate Policy created a regulatory sandbox framework with introducing inefficient or otherwise problematic regulations
the Dutch “Experimenten Elektriciteitswet”. It allows dero- nationwide.
gations from specific articles within the national Electric-
ity Act to test new technological solutions, or products, The evaluation of regulatory options is at the core of these
services and tariff models around energy products and projects, regarding such criteria as effectiveness, efficiency,
services. The regulator grants a limited number of exemp- justice implications, acceptance and unintended side ef-
tions (but not funding) every year, for which interested ac- fects. The knowledge gained from real-world experiments
tors can apply. With these experiments, the regulator aims can complement other ex-ante policy impact assessment
to identify necessary adaptations of the Electricity Act, as methods like modeling. RITs should not be used without a
well as the need for new policies. serious intention to learn, i.e. neither to justify policy choices
In Germany, SINTEG-V is a statutory ordinance based on already taken nor to delay policy decisions for tactical rea-
the energy law and applied to the R&D program “Show- sons.
case Intelligent Energy – Digital Agenda for the Energy
Transition” (SINTEG). The ordinance makes it easier for To compare different policy options, public authorities may
program participants to test new technologies, proce- even set up several parallel RIT in different regions or city
dures and business models in practice by reimbursing quarters, or sequential RITs in the same community.
them ex-post for costs that they may face under current
regulation as a result of their demonstration projects. The Examples of RITs range from local to national governance
ordinance clearly defines the situations for which such a levels.
retrospective reimbursement can take place.
Examples of RITs:
The regulatory exemption or adaptation of norms is usual- A permanent “green arrow” traffic sign for cyclists at city
ly not in the focus of an evaluation of the experiment, al- crossroads (so cyclists can always turn right) has been
though one could assess its cost-effectiveness, for example. tested in various European municipalities over recent
A proper evaluation of the experiment may lead to insights years, including in Paris, Berlin and Basel. In Germany, the
into whether and which longer-term policy changes would Federal Transport Ministry took up the idea and conduct-
be appropriate. However, in order to determine optimal ed pilot projects in nine cities. Based on the experiences
regulation around an innovation, one should switch to the gained, it finally adapted road traffic regulations that now
second type of regulatory experiments (yet not limited to generally allow green arrows for cyclists across the coun-
regulation around socio-technical innovations), as present- try.
ed in the next section. In Finland, a basic-income experiment was carried out in
2017-18, in which a sum of 560 € per month was paid to
a randomly-selected group of 2,000 unemployed Finns.
2.2 Regulatory Innovation Trials (RITs): experi- The basic income replaced the existing unemployment
menting with new regulatory options benefits and was paid even if participants took up jobs.
The evaluation revealed small employment effects but
In contrast to Regulatory Sandboxes focusing on socio-tech- better perceived economic security and mental well-be-
nical innovations (see previous section), regulation itself is ing among the participants.
the core of RITs. They are about testing new or substantial- China is known for using policy experiments quite system-
ly-modified policy instruments (“regulatory innovations”) atically as a tool for national policy-making. In the case of
in a real-world setting to find the optimal regulation to carbon emissions trading, China started in 2011 by select-
achieve certain political goals. ing two provinces and five cities as pilot regions. Given
leeway to design their schemes, they varied regarding –
Being standard in the case of technical innovations, testing inter alia – sector coverage, the allocation of allowances,
regulatory innovations before being “rolled out” can help price uncertainty, market stabilization, and enforcement.
to design better regulation from the outset. When experi-
How to design and evaluate a Regulatory Experiment?
8
A Guide for Public AdministrationsTo obtain substantial evaluation results from experiments, background characteristics. However, controlling potentially
(natural) scientists often use Randomized Control Trials. intervening variables is a challenging task in real-world set-
Such experiments randomly allocate individuals to two or tings and only possible to a certain degree. In the Finnish
more groups, where some receive the intervention studied case, the evaluation was further complicated by a general
(treatment group) and some do not (control group). Re- policy reform of entitlement criteria for unemployment ben-
searchers then compare the two groups with respect to a efits in 2018.
measured response of interest. By relying on randomization
and only varying one factor at a time, using a Randomized Nevertheless, learning can also occur without explicitly
Control Trail design significantly increases the chances of randomizing subjects into treatment and control groups,
producing valid cause-effect relationships. e.g. learning about practical implementation issues, stake-
holder reactions, and politics. In these cases, the interaction
Randomized control-like experiments can also be used to between different actors is important to facilitate learning.
study targeted policy interventions, as the Finnish basic in- What all forms of experimentation have in common is that
come case (s. Textbox above) shows. The group receiving monitoring and evaluation is important to maximize knowl-
the basic income (“treatment”) was similar to the rest of un- edge gains from the experiment.
employed people in Finland (“control group”) in all relevant
photo: istockphoto / skyNext
How to design and evaluate a Regulatory Experiment?
9
A Guide for Public Administrations3 Key issues to consider for regulatory experiments
Here, we present key issues to consider when designing and decisions that remain relevant even when the experiment is
implementing regulatory experiments. finished? Moreover, how can unintended consequences of
the policy instrument be dealt with within the experiment?
3.1 Learning & evaluation throughout the experi- If these issues are not adequately reflected in the initial de-
ment sign of the experiment, they can hardly be fixed at a later
stage and will thus weaken the validity of any evaluation of
Experiments are about learning: It is important to conduct the experiment.
experiments with this objective in mind. The experimental
design should enable as much learning as possible. One can It should also be an explicit option that the experiment fails
learn about the effects of the instrument on various stake- and that the tested instrument is not useful. This can be an
holders, as well as learning about processes, practical imple- important learning result. Moreover, this option can help to
mentation issues, stakeholder reactions, and politics. A key address concerns that the experiment prejudges future reg-
question – even before the experiment is set up – is whether ulation.
an experiment is the most appropriate tool for learning in
the specific case (see Chapter 4.1). Learning should also be a guiding principle in the course of
the experiment. This also helps to deal with the uncertain-
Type of regulatory experiment ty that is inherently part of experiments. Such continuous
Learning is particularly relevant in the case of RITs, which are learning helps to better understand the tested policy instru-
explicitly set up to learn about new regulatory options. None- ment. It can also be important to fine-tune the experiment
theless, it should not be neglected in the case of regulatory in the process and adapt it to new insights. Experiments are
sandboxes either, even though these experiments focus on also about learning to learn.
non-regulatory innovations. Even in these cases, it is benefi-
cial to draw up the regulatory exemptions and the design of When experiments are set up to test future regulatory op-
the experiment in a way that allows for as much regulatory tions, it is highly recommended to develop an understand-
learning as possible. Regulatory sandboxes with regulatory ing of what the socio-technical context could look like in
exemptions that have no chance of being implemented in the the future, which regulatory challenges are likely to arise,
future are therefore not recommended. and which regulatory options may become relevant. This
is particularly important in the context of sustainable de-
Learning throughout the experiment: Learning is not velopment, where dynamic changes can be expected. Ex-
something that starts once the experiment is finished and periments need to be coordinated with expectations and
evaluated. While an explicitly planned evaluation after the emerging requirements, i.e. the future context in which the
experiment is highly important, learning should start well tested policy instrument needs to function.
before that and play a key role throughout the experiment.
Indeed, how successful learning can be already depends on Type of regulatory experiment
the design of the experiment. It should be clear from the This also points to a major difference between exemptions
outset which questions the experiment is supposed to an- from existing legal rules (Regulatory Sandbox) and testing
swer, and the design of the experiment needs to be geared new regulatory options (Regulatory Innovation Trial). Learn-
towards answering these questions. ing ideally requires testing new solutions, and not simply an
exemption from existing solutions.
For example, it needs to be decided at the very beginning
whether different policy options are to be tested in parallel, Upscaling and transferring the results: Importantly, learn-
and if randomization and control groups are needed. How ing does not only refer to the experiment as such. The ex-
should results be dealt with that may be biased through the periment ultimately aims to improve the policy instrument
experimental setting and the limited timeframe of the ex- and apply it in a broader context. This means upscaling it
periment? How can long-term effects of the tested instru- from the restricted experiment to e.g. the national policy
ment beyond the duration of the experiment be anticipat- level, as well as transferring it to other contexts. The learning
ed, and how does the tested instrument affect investment design of the experiment should be in line with this objec-
How to design and evaluate a Regulatory Experiment?
10
A Guide for Public Administrationstive. Again, experiments should consider this from the very more about regulatory options. Indeed, especially if the
beginning. experiment also includes the question of how a new instru-
ment can be implemented in practice, regulators should not
In terms of upscaling, the experimental design ideally also only be observers, but rather they should become directly
includes a procedure concerning how to deal with the learn- involved (learning by doing).
ing results of the experiment: Does the experiment simply
finish, or is there a decision procedure for how to take the An involvement of regulators or law makers is relevant for
results further? another reason, namely involving those who decide about
how to use the results and whether they are applied more
Overall, when conducting experiments, one should invest broadly (upscaling) after the experiment can help to in-
in a robust methodological design and useful learning crease the impact of the experiment.
conditions, even in the face of higher resource require-
ments and potential resistance. This leads to the politics of experiments. Experiments are
not only about creating new knowledge, but also about de-
veloping support and an actor network for new regulatory
3.2 Stakeholder involvement options. Especially if new ideas are tested, experiments can
help to build up public support. This requires broad partic-
The outcome of an experiment depends on who partici- ipation and is also a prerequisite for learning. The involve-
pates. While experimenting with new regulation primarily ment of key actors in relevant networks can support the
affects the regulator and the regulated, the group of actors recruitment of participants and distribution of information.
who should be involved in an experiment typically needs to
be broader. For the actors involved in designing, steering and evaluating
the experiment an open-minded attitude, based on a “cul-
Type of regulatory experiment ture of analysis” is key. This needs to be defended against
This is the case especially in Regulatory Innovation Trials, any attempts to “capture” the process due to political prefer-
where the effect of new regulation on various stakeholders is ences; a dispute that might occur at any time.
relevant. If the experiment is more about regulatory exemp-
tions to test technical innovations, this may be less critical
from a regulatory perspective. However, it is still important to 3.3 Legal issues
include the actors who are affected by this innovation.
From a legal perspective, regulatory experiments face four
Appropriate participation needs to be organized through- general questions:
out the experiment. As with learning, participation is not
something that happens automatically or “on the side”, but First, are exemptions from the given legal framework
rather it requires explicit attention, appropriate tools and necessary to carry out the experiment?
instruments to foster exchange between participants. Ex- Second, does the experiment itself require the enact-
periments are about learning, but also about politics. Broad ment of new legal regulations?
stakeholder involvement is relevant for both. Third, which authority has the legal competence to issue
the relevant provisions to permit and/or perform the ex-
If only the regulator and regulated take part, other stake- periment?
holders will not learn about the new regulation, and it will Fourth, are there any potentially colliding legal rules or
be difficult to ascertain how it affects them. Broad stake- principles – apart from the norms to be potentially sub-
holder involvement is one way to deal with the unintended stituted as a part of the experiment – that might prevent
consequences of a new instrument. Moreover, a heteroge- the experiment or lead to unwanted legal consequenc-
neous set of participants may be a prerequisite for drawing es?
universally applicable conclusions about the effect of the
regulatory options. Although the answers may diverge, these general ques-
tions are relevant for both types of regulatory experiments
An active role of the regulator in the experiment cannot be explicated in Chapter 2. The necessity of exemptions from
taken for granted, as regulators are used to regulating, but existing legal rules is given – by definition – in regulatory
not to experimenting. Nonetheless, regulators should con- sandboxes. In these cases, specific given rules or standards
sider experiments as a useful tool for themselves to learn have been identified as obstacles for social or technological
How to design and evaluate a Regulatory Experiment?
11
A Guide for Public Administrationsinnovations and shall be removed for specific cases during a According to the aforementioned third question, in cases in
limited time to permit experimentation. However, the ques- which the adaptation of the legal framework is necessary to
tion concerning the way in which the adaptation of the ex- carry out a regulatory experiment, it has to be clarified which
isting legal practice can be performed can be answered in legal entity has the legal competence to issue the relevant
different ways: in the regulation in question, experimenta- provisions to permit and/or perform the experiment. Com-
tion clauses or other legal options that allow an experimen- petences rules differ from country to country.
tal deviation from the given legal practice may already exist.
Only if such leeway does not yet exist is an exemption from Examples of competences concerning experi-
the relevant regulatory statutes necessary, with the conse- mental legislation in Germany
quence that the competent institutions have to adapt these Experimental legislation at the federal level in Germany in
norms or issue new regulations in accordance with the spec- accordance with the Fundamental Law (i.e. the German Con-
ified procedures. stitution) is conceivable – for example – in the area of air
transport (possibly for drones), (autonomous) railways, com-
In the case of RITs, this question is slightly more complex. On mercial law and road traffic (autonomous driving). On the
the one hand, such a regulatory experiment itself comprises federal level, the “Länder” may also enact experimental leg-
the modification or enactment of norms, which may or may islation for themselves, as well as experimentation clauses to
not be statutory norms. On the other hand, issuing such an deviate from the standards provided for in their law. Relevant
experimental norm again can require an adaptation of supe- legislation of the state level already exists. For example, there
rior law: just as in the case of regulatory sandboxes, superior are experimental clauses under municipal law to promote
law can impede experiments with innovative rules if it does possible reforms/innovations at the local level (Wittig et. al.
not provide the relevant leeway for deviation from the given 2020). Finally, experimental legislation can – under certain
legal practice. An experiment with innovative rules in a field conditions – also be enacted by the government: in Germany,
of law that already is densely regulated thus also requires in accordance with Article 80 (1) of the Basic Law, the Federal
adapting the respective laws by the competent legislator, Government, the Federal Ministers and the Governments of
e.g. by means of establishing experimentation clauses. the States may issue statutory ordinances, if provided for in
a federal law.
Even if no colliding laws exist, testing new or substantial-
ly-modified policy instruments may presume enacting new As a final general point, experimenters also must consider
legal statutes to allow for the RIT. Rule of law principles can rules and principles outside of the experimental scope. For
presuppose that the administration has a legal basis for its example, this applies to the fundamental principle of equali-
activities, in particular if these activities affect fundamental ty, which may become relevant when people are confronted
rights or hold relevance for other essential issues of the com- with different legal or procedural regulations due to spatial-
mon good. RITs that aim at testing policy instruments that do ly limited experimental regulations. The principle of equal-
not affect fundamental rights and other important policies ity also is concerned when regulatory experiments only
therefore may be possible without a specific legal basis. For give individual companies the opportunity to test certain
example, the existing leeway for legal experiments can be innovations. However, as has been found for the German
wider with respect to changing existing policies regarding context, legal objections regarding unequal treatment may
subsidies or changes of public institutions and governance be rebutted if experimental clauses are substantiated with
mechanisms. Thus, it can be possible to perform regulatory foresight and designed in such a way that decisions at the
experiments without a specific legal authorization. Howev- administrative level are comprehensible in terms of both the
er, in many cases, regulatory experiments will require a spe- substance and the procedure. (See the following publica-
cific legal basis. In principle, this will be the case when new tion: Holger Schmitz, Christian Alexander Mayer, Carl-Wen-
regulations are supposed to comprise a legal mechanism delin Neubert, Ines Reiling, Umsetzung der BMWi-Strategie
to ensure compliance. Such a necessity has been observed „Reallabore als Testräume für Innovation und Regulierung“:
with respect to different legal systems: For example, there Erstellung einer Arbeitshilfe zur Formulierung von Experi-
was no law at the national or regional level for a regional mentierklauseln (Los 1), BMWi., 2020)
emission trading pilots in China, only an ordinance by the
central government. This later proved to be a problem when
regional governments had to design compliance systems for
the experiment, as possible fines had no legal basis due to a
regulatory gap in the ordinance.
How to design and evaluate a Regulatory Experiment?
12
A Guide for Public AdministrationsLarge variety of legal issues in concrete cases
Besides such general issues, the formulation of an experimen- relevant in the context of the experiments studied in our proj-
tation clause can lead to a variety of particular questions. For ect: the Finnish basic income experiment had to address con-
example, liability risks should be clarified in cases in which tradictions to the Finnish constitution, in addition to possible
experiments involve testing real-life applications or technol- conflicts with other national legislation, e.g. social laws and
ogies that could affect the legal interests of the involved en- European legislation. In the Dutch electricity sandbox, the Eu-
terprises or third parties. Regulatory experiments should thus ropean consumer protection framework and the Dutch tax act
systematically consider such legal issues. Recent research has had to be taken into account. Similarly, the Italian electricity
therefore developed detailed guidelines to ensure that such sector experiments had to consider the market anti-discrimi-
clauses can be formulated in a permissible and practicable nation and consumer protection legislations when designing
manner (cf. Schmitz et. al., 2020). Such questions also became derogations.
photo: istockphoto / sommaiphoto
How to design and evaluate a Regulatory Experiment?
13
A Guide for Public Administrations4 Design, implementation and evaluation of regulatory
experiments
This chapter provides detailed step-by-step options for de- back into the policy process, which then lea ds to new ex-
signing and implementing regulatory experiments. Experi- periments.
ment can be divided into four ideal-typical phases.
The following table shows the key tasks for each of these
While an experiment is a fixed-term project with a clear start steps, which will be explained in more detail in the following
and end (as depicted above), ideally the experiment is part sections.
of a broader circular learning process where results are fed
Clarification:
Goal and type Preparation Implementation Evaluation
Is an experiment Embed experiment Monitor the Evaluate the experi-
needed at all? in broader context implementation ment to learn about
Clarify: goal of the Obtain political Be ready to fine-tune the tested regulatory
experiment support, sufficient the experimental innovation
Appropriate type: budget and clarify design how it can be
Regulatory Innovation legal issues If needed, even adapt upscaled
Trial or sandbox? Organize stakeholder instrument design the experimental
Is there uncertainty involvement during the implemen- design
concerning the future Determine the con- tation Recommend
system and its regu- crete experimental Manage stakeholder regulatory option(s)
lation? project involvement Adapt regulatory
Define clear regulato- Prepare evaluation option based on
ry question and learning evaluation results
Decide on size and Prepare the upscaling
duration of the project of results
4.1 Phase 1: Clarification: Goal of the experiment
and determination of type
Clarification: Goal and type
Is an experiment needed at all?
Clarify: goal of the experiment?
Appropriate type: Regulatory Innovation Trial or sandbox?
Is there uncertainty concerning the future system and its regulation?
Define clear regulatory question
Regulatory experiments can be a powerful learning tool sidering the specific context. Therefore, before a regulatory
that should be applied more often. However, this is not to experiment is set up (see next phase “preparation”), the fol-
say that they should be applied in any case and without con- lowing questions need to be addressed:
How to design and evaluate a Regulatory Experiment?
14
A Guide for Public Administrations1) Is an experiment needed at all, and of added value, in this a. Regulatory sandbox: Regulatory exemption to test
case to meet the goal? socio-technical innovations
a.
Regulatory sandbox: Is there really a regulatory b. Regulatory Innovation Trial: Testing a regulatory in-
problem that prevents testing the socio-technical novation
innovation? Or can it be tested within the existing
regulation? It can help to take a closer look at what is Depending on the type, the design of the experiment –
possible within existing regulation before setting up which is further explained in the next phases – differs.
an experiment.
3) Before setting up a regulatory experiment, it is important
Case example to scrutinize whether there is really a regulatory problem
The UK Electricity and Gas Sandbox showed that af- that needs to be addressed to enable sustainable solu-
fected businesses tend to be more in need of advice tions for the future. Is there regulatory uncertainty con-
on possibilities within the existing legal framework cerning the transformation to a future system that meets
rather than applying for a regulatory sandbox, as they the Sustainable Development Goals? In this case, a regu-
are mostly unaware of what they are already legally latory experiment can be particularly useful. Alternatively,
allowed to do. Similar situations occurred within the is there simply a request for a regulatory exemption to
Dutch energy sandbox. enable certain business models that are not viable oth-
(see Annex for list of cases) erwise and where it is not clear why this business model
should be promoted from a societal perspective? In the
b. Regulatory Innovation Trial: Do the benefits expect- latter case, an experiment should typically not be set up.
ed from the experiment exceed the costs? Costs not
only refer to monetary costs but can also mean that 4) As a result of this phase, there should be a clear regulato-
the introduction of a new regulation may be delayed ry question.
through the experiment. Are there knowledge gaps
that can be expected to be closed through learning Type of regulatory experiment
in an experiment? Or would it be possible to directly The regulatory question will be less pronounced in the
opt for a full-fledged implementation? case of regulatory sandboxes. However, even here, exper-
imenters should have an idea about how they can benefit
2) What should be the goal of the experiment and what from the experiment for developing future regulation.
type of experiment would be appropriate to meet that
goal? Please note that this phase can bring about useful learning
insights, even if an experiment is not started as a result.
photo: istockphoto / Elmar Gubisch
How to design and evaluate a Regulatory Experiment?
15
A Guide for Public Administrations4.2 Phase 2: Preparation of the experiment
Preparation
Embed experiment in broader context
Obtain political support, sufficient budget and clarify legal issues
Organize stakeholder involvement
Determine the concrete experimental project
Prepare evaluation and learning
Decide on size and duration of the project
A thorough preparation of the experiment is highly import- should not be limited to current problems, but needs to
ant to maximize the learning effects that can be achieved. include potential future issues, for which a regulatory op-
On the one hand, this includes measures regarding the con- tion can be tested now.
text of the experiment like embedding the experiment in a Complementary instruments that are available to fill the
broader learning context and getting political support. On knowledge gap; for example, modeling studies, labora-
the other hand, the setup of the experiment as such needs tory experiments, role plays.
to be prepared.
Understanding the potential contribution of a regulatory
Overall, the preparation of Regulatory Innovation Trials is experiment requires analyzing the regulatory problem. This
more demanding than regulatory sandboxes to achieve the in turn requires an understanding of why current regulation
desired regulatory learning. Nonetheless, given that regula- does not achieve the desired outputs and which alternative
tory sandboxes can also generate regulatory learning, care- regulatory options are available.
ful preparation is important in this case as well.
This can be based – for example – on a delta analysis, includ-
Previous experiments show various complications that ing the following questions:
experimenters can face when planning regulatory experi-
ments, e.g. due to political conflicts, diverging interests of What is the regulatory problem?
stakeholders and legal uncertainties. In the case studies an- What do we want to achieve with regulation (target state)?
alyzed, experimenters have addressed these complications What prevents the actors in the current state from mak-
by using various preliminary measures, e.g. by analyzing ing contributions towards the target state? Why do the
prior experiments to identify best practices or consulting actors in the status quo not behave in the way that is re-
scientific and political experts as well as stakeholders and quired from the perspective of the regulator?
the public. Sound preparation of the experiment has proven What are the regulatory options to overcome this delta?
useful to determine what the objective of a specific regula-
tory experiment should be, how it should be designed for Before setting up the experiment, it is useful to gain an over-
that purpose, and to secure the framework conditions for view of the knowledge that is already available to answer
the experiment. these questions. It can also be helpful to fill specific knowl-
edge gaps before the experiments is set up.
4.2.1 Embed experiments in a broader learning This includes analyzing the situation in the jurisdiction at
context hand, as well as gaining a good understanding of the experi-
ences elsewhere with the regulatory instruments that are to
Regulatory experiments should not be stand-alone instru- be tested, or even the experiences with similar experiments
ments but rather part of a broader portfolio of measures to that have already been carried out. All of this should be done
improve regulation and facilitate regulatory learning. Exper- anyway before regulation is changed, but it can also help to
iments need to be embedded in this broader learning con- identify the specific question and setup of the regulatory ex-
text, which includes: periment.
An analysis of the regulatory problem as well as the This approach shows that a regulatory experiment needs to
knowledge that is available to address the problem. This be embedded in a broader process of generating regulatory
How to design and evaluate a Regulatory Experiment?
16
A Guide for Public Administrationsknowledge. An experiment can complement such a process, 1. Who should participate?
rather than replace it.
Explicit mapping of relevant stakeholders and their interest
can help to answer this question.
4.2.2 Obtain political support, sufficient funding
and clarify the legal basis The case studies have shown that having a heterogeneous
group of participants is beneficial (or even a prerequisite) for
Regulatory experiments require three key resources that learning from regulatory experiments. Nonetheless, there is
should be secured in time: political support, a sufficient bud- also a trade-off between “leaving the door open” and target-
get and a clear legal basis. ing specific actors if the composition of participants needs
to be representative.
In general, the preparation of regulatory experiments is facil-
itated if all political actors involved in the legislative process 2. How to build a broad consensus between stakeholders
have understood that there is a need for regulatory learning that the experiment is useful.
and consider regulatory experiments as an instrument to
facilitate this. Consultations with other experts, political ac- 3. How to consider the knowledge and interests of stake-
tors and the public may help to pinpoint different positions holders, and how to transfer learning results to stake-
and interests towards the foreseen regulatory experiment holders.
(see also Chapter 4.2.3: How to involve stakeholders?). Co-
ordination between and within governmental (municipal) 4. How does the role of stakeholders change during the ex-
departments and with regulatory funding agencies can also periment? In which phase should they participate, and in
be highly relevant. which way?
In order to increase public legitimacy, not only the goal and 5. Which stakeholders should be on board at a very early
reasons but also the costs of experimentation should be stage to better understand their perspective and – at
transparent. There should be sufficient budget to cover all least informally – involve them in the design of the ex-
necessary steps as described in these guidelines, including perimental space?
the evaluation and upscaling phase; otherwise, the experi-
ment may end without satisfying results, which could even 6. How to incentivize and motivate stakeholders.
be counter-productive.
7. How to organize concrete stakeholder involvement and
Finally, a sound legal basis is a prerequisite. Deviating from exchange between stakeholders. There should be explic-
the legal framework in place or enacting new, innovative it formats for stakeholder participation and interaction.
rules is not trivial and frequently requires putting legal pro-
visions in place. General issues to be addressed at an early Case example
stage concern the initial analysis of which legal stipulations For a shared space pilot project (“Begegnungszone”) in a
may be subject to the regulatory experiment – i.e. should street in Berlin, a “steering group” comprising representa-
be suspended or enacted – and which legal body is com- tives of local administration, associations, local business-
petent to carry out the adjustments of the legal framework. es and a residents’ initiative was established based on
A preliminary examination of more specific potential legal the suggestion of the latter. Their task was to discuss and
hurdles is also recommended at an early stage. Such legal agree on concrete steps for public participation around
issues can be numerous and diverse. Addressing these is- the project. Concrete participation formats included on-
sues early and with the help of specific expertise can help to line and paper questionnaires as well as townhall meet-
avoid legal complications at a later stage. ings and planning workshops, The steering group also
agreed to precede the permanent reconstruction of the
street with a “test phase”, namely an initial time-limited
4.2.3 How to involve stakeholders experiment as part of the locally-limited experiment.
How to involve stakeholders requires explicit attention in (see Annex for list of cases)
the preparation phase, which includes considering the fol-
lowing issues: Whether participation is compulsory or voluntary is a key
question for experimenters; for example, in the case of
How to design and evaluate a Regulatory Experiment?
17
A Guide for Public Administrationselectricity customers and the regulation of electricity tar- previous experiments, the internal organization of the team
iffs (see also the case example on the basic income exper- that conducts the experiment including the interdisciplinary
iments in Finland and Ontario in Chapter 4.2.5). If it is vol- research team, intra-administration organization, and the
untary, participation may be distorted and only innovative coordination and setting up of processes between actors
customers or those who stand to benefit from the regula- from the administration and research team have been pivot-
tion may take part. If participation is compulsory, this may al. In particular, clarity of decision-making processes and the
allow for more representative results. Nonetheless, exper- stringent application of formalized a-priori-agreed rules has
iments may need to ensure that customers in the exper- been helpful to establish trust between the various actors.
iment are not disadvantaged, which again may lead to a
distortion of results.
4.2.4 How to determine the concrete experimen-
Especially if participation is voluntary, experiments largely tal project
rely on the motivation of the target group to participate.
Consequently, the experiment needs be to transparent, There are different ways to select the concrete experimental
hurdles for participation need to be low and incentives like project and the specific question to be addressed. The selec-
reimbursement of costs can be important. If there are high tion process can be top-down or bottom-up, or a combina-
prequalification requirements or cumbersome administra- tion of both.
tive procedures, this may deter potential participants who
might have been relevant for the overall success of the ex- Type of regulatory experiment
periment. Experimenters might guide actors through the In regulatory sandboxes, the selection process is rather bot-
application process. If the new regulation has a realistic tom-up: innovators face certain regulatory barriers for which
chance of being adopted after the experiment, this can also they request a regulatory exemption. However, in this case
incentivize participation. it is recommended that there is also a top-down process to
assess whether the requested exemption fits into the “bigger
Case example picture” of where regulation should be going.
The experiments with voluntary participation have shown
several measures to incentivize stakeholders to participate. In RITs, experiments should always start from a regulato-
The energy sandbox in Singapore provides a perspective for ry problem – and thus a systemic question – rather than a
the time after the experiment: exemptions may be transferred problem for individual innovations or business cases. Here,
to the legal framework so that the tested innovations can be the regulator is the one who wants to ascertain how certain
introduced on a larger scale. The experimental framework in- regulatory options work to address societal challenges. This
tegrates a decision procedure and pre-defines transition con- regulatory question still needs to be developed with stake-
ditions for the end of the experiment. holders, so a bottom-up approach should not be neglected
here either.
(see Annex for list of cases)
The following table presents several options for how to de-
As for concrete methods for stakeholder involvement, the termine the concrete experimental project, especially for
rich experience from transdisciplinary research projects can RITs. (Table 3)
be used.
While a broad stakeholder involvement is important, the
coordination of the core team should not be neglected. In
How to design and evaluate a Regulatory Experiment?
18
A Guide for Public AdministrationsYou can also read
