Usability testing and the social analysis on online counselling system for recommendations in technical vocational schools
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Journal of Physics: Conference Series
PAPER • OPEN ACCESS
Usability testing and the social analysis on online counselling system for
recommendations in technical vocational schools
To cite this article: I M Pageh et al 2021 J. Phys.: Conf. Ser. 1810 012022
View the article online for updates and enhancements.
This content was downloaded from IP address 46.4.80.155 on 11/09/2021 at 04:55
IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
Usability testing and the social analysis on online counselling
system for recommendations in technical vocational schools
I M Pageh1, A A J Permana2 and K Suranata3
1
Historical Education, Faculty of Law and Social Sciences, Universitas Pendidikan
Ganesha, Singaraja, Indonesia
2
Informatics Engineering, Faculty of Engineering and Vocational, Universitas
Pendidikan Ganesha, Singaraja, Singaraja, Indonesia
3
Guidance and counseling, Faculty of Science Education, Universitas Pendidikan
Ganesha, Singaraja, Singaraja, Indonesia
Email: made.pageh@undiksha.ac.id1, agus.aan@undiksha.ac.id2,
kadek.suranata@undiksha.ac.id3
Abstract. With the great numbers of vocational schools in Indonesia, there should be plenty of
jobs available and to decrease the number of unemployment. The purpose of this research was
to help schools to assist students in choosing major courses that suit their skills and interests so
the students could feel motivated to participate in the learning process and in the end, they
could understand the lesson well. Placement test was carried out to determine the prior
knowledge of new students so that they could easily be directed to the appropriate courses. The
results of the test were analyzed as a basis to conduct counselling to students. This research
was developed by using the R&D method and the final product was in the form of a computer
application. This application was tested at SMKN Bali Mandara. The results showed that the
application can help students to choose major courses that fit their interests, and have
counselling based on the results of the test. It needs to be tested again regarding its usability by
using the System Usability Scale (SUS) to measure the level of user satisfaction. The test
results in the range of 66,125, this means it is good.
1. Introduction
The number of labor force on February 2020 was 137.91 million. It increases 1.73 million compared
to February 2019. The statistics for the last year show that unemployment has increased by 60
thousand. In contrast, the level of open unemployment fell to 4.99 percent on February 2020. Looking
at the education level, Vocational High School (SMK) is the highest contributor to unemployment
among other education levels, which is 8.49 percent [1].
The value of Indonesian human resource is 0.53 placed in the 87th position out of 157 countries.
The results are based on the level of educational and health of Indonesian children. It is estimated that
the 18-year-old young Indonesian generation can only reach 53% of their maximum potential
productivity. In terms of human resources, Indonesia still lags behind Vietnam at 48th, Philippines 84th,
and Thailand 65th. It is slightly better than Cambodia 100th, Myanmar 107th, and Laos 111th. The first
rank is still held by Singapore [2]. Indonesian human resources are still underrated compared to
Southeast Asia countries, such as Vietnam, which just has trough a bitter civil war. The low quality of
Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
human resources not only results in incompetent worker who can not compete at the international
level but also in their own country [3].
Indonesia is the largest population in ASEAN in 2019. The total population of ASEAN is projected
to be 647.45 million. This number is much larger than the population of other countries, especially
Singapore which has the least population [4]. The total population of productive age in Indonesia in
2020 is 185.3 million out of a total of 269.6 million people. It takes the right solution to deal with the
high number of productive populations.
The solution that has been taken to improve the skill of the productive age group was through 9
years of compulsory education. Indonesia is in the top 4 in terms of human resource skills in ASEAN.
This should be improved through the 9 years compulsory education program. The prospective human
resources in the future are those who have strong and creative characters and are able to compete in
the labor market. Countries in the world must be able to work together to create better human
resources through vocational education.
Vocational education becomes favorable among Indonesian. Parents want their children to have
adequate skills so that they can easily find suitable jobs. Skills are also closely related to student career
plans [5]. In addition, vocational school graduates will also be able to continue their higher education.
For students who were graduated from technology and information vocational school, they are more
likely to get a job than any other field. The areas of expertise in software engineering, multimedia, and
networks are on the rise.
Considering the great opportunities for vocational education in the field of engineering, this
research was conducted to develop better human resources whose competencies fit the need of the
industry. The research has been started from 2017 which implemented Research and Development
method. The final product was applications applied at SMKN Bali Mandara as a Vocational School.
This school has a strong commitment to improve the competence of its graduates. SMKN Bali
Mandara is a new school in Singaraja-Bali, but has a big vision by targeting 70% of graduates
absorbed by industry. If all schools in Indonesia have the same vision and agree to achieve the vision,
then the future of vocational education in Indonesia will be promising.
In realizing the vision of SMKN Bali Mandara, there are many things that must be addressed. The
improvement starts from recruitment of the student, the learning process, internships, and
collaboration with the industry. Based on the results of interviews and seminars and continued with
group discussions, information was gathered that many students were unable to continue their study.
Actually, students have got counseling process and been assisted in choosing their major courses. The
absence of adequate measuring tools for choosing the right course and counseling are the source of the
drop-out problem. Through this occasion, the team collaborated with the school conducted a test on
the course placement system and counseling for new students of vocational school. New students in
vocational high schools experience major changes in their lives and habits. Almost all students at
SMKN Bali Mandara come from underprivileged families, so in testing this system they experienced
main problem because they do not have a smartphone. In addition, they could not use the lab at school
because they have to stay at home due to the spread of the corona virus.
This issue was solved by the team thus all students could use the system and follow all counseling
questions through the online system properly. Therefore, the test could run well and the most
important thing is to determine the level of student satisfaction in using the course placement system.
The results of this test were used as material and basis for counseling students and parents in choosing
the appropriate course.
The research needs to be tested quickly and disseminated immediately so that it can be used by
SMK in the course-placement process for prospective students during the selection process. During the
testing process at SMKN Bali Mandara, the team received a lot of input from students who used the
course-placement specialization test application and the problems encountered were immediately
resolved. The level of user satisfaction with the system is important in developing such system. Many
methods can be used to develop this kind of measurement, one of them is by using the System
Usability Scale (SUS). This method is widely used to perform software testing because it is easy to use
2IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
and the results can be used as a consideration. The reason for using this method is because it is easy to
understand and in its implementation, it is undemanding to use so that this method is popularly used
in conducting Usability Testing tests. With the help of this method, the level of user satisfaction can be
measured into quantitative values.
2. Method
This study applied a quantitative descriptive research which described the existing phenomena based
on data in the form of numbers obtained in the field. The data collection method used in this research
was in-depth interviews and questionnaires distributed to 20 respondents. The questionnaire used was
a 1-5 scale SUS and this is a standard questionnaire to perform Usability Testing. Respondents came
from students and teachers of SMKN Bali Mandara who were selected by random sampling technique.
Application
Majoring Recomendation & t
Counseling rne
t Inte
rne
Inte
Internet
Access via Smartphone Access via Personal Komputer /
Computer Lab
Access via Laptop
Figure 1. How to access the application.
2.1. Data Collection
The data collection process was carried out for 3 months from June to August 2020 after the
development of the majoring system website (http://simajuganesha.com/) was completed.
Simajuganesha can be accessed online trough http://simajuganesha.com/. The next stage after the
development of application was testing. The questionnaire used was validated for conducting SUS
testing. The questionnaire can be downloaded directly on the official SUS website online.
The process of distributing questionnaires was carried out by a team and assisted by teacher staff.
Respondents were the tenth-grade students of SMKN Bali Mandara and teacher who are members of
the new student recruitment committee. Students come from three major courses, namely: Computer
and Network Engineering, Building Drawing Engineering and Light Weight Vehicle Automotive
Engineering. The data were collected by means of a questionnaire on the SUS questionnaire sheet. The
questionnaires were distributed to 20 respondents who were selected randomly.
2.2. Majoring Application Development
The system implemented was web-based which consists of four modules, namely: an instruction
module, a questionnaire, a score, and a test participant profile module. The questionnaire module has
four parts, namely the type of activity, competency expertise, type of occupation, and field of
expertise. After students took the test, they were given recommendations about major courses
according to their competence. Students could also consult with a school counselor to ensure their
choice.
Further development was focused on improving the four system modules of the online version.
Some data were also added such as: school modules, course, evaluation results, and the option to
3IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
export the test results into excel. The development aimed to help teacher to analyze test results and
conduct counseling to students. The system was equipped with a manual for school counselor teachers
and students as system users. The collected data was reliable because it has been tested in a wide
scope, namely the vocational course in Bali. The way the system works can be seen in Figure 1.
During the application development process, there were several stages carried out. They were
Communication, Planning, Modeling, Implementation, and Deployment. The model used in this
research was Waterfall. The waterfall method has several stages and can be seen in Figure 2. This
method is a method with a classic and sequential approach it a clear sequence [6]. The stages of the
waterfall method are communication, planning, modeling, implementation and deployment. After the
implementation of the system can, the real data were then inputted into the system. The data used in
the system came from all SMKN with various fields of expertise, collected using questionnaire.
Figure 2. Waterfall Model Pressman.
Figure 3. Holland's Personality Theory.
2.3. Counselling Using Inventory Personal Survey
Counselling in vocational schools requires unique skill with the main mission is to support all students
in their academic, career, social and emotional development [6]. Long before humans were acquainted
with measurement tools currently known as personality inventories, various attempts to obtain a
person's personality characteristics were carried out by palm reading, interpreting facial expressions,
calculation of date / day of birth, and other means [7].
Personality measurements are important in predicting career of and individual and have been used
in the last 75 years since 1918s. Although many have focused on abnormal personalities, the
personality measurement has also been applied on normal personalities, especially to help someone on
selecting and making choices about jobs and positions [7].
Holland (1985 in [7]) classifies six personality types, namely: realistic, investigative, artistic,
social, entrepreneurial, and conventional (RIASEC). The harmonious relation between person and the
environment can be used to predict and understand knowledge about personality types and
environmental models. This will be very useful for job selection, vocational stability and achievement,
4IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
selecting course and achievement, individual competence, social behavior, and for developing of
individual sensitivity.
Simajuganesha.com as an application developed in this research, is an online test which is adapted
from a Self Direct Search Inventory (SDS) inspired by Holland's personality and career theory. SDS is
a type of inventory developed by Holland through the adaptation and refinement of the Vocational
Preference Inventory (VPI) which was developed in 1958. SDS has undergone three revisions and the
last revision was done in 2001. SDS has been used and adapted in more than 20 nations and available
in more than 15 different languages. This SDS has been proven well through various studies in
measuring personality types and providing predictions about suitable careers for students as well as for
those who are already working. The structure of the personality test which includes RIASEC can be
seen in Figure 3. Personality scores are divided into primary, secondary, tertiary personality scores [8].
2.4. Vocational School for Engineering
Vocational schools equip graduates with competence in various fields of expertise. One of them is
engineering. If other fields of study adapt to technological developments, then the engineering field
itself becomes a producer and developer of technology-based products. There are several expertise in
engineering which includes: Informatics Engineering, Electronic Engineering, Automotive
Engineering, Electrical Engineering, Mechanical Engineering, Naval Architecture and Shipbuilding
Engineering, Textile Technology, Industrial Engineering, Chemical Engineering, Aircraft Technology,
Construction and Property, Geometrics and Geospatial [9].
Engineering is one of the world oldest branches of science. It plays a major role in the development
of human civilization. With the help of engineering, humans can build amazing architecture such as
the Coliseum in Rome and the Pyramids in Egypt. In the 1800s, new branches of engineering
emerged, such as mechanical engineering, mining engineering, metallurgical engineering, electrical
engineering, and chemical engineering. Then in the 1990s other branches of engineering emerged such
as industrial, nuclear and aviation engineering.
Figure 4. Administrator page on website.
5IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
Figure 5. Graph of test value.
2.5. The Social Perspective
Humans are social beings that cannot be separated from social values. Social issues are closely
connected to culture. The socio-cultural system is the relationship of an individual or group in society
that is accepted by society members because it is originally from cultural values that have become part
of the community itself. Rural communities have different socio-cultural system from the urban
communities. The solidarity value of rural communities is higher than urban communities. The values
and habits that live in rural communities will stick to the hearts of each individual or can change due
to certain imperative factors such as the current Covid-19 pandemic [10].
The Indonesia word “budaya” comes from the Sanskrit language of “Buddhayah”, the plural form
of the word “buddhi” which means mind or intellect. On the other hand, the word culture originally
comes from the Latin "colere" which means processing or working. Thus, the word culture in
Indonesia is often translated into “budaya”. Culture is the real application of various collective
agreements that have become a way of life. Culture equips community with a set of skills, values, and
attitudes dealing with problems in their daily life.
According to Koentjoroningrat, there are seven elements of culture, namely: religious systems and
religious ceremonies, social systems and organizations, knowledge systems, language, art, occupation,
and technology systems and equipment. It is interesting to note that social value cannot be separated
from technology. Technology is a container, the producer of production tools, weapons, and other
practical tools [11]. Technology is the result of culture, so that social values cannot be separated from
the field of technology.
2.6. System Usability Scale (SUS)
The usability measurement process includes effectiveness, efficiency, and satisfaction [12]. The
evaluation of the usefulness of a product must use these three parameters. One method that can be
used to test usability is SUS.
This method can be used to perform testing on the independent technology on hardware, software,
websites, and even mobile devices (Sauro, 2011 in [13]). In SUS testing, there are ten questions. Each
question has a score of 1-5. Score 1 is for ‘strongly disagree’ while score 5 is for ‘agree’.
The score calculating guidance for the 10 questions is that question with an odd number, the
answer from the respondent score is (Respondent's answer score - 1). Question with an even number,
the score for respondent's answer is (5 - Score of the respondent's answer). Then add up the scores per
respondent (Total Respondents Score x 2.5). After all respondents' scores are multiplied by 2.5, then
add up and divided by the number of all respondents.
6IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
Figure 6. Graph of respondents' SUS values.
2.7. Competency and Interest Tests
In assisting students to get the right courses, it is necessary to conduct a test. The specialization test
referred in this study is a measurement to understand psychological aspects of students. This was done
to assist student to continue their higher studies and select right occupation, as well as to develop the
success and suitability of their achievements in the future (study while working) [14]. Competence is
formed by behaviour, knowledge and skills [15].
New students expected to show high interest upon the expertise program that they chose before.
After the recruitment process, the next process was teaching and learning process. The process of
specialization of expertise was carried out by considering several variables, including student’s
evaluation report and teacher or school counselor recommendations. In addition, the process of
specialization in the expertise program can be done with a placement test conducted by a psychologist
[16]. This is where the role of a school counsellor is to counsel students. This also refers to the rule in
Indonesia that school counsellors are required to handle 150 students per week [17].
Table 1. Question List on SUS.
Number Questions SUS Score
1 I am thinking of using this system again. 64
2 I find this system complicated to use. 37
3 I find this system easy to use. 68
4 I need help from other people or technicians in using this system. 48
5 I feel the features of this system are working properly. 65
6 I feel that there are a lot of things that are inconsistent (not compatible with this 57
system).
7 I feel that others will quickly understand how to use this system. 71
8 I find this system confusing. 56
9 I feel there are no obstacles in using this system. 51
10 I need to familiarize myself first before using this system. 12
3. Result and Discussion
Based on the results of testing using the SUS method, it can be shown that there were 20 respondents
who filled out a questionnaire with questions as in Table 1, users use a system that is managed by an
administrator as in Figure 4, and the system consists of several modules as described previously.
Respondents filled in each item of the question according to what they had experienced in using the
system. The experience is then included in the score items on the questionnaire. The score of the
7IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
testing is 66,125 (Figure 6) which means that the user is good enough in using the system. This
assessment is still on a good scale considering that students who attend SMKN Bali Mandara come
from economically disadvantaged families. During the test they did not have a smart phone, moreover
in the midst of a Covid19 pandemic, students could not use the school lab because school was
temporarily closed. Internet access during the test plays an important role, while many homes of the
students are still not in blank spot area. The results of student testing on the specialization system are
quite unique, based on Figure 5 where the first highest result is Realistic, and the second is Social.
Supposedly in technical schools, social value is normally on the third position. The current test results
happened in several other technical schools. Based on the results of discussions with experts, it was
acknowledged that social values can indeed emerge in the engineering field because technology is an
element of culture. So that the conclusion is still within the limits of the tolerance value because the
first value is Realistic which is the characteristic of engineering department.
4. Conclusion
The present research implemented the R&D method. The final product of the research was a
specialization application called simajuganesha.com. Based on the result of the questionnaire
distributed to 20 respondents, it could be said that the level of customer satisfaction was 66,125, which
means that the application is good. This value is obtained by distributing questionnaires using the
System Usability Scale (SUS) method in data collection and data analysis. Field implementation of the
application showed that the application can be accessed and used properly, even though there are
several issues, but those issues can be overcome. The application developed is substantially helpful
especially during the Covid19 pandemic. The results also found that for students who take engineering
course, it turns out that social values are still in second place. It can be understood that social valuess
cannot be omitted because technology is an element of culture.
References
[1] BPS 2020 Tingkat Pengangguran Terbuka ( TPT ) sebesar 4–5
[2] Agustiyanti 2020 Bappenas: Kualitas SDM Indonesia Masih Ketinggalan Jauh dari Vietnam
1–8
[3] Yulifar L 2009 Pemberdayaan Sumber Daya Manusia Indonesia Di Tengah Krisis Global J.
Geogr. GEA 9 1–8
[4] Jayani D H 2019 Jumlah Penduduk Indonesia Sepertiga Penduduk ASEAN
[5] Irwan et al 2017 Counseling Model Application : A Student Career Development Guidance for
Decision Maker and Consultation Counseling Model Application: A Student Career
Development Guidance for Decision Maker and Consultation IOP Conf. Ser. Earth Environ.
Sci. 97 012045
[6] Pressman R S 2010 Software Engineering ed S EDITION (McGraw-Hill)
[7] Suratana K 2009 Hubungan Antara Kesesuaian Tipe Kepribadian dan Model Lingkungan
dengan Kematangan Arah Pilihan Karier (Studi pada Siswa Kelas XII di SMK N 1 Padang)
(Padang: Program Studi Bimbingan dan Konseling, Program Pascasarjana Universitas
Negeri Padang, (Thesis : Unpublish))
[8] Permana A A J et al 2017 Recommendation System for Selection of Majors and
Apprenticeship on Vocational and Training Education Based on Competency to Produce
Demand Driven Graduates 134 153–8
[9] Rencanamu.id 2020 Bidang SMK: Teknik dan Rekayasa 26 1–12
[10] Fajri M 2020 Sistem Sosial Budaya Masyarakat Pedesaan di Tengah Pandemi 1–8
[11] Thabroni G 2020 Pengertian Budaya, Unsur, Wujud & Fungsi Menurut Para Ahli Pengertian
Budaya
[12] Peppa V, Lysikatos S, and Metaxas G 2012 Human-computer interaction and usability testing :
application adoption on B2C Web sites 14 112–8
[13] Ependi U and Panjaitan F 2018 System Usability Scale Antarmuka Palembang Guide Sebagai
8IConVET 2020 IOP Publishing
Journal of Physics: Conference Series 1810 (2021) 012022 doi:10.1088/1742-6596/1810/1/012022
Media Pendukung Asian Games XVIII 3
[14] Sulystiyawati E W and Purwaningsih I E Peran hasil tes penjurusan studi terhadap pemilihan
jurusan pada siswa sma 5 35–47
[15] Permana A A J and Pradnyana G A 2019 Recommendation Systems for internship place using
artificial intelligence based on competence Recommendation Systems for internship place
using artificial intelligence based on competence IOP Conf. Ser. J. Phys. Conf. Ser. 1165
1165 012007
[16] Asmara A 2015 Pengembangan Tes Minat Dan Bakat Dengan Metode Jaringan Syaraf Tiruan
(JST ) Untuk Memprediksi Developing Interest And Talent Test Using Artificial Neural
Network ( Ann ) To Predict Students ’ Aptitude J. Pendidik. Vokasi 5 273–86
[17] Suranata K S, Rangka I B, and Permana A A J 2020 The comparative effect of internet-based
cognitive behavioral counseling versus face to face cognitive behavioral counseling in terms
of student ’ s resilience Cogent Psychol. 7 1751022
9You can also read
