لخّصلي

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/339439281

Measurable performance indicators of student learning outcomes: a case
study

Article in Global Journal of Engineering Education · February 2020

CITATIONS READS
20 11,498

4 authors:

Tahar Ayadat Danish Ahmed
PMU University Prince Mohammad University
94 PUBLICATIONS 832 CITATIONS 35 PUBLICATIONS 85 CITATIONS

SEE PROFILE SEE PROFILE

Saidur R. Chowdhury Andi Asiz
Prince Mohammad University Prince Mohammad University
29 PUBLICATIONS 491 CITATIONS 38 PUBLICATIONS 148 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Danish Ahmed on 23 February 2020.

The user has requested enhancement of the downloaded file.
Volume 22, Number 1, 2020 © WIETE 2020

Global Journal of Engineering Education

Measurable performance indicators of student learning outcomes: a case study
Tahar Ayadat, Danish Ahmed, Saidur Chowdhury & Andi Asiz
Prince Mohammad Bin Fahd University
Al Khobar, Kingdom of Saudi Arabia

ABSTRACT: Determining student learning outcomes is a crucial step in maintaining and improving teaching and
learning quality in education. Accrediting bodies require educational institutions to develop assessment systems to
analyse students learning outcomes. The main objective of this article is to discuss learning outcomes using performance
indicators within rubric-based assessment systems. This article complements various studies about assessment and was
intended for newly established engineering programmes seeking international accreditation. In this study, an example
was given of a grading scale mapped to performance indicators. The student learning outcomes at the programme level
were determined by combining and averaging the performance indicators of programme courses. Comparative analysis
was conducted between the direct assessment results and those of indirect assessments based on surveys distributed to
programme constituents including students, alumni and employers.

Keywords: Learning outcomes, teaching assessment, accreditation, continuous improvement, engineering education

INTRODUCTION

Student learning outcomes can be defined as a student’s ability to demonstrate a set of skills after completing
a course. Educational experts distinguish between student learning outcomes at the course, programme and institutional
levels [1]. They are interlinked and can be assessed qualitatively or quantitatively using measurable performance
indicators for each outcome [2-4].

Student learning outcomes at the course levels must cover the learning hierarchy as proposed by Bloom’s taxonomy,
starting from attainment of knowledge, comprehension or understanding, application, analysis, synthesis and
evaluation [5][6].

At the programme level, interpersonal (behavioural or attitude) skills including written and oral communications, ethics
and professionalism, teamwork and leadership must be incorporated to ensure students also possess the skills to succeed
in a professional setting. Student learning outcomes at the programme level can be derived from the learning outcomes
at the course level, and the learning outcomes at the institutional level can be derived from the learning outcomes at
the programme level.

In engineering education, students are expected to achieve these technical outcomes: to design engineering components
or systems; to apply knowledge of science and mathematics in engineering; and to conduct and interpret engineering
experiments. Other outcomes include interpersonal skills, such as written and oral communication, teamwork and
leadership, and lifelong learning [7][8].

At the Prince Mohammad Bin Fahd University (PMU), student learning outcomes for the civil engineering programme
follow the Accreditation Board for Engineering and Technology (ABET) student outcomes, which cover technical and
soft skills. The ABET is an international accreditation body with headquarters in the United States, and most of ABET-
accredited engineering programmes use ABET-prescribed student outcomes. However, ABET suggests that programmes
seeking accreditation can develop their own student learning outcomes at the programme level, provided they are in line
with the outcomes below:

(a) an ability to apply knowledge of mathematics, science, and engineering
(b) an ability to design and conduct experiments, as well as to analyse and interpret data

40
(c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as
economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability
(d) an ability to function on multidisciplinary teams
(e) an ability to identify, formulate, and solve engineering problems
(f) an understanding of professional and ethical responsibility
(g) an ability to communicate effectively
(h) the broad education necessary to understand the impact of engineering solutions in a global, economic,
environmental, and societal context
(i) a recognition of the need for, and an ability to engage in life-long learning
(j) a knowledge of contemporary issues
(k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice [9].

Out of 11 outcomes, six of them are related to interpersonal skills, such as communication, leadership, teamwork, ethics,
professionalism and lifelong learning. The remaining five learning outcomes ((a), (b), (c), (e) and (k)) are focused on
engineering design and analysis. At the higher level (i.e. institutional or university), student learning outcomes arguably
can be called graduate competencies or attributes [10].

As at the programme level, student learning outcomes at the institutional level must cover basic learning skills, such as
knowledge, cognitive and interpersonal (behavioural) skills. At PMU, student learning outcomes at the institutional level
have six learning competencies:

I. Communication competency: ability to communicate effectively in English and Arabic in professional and social
situations.
II. Technological competency: ability to use modern technologies to acquire information, communicate, solve
problems, and produce the intended results.
III. Critical thinking and problem solving: ability to reason logically and creatively to make informed and responsible
decisions and achieve intended goals.
IV. Professional competency: ability to perform professional responsibilities effectively in both local and international
contexts.
V. Teamwork: ability to work effectively with others to accomplish tasks and achieve group goals.
VI. Leadership: ability to be informed, effective and responsible leaders in the family, the community and the Kingdom [11].

The six competencies practised at PMU are unique among Saudi Arabia’s universities. They were commended as one of
the key institutional strengths by ABET team evaluators. Student learning outcomes at the programme level must be
[12] .
compatible with those at the university level

It can be shown that the learning outcomes at the programme level can be fully correlated to the learning outcomes
(competency) at the university level (Table 1). Furthermore, assessment of student competency at the university level
can also be determined quantitatively. Presenting students’ rating with respect to the university competency or graduate
attributes would enhance a student’s ability to work at a professional level. Because of this, PMU has started issuing
a competency rating to each graduated student in addition to an academic transcript. Arguably, the graduate attributes
can be extracted directly from some of the key courses that focus mostly on the development of student interpersonal
skills, such as internship and senior design project courses.

[13] . However, in
A separate rubric with key performance indicators can be developed for each university competency

this article, quantification of graduate attributes was generated based on student learning outcomes at the programme
level, with the intention of showing outcome consistency between the university and programme level.

Table 1: relationship between PMU learning outcomes (competency) and CE student learning outcomes.

CE student learning outcomes
PMU competency

a b c d e f g h i j k
I X
II X X X
III X X X X
IV X X X
V X
VI X X X

Learning outcomes should be distinguished from learning objectives. The objective refers to the teacher or programme
perspective rather than the student. Student learning outcomes and learning objectives can be stated at various levels.
At the programme level, they is called programme educational objectives (PEOs) and are one of the important
accreditation criteria that need to be assessed and evaluated. The PEO is intended to be achieved by engineering
graduates within five years of their graduations, which is different from the learning outcomes that are intended to be
achieved at the end of a course (e.g. after four years). However, PEOs must be supported by the student learning

41
outcomes or in other words they must be mapped congruently to each student learning outcome. The Department of
Civil Engineering (CE) at PMU has the following PEOs:

• PEO1: graduates have successful and professional careers in civil engineering and related industries, and meet the
expectations of the prospective employers.
• PEO2: graduates demonstrate leadership and effectively undertake services within their profession and contribute
to sustainable development in their communities.
• PEO3: graduates pursue their professional development through continuous lifelong learning, advanced studies and
membership in professional societies.

At present, the PMU Department of Civil Engineering has enrolled about 160 students supported by five teaching
professors, two laboratory instructors and one laboratory technician. The Department has four modern engineering
laboratories (materials, geotechnical, surveying and hydraulic) to support teaching and learning. Civil engineering
students are required to complete 139 credit hours to earn a Bachelor’s degree. This 139-credit-hour accomplishment
comprises 17 hours mathematics, 17 hours sciences, 57 hours engineering topics and 48 hours general social science.

The curriculum was designed in accordance with ABET specifications, with respect to math, science and engineering
course requirements [9][14]. The PMU CE Department has been accredited by ABET Engineering Accreditation
Commission and the next cycle of visits will be conducted in the next two academic years.

In this article, student learning outcomes at the programme level and competency at the institutional level will be
assessed and analysed. It is obvious that students can be assessed straightforwardly at the course level by a standardised
numerical scale measurement (e.g. 0-100) or letter grading (e.g. A to F). At the programme and institutional levels,
the assessment requires different indexing to indicate learning outcomes. The assessment of 1 to 100 is too refined to
indicate outcome achievement, and in this article four-scale rating is used to assess learning outcomes [15].

ASSESSMENT STRATEGY

Key performance indicators (KPIs) and rubrics for each programme outcomes were developed based on the applicability
and practicality of assessing courses against the outcomes. The rubrics considered basic criteria, weight for each
criterion and level [16]. Table 2 shows an example of outcome (a), its KPIs and rubric. The remaining programme
outcomes, KPIs and rubric can be seen in a document issued by the Department [17]. Most of the KPIs are limited to
four (e.g. outcome (c) and some have only one indicator (e.g. outcomes (i) and (j), which are interpersonal skills
related). Outcome (c) is designed to have the most indicators because of its design-oriented outcome.

Table 2: Rubric for outcome (
a).

Outcome (a): ability to apply knowledge of mathematics, science and engineering

Criteria Low (1) Needs improvement (2) Good (3) Excellent (4)
Fails to understand Shows limited and less Demonstrates Understands and
and apply proper than adequate satisfactory applies proper and
a1: Apply
linear algebra and application of linear application of linear accurate linear
mathematics to
differential calculus algebra and differential algebra and algebra and
solve engineering
in solving calculus in solving differential calculus in differential
problems.
engineering engineering problems solving engineering calculus in solving
problems problems engineering
problems
Fails to apply Shows limited and less Demonstrates Understands and
fundamental than adequate satisfactory applies proper and
a2: Apply
concepts and understanding of theory application of proper accurate concepts
concepts and
theories in solving and concepts in solving concepts and theory in and theories in
theories of science
science and engineering problems solving engineering solving engineering
and engineering.
engineering problems problems
problems
Fails to transform Shows limited and less Demonstrates Understands and
science and than adequate satisfactory applies proper and
a3: Convert
engineering transformation of transformation of accurate
science and
problems into science and engineering science and transformation of
engineering
solvable problems into solvable engineering problems science and
problems to
mathematical models mathematical models into solvable engineering
solvable
mathematical models problems into
mathematical
solvable
models.
mathematical
models

42
The assessment tools included quizzes, examinations, term projects, laboratory experiments, internship training reports
and senior design project exercises. The Faculty maintains a numerical value for each course, a course assessment report
and a teaching improvement strategy. It should be noted that not all KPIs are applicable to each course.

For example, courses without laboratory experimentation will not mention outcome assessments about conducting
experiment and experimental data analysis, which is one of the performance indicators for outcome (b). Normally, in
one semester three measurements are taken for each course covering one mid-term examination, one final examination
and a group term project. The examination and project questions reflect the KPI in the assessment. No homework
assessments are required since they are only intended for student practice.

Table 3 shows an example of a student assessment for an engineering course, for outcomes (a) and (e). Each question
was devised in accordance with the KPIs including the maximum grade students can attain. The rubric was developed
based on a four-scale assessment and so the student grade needs to be converted. Each student will have a unique set of
performance value, and averaging outcomes values for all students will result in the overall performance for that specific
course.

Table 3: Sample of rubric assessment for a civil engineering course.

ABET Maximum Student ABET KPI Student
Question Maximum
KPI grade grade (1-4 scale) grade
a1 20 10 2
1 a2 5 4 3 35 24
a3 10 10 4
e1 20 16 3
2 e2 35 30 3 65 46
e3 10 0 1
Total 100 70
Note: KPI = key performance indicator

The teaching faculty checks whether the outcome is below a threshold value (e.g. 2.5) and identifies the teaching
improvements if they are required. If the outcome values for one course are averaged with the other courses with
the same outcome assessments, this will result in the overall student learning outcomes at the programme level. This is
the focus of this article.

Table 4 shows 10 courses used as indicators for outcomes assessment at the programme level. The main justification for
not including all engineering courses is that the measure must reflect the ability of civil engineering students to master
key design courses. If all engineering courses are included in the assessment, this would under-represent performance
because students at the early engineering level do not have ability in design and analysis.

Table 4: Selected 10 courses for student learning outcomes assessment at the programme level.

Code no. a b c d e f g h i j k
No. Course name
1 Materials in Civil Engineering CVEN 3322
2 Reinforced Concrete Design CVEN 3312
3 Engineering Measurements CVEN 3341
4 Environmental Engineering Fundamentals CVEN 3331
5 Hydraulic Engineering CVEN 4432
6 Intro to Geotechnical Engineering CVEN 4423
7 Design of Steel Structures CVEN 4313
8 Construction Management CVEN 4314
9 Water and Waste Water Treatment CVEN 4333
10 Learning Outcome Assessment III ASSE 4311

The Civil Engineering Department Council decided that the 10 courses selected must cover major civil engineering
disciplines, including structures, environment, geotechnical and construction management. Apart from the
transportation engineering course, courses were selected at the junior and senior level that are design and/or laboratory-
based.

e) is measured the most (six times). This is because outcome (e) is related to basic
It is seen from Table 4 that outcome (
engineering skills for students to identify, formulate and solve engineering problems. This is followed by outcomes (a),
(c) and (k), which relate to the design and analysis of engineering problems. With respect to outcome measurement
frequency, course Learning Outcome Assessment III (No. 10), also known as the senior design project, has the greatest
frequency (nine) from outcomes (c) through (k). This is because the senior design project demonstrates the mastery of all
the important skills that students are expected to acquire by the end of their undergraduate studies [18-20].

43
Outcomes (d), (g), (h) and (j) were taken twice; this is due to the nature of interpersonal skills assessment that can only
be measured through non-conventional ways; for example, by using self-evaluation in teamwork exercises, peer review
and evaluation of lifelong learning.

ANALYSIS RESULTS

Assessment of the programme outcomes has been conducted for six semesters within three academic years (2016 to
2019). Student learning outcomes (a) to (k) can be quantified by averaging the KPIs for each outcome. Summarised in
Figure 1 is the result for the 11 outcomes since the academic year 2016 to 2017. A line indicating threshold value (2.5)
is also shown in the graph.

Unlike the assessment conducted for each course, the numerical values for the programme outcomes are presented up to
one decimal point to indicate a refined level of attainment. This seems to contradict developing a rubric with a rounded
value of 1 to 4. However, the refined outcome values will be useful later to support justification for continuous teaching
improvement. It would be difficult to observe trends for outcome improvement if they are presented only in rounded
numerical values.

Figure 1: Student learning outcomes at the programme level.

In general, it can be observed that the target outcomes were achieved, all values are above the threshold value (2.5).
Looking closely at the average values, outcome (a) has the lowest value relative to the other outcomes. Although it is
higher than the target value, this could indicate that teaching staff should pay more attention to students’ abilities in
applying knowledge of mathematics and science in engineering. Except for outcomes (a) and (e), the other outcomes
showed improvement in the past three academic years.

Figure 2: University competency (graduate attributes).

44
The