Another aspect of making assessment relevant is to focus yearly assessment efforts on small pieces of assessment and/or problems that many program faculty agree are urgent and important. Simply recording the department discussions serves important assessment purposes: including clarifying the problem and the evidence for it, and deciding on a course of action and evaluation. These examples show the programs (and us) how they approach student learning. They allow us to recognize student learning issues shared by many departments and programs.
For 2009-2010, we decided to assess the first learning goal from our assessment plan:
- Students should be able to write software in several programming languages, including intermediate to expert proficiency in at least two languages of distinct paradigms. Look at final projects for Software Design, and some programs from Programming Languages. Classify each as indicating whether or not it shows intermediate-to-expert proficiency.
How we assessed the goal
Since all of the CS majors in the class of 2010 took Programming Languages from Dave Musicant and Software Design from me, Dave and I were responsible for this year’s round of assessment. Here’s what we did:
- We selected at random one third of the CS majors in the graduating class. This sample turned out to include a good range of students, including our weakest senior, one of our strongest, and a bunch in the middle.
- We had a preliminary discussion of the criteria we would use to determine “intermediate” and “expert” proficiency. This turned out to be tricky. We both found ourselves in one of those “we know it when we see it” situations familiar to assessors of great art and obscenity. So we sketched a few notes about particular constructs we would like to see in the students’ Python and Java programs, and moved on to the next step. During this meeting, we also decided to skip the programs written by the students for the Programming Languages course. That course is notable for giving the students the chance to study several programming languages (in addition to learning about programming language theory), but does not attempt to guide the students beyond beginner-level proficiency in those languages.
- We studied the assignments from Software Design submitted by these students. The assignments came from three different sections of the Software Design class, so there was no single assignment done by all the students. Each student had submitted at least two assignments written in the Python programming language, and two written in Java.
- We met to discuss each student’s work. This interesting meeting led us to the following observations.
- The best programmers often avoid complicated language structures. Coherent and effective design is often best implemented via simple code. Thus, you can’t assess beginner/intermediate/expert proficiency in a programming language simply by looking for fancy language features.
- That said, there are some intermediate-level language features in Java and Python that were appropriate in the assignments we read, and the best programmers used those features. (For example students who used regular expressions, built-in date and time processing tools, convenient graphical user interface classes, etc. generally wrote better code overall than did the students who used less specialized language features to achieve the same goals.)
- Some language features stick out as red flags in student code (usually through their absence), and thus might be especially valuable in assessment of language proficiency. (Notably, only the strongest students made reliable and appropriate use of “final” and “static” in Java, list slicing and dictionaries in Python, and exceptions in both languages.)
- One student (whose code was certainly of intermediate proficiency) used advanced language features, but not entirely appropriately. So is his proficiency higher because he used tools many students didn’t know about? Is it lower because he didn’t notice the inappropriateness of his usage?
- Overall, our feeling was that all but one of the assessed students had intermediate-or-better proficiency in Python, and about half also had intermediate proficiency in Java. This conclusion was reassuring in the sense that we’re nearly reaching our goal (and thus don’t need to panic), but also leaves us with plenty of room for improvement.