How do I write an effective Test Suite in Cody Coursework?

With the introduction of Cody Coursework, I'm often asked how to best use the power of MATLAB to effectively challenge students' code.

[Ben Hawkins]

This answer is very outdated. Unfortunately, it shows up really high on Google search results. None of these are necessary with recent updates to Cody, in my experience.

Answer 1

[MathWorks Community Team]

First, we need to understand why MathWorks has made the Test Suite visible to students. Students are encouraged to use MATLAB on their computer to solve problems. To make this possible, students must be able to copy and paste the Test Suite code into a MATLAB script which, in turn, calls their solution function. Once satisfied, they then copy their solution function into Cody Coursework for assessment.

With this in mind, consider how a student might "game the system" presented with the following Test Suite from the Times2 problem on Cody:

   assert(isequal(times2(1),2));
   assert(isequal(times2(11),22));
   assert(isequal(times2(-3),-6));
   assert(isequal(times2(29),58));

Here, the Test Suite simply calls the students' function, times2, using explicit data values to confirm the students' code returns the expected result. This approach is morally hazardous, as a student can simply write a case or series of if statements to compromise the approach.

Each Test Suite is a MATLAB script. Here we draw on this power to more effectively assess students' code in response to the Times2 problem,

   X=rand(1);
   Y_answer = x*2;
   assert(times2(x),Y_answer)

This still poses two issues, 1. it reveals the answer the students' code needs to implement (i.e. return x * 2), and 2. it does not apply well to complex programming problems.

Now that we've examined what not to do, let's discuss a more effective approach for assessing students' MATLAB code. Faculty already have a reference solution to each programming problem assigned, traditionally offered to teaching assistant staff for manual grading. Protected MATLAB code offers the key of how we use these reference solutions as part of a generalized recipe to write an effective Test Suite.

First, compile your reference solution as Protected MATLAB code.

Second, post this .p file somewhere online accessible by Cody Coursework. This must not be behind the school's firewall, it must be publicly accessible. One option is to use the public folder of a free dropbox account. If using DropBox, it will help to " copy the public link " using their web interface.

Third, implement a test case of your test suite to use your protected reference solution code, for example:

   %%Test Functional Correctness of Students’ Code
   urlwrite('http://url/ProfsFunction.p','./ProfsFunction.p');
   addpath('.');
 
   for i=1:10
     data_vector=rand(1000,1);
     prof_answer = ProfsFunction(data_vector);
     student_answer = StudentsFunction(data_vector);
     assert(isequal(prof_answer,student_answer), 'student code gave an incorrect answer');  
   end

Adapting this for more complex problems will require you adapt the data_vector randomization to something appropriate for your subject matter. Alternatively, you might wish to incorporate a data file from which your Test Suite randomly selects input/output values.

Fourth, add a test case to prevent student attempts to call on the .p file themselves, for example by parsing the students' solution file directly,

   %%Test for attempts to cheat
   found = 0;
   fid = fopen('StudentsFunction.m');
   while(~feof(fid))
      s = fgetl(fid);
      k = regexp(s,'urlwrite\(|ftp\(|mget\(|web\(|fprintf\(|fwrite\(|sprintf\(');
      if ~isempty(k)
          found = 1;
      end
   end
   fclose(fid);
   assert(isequal(found,0))

Fifth, use the solution map as a "line of last defense." Typically, honest solutions which are correct follow similar patterns. Horizontal bars of these patterns form on the solution map. The few stray solutions below these bars which Cody Coursework has marked correct merit manual code inspection.

There are many possibilities the power of the MATLAB language offers for implementing test suites. We will add more information and examples in future, and I encourage you to share your own insights as answers or comments below.

[C Lira]

Can you explain the line

k = regexp...?

does not the line

k = regexp...

cause any code the student has written that has formatted output fail the test?

Many of my templates will include these statements unless I edit them heavily.

[Jan]

Searching for "web" will cause false hits, e.g. when it occurs in a comment. It will be very frustrating for a student, when his solution fails, because his name contains this character sequence.

After several discussions in this forum it is clear, that the contents of P-coded functions is not hidden securely, because the debugger reveals almost all details. I would encourage any teacher to read the code of the students personally, while the automatic assessment can and will be cheated.

[EDITED] You can let Cody's virtual machine load and run code from external links? Isn't this a massive security problem?