Summary

The Non-Profit FOSS Institute (NPFI) provided Steven Huss-Lederman with a grant for the 2018-19 academic year to implement an HFOSS project in a senior capstone course that spanned the entire academic year. This course differed from the CO-FOSS model in several ways and thereby offered an opportunity to analyze how this model might be applicable in fostering HFOSS projects at academic institutions. This report to NPFI summarizes the techniques used, analysis of outcomes and ideas/steps for using this model.

Overview

This report contains the following sections:

1. Overview of Open Energy Dashboard Project
2. Overview of Course and Beloit College
3. Overview of Model
4. Tradeoffs & Experiences
5. Why Use This Model
6. Timeline of Actions
7. Course Outcomes
8. Course Materials

Many of the thoughts in this report mirror what others have written about using open source projects in a course. Thus, the discussion is not necessarily unique but is designed to show how these ideas apply to this model.

Overview of Open Energy Dashboard Project

The Open Energy Dashboard (OED) is a Humanitarian Free Open Source Project (HFOSS) aimed at supporting sustainability by allowing an organization to acquire, store, analyze and display energy usage. The user interface is via a web browser where significant code runs on a back-end server. The software is designed to be usable and customizable by each organization to allow for easy portability. The OED project has existed since the fall of 2016 and, as of spring 2019, is in use in about half a dozen colleges and universities. While it can be used by any organization, higher education institutions have been the users to date.

An important aspect in terms of its use in a course is that it was begun by a college (Beloit College) and continues to be driven by academic institutions. This means that while the project has important goals in terms of sustainability, it also wishes to support its use in computer science education. Thus, the OED project offers mentoring to interested developers, links to resources to learn about its technologies, and tries to organize its work so it is accessible to a range of potential developers. It also has a tradition of helping support the advancement of developers for both the advantage of the student and the project.

OED is a modern application that utilizes a large number of technologies and frameworks. These include Javascript with promises, React, Redux, PostgreSQL and general SQL. The application is designed to rapidly acquire/store new data and provide graphical displays of information with sub-second response times. These goals mean that while functionality is important, it is generally critical that efficient implementations be used. This means the code is more complex than a less responsive system. The code also needs greater complexity to meet its goals of portability, testing for reliability, and other goals. This does not mean that it is beyond the reach of student developers but it does mean that a solid base of computer science knowledge is needed to engage with the OED project.

It is noted that while OED was used in this course, the general idea is using an existing open source project in support of educational goals in a computer science curriculum. Thus, the model should apply to a large number of potential projects and can even start as a new project for the initial offering of a course using this model.

Overview of Course and Beloit College

The Open Energy Dashboard project was used as part of the Database Capstone I & II course at Beloit College. This pair of courses runs over two semesters in one academic year where it generally meets three times per week for 65 minutes for each class. Students normally take both semesters with limited exceptions for student who are not in residence in a given semester (such as during study abroad). This report is about the 2018-19 academic year when 17 students were enrolled in the course and four students only did one semester (two first semester and two second semester so enrollment was constant at 17).

The course has a number of objectives that directly relate to the use of OSS:

• Learning databases. This course must teach students the fundamentals and practical usage of databases. Some students get experience with certain parts of databases via their work on OED. However, many would not due to the extensive front-end parts and the fact that significant parts of the back-end database code is complete. Thus, the course exposed students to the database aspect by having them create a database to analyze data for the local government. This part of the course was separate from the OED work and not directly related to this model.
• Learning real-world development skills. Computer Science students at Beloit College take a separate software engineering course. However, this course is designed to let them apply practical development skills to a real problem. As such, the students are exposed to coding including learning new languages/technologies, development environments and shared repositories. The OED project naturally allows for all of these goals.
• Real-world experience. Beloit College has a Liberal Arts in Practice (LAP) requirement where every student must do significant work outside the traditional classroom. By working with the OED project, students get exactly such an experience. Furthermore, doing a humanitarian project fits well with the goals of Beloit College and the interest of many of our students. This model does not require the use of a humanitarian project.
• Teamwork. The Beloit College Computer Science Program is committed to exposing students to teamwork and making them effective members of teams. One reason all students in the course worked on the OED project was to get them involved in a large team effort that was accomplished through a series of smaller sub-teams.
• Mentoring. This course sequence deliberately mixes 3rd and 4th year students. The goal is to have more advanced students mentoring less advanced students. While this often means 4th year students are mentoring 3rd year students, there are many times where a 3rd year student is more knowledgeable in an area so is mentoring seemingly more advanced students. This is exactly what is desired so that students consciously think about both sides of the mentoring relationship: being a mentor and being a mentee. It should be noted that this model of mixed year students is also due to curricular and staffing considerations and is not essential to this model.

Some of the setup of this course directly relates to its use for supporting an HFOSS project and were desirable in supporting this model. These were:

• Grading. As mentioned above, the course had multiple objectives. The percentage value of different aspects of the course were designed to reflect these objectives. They were (first/second semester):
  • 10%/10% for class participation
  • 25%/18% for assignments
  • 40%/47% for project
  • 25%/25% for examinations
• Weekly wiki. To keep track of all the students in the course, each student has to input a wiki entry by the end of each week. This was essential to know about how each student was doing along with understanding progress on the project. By early the next week, the instructor would review each entry and put comments in the wiki for the student. The comments included if the work was satisfactory, suggestions on issues encountered, and, sometimes, a request for a meeting with the student or team. The weekly entries requested the student to address each of these items where only work related to the project was included:
  • What are the items you accomplished?
  • Did you work with others (by item if it varied)?
  • How much time did you spend on each item?
  • How were you a good mentor/mentee?
  • How is it going with your team?
  • Do you need or want something to help you participate in the project?
  • List what you did for any times that class time was made available as a separate entry from other work with the time spent listed separately.
  • Do you have any questions or thoughts?
  • Anything else you think is important.
• Project discussion. Whenever possible, class time was provided on Fridays to discuss the project. It began with a whole class discussion of general project progress and global issues. It was followed by the instructor meeting with teams/individuals as appropriate. These were found to be invaluable as big picture items along with some items students were loath to put in the wiki entry were brought up.

Overview of Model

An important model for supporting HFOSS in higher education is the CO-FOSS model. The techniques used in the current course has similarities and differences from the CO-FOSS model as outlined below:

• Existing versus homegrown project. This is probably the most important difference. The current course continued development of the Open Energy Dashboard (OED) that is an ongoing HFOSS project to develop a freely available energy dashboard. This is a large and complex project involving multiple technologies that had been ongoing for two years and is expected to continue development for many years to come. This contrasts with the CO-FOSS model where the team works with a client to design and develop a solution in the time frame of the course. It also means that the final work is integration into the project's repository rather than delivery to the client for final deployment. Thus, unlike CO-FOSS, there was not a need to find a source of ongoing support as the overall project provides this framework.
• Large community versus single/limited client. OED does not have a single client but is designed to be flexible and customizable for use by any organization. As a result, project requirements come from listening to the larger community and filtering those needs/requests into general features for future project releases. This contrasts with the CO-FOSS model where a single entity provides the requirements and there is one person or a limited number of people who discuss requirements and review progress. Combining with the fact that OED is an ongoing project, the students experienced the design process in a very different way than the CO-FOSS model. With OED, the students were working with requests/issues that already existed within the project. Therefore, they did not do a complete system design but worked within the existing design with much more limited scope. Furthermore, the students did not tend to interact directly with clients but did so indirectly by working with the OED project manager (also the teacher of this course) to understand and refine requirements. This does not mean that a client-oriented design focus was lost but it does mean that students did not experience the creation of a new project and did not have the traditional client of the CO-FOSS model. It should be noted that somewhat over half the students in this course had already completed a software engineering course so they already had the more traditional exposure in that course.
• Learning project technologies verses picking them. Since OED already existed as a modest size project, the main technologies had already been selected. These included JavaScript using promises, React, Redux & PostgreSQL and many others that had more limited usage. Depending on what aspects of the OED project were being worked on by a student, they needed to learn a number of these technologies well and have a basic understanding of the others. This required allocating many weeks of the course to learn what was needed. This contrasts with the CO-FOSS model where the project can pick appropriate technologies to meet the user requirements and also take into account student knowledge/ability to learn. The CO-FOSS projects that have been discussed to date required fewer and simpler technologies. This means that having students jump into the OED project was much more complex and this has been noted by others who have worked on existing OSS projects.
• Learning verses designing an architecture. OED is a non-trivial project with an existing architecture. This necessitated students learning the OED software design and understanding the interrelationships of parts of the code. As with most existing projects, but esp. as one that is only a few years old, there were quarks and deficiencies in the design of OED. This made this process even more complex. This contrasts with CO-FOSS where a new project is begun or a modest size project is reworked. With OED the students had much more expose learning to understand existing code than CO-FOSS where they get a strong overall design experience. The OED students needed to learn how to be "productively lost" (a phrase in the OSS community) where they balanced a complete and detailed understanding and what was needed to get the job done in a reasonable period of time. The OED project, in general, uses more real-world technologies that are utilized in larger projects outside of academics and students appreciated that applicability. On the other had, the instructor had to monitor and deal with student frustration in feeling overwhelmed by what needed to be understood.
• Student goals and timelines.The goal was for each student to contribute code that was integrated into the OED project. This might happen multiple times over the course and could happen at any time. Also, given the size of certain requirements and uncertainty of the scope of development, students might not complete the work and leave a fork of partly developed code. Clearly this was less desirable but a reality, esp. given a wide range of student abilities. Students could also complete coding until near the end of the course. (This did raise issues when changes were needed with no time to complete but this did not mean the work was lost or could not be used.) In the CO-FOSS model, the final goal is a working application so success is defined differently. Also, the application needs to be given to the client with documentation so time is left at the end of the course to work with the client. This leads to somewhat different student experiences where, in hindsight, the OED experience would have been enhanced by stronger deadlines.
• Year-long verses semester project. Anyone working with students to create software in an academic course appreciates the fact that semester (or quarter) courses impose significant constraints on the work done by students. The current course runs as ½ a standard course per semester for two semesters. Having a longer time frame makes it much easier to deal with software issues and questions. It also allows for larger chunks of work to be accomplished. The break between semesters does represent a scheduling challenge as the gap allows students to forget what they were doing if a task remains incomplete. In contrast, the CO-FOSS model has been run in a traditional one-semester course. The issues of this time constraint has been discussed by CO-FOSS users. The difference of one vs two semesters does not seem fundamental to either model was they both could be done on different timescales. However, the fact that there was a difference in the courses to date does mean that the experiences were influenced by this.

Tradeoffs & Experiences

Several of the tradeoffs and experience have been mentioned in the previous sections. Here greater details are given with an honest assessment.

• Development environments. As others have noted, getting students working within the development environment of a OSS project can be a daunting task. OED tried to address this issue by investing in Docker setups to allow for more automated creation of development (and production) environments. A significant issue encountered was Microsoft's decision to discontinue support for Docker on platforms other than Windows Professional. At least ¾ of the students used Windows but none had a professional grade installation of the OS. This necessitated doing manual installs that were more difficult, error prone and required help from more experienced developers on the team. The time spent and aggravation experienced by students was a distraction to the real goals of the course and project. This was quickly realized and a full development environment (under Linux) was created in the student lab. Unfortunately, students continued to spend many hours trying to get OED working on their own Windows machines rather than use the one set up in a lab. In hindsight, the course should have made it explicit that only Linux-based systems were supported (including Mac OS) and students without this OS needed to use a provided machine or a containerized/dual boot system. This would have saved student frustration as well as a drain on the time of more experienced developers. It seems a good practice to consider not just the ease of installing the development environment but the range of machines that students can use.

  Another issue encountered was the command prompt. Many students in this course had no or limited experience in working with the command line. Even though directions included explicit commands to run, students had issues and were unable to overcome seemingly simple issues. This situation was magnified on Windows machines where manual installs via the command line were needed. In hindsight, the course should have had an early lab on command line usage to avoid many of these problems.

  Finally, a few students had old machines or ones that were in a poor state in terms of installed software. First, this lead to operations taking much longer than should have been the case. Second, students would sometimes skip installing a software package because they found it was already installed. However, if the installed version was old, the student would encounter errors that were difficult to understand. Again, this was more of an issue on Windows machines where the Docker install should have taken care of these problems. (Note it does not fix all issues as the development IDE used might not support recent versions of the languages being used.) There is no trivial fix for this situation but students were given information on the minimum/desired versions of software. Finally, if students had used the provided lab machine then this would not have been an issue but they did not wish to do that.

• Learning technologies. As noted in the overview of the model, OED uses a number of modern technologies/frameworks. These often went beyond what students had already experienced in previous courses. This was a known concern and students were provided with online links to overview, learning and reference materials for all the main technologies. Furthermore, they were given three weeks of class time to learn in a self paced structure where questions could easily be asked. In several cases, enough students had questions about an area that a mini-lecture was given on that topic. It is frustrating to report that this did not work out well. About ⅓ of the students who were self motivated and/or had more experience took advantage of this time to get a basic understanding of the OED technologies. The remaining students did not with most not seeking any help. As would be expected, these students had ongoing issues with the project. It is clear that the issue was not a lack of course time dedicated to the process but how it was done. In hindsight, the course would have benefitted from these steps:
  • Regular mini-lectures to introduce a topic. This would serve as a chance to frame the needed technology and made it easier for students to ask questions.
  • Short quizzes on a frequent basis. This would have reminded students were they should be in the process and given the instructor more timely feedback. The instructor mistakenly thought that students in the capstone were mature enough to self-regulate and doing this was an important capability in preparation for working outside of classes. While this probably should have been the case, it was not and external motivation was needed.
  • Every student should have been made to do a simple pull request to the project such as a change in wording to documentation. This would have helped guaranteed they had their development environments set up. Possibly having to specify the change on the development branch or some other activity might have help. The trick is something that would be hard to do without a true development environment (something harder due to the nice interface from GitHub). Some students tried to work without this, such as used a text editor, and it naturally lead to significant issues.
  • It may also have made sense to require students to turn in short program utilizing the needed technologies.

  When the depth issue was recognized half way through the course, the students with issues were required to go through sections of the code to add/modify the documentation. The goal was to force them to engage with the technologies. This was done by setting up small teams (2-3 students) where some teams could interact due to overlap in the technologies being engaged. The OED project was broken up in two dimensions: technologies & features where there were three main areas for each of these. This lead to a grid of 9 items where some overlapped so significantly that they were combined. This ultimately lead to 5 areas of technologies/features. The twelve students who needed this engagement were assigned to one of the areas and a few were assigned to a primary and secondary area. Each area was also assigned a mentor who had a good understanding of the area (from the ⅓ of students who had a good understanding of the project). The teams then worked on their area. Once documentation was done, a team working on the same technology and/or feature was notified so they could review the new documentation. This served to both check the proposed documentation and force students in one team to stretch into related areas. After this step, teams submitted a pull request to get the updated documentation included into the OED project. Each team was also to include an overview of their area in a master document.

  Students uniformly liked this activity and having done it earlier would have been much better. The reality is that about half the students (7 of 12) engaged successfully in the sense that they were then able to engage with coding activities in the OED project. The other half continued to struggle but almost all of those students did not reasonably engage in this process.

  It is noted that it takes a good technical understanding of a project to separate out the directories/files into different boxes. For reference, OED was separated as follows:

  		Features
  		graphics	Data acquisition	Admin
  Technologies	DB	team 5		team 4
  	API/server		team 2
  	Front-end	team 1		team 3

• Background knowledge. With about half the students being 4th year having taken a software engineering course and the other half being 3rd year without software engineering, there was a greater range of background. As previously mentioned, this is partly to allow for exposure to mentoring. While there are advantages, it was certainly trickier to run a large project with this many students when some did not appreciate software engineering and had limited experience in working on a large software effort. It was clear that the less advanced students had greater difficulties in effectively working on the project. If possible, it would be best if students involved in a larger, real-world project already had experience in developing at least a medium size project and working on teams.
• Tracking student progress. It can be trick to know how much time and effort is needed to address an issue or feature in a project. As a result, it was often difficult to know if a student or team was applying sufficient time and being effective. The weekly wiki helped but it was still difficult. Augmenting these with regular meetings, probably being once a week would be ideal, would have helped in this matter. A combination of group and individual meetings would likely be best. The reason these were not done was a time issue when nearly 20 students were involved. However, in hindsight, they should have happened on a regular basis and not been more at the request of students.
• Deadlines. A modest fraction of the students did not make sufficient progress. Having clearer deadlines would have helped. Being more rigorous in the agile deadlines where everyone did a sprint for a specified period (say two weeks) would have helped. As long as students understood that they did not have to finish in one sprint but needed to discuss why not and the plan for completion then it would probably work. It was difficult to be reasonable and flexible in deadlines while still holding students accountable in work where the difficultly of work was tricky to judge in advance.

  An important consideration in missed deadlines is the impact on other students and the project. Balancing the educational needs with project goals was a major concern throughout the course. At times it required shifting students off a team or adding a team member so progress was made. As skills and effort for each student became clearer, the more capable students were placed on more critical tasks and others were placed on desirable tasks that would not impede the project if deadlines were missed. This balancing act took the instructor more time than desired but no easier alternative was found.

• Evaluation. The instructor mistakenly thought that the feedback provided on each weekly wiki along with periodic discussions would be sufficient for the students to know their standing given this was an advanced course. Student feedback clearly indicated otherwise, esp. for students who were having issues. Thus, in the second semester a clearer set of criteria were provided and more regular feedback given. The project was worth almost 50% of the course grade. The criteria for the second semester were:
• 15% for wiki and check-ins. Students received written feedback on each wiki entry and verbal feedback at meetings.
• 10% for quality of community membership (helping others, working well with others, meeting deadlines). Students were instructed to put information about this in their weekly wiki. The input of the student and other students in the course were used to determine the situation. Entries in project spaces such as comments on pull requests counted toward this area.
• 50% for student efforts. This was based on weekly wiki entries that required accounting of time spent on each task and the entries of other students. The number and size of pull requests or code in a branch was also used. A discussion of this area was an important part of the final student meeting.
• 25% for the quality of student work. This was primarily based on code in the project repository where care was taken for team work where the id associated with a pull request might be another student. The wiki entries and discussions helped with this but students were also informed that all team members should be doing commits at points in time. This was an important part of the final student meeting where the instructor explicitly talked to other team members to get their take on the work of other students.

It was found that a non-trivial percentage of students had issues in getting in a substantial pull request to the project. This is why such a large percentage was allocated to effort compared to quality of work. The community membership was important to acknowledge the importance for the project and others in working as a team.

Why Use This Model

The model described was chosen because it was a good fit for the objectives of the course. Here are a few thoughts on when and how this model might be appropriate for others.

• If you want to give your students a more open and potentially shifting experience then using an existing open source project can help. Students need to learn the tools and technologies of the project and these often mirror those of the real-world developer. Students are also faced with a large code base so they learn how to jump in and move forward without understand or learning about all the parts.
• Knowing the open source project is helpful (but not necessary). If the department/institution, instructor or students have experience with a software project then this is a clearer avenue to its uses in a course. It is still important to be sure it will be viable for your course objectives. If you don't have an existing project, picking one that aligns with student or instructor interest can help with motivation and general knowledge of the application area.
• Having a project that is used more generally within a department means you build on the use outside your course. It also gives students an avenue to engage with the project after (and even before) your course. For a number of students, the deepest knowledge and experiences happened after the first offering of this course when they continued work as a special project. Building on their existing knowledge gave them a very rich experience. In addition, having a range of students working on and knowing about the project on an ongoing basis created an ecosystem where students could help each other and felt part of an important and larger effort.
• One issue with an extant project is engaging students with more limited experiences. In the case of OED, having students do documentation, giving them class time to learn the needed technologies and having tasks that were easy for a beginner all helped with this issue. This issue can be magnified when you try to integrate in multiple ways within a department. Thus, knowing a project can support your range of students is important.

Timeline of Actions

Similar to doing any large project, utilizing an open source project means doing items at appropriate time. Here are some of the more important ones for this model.

• The biggest choice is deciding on an open source project. As written by others, you need to consider the size, scope, technologies, friendliness, etc. of the project. Some ideas on considerations for the project are elsewhere in this report. Starting early cannot be stressed too much. It takes time to evaluate projects and one often finds that a promising one later turns out to have issues. Thus, starting at the high-level, narrowing the field, and then looking more closely can be a valuable way to go about this. If at all possible, starting months in advance is the way to go.
• You want to begin interfacing with the project early in the process. Initially it is to understand how welcoming the people are to students, how do they deal with tasks including submitting proposed code, and do they answer questions in a timely fashion. Later on you need to see the documentation for the project and technologies, the tasks that are available and how to determine appropriate ones for your course. In an ideal situation, you can get contacts/mentors for your students and the project will listen to the timeline/goals of your course so you know everyone is on the same page. The ease of working with a friendly project that offers support cannot be underestimated. Having this in place at least a month before your course begins will make the process go much smoother.
• If you want to expose your students to the clients involved and it is possible for the chosen project, then you need to contact people in advance of the course to make arrangements.
• Before the course begins you need to have a list of appropriate tasks where they are ideally separated into types (newbie, modifying code, new code, design, etc.). In some cases the project may not clearly articulate the larger picture of why this feature is needed. Getting that information so students can appreciate the work is both valuable and motivating. In the case of OED, the GitHub issues page was useful but insufficient. The project organizer provided more information and this was then used to create a course specific document on the features that would be addressed, why they were being addressed, and links to needed resources to engage with these tasks. An ideal project might have this but generally they do not.
• Once the semester begins you will need to form teams to work on the project. There are different ways to do this: student interest, grouping students with similar or a range of experience, randomly, rotating, etc. It isn't clear that one way is superior but having a plan and being flexible is important. With the uncertainty of engaging in an ongoing project, this flexibility to shift students around is often critical as the situation unfolds. For the OED experience, it was important to get the teams set up early so students clearly understood their work and had other students to work on issues they encountered. Generally, lone students tend to spin their wheels more and flounder.
• Keeping track of what is going on with individuals, teams and the overall project is important. As discussed elsewhere, students did a weekly wiki as input. Also, every Friday was a day for team reports, to raise issues and track the overall project. The OED experience over the years has lead to more rather than less oversight. It was found that fewer students ran into issues and the project was more successful with greater oversight. This was even the case with better/motivated students who worked on OED in special projects.
• Having a timeline for work to be completed and track the progress. Students often don't appreciate the time needed to accomplish tasks and when they are starting to get into trouble. The tracking just mentioned should help the instructor appreciate the situation. However, students should get a rough schedule at the beginning and it should be revised, as appropriate, throughout the course. In the OED experience, students who had issues had a tendency to give up and needed redirection. On the other end of the spectrum, students who experienced few issues sometimes slacked when they should have begun focusing on new tasks. Only though oversight by the instructor were these issues addressed.

Course Outcomes

Overall, the course evaluations showed that students had a good experience. They appreciated being exposed to real-world software and development. As seen by others, they were motivated and excited to work on a humanitarian project. Another important outcome is that a significant fraction of the students report that their work on OED was discussed during job/internship interviews and/or they felt it was important in securing a position. It is also noteworthy that this work/project was noticed by the institution and its leaders.

Course Materials

Course materials that might be of value to others are:

• Course overview: CSCI 365/first semester
• Course overview: CSCI 366/second semester