In academia, at least, it’s a rare privilege to write the criteria by which your work will be evaluated. That was, in fact, one of my major tasks for the 2015-16 academic year, and part of my backlog of blog topics. Why is it on my mind right now? While I am being reviewed for tenure at Whitman, I also just submitted my first letter as an external reviewer for a tenure case at another institution. That other institution’s departmental evaluation guidelines look quite different from the ones I wrote for Whitman, and that got me thinking back on the process of developing guidelines.

I tell the story of Whitman’s CS Scholarship Guidelines below.

---

Whitman College requires every academic department to to “develop and submit to the Committee of Division Chairs scholarship guidelines that are appropriate to specific areas of academic inquiry.” These documents are intended to provide guidance to the candidate, to the Personnel Committee, and to letter writers both internal and external.

These disciplinary scholarship guidelines were a major topic of discussion during my on-campus interview in fall 2014, and I left campus with several examples given to me by Sharon Alker, who was responsible for creating the policy as Personnel Committee Chair.

I was happy to learn about Whitman’s policy requiring these documents. At Grinnell, my colleagues drafted a similar document to provide evidence supporting the primacy of conference proceedings in CS (a quirk of how the field evolved, and quite different from other fields). The document was shared with the Dean of the College but, to the best of my knowledge, never formally ratified, perhaps because no other department has a document like it. Therefore, I was pleased to learn that at Whitman there is a formal structure for departmental input on criteria for the evaluation of disciplinary scholarship. That formal structure means the guidelines I wrote were approved by Personnel Committee¹ and are now posted on the Provost’s web site.

---

The process of preparing the Computer Science Scholarship Guidelines took several months. I started in December 2015 after on-campus interviews were over and I had sent the search committee’s recommendations to the Committee of Division Chairs. It was a welcome distraction from waiting.

I looked to Grinnell’s CS scholarship guidelines for guidance, as I had been part of their preparation and I generally agreed with their principles. However, I faced a strict 3-page limit, while Grinnell’s document is 21 pages long ², excluding references. After my first experience as an external reviewer, I can definitely appreciate the 3-page limit.

I also looked at several other departments’ guidelines to better understand the structure of such documents, the language used, and the relationship to Faculty Code.

I shared my first draft with Lydia McDermott, the director of Whitman’s Writing Center and also a tenure-track faculty member. She generously shared her feedback, which resulted in a complete reorganization of the draft. The first draft had subheadings that matched the types of professional activities listed in Faculty Code; the second draft emerged in its current organization. Lydia and I also commiserated over the ambiguous standing of disciplinary education research in current Faculty Code.

I also shared the first draft with my faculty mentor, who had relatively minor feedback by comparison. However, she also pointed out that my statement on disciplinary education research could be at odds with faculty code: “As we value excellence in teaching, we recognize CS education research as a primary area of scholarship.”

By the time this major revision was complete, John Stratton had signed a contract with Whitman. I shared the second draft with him just after New Year’s Day 2016. I asked him, “Are you satisfied with how your scholarship would be evaluated according to these guidelines? If not, what are the problems?” The problem John pointed out was in my language around workshops. In my subfield, HCI, “workshops” are typically informal meetings or working groups with review processes that are much less rigorous and competitive than those of conference paper tracks. But in John’s subfield, systems, “workshops” are often full-blown conferences with refereed proceedings. The funny thing is that I already knew this from my first three years of grad school, when I worked in networks and ubiquitous computing, but I had completely forgotten about it. I wrote a new paragraph to address this variability.

John then raised two other concerns. Like my faculty mentor, he wondered how the statement on CS education research would go over. I was stubborn and left it in. He also pointed out, based on his own experience, that software artifacts can be published such that it is possible to collect data about their usage, which is an indicator of their impact. This point I was able to address through minor revision. However, I also had to make other revisions to keep the document under the 3-page limit.

I asked the CS Steering Committee for feedback and got information about the approval process rather than requests for changes to the document itself, which I took as a good sign.

Next I shared the document unofficially with Lynn Sharp, then chair of Personnel Committee. She had substantive feedback:

• She told me she was “uneasy” with frequent footnotes to Grinnell’s CS scholarship guidelines. In looking at the document again, I agreed: Citing Grinnell’s document was mostly laziness on my part. I was able to chase most information back to original sources and cite those instead. However, my colleagues at Grinnell did some original research on the impact of journal articles versus conference papers (see Appendix A), which I still wanted to cite, so I explained the reason for that reference.
• Fortunately, Lynn also told me the 3-page limit was exclusive of references, which gave me some breathing room for both the text and the reference list. (In fact, I do not think many other departments cite sources in their scholarship guidelines.)
• Based on her Personnel Committee experience, she pointed out a number of points that were unclear. I clarified them. Interestingly, some of the unclear language was lifted from other departments’ published scholarship guidelines. I understand that all these documents were approved over a short period of time after the policy requiring them was instituted, and I wonder if they will receive similar scrutiny when they are reviewed in the future.
• Lynn also raised concerns about the sentence on CS education research, similar to those raised before. I finally removed it.

By this point—the end of March—we had hired Andy Exley. He reviewed the draft and had nothing further to add.

I formally submitted the document for review by Personnel Committee on April 8. Lynn relayed some requests for minor revisions from the Committee:

• Given a statement about the variety of customs regarding author order in various subfields of CS ³, the Committee asked for consistent language throughout the guidelines directing candidates to clarify their contributions to published works.
• The Committee insisted on a clarification that grant proposals, even successful proposals to highly competitive programs, are not equivalent to peer-reviewed publications. I agreed this clarification was necessary to comply with Faculty Code.

I received notification that the CS scholarship guidelines were approved on May 3, six months after I started writing them. I concluded in my final email to Lynn: “I’m glad to put this to rest. It’s been a scholarly effort in itself!”

---

The final guidelines can be found at on the Whitman College web site at https://www.whitman.edu/Documents/Computer%20Science%20Scholarship%20Guidelines%20Apr2016.pdf

¹ Re-reading what I just wrote, I hope it’s not a problem that Faculty code says the Committee of Division Chairs should approve the guidelines and Personnel Committee actually did.

² My senior colleagues at Grinnell are not known for their concision.

³ The great thing about standards is that there are so many to choose from!