Lessons Learned Writing the CAREER
So, I won the NSF CAREER award. To say I’m excited about this would be an understatement — my first Ph.D student has support locked in, I get to actually do the work I’ve been building towards for years now, and we’re going to have a much better understanding of how recommender systems (mis)behave in response to their individual and social human contexts.
One of the things I found useful while planning and writing was hearing a variety of ‘path-to-the-CAREER’ stories and trying to take from them the things that would work for me. So here’s mine, for what it is worth. There are many paths to success; the opening line of Anna Karenina does not apply to grantwriting. My road is neither necessary nor sufficient.
This post is adapted and heavily expanded from notes I wrote in preparation for the successful applicant panel at Boise State’s CAREER prep workshop this spring. But don’t listen to me Academic advice writing, panels, etc. are plagued by the problem of survivorship bias. When we put together a panel of successful grant applicants or people who landed good jobs post-Ph.D, we sample success. Success is great, but looking only at successful cases often obscures the factors that actually made a difference between success and failure. Keep that in mind whenever you are reading or hearing advice. That doesn’t mean observing success, or listening to people who have succeeded at something is entirely useless; it just means that, in order gain generalizable, applicable knowledge, you need to interpret the data in light of that bias. I started my faculty career in Fall of 2014. I immediately submitted my first attempt at the CRII, building on my dissertation work; this proposal was declined. In summer 2015, I submitted my first CAREER attempt; this went down in flames, as I had no idea what I was doing — I submitted because ‘submitting CAREER is what junior faculty do’. In Fall 2015, I decided to focus my energies on understanding fairness, bias, and discrimination in recommender systems. I submitted my second (unsuccessful) CRII attempt on the key idea that would later be the heart of my CAREER grant, and had a couple of my M.S. students work in earnest on projects that would produce preliminary results for future grant proposals. In Fall 2016, I moved to Boise State, attended my first FATML in New York, and began to make in-person connections to the algorithmic fairness research community. I submitted an unsuccessful proposal to the Google Faculty Award program for a project that would become Research Objective 1 of my CAREER. I also began working informally on planning out my CAREER proposal’s research scope and getting feedback on the ideas. In spring 2017, I began working specifically on the grant. I attended the NSF CISE CAREER workshop, scoped out collaborations and support across the university, and drafted my project summary for program officer feedback. In April and May I did the bulk of the writing, and delivered a complete draft of the project description in late May. I then went on conference trip & drank terrible wine1 while our college grant specialist reviewed it. I spent the first half of June revising, securing letters, and preparing the rest of the application package. With a complete application package ready, I sent it to our grant specialist and friends & colleagues for review, and took 2 weeks off to visit Belgium and the Netherlands. Didn’t touch it to clear my mind for revisions. When I returned, I spent 2–3 weeks revising, finalizing, and submitted in mid-July. This probably deserves a blog post of its own, but the purpose of startup funding at a research-intensive institution is to get your research program started while you apply for grant funding to keep it going. CAREER is just one of the biggest and most signpost-y of those grants. My top priority for my Boise State startup was laying the groundwork for the work that I would propose in CAREER. I focused this on the work itself, not on the grant — that would put a few too many eggs in one basket — but I put my student to work on projects that would advance my recommender fairness work and produce results that I could cite in CAREER and other proposals as preliminary findings. Loose ends from previous things I mostly worked on myself, with external collaborators, or put on the back burner. Other things I paid for that I found helpful: I am not a solo researcher; I do my best work in a vibrant intellectual community, where I can regularly have both serious scholarly discussions and informal banter. The support of my great colleagues at Boise State Computer Science played a major role in getting me ready to write & carrying me through the process. Boise State is particularly good at providing resources and helping prospective CAREER applicants locate them. Each spring we have a 2-hour workshop for prospective applicants that provides pointers to many campus resources (along with a panel discussion on successful proposals), and a faculty learning community focused on the educational and outreach component. Relatedly, I found it useful to connect my work directly to local and regional needs, and several people at Boise State were helpful in identifying those as well as securing collaboration commitments to address them. And finally, I documented this support in Letters of Collaboration. Strategically, I saw this as demonstrating that my university is behind me (beyond the letter of commitment from my department chair), and that I already have some of the connections necessary to successfully carry out the work you are proposing. It’s easy to say I’m going to work with libraries on improving public information literacy with regards to recommender systems and similar modern information technologies; having letters from my first library and one of our Boise State librarians with connections to libraries across the state shows that the project has a decent chance of success. My full application included 9 letters of collaboration (and a table on the last page summarizing the role of each collaboration). Part of a research proposal is demonstrating that the work is worth funding. But another part, that seems particularly important for CAREER, is demonstrating that you should carry out the work at your institution. I spent about two-thirds of a page on how the work I was proposing connected to my career and professional preparation and goals (including such details as a rural public library’s role in planting the seeds of my career, connecting this to my plan of working with libraries on information technology literacy) as well as to my department’s stated learning objectives and ongoing curricular enhancement activities. Part of my educational activities involved extending a current department effort in our undergraduate program to our graduate artificial intelligence classes. The single most useful source of wisdom I found was the NSF CISE CAREER workshop; slides from the talks are still available. It often isn’t terribly well-advertised, so contact your program officer in January or so if you want to go. These workshops are one way to break the survivorship bias bubble. The one I attended had a talk on Top-Ten Mistakes that was very valuable. Breakout sessions with specific divisions and clusters were helpful in figuring out where to target my proposal. The CAREER solicitation provides a minimum budget, but does not provide much guidance for actual expected budget; budget was a place I got in trouble in my initial attempt. At the CISE workshop, they provided a lot clearer guidance: one student + one month of summer per year, and associated research expenses. This obviously helped with the budget, but also helped me scope the work in general. I approached the research components of the proposal as if I were writing an overly-ambitious version of my first Ph.D student’s thesis proposal. Somewhere along the way, I received some advice that said that you need to be explicit about a lot in a proposal — if you leave gaps, the reviewer will fill them in with something you didn’t intend, and judge the proposal on the basis of their assumptions. The way to avoid this is to explicitly say things, even if you find them obvious. I had the particular problem of using somewhat unusual methods for my field: simulation studies. Current recommender systems research does not have a lot of simulation work, for good reason: it has a lot of weaknesses, and there are better ways to answer the research questions that a lot of research to date has asked. I tend to find the common questions less interesting. And it turns out that simulation is the only way, with current technology and data, to answer the questions I want to answer. I handled this by taking time (about ⅔ page) to lay out the space of recommender system research methods and explain how my choice of simulation studies is necessary for my questions, and that it will enable more generalizable results. It worked in so far as none of the reviewers had a problem with my methods and the program officer called them out as one of the fundamental expected contributions in the abstract. With any of my high-stakes writing, I would rather have my friends tear it apart so I can rebuild it better than to have to wait for a round of reviews to point out the flaws. It’s like a free review cycle — every objection I can identify and address (or decide to ignore) before submitting increases the likelihood of success. And I can’t find them all myself. There are several classes of feedback that I made use of (and I am immensely grateful to everyone who took time to help me): It’s difficult to overstate the value of the candidate panelist review — if you can find such a person, I highly recommend it. Such reviews are one of the services provided by some grant prep consulting firms. Bear in mind that whoever you ask to do this will have a conflict-of-interest when it comes to actually reviewing, and research ethics demands you list them either in your COI form or in Reviewers Not To Include if they don’t fit in one of the defined COI categories. Long-time collaborators aren’t necessarily the best pick if you want to maximize the likelihood of finding one of your blind spots. I wrote my way out At the end of the day, with a lot of prep and planning and logistics and feedback, you still need to write. There’s no one right way to write a winning proposal; we have a successful proposal library at Boise State, and they’re all over the map in both the low-level writing and high-level organization. Here’s some things I did: Win by Page 2. Based on advice & tips I read and discussions with colleagues with proposal reviewing experience, I figured that if I didn’t win the grant by page 2, I wouldn’t win. Details later in the proposal could sink my proposal, but they wouldn’t persuade a panelist or program officer to recommend funding. This governed a lot of my decisions in proposal structure, such as summarizing my research & educational objectives in second page forward references for the details. Write in first person. Certainly not necessary, but it’s the approach I took. We like to think about dispassionate interest in science, but a CAREER proposal is asking for financial backing for your career of research and education. Use active voice. The passive voice should be avoided. Write what I believe. This is generally a good practice, but it’s worth restating. It’s easier to be convincing when you are convinced. Write what I want. Very related to the previous point, but I took the approach of just writing what I wanted to do. I have been thinking about the next steps of my research for a while, and was planning to pursue this rough agenda regardless of this grant. I went in to this proposal at the point where I believed the work I was proposing was the most important thing for me to work on over the next few years, so I wrote it all down. Help the reviewer. The easier you make it for the reviewer to like your proposal and find ways to defend it, the more likely they are to do so. Explicitly and concisely tell them why it is a good idea. I also used a larger font than required (11.5pt Times instead of 11pt) and slightly relaxed line spacing to make the proposal easier to read. Reviewers are people subject to psychology; I gambled that what I lost in room for exposition I would recover in reducing fatigue and frustration reading the proposal. Pilot pen in pocket, I’m riding instinct and ink jets. I got feedback, I took advice, but at the end of the day, I also trusted my instincts. I figured that if I didn’t fully buy in to what I was doing, either in the work itself or in how I presented it, that would come through and make the proposal less credible. I can also be stubborn, especially when I’m convinced of something. I came to Boise State with a fresh sense of purpose, determined to make information systems good for people or lose my job trying. I deeply believe that understanding recommender systems’ social impact and consequences is the most important thing I can work on right now, and that my proposed methods will illuminate the subject. I have to believe that conviction came through in the proposal and made it more persuasive. And when I’m writing, hip-hop is my psych music. Dessa goes on heavy repeat when I’m working on a grant proposal, and Hamilton has helped a lot. The rap I like best walks a fine line, balancing concision and repetition, storytelling and summary. Its rhythms and rhymes give it flow, building materials for working out the principle that good writing sounds good. Dessa’s urgent, confident blend of lyricism and boxing fueled my ‘Win by Page 2’ approach. ‘My Shot’ keeps me inspired to seize the day and try for the next project. The prophetic assessment of society that drips from the pens of socially-conscious rappers keeps me mindful of why I do this work. I just wish I could have called John the day I got the news. F*** the plan, man Merlot, that tasted like it was poured from an old boot.↩︎Preface: Don’t Listen to Me
We just met
What do I know?
— Propaganda, ‘Don’t Listen To Me’, Excellent (2012)Context: Timeline
I’m willing to wait for it.
Startup is There to be Used
Every morning when I wake up, uh
Money on my mindGood Community
I built more than a rap career
I got my family hereGood Institutional and Collaborator Support
Connect to My Situation
Workshops
Scoping The Work
Explain Everything
Feedback
Writing
Wrote everything down as far as I could see
— Lin-Manuel Miranda, ‘Hurricane’, Hamilton (2015)Winging It
— Dessa, ‘Fighting Fish’, Parts of Speech (2013)
I’m tryna call an audible
Probable lost cause
But I got a thing for long shots
— Dessa, ‘Warsaw’, Parts of Speech (2013)