Research Taste Exercises

One of the most important aspects of growing as a researcher is developing research taste -- roughly, the ability to chose good problems to work on. But it can be hard to explicitly work on developing taste, so I wanted to share some concrete, actionable exercises.

Before we start with exercises, it's worth think about why it's hard to develop research taste. I think the fundamental issue is that actually testing whether a research idea you come up with is good is very expensive. Often it takes months, so you only really get a few pieces of feedback on your taste every year. Many of the following exercises are really strategies for getting (proxy) feedback on more research ideas faster. The feedback you get is generally lower quality than actually executing the idea yourself, but these aim to get you orders of magnitude more of this noisy feedback.

Take all these exercises and suggestions with a grain of salt. I think I have better taste than I used to and doing some things like this helped. I also think these exercises have also helped the occasional person I mentored or managed. But I'm hardly an expert. If you have better ways to build resarch taste, please share them in the comments or on twiter!

One other point: I don't recommend forcing yourself to work on something you don't find exciting, even if you intellectually think it's more important. You'll be a lot less effective. Ideally, you want to get to a point where the problems you think are most important also feel most exciting.

Exercises

Exercise 1: Write down a list of research ideas. Have a mentor you respect rate each idea 1-10. Discuss ideas where you disagree with them after reflection.

It often takes several months of work to truly test an idea. Asking a mentor is a cheap proxy.

Exercise 2: Pay attention when other people try ideas you’ve had. How did the results compare with your expectations?

If you’re investing in brainstorming, you’ll have tens or hundreds of research ideas for each one you try. It can be emotionally uncomfortable when someone publishes an idea you had, but it’s an opportunity to get precious feedback for free.

Exercise 3: Interview researchers around you on their taste. Why do they work on the problems they do? How do they pick problems? What’s their “big picture” of research?

Bonus: If you have a great interview, you might consider writing it up with their permission.

(PhD students often have short meetings with researchers visiting their lab. One pattern is pitching their project to the visitor, which may not be very productive if they don't have similar interests. I suspect they’d get more out of talking about meta-research.)

Exercise 4: Read books about the history of science. Reflect on why some researchers focused on important directions their contemporaries ignored.

(Kuhn’s “The Structure of Scientific Revolutions” is one of my favorite books, and you can get an audio book!)

Exercise 5: Critically consider your research taste, and the community taste around you. Your taste is likely very influenced by your research cluster (your collaborators, advisor, etc).

In what ways has your own research taste or your community's taste been wrong over the last few years?

Are there adjacent research “schools” with significantly different research taste? If so, try to articulate the strongest version of their view, and why you agree or disagree.

In what ways to you disagree with the research taste of your own community? What are ways in which you think your community might be wrong even if you aren't confident in that view?

What are the underlying goals of your research taste? This could simply be intrinsically wanting to understand, or just finding research fun. But there might also be ways you want to make the world a better place. Are the problems you are working on (or think are intersting) algined with those goals?

Failure Modes

I also wanted to go through some research taste failure modes:

Failure Mode 1: Getting overly attached to one research direction / falling into sunk costs.

A lot of researchers start working on one problem (often more due to circumstance than contemplated decision) and then find it hard to move on.

Potential Antidote: Set aside a week or two to step back and create a list of the other research directions you think would be most promising to work on (look at new research, talk to others, etc). Then ask what you’d want to work on if you were starting from scratch. Repeat every 1-2 years as needed.

Failure mode 2: Lack of research knowledge / intimacy.

Theoretical knowledge is table stakes for research taste. You can’t have research taste in a vacuum.

Sometimes people fall into a trap of trying to pick the perfect problem or have a brilliant insight before getting their hands dirty. You can’t have research taste in a vacuum. You need theoretical knowledge and research intimacy.

Potential Antidote: Help someone else with their project in a space you’re interested in. You’ll learn a lot and get your hands dirty without becoming overly committed to working on something long term. Ideally, by not leading the project, this should also be lower stress. Alternatively, do your own short-term projects to get your hands dirty.

Failure mode 3: Environment not aligned with your interests.

I sometimes talk to PhD students who have different interests than their advisor, and are trying to contort their research interests into something their advisor will find palatable. This generally leads to ideas optimized for compromise rather than quality. This leads to suboptimal resarch, and starves you of an opportunity to build taste.

This can happen even when those around you nominally support you working on whatever you’re interested in. It can be challenging to pursue a direction when people around you are unenthusiastic.

Potential Antidote: Either (1) move to an environment which is aligned with your interests, or (2) temporarily buy into the interests of your group and try to exercise good taste within those interests. I’m generally pro moving to another environment once you’ve reflected a lot on what you’re interested in and have something you feel very actively excited about.

Suggestions from other people

In the twitter discussion (and especially, a thread by Andy Matuschak), a number of people suggested other ideas. Note all of these are quite "exercises for building taste" (many are more "strategies for exercising good taste"), but I think they're very interesting:

• Rachel Prudden suggests
  The heuristic of imagining that another group has published the paper you have in mind. Are you excited to read it?
  Chris comment: I think this is a pretty good heuristic.
• Andy Matuschak suggests
  ask the anti-Hamming question about your ideas. What are the most interesting (not important!) problems in your field? Why aren't you working on them?
  Chris comment: This is a great question. See also Hamming's famous "You and Your Research" talk.
• Andy also shares a failure mode:
  running to make an idea happen as soon as it seems tractable. It's exciting to have a viable approach—makes me wanna go do it! But if I sit with an idea longer I can often evolve a deeper variant and run with that instead.
• Michael Nielsen's Principles of Effective Resarch (via Shrey Jain) notes the value of paying attention to messes:
  When you identify such a mess, the natural inclination of many people is to shy away, to find something that is easier to understand. But a field that is a mess is really an opportunity. Chances are good that there are deep unifying and simplifying concepts still waiting to be understood and developed by someone - perhaps you.
• Arthur Allshire
  What are the bottlenecking problems in your field? Eg. in a lot of reinforcement learning research, a lot focus on slight tweaks to algorithms, but the biggest improvements have come from improving simulation for RL.
• Preetum Nakkiran
  I've found that writing "project proposals" in the short term, and "research statements/plans" in the long term, are good exercises to think about motivations & goals (at diff timescales). By "project proposal", I mean: Once you decide to commit the next >1 month of time to something, write up a few pages explaining: why the question is interesting, why the answer will be valuable, and what you expect to find (w/ initial evidence). And then get feedback.
  ... Also seconding the advice to ask everyone you meet about their "big picture" research goals. This is almost never written in papers, but is much more interesting/important.
• Christian Szegedy
  In hindsight though, most of the errors are made in my research was when I listened to the criticisms of my respected mentors and most of the cool stuff I did were the ones those everybody called initially useless, uninteresting or "good luck!"... Generally, ideas in research need to be either ahead of the others or should defy conventional wisdom. Being in the latter camp requires less skill, but more courage.
  Chris comment: I think there's a lot of truth to this, but beware survivorship bias and remember that different people may have different levels of scientific maturity / taste at different points in their careers. I suspect that many PhD students would find charting their own research agenda from the start very challenging, while others would thrive.
• Sergey Gorbunov
  My only advice on research taste to grad students or junior researchers is *not* to listen to your advisors/mentors/established researchers... Grad school is the only place when you get to try and fail as many times as needed. Grad students should come up with outside the box problems/solutions. (Established researches have their own agendas/way of thinking😂). The advice can similarly be used to overturn itself. 🧐
• Boaz Barak
  I’m fond of “20/80 projects” where you do 80% of work to get 20% of goals:
  Rather than asking “what’s coolest result I can get with smallest effort?” ask “can I find problem that distills the main obstacle even if end result doesn’t sound cool?

Acknowledgments

I'm grateful to the countless people who've talked to me about research philosophy over the years.

Thanks to Nick Cammarata for discussion of these exercises.

Thanks to Ryan Saxe and Abinav for correcting typos.