Claude Code for Research Papers [R]

[u/NeatFox5866]

Third-year PhD student, NLP / interpretability. I want a reality check from people doing similar work.

I started using Claude Code for the boring parts: argparse boilerplate, plotting, config wrangling. Over the last few months the scope has crept. It now writes most of my experiment scaffolding, refactors my dataloaders, does first-pass debugging on training runs, and drafts the analysis scripts. I mostly read diffs and say yes.

The output is fine. My throughput is up. The thing bothering me is that I no longer hold my own codebase in my head. When a result looks off, I used to have an instinct about which line was lying to me. Now I go hunting like it’s someone else’s repo. I catch bugs later than I used to, and I catch them by reasoning about the numbers rather than by knowing the code.

I don’t think the tool is the problem. I think I delegated a layer that was doing more for my understanding than I gave it credit for.

Questions for people further along or in the same spot:

1. Roughly what fraction of your research code do you write yourself now?

2. Is there anything you deliberately refuse to hand off? (For me I think the eval harness and anything defining a metric should stay mine, but I keep breaking my own rule.)

3. Does anyone have a workflow that keeps the speedup without the detachment? Reading the diff line by line is not cutting it.

Not looking for a “tools are just tools” answer. I’m asking about the specific feeling of not owning your own experiments anymore.

[u/Specialist-Manager67]

I noticed this during my research internship as well. I feel like I wasted 2 months due to this. I found a really novel idea but was addicted to auto-research and just told Claude Code to implement it, and when it didn't work I didn't really have an idea of how to make it work or what to tell my professor.

Of course I tried to reason with numbers, high level intuition etc but it didn't work. The codebase was stranger to me too ofcourse.

Since then, I have set it to manual mode and manually read each and every thing like a bottom up approach. However, that gets very slow sometimes so I still set it on auto for small tasks. Like when I need to brute force search through something.

But yeah even I don't know what to do about these tools like the best way to use them or not to use them.

[u/needlzor → u/Specialist-Manager67]

I suspect this is what is happening to a few of my PhD students. If you don't mind me asking, has your professor mentioned anything regarding this, and if not what would have been the best and less accusatory way to broach the subject?

[u/Specialist-Manager67 → u/needlzor]

He never mentioned anything, but I feel like it becomes painfully obvious. When you are having a research discussion and you can't explain things in depth when asked, need to say things like "yeah from what I remember/think thats what it should be i'll check again", or pull out those claude written latex reports. For example, the student might know on a high level that they told the AI to "train the model to beat the baselines", but they might not know anything about what data it was trained exactly, how it was trained on etc when asked. They abstract it to AI, but the issue with that is ofcourse that your professor has no way of helping you if you yourself don't know what you are doing.

Best way to broach the subject, depends on your student. I would say just tell them directly that you prefer quality over quantity of work, that they shouldn't be pressured to use these tools to increase their output ( because I think thats why most students are doing it ). Make them understand that this is making them lose their ability to think and a PhD probably isnt just about being a paper mill.

I do feel like that having 1 paper accepted that you truly understand and are confident in will be worth more than ten AI auto-researched papers in the future.

[u/needlzor → u/Specialist-Manager67]

Thanks, I already try to make them feel less under pressure but I think a lot of it comes from seeing and talking with other PhD students from much harsher advisors. I need to sit down and talk this out with them.

[u/Specialist-Manager67 → u/needlzor]

Yep, having a proper talk about this with them would probably be the best, and depending on their personality they might appreciate having this pointed out