r/MachineLearning • u/NeatFox5866 • 2d ago
Research Claude Code for Research Papers [R]
Third-year PhD student, NLP / interpretability. I want a reality check from people doing similar work.
I started using Claude Code for the boring parts: argparse boilerplate, plotting, config wrangling. Over the last few months the scope has crept. It now writes most of my experiment scaffolding, refactors my dataloaders, does first-pass debugging on training runs, and drafts the analysis scripts. I mostly read diffs and say yes.
The output is fine. My throughput is up. The thing bothering me is that I no longer hold my own codebase in my head. When a result looks off, I used to have an instinct about which line was lying to me. Now I go hunting like it’s someone else’s repo. I catch bugs later than I used to, and I catch them by reasoning about the numbers rather than by knowing the code.
I don’t think the tool is the problem. I think I delegated a layer that was doing more for my understanding than I gave it credit for.
Questions for people further along or in the same spot:
Roughly what fraction of your research code do you write yourself now?
Is there anything you deliberately refuse to hand off? (For me I think the eval harness and anything defining a metric should stay mine, but I keep breaking my own rule.)
Does anyone have a workflow that keeps the speedup without the detachment? Reading the diff line by line is not cutting it.
Not looking for a “tools are just tools” answer. I’m asking about the specific feeling of not owning your own experiments anymore.
23
u/MayeeOkamura17 1d ago
For Point 3, you are saying how can you keep the understanding / intuitive mental picture of the code, without the slow process of ingesting the code into your head? You can't outsource understanding because your brain would be the ultimate bottleneck to any process / workflow. I still think reading diff line by line is as good as it can get
11
u/allenasm 1d ago
I don't use it for research even to propose things because it can spread the table with too many things (many of them worthless if its research related because you know... its research). So while I'll use it to execute tests and such, I keep it on a short leash for trying to help me work through things. I wasted a few weeks a while back with it leading me down a rabbit hole that if I had just thought it through beforehand I might not have dug in.
43
u/milesper 2d ago
Fifth year PhD and current research intern— I use it extensively to debug code and suggest ideas, but the code it actually writes is a very confined scope—data analysis and visualizations. For boilerplate like configs, I have a few files I reuse across projects that I can quickly modify for the new project.
20
u/milesper 2d ago
My feeling is that while agents can certainly write the code fine in most cases, my goal as a researcher is to understand the problem and method as well as possible, and one great way to do that is to actually implement everything. And frankly most ML experiments are pretty easy to code up nowadays so it’s not really a big productivity loss.
39
u/PossiblePossible2571 2d ago edited 1d ago
I also work in Interp / NLP.
If you are using some of the latest frontier models (e.g. GPT 5.6 Sol or Opus 5), I think you should rarely need to worry about wrong implementations. If you aren't sure just let it double-triple check.
I don't write any code myself now because it's not useful. Unless you had an IOI medal chances are it may make less mistakes than you do.
For me at least, as long as I know what experiments are being implemented in what way and the results are accurate, I do feel like I have ownership. In the same spirit, I would not let it control the research direction and experiments and analysis, both because of ownership but also because they fail at doing these very badly.
6
u/clonea85m09 1d ago
I KNOW implementation is fucked a lot of times, not because it is not good (I have Claude science with Opus5 now) but because it tires to I don't know how to say it, but impress the user probably? Like inventing whole analysis with extremely positive results and I am SURE that Clause science has guardrails against this. Or like developing a synthetic dataset to test A method, instead of using one of the bolerlate easy ones (that I linked) it decided to build one from scratch in a way where my method would work and the other wouldn't. I have done three research projects like this but every time there is something sketchy going on. The last one had some wrong assumption in it that we t on undiscovered for days untill I reviewed every number. If I hadn't taken on these extra project BECAUSE I counted on LLM research assistant I woul have ditched it and gone back to build configs and write boilerplates.
11
2d ago
[removed] — view removed comment
3
u/PossiblePossible2571 1d ago edited 1d ago
History shows you are an LLM bot. What's wrong with people agreeing over your point about LLM coding while not realizing you are an LLM.
4
u/9I54492AB6F9I 1d ago
For me at least, as long as I know what experiments are being implemented in what way and the results are accurate, I do feel like I have ownership. In the same spirit, I would not let it control the research direction and experiments and analysis, both because of ownership but also because they fail at doing these very badly.
For me, controlling research direction, experiments and analysis has become harder when using opus 5. It drove me insane with its verbosity. I figured claude code is built for software engineering work, not AIML/data science work. I started messing around with the system prompts, etc but didn't really see any improvements. I just shifted to using opus 4.7/4.8. Do you think a slightly different harness more suited to AIML research is something that would solve this issue?? Or do you think this is a model behaviour issue?? (maybe a model that has been RL'ed to listen to user feedback and using less complicated language than going off the rails with weird, complicated suggestions would help researchers)
2
9
u/dsmirza 1d ago edited 1d ago
- ~100%. I see inline suggestions and many times use them. If you don't count suggestions, then it's almost 100%.
- Almost everything. Except that I use it to debug and find errors, digest the entire repo to find flaws etc.
- I don't. My code is my little creation. Its beauty and its flaws are due to me.
I am not saying all this in an arrogant manner. I sometimes think maybe I'm behind if I don't do coding agents. Maybe I'll switch soon. "Tools are tools" logic. So far it's like writing maths or writing English. I want to write my own maths and my own English. Sometimes it could be an exact copy of the AI agent, but I have to give it just as much thought as I would my own writing. E.g. maybe I want to use a certain symbol in a maths equation, I will use what I want not what AI generated. So far I refuse to budge on this.
7
u/nonotan 1d ago
I'm a dev first, with ML more of a side thing than my main focus, so I'll offer my perspective from that POV.
You are right. It is no longer your code. It is literally no different than if you hired some guy from India (or wherever) to write it for you for a few dollars. And the cost is not just that you have no intuition about how it works, that it will be a pain to debug any issues, etc... but perhaps even more importantly, that you aren't learning a single thing.
Now, perhaps you just aren't interested in getting better at coding in the first place. It's not my place to make a subjective value call there. But I'll tell you one thing for certain: if you do it yourself, over the years you will not just get faster, but also write higher quality code that is easier to maintain, less likely to have bugs in the first place, etc. Personally, at my skill level, if you add up all the costs, from the point when there is no code, to me delivering a stable piece of software that I feel confident I can provide support for if needed, I feel quite confident I can do it faster and end up with a better product, if I do it myself.
If you delegate it (especially if you delegate the tricky bits that actually require thinking through, boilerplate is less relevant), well... let's just say you better hope you don't run out of tokens at an inopportune time.
A good parallel is natural language, as in learning a foreign language (you could argue programming is just translating the concepts in your head into an artificial language that the computer understands). Yes, running something you wrote in your native tongue through an LLM is going to blow anything a beginner or even intermediate learner can write out of the water, and do it much faster, too.
But picture two possible futures: in one, you did the hard work and you're now fluent. You can write in this foreign language just as fast and effortlessly as in your own tongue -- going through an LLM would just slow you down. Even if the output quality was maybe marginally higher, it wouldn't be worth it. Not just that, but you can also use the skills you acquired in situations where relying on an LLM would be impractical (imagine you're in a party and trying to chat with somebody through ChatGPT...)
In the other future, you relied on an LLM all the way through, and you have learned nothing. You're just as dependent on a tool as you were on day 1 -- probably more, as breaking the habit now would be harder than not forming it in the first place.
Again, it's not my place to judge if the additional effort it would take to write the code is worth the various upsides. I know for many researchers, coding is something they only touch because they have no choice, and they'll be more than happy to make whatever sacrifices necessary to make it "somebody else's problem". I'm just saying that yes, those upsides do indeed exist. It's up to you what to do with that info.
8
u/Melodic_Reality_646 1d ago edited 1d ago
TLDR: Plan Mode + Architecture Diagrams + Persistent State
The best immediate, cheap, off-the-shelf fix is to use Plan Mode and actually read the plan before letting Claude execute it. That alone helps keep the architecture in your head.
Also, have Claude maintain documentation and architecture diagrams as the codebase changes.
Finally, don’t rely on either your memory or Claude’s for vision, key decisions, critical issues, or the roadmap. Externalize that context into structured, persistent state that gets updated with the code, reviewed regularly, and archives stale information for retrieval when needed.
8
u/Gusonn 2d ago
My experiences with claude code has been very similar to yours
- I write veeery little currently. I usually have clear ideas on what I want to implement, and explaining it in detail to the agent works out for me very often. Lately, most manual changes that I have done have only been config changes between experiments...
- Not sure if you meant it like this, but like I said in the previous point, I don't really hand off too much thinking to it. All main ideas and, I would say, almost all implementation details, come directly from me, and I just want it to implement it. Like you, I read diffs and make sure that it's generating what I would expect. As for what I am still coding myself, well nothing specifically, I try to explain everything I want, and if it fails somewhere, I would clean up
- Can't help you here, since I only read diffs as well. Maybe just one thing: I specifically guide it to a certain code structure, one that I have been using for most of my projects (I am also PhD, and my projects have a lot of common points since they are tackling similar issues), and in that way, it is easier to adapt to whatever it generates, since you know the structure and know where certain things should be. Not a solution for sure, but it helps slightly for me
11
u/JimmyTheCrossEyedDog 1d ago
This post itself clearly being written by AI is all too ironic.
5
u/MayeeOkamura17 1d ago
Can you point to me what parts of this post makes you think so? I'm curious because OP's sounds similar to how I would write too
7
u/whyVelociraptor 1d ago
The whole thing really lol, but one dead giveaway is the end. All the AI posts have this little couple line wrap up at the end that’s something like “Looking for people who have actual experience with…”. or similar. Bonus points for “Not looking for X, looking for Y” bit of this.
5
1d ago
[deleted]
2
u/JimmyTheCrossEyedDog 1d ago
It's different, though. I too use phrases that sound like AI, and I refuse to give up the em-dash. But AI posts just don't sound quite like a similarly well-written human post. They're consistently sterile. They live in a particular dimension of how text can sound and they almost never stray from that manifold. The person who responded above pointed out some of the particular phrases, and that's part of it, but like they said, it's the totality of the post that sets off the AI red flags. I'm sure some amateur AI detectives might get tricked by your writing style, but I feel pretty confident that I and many others wouldn't be (most of the time, at least).
1
u/whyVelociraptor 1d ago
I wouldn’t worry about it. My guess is that even though you may write similarly, it will be different enough from the AI voice that folks won’t think that. The LLMs do generate text that is pretty well-structured and free of errors, so it is not always bad to be similar to that.
1
u/alexbaas3 1d ago
Feel like people also write more like AI even if they didn’t mean to because of how much we use the tools
8
u/NeatFox5866 1d ago
I literally spent 30 minutes on this… also I am not a native English speaker
2
u/JohnathantheCat 22h ago
I find it endlessly frustrating that afyer 35 years of writing with hyphens, semi colans and using large complex sentences I have to start not using them, because these are things AI uses consistantly. It turns out I am just, you know, literate. (Cant spell, but I xan read.)
5
u/Krugger_Correctly 1d ago
Honestly losing ownership of the code is not the hell we fear it is. Naturally, it is a downside to be weighted but at the end of the day... if Claude can't solve it and you need to interfere, the onboarding cost is the same as joining a late-stage open source project. Not at all impossible to map out what is where and contribute, just annoying. The time you save using Claude still (imo) makes up for any on-boarding you might need later.
I'd only really worry about it if this is your magnum opus. A topic or idea you intend to build your career around and spend your life expanding. Mandatory contributions, short papers, etc, I wouldnt hesitate.
2
u/k3nal 1d ago
Start reading your code again!! And it would be probably best if you start writing your code yourself as well again.. depends on you of course. For example: me personally I am able to write much better code than the AI. So for me I have no choice: I need to write my code myself. As I am „lucky enough“ to still be better in that department than the AI is, and probably will ever be.. but if your code quality is not so great, maybe start using the AI differently? Maybe let it do even more of your work and deploy agentic workflows and more automation into your workflow to get yourself even further distanced from the code itself? As it can debug it for you as well, depending on you prompt quality of course. Or start using more than in AI: start using OpenAI ChatGPT Codex as well an use them back-to-back, so that they catch each other’s mistakes and also enhance each other’s ideas and even introduce new ones into your code. I have good results with that using smaller models on my local machine, on a PNY NVIDIA RTX A6000 that I like to use for these type of things, also have good results on bigger models on our university cluster using H200s there with much fun. So I think that is the winners approach of doing things if you like to cooperate with others.
5
u/marr75 1d ago edited 1d ago
I'm an experienced software engineer and lead our software and data teams. We're a lot more productive now but have all faced about what you're talking about.
I highly recommend reading content from the HumanLayer consulting team, start with Getting AI to Work in Complex Codebases. Fundamentally, you should systematically manage context between defining an issue/task, researching it, designing it, and then planning the implementation. You'll actually end up with more documentation about your project in a more digestible format. During the implementation, I recommend asking the agent to break up the work into commit-sized phases and reviewing each commit personally (and committing yourself after reading it).
You still won't have had enough time to "stew" in the code so you can memorize it. But, you'll have generated and read design documentation, read and signed off on every commit, and (if you're smart) insisted on maintenance and quality helpers like linting, tests, design docs, diagrams, etc.
4
u/madkimchi 1d ago
Avoid anything Anthropic like the plague, especially opus.
Their models output so much AI slop is honestly going to get your papers blacklisted
1
u/Specialist-Manager67 1d ago
on a serious note what do u suggest using? chatgpt limits arent that great since claude code is boostoed rn and also deepseek doesnt seem to be better than claude
0
0
2
2
u/howtorewriteaname PhD 1d ago
in my research team we don't write ANY part of the code anymore. and the scientific throughput and quality of the research is high (we do efficiency LLM research). yes of course you understand less about the code and run on a higher level of abstracion, but that's the whole point? we are all faster, even factoring in that at times, unadverted bugs happen
2
u/mofoss 1d ago
Dont see whats the issue, the whole world is using it. Whether you use it to produce slop or not is something manual verification requires, if it produces questionable results as well.
AUDIT. AUDIT. AUDIT.
Create new chats, wipe memories and always ask it to audit/verify whatever older results were made. I've had research papers take 5-6 months to write/produce even with extensive AI usage.
3
u/Specialist-Manager67 1d ago
Auditing is fine but I feel like its still hard to have a deep understanding of something you completely auto-researched.
Like if you were reading every diff and approving things manually you would have a bottom-up view of how things were built.
But if you first wait for it to get good results, audit and then ask you to explain things deeply only if it works, I feel like I'm never truly able to understand things in that top-down approach as I would in the bottom-up approach.
1
u/d0esthismatter 1d ago
Slightly off-topic, but how are you reading PDFs in Claude Code? Do you convert them to MD first? - Whats your workflow? or just ingest the entire paper including the supplementary stuff ~ because that consumes a lot of tokens, and then after a few chats, you're done for the day.
1
u/BackgammonEspresso 1d ago
I do commercial research for AI tutoring in board games, and find that I get the best results by actually using it in the browser. Otherwise my projects just get eaten and I end up with thousands of lines of code, dozens of files over too many directories... I tend to end up stuck in the mud.
1
u/ComprehensiveTop3297 1d ago
I mostly use it for generating the implementation details and testing the constraints for my architecture. My workflow now looks something like this.
- I come up with the idea myself. I am a PhD student, so it is usually easy to come up with the idea as I've read tons of research on the specific topic I am working on, and I kind of have a sense of a good direction to try.
- I find suitable datasets and baselines to test my idea against (baselines with open code are the priority; they also come with the eval harness usually)
- I write the idea and the motivation for it, and use Fable 5 to generate the code and the test (manual approval; I usually catch 2-3 things that look off per day with this, and when I tell it to fix it, Fable usually does a good job of doing so)
- I check the test cases manually and usually use another AI (like GPT 5.6-Sol to stress test the code that Fable generated; it usually catches some things that are off)
- I run the baselines on the data, check the numbers, and I try to reproduce their results. If I do, then I run my models on this dataset as well.
- After the run finishes, I check the graphs and too-good-to-be-true results. If I catch amazing results, I actually become skeptical of the implementation, and I look at it myself deeply. I do not trust the results until I've checked the code and confirmed that it does what it's supposed to do without any weird optimization tricks. This usually also surfaces when I try a different dataset, or an ablation. So, I usually have many ablations, and more than one dataset that I test my models in.
- I repeat until I am satisfied with the outcome of the research (not talking about beating the baselines here, but understanding the limitations of the idea, and what that gives me in terms of the research)
So, I'd say I mostly review the code and catch off-looking things rather than writing my own code. My input mostly comes with the baselines, datasets, eval harness, and the idea of course. To be honest, I find Fable 5 satisfactory, and I think it is a clear step forward. I have approximately 10 years of programming experience, so I think I've developed some taste for what good code should look like. Though I can imagine it is different for people who have not programmed before, and learnt it with AI only.
1
u/AnOnlineHandle 1d ago
I've migrated some personal projects from messy unplanned versions to more concrete rewrites using Claude Code over the last two weeks, as my first time using it. Then have been making small incremental upgrades as I reason that they're needed.
Something which I felt has helped is it was my code to start with in the initial messy version, and most of the logic remains the same. I also had Claude write a file-by-file brief summary and description of any non-obvious methods, which I've read through multiple times making sure I understand and touching up explanations which I think could be better, which both helps subsequent models working on the project and helps me keep a mental map of the model in my head.
It's not perfect, I've lost tracks of bits of it, but overall I know how the code works because it's either a rewrite of what I did or is parts I've requested after thinking about how to solve specific problems.
1
u/eldrugo85 1d ago
Not a researcher, I run agents in production. Same creep here, and it bit me: I caught myself writing counts into a report that no command had ever produced. Fix was dumb, a pre-commit hook that rejects numbers and ids in artifacts unless they came from a command run in that session. Reading diffs never caught that, only the provenance check did
1
u/Envoy-Insc 1d ago
Why does this post have that AI written cadence
2
u/Equidissection 1d ago
OP has said English is his second language, common for formal sounding text from ESL speakers to sound like ai
1
1
u/AllNurtural 13h ago
Sounds like you've accrued some "debt" in terms of understanding. Reading diffs may not be enough. Also step through things with a debugger. Rewrite parts that don't make sense. Take ownership of the code. Let yourself learn from the style and methods the agent implements.
This can all still be a speedup and it can be a mechanism for you to continue learning. But where you used to spend an hour coding you now need to spend a sizable fraction of that time reading + comprehending + debugging. You may need to spend extra time now catching up to repay that debt.
And BTW getting good at reading others' code is absolutely a necessary skill after you graduate. This is good practice.
1
u/pastor_pilao 2d ago
0% but I take care of painstakingly describe every small detail of what I have in mind and make sure I understand at least what is the expected output of every function created.
Reading and writing papers.
Hard to describe in words but you have to tell claude to generate the code in a way that it makes easy for you to understand what is going on and check the diffs. As more experience as a software engineer you have as easier it is for you to do that
0
u/ScholarImaginary8725 1d ago
1) it's close to 80% for me
2) anything that I cannot confirm, i.e has no output
3) I don't bother honestly.
My reasoning is that I'm not a programmer, honestly programming is just a tool for me, I do computational science research and so while coding is valuable as a skill to have, it's more valuable to have the best research you can have. I could probably eventually do everything that one done with AI myself, but it would take 2-3x and it would be lower quality. I do think my coding ability has nosedived and it's something I'll look to practice but I'm not too worried overall.
0
u/22cristobal22 1d ago
Postdoc in NLP; been in the field since 2019. I can tell you used an LLM to generate this question too. People can and do notice these things, and it doesn’t reflect positively imo. When I can tell something I’m reading (such as a paper I reviewed) sounds exactly like copy-pasted or unedited LLM output it makes me wonder to what degree the author stands behind and rigorously evaluated the statements in the text. For example, your post says “the output is fine”, but when I’m wondering if that sentence is also LLM-generated, it calls into question whether you, the human taking credit for the piece of work, would actually say that yourself.
TL;DR: don’t outsource your writing (light editing can be okay) because it undermines your credibility
129
u/Specialist-Manager67 2d ago
I noticed this during my research internship as well. I feel like I wasted 2 months due to this. I found a really novel idea but was addicted to auto-research and just told Claude Code to implement it, and when it didn't work I didn't really have an idea of how to make it work or what to tell my professor.
Of course I tried to reason with numbers, high level intuition etc but it didn't work. The codebase was stranger to me too ofcourse.
Since then, I have set it to manual mode and manually read each and every thing like a bottom up approach. However, that gets very slow sometimes so I still set it on auto for small tasks. Like when I need to brute force search through something.
But yeah even I don't know what to do about these tools like the best way to use them or not to use them.