r/learnpython • u/LiteratureTypical982 • 13h ago
Client wants 40k emails from a directory. Each one requires a click. What do I do?
CLint need a list of 40,000+ members from the ACR directory. Need Name, City, State, Zip, Specialty, Email, and Phone.
The directory/search results give me some of this information, but email, phone, address, Member Since, etc. appear to be available only on the individual member profile.
So I'm trying to figure out the best way to approach this in Python.
If I literally open/request every profile, that's 43k+ profile requests, which obviously makes me concerned about rate limits, blocking, or getting the account/IP banned.
I'm considering Python requests/BeautifulSoup or Playwright, but before attempting something this large I'd like some advice.
8
u/Skiamakhos 12h ago
I think this needs more information. What is it you're trying to achieve? You want to send 40,000 emails or you want to extract the emails from a bunch of profiles? ACR is what - Azure Container Registry? Be clearer & either someone will be able to help or you might find the way forward reveals itself. Often all it takes is explaining the job clearly and methodically.
-6
u/LiteratureTypical982 12h ago
I need to extract a list of members from the American College of Radiology (ACR) directory. The client wants:
- Name
- City, State, Zip, Address, Country
- Phone
- Specialty (e.g., Diagnostic Radiologist)
- Member Since
- Profile URL
Total records: ~43,744 members
Emails and phones are NOT visible on the listing page.To get emails/phones, I must click on each profile.
The site will likely detect and block me.
any solution10
4
u/lilovia16 12h ago
Perhaps a webscraper? You can build one easily with Selenium
1
u/LiteratureTypical982 12h ago
The problem isn't building it. It's running it.43,744 clicks means 43,744 popups.
The site WILL detect this. My IP will get blocked. My account will get suspended.Any way out?
8
0
u/greatsmapdireturns 12h ago
Build the scraper so it can resume on failure or drop and use vpn so when one ip gets blocked you can just resume with a new ip..or just super rate limit so you're not hammering it and getting detected
0
u/LiteratureTypical982 12h ago
With VPN rotation, I might avoid an IP ban. But the account itself is still doing 40k clicks.
So the account ban risk is still there, even with VPN.Unless I use multiple accounts and rotate them too. But that's a whole other level of complexity.
4
u/esituism 12h ago
how do you know they will ban you for this many clicks? I'll be honest and say I can't imagine the college of radiology has hyper-intense security measures. Most saas apps don't track clicks persay, but rather just API usage, and only when their servers are getting hammered or their bills went up significantly. 40k clicks is a miniscule amount of data to be passed and would be well below any rate threshold anyone would care about.
Add a little bit of rate limiting and you will be fine. Or, contact the owner of the db - perhaps the college.
That said, it seems to me like you're getting into some shady shit and you know it.
2
u/Skiamakhos 6h ago
This, yep. I'd say be open about what you want, contact the college, see if they're OK giving you the list. Chances are though that they will be absolutely not OK with it. It sounds like a privacy nightmare. Gonna be some pissed off radiologists.
2
1
u/greatsmapdireturns 10h ago edited 10h ago
Yeah op trying rip a members list without express permission
1
u/NoForm5443 12h ago
Add sleep(30) in between? :)
Run it from different machines?
BTW, there's probably no need for that many clicks; I'd assume there's a url pattern that the click takes you to
3
u/jmooremcc 11h ago
Your time is more valuable.
Pay for access and charge the client appropriately. Trying to do this on the cheap to save the client money will actually cost you more. If you factor in your time trying to get free access, you'll find yourself working for less than minimum wage.
0
u/LiteratureTypical982 11h ago
How much should I charge?
3
1
u/jmooremcc 11h ago
In order to answer your question, you’ll need to find out how much the website will charge you to access their data. Then you can calculate how many hours it will take to get and format the data for your client and multiply that by your hourly rate.
The total charge to your client should be the access charge x your markup + your fee based on the number of hours you estimated it will take.
7
21
u/socal_nerdtastic 12h ago edited 12h ago
The first thing to do is ask the owner of the database. Most of them have some kind of api and you can pay some amount of money to access it. Scraping it will be weeks of work, so any amount of money that is less than paying you for a week is worth it. Selling member lists for marketing purposes is of course very common and they are probably very used to this request.
If you are forced to scrape it, you should try first to use something like requests, which does not use a browser interface. This will be much faster and smoother. You many need to do some sleuthing to find the internal api that the website uses. Sometimes that does not work, and you are forced into using browser automation like selenium or playright.