r/PythonLearning • u/LiteratureTypical982 • 18h ago

Client wants 40k emails from a directory. Each one requires a click. What do I do?

CLint need a list of 40,000+ members from the ACR directory. Need Name, City, State, Zip, Specialty, Email, and Phone.
The directory/search results give me some of this information, but email, phone, address, Member Since, etc. appear to be available only on the individual member profile.

So I'm trying to figure out the best way to approach this in Python.

If I literally open/request every profile, that's 43k+ profile requests, which obviously makes me concerned about rate limits, blocking, or getting the account/IP banned.

I'm considering Python requests/BeautifulSoup or Playwright, but before attempting something this large I'd like some advice.

1 Upvotes
  • permalink
  • reddit

67% Upvoted

5 comments sorted by

2

u/johlae 18h ago

What's the ACR? American College of Radiology?

4

u/HotPersonality8126 17h ago

It’s worth spending 6 hours or so investigating whether there’s an API you can access, or get access to; that could include phoning someone at the company to check.

Otherwise you’re going to scrape the page, and you take on all the technical overhead of attempting to evade their anti-scraping measures.

2

u/recentAd1666 15h ago
  1. Ask them if they sell the data ?
  2. Try with python yes, maybee with delay between requests

2

u/FoolsSeldom 15h ago edited 15h ago
  • contact them to ask if there's an API
  • check terms and conditions and see if web scraping breaks their rules / is blocked
  • explore PyAutoGUI for automation to do what you would do (alternative to web scraping)

NB. Last two both risk your device being identified by IP address and being blocked - you might need to find ways of spinning up containers and spoofing IP addresses (possibly paying for a legitish bot farm)

Of course, if this has to be done from a signed in account ... well API is really the only way.