Toggle Dark Theme
logo
234,387 posts archived


[–]Modredpillschool[M] [score hidden] stickied comment (5 children)

Thanks. So did we. https://www.forums.red/i/theredpill

We will always welcome more backup points, so thank you for helping out.

Our backup includes 3,176 Posts between TRP and AskTRP. With over 519,619 individual comments.

Counting RPW and ThankTRP, our total archive is 4,573 posts.

Instant search can be found here

In case of emergency we will release the entire database as torrent. FYI.

Forums opening soon with new mobile-friendly design.

[–]SpiderAlpha33 2 points3 points  (0 children)

Yeah BeautifilSoup is fast and reliable, and for dynamically generated websites I use Selenium with headless Firefox.

[–]ThePantsThief 2 points3 points  (5 children)

Why scrape it when they have an API? Fellow developer here. Curious in case the scraping solution is better somehow

[–]Modredpillschool 3 points4 points  (0 children)

The https://forums.red/i/TheRedPill archive was done via API

[–]needz 0 points1 point  (3 children)

Sometimes if you already have your favorite tool and it works, there's no need to learn another tool (or in this case, API).

[–]SilkTouchm 4 points5 points  (2 children)

Scraping is always harder than just using the api.

[–]needz 0 points1 point  (1 child)

Never say never or always.

So if I've been using BeautifulSoup since it came out, have an entire framework and dev environment dedicated to scraping websites and have little to no experience with APIs, which is gonna be harder?

[–]SilkTouchm 1 point2 points  (0 children)

which is gonna be harder?

Scraping, by far. It doesn't matter how much experience you have, you still have to do the legwork. An API is just using a few pre-packaged methods, you don't need experience on it.