Downloading request data from Network Tab

I was making a crawler for some site written in Angular. This was my first time dealing with such site.

I could not use requests/BeautifulSoup as the page components loading needed JS to work. So, I chose Selenium to write crawler.

I realized the links were not in the html page but instead were only visible in Network tab in browser debugging tools.

So, I looked up and found the solution using browsermob-proxy to access those links.

And that’s all.

Thanks for reading through!

Related posts:

Welcome to The PATH — a weekly reflection with three insights for daily life. This week’s reflection searches for ancient insights on discovering true joy. To do so, we reflect on — solitude, love…

I remember the first time I walked into a Christian bookstore. It was honestly one of the most bizarre places I had ever visited. The old lady at the front counter welcomed us in like a librarian…

Hello dear followers, today we will share with you a short content on how to look at the transaction history via Yobit.net Exchange Service. Here is more detailed information about the transactions…