Delegation Pattern In Python To Crawl Data With API

An effective way is widely used in the mobile development sector to build APIService, which I now bring to the data-related sector.

• Delegation Pattern In Python To Crawl Data With API
• The problem it solved
• Knowledge Base
  • Catch that API call
  • Delegation Pattern
• Structure
• Application
  • Shopee
  • Artstation
  • Tinder
  • Pexels

The problem it solved

• Crawl data easily
• Collect any possible data you want (check robots.txt to make sure you are not violent with any rules), even Tinder!
• Wide range use

Knowledge Base

Catch that API call

I have a demonstration in this repo (Crawl data section).

Delegation Pattern

Structure

• APIService
• ItemModel
• GetData

Application

Shopee

📢 Up-to-date: 07/10/2022

Artstation

💡 TODO: This is the code from November 2021, now October 2022, for some reason, looks like it crawls all the data available with per_page = 3, max_page=1, so:

• Check the API call
• If you decide to use this code, make sure to check attribute all_items to not have any duplicate value, and limit it, because calling the method crawlImage() will take time.

Tinder

Should I publish it 🤔

Pexels

Not done yet! Now it works on a JSON file you already downloaded, use the function on GetData, need to change the APIService.