Day013_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PTT 網路爬蟲實作練習\n",
    "\n",
    "\n",
    "* 能夠利用 Request + BeatifulSour 撰寫爬蟲，並存放到合適的資料結構\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標\n",
    "\n",
    "根據範例 ，完成以下問題：\n",
    "\n",
    "* ① 印出最新文章的「作者」「標題」「時間」\n",
    "* ② 印出第一頁所有文章的「作者」「標題」「時間」\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ① 印出最新文章的「作者」「標題」「時間」"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "最近文章\n",
      "作者: bow231 (魔鬼的新歡) \t 時間: Mon Jun  7 19:07:14 2021\n",
      "標題: Re: [討論] 獨行俠簽KP真的是錯的嗎\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Your Code\n",
    "'''\n",
    "import requests as rq\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "url='https://www.ptt.cc/bbs/NBA/index.html'\n",
    "\n",
    "r=rq.get(url)\n",
    "soup=BeautifulSoup(r.text,\"html5lib\")\n",
    "d=soup.find(class_='title')\n",
    "r = BeautifulSoup(rq.get('https://www.ptt.cc'+d.find('a')['href']).text, \"html5lib\")\n",
    "\n",
    "print('最近文章')\n",
    "print('作者: ' + r.find(class_='article-meta-value').text, \"\\t 時間:\", r.find_all(class_='article-meta-value')[-1].text)\n",
    "print('標題: ' + r.find_all(class_='article-meta-value')[2].text )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ② 印出第一頁所有文章的「作者」「標題」「時間」"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "作者: bow231 (魔鬼的新歡) \t 時間: Mon Jun  7 19:07:14 2021\n",
      "標題: Re: [討論] 獨行俠簽KP真的是錯的嗎\n",
      "\n",
      "\n",
      "作者: WIGGINS22 (圍巾絲22) \t 時間: Mon Jun  7 20:17:29 2021\n",
      "標題: [情報] Simmons：如果裁判不吹那麼多犯規，我希望開局就防守Young\n",
      "\n",
      "\n",
      "作者: Yui5 (唯控) \t 時間: Mon Jun  7 20:40:23 2021\n",
      "標題: [新聞] 總冠軍賽組合新面貌 不見湖人馬刺熱火勇\n",
      "\n",
      "\n",
      "作者: thnlkj0665 (灰色地帶) \t 時間: Mon Jun  7 21:15:07 2021\n",
      "標題: [花邊] Barkley 認為金塊隊將在第二輪擊敗太陽隊\n",
      "\n",
      "\n",
      "作者: sezna (sezna) \t 時間: Mon Jun  7 21:17:05 2021\n",
      "標題: [情報] 老河:最後階段不希望讓鉛筆上場拼搶，可\n",
      "\n",
      "\n",
      "作者: s66449 (老實樣) \t 時間: Mon Jun  7 21:26:25 2021\n",
      "標題: [新聞] 「跟父子檔交手後」詹姆斯嘆老了　若碰\n",
      "\n",
      "\n",
      "作者: aaagun (A槍大叔) \t 時間: Mon Jun  7 21:33:23 2021\n",
      "標題: [情報] 明日裁判名單\n",
      "\n",
      "\n",
      "作者: GhostFather (鬼父Oni Chichi) \t 時間: Mon Jun  7 22:05:26 2021\n",
      "標題: [討論] 2020-21 NBA季後賽冠軍&第二輪賠率\n",
      "\n",
      "\n",
      "作者: aaagun (A槍大叔) \t 時間: Mon Jun  7 22:16:11 2021\n",
      "標題: [情報] 第二輪裁判名單\n",
      "\n",
      "\n",
      "作者: adam7148 (Adam) \t 時間: Mon Jun  7 22:19:51 2021\n",
      "標題: [情報] KP對自己的角色感到沮喪，認為077吸走所\n",
      "\n",
      "\n",
      "作者: laptic (靜夜聖林彼岸花) \t 時間: Wed May 19 00:35:47 2021\n",
      "標題: [情報] Playoffs Schedule 20–21\n",
      "\n",
      "\n",
      "作者: laptic (靜夜聖林彼岸花) \t 時間: Mon May 17 12:01:32 2021\n",
      "標題: [情報] 2020-21 NBA Playoffs 對戰圖\n",
      "\n",
      "\n",
      "作者: Induction (愛人愛己) \t 時間: Fri Jun  4 04:40:51 2021\n",
      "標題: [公告] NBA_Picket 檢舉板上線 \n",
      "\n",
      "\n",
      "作者: Induction (愛人愛己) \t 時間: Mon May 24 04:02:16 2021\n",
      "標題: [公告] 季後賽條款與置底檢舉區\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Your Code\n",
    "'''\n",
    "import requests as rq\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "url='https://www.ptt.cc/bbs/NBA/index.html'\n",
    "\n",
    "r=rq.get(url)\n",
    "\n",
    "soup=BeautifulSoup(r.text,\"html5lib\")\n",
    "\n",
    "for d in soup.find_all(class_='title'):\n",
    "#     t=d.text.replace('\\t','').replace('\\n','')\n",
    "    try:\n",
    "        r = BeautifulSoup(rq.get('https://www.ptt.cc'+d.find('a')['href']).text, \"html5lib\")\n",
    "        print('作者: ' + r.find(class_='article-meta-value').text, \"\\t 時間:\", r.find_all(class_='article-meta-value')[-1].text)\n",
    "        print('標題: ' + r.find_all(class_='article-meta-value')[2].text )\n",
    "        print('\\n')\n",
    "\n",
    "\n",
    "    except:\n",
    "        continue\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ③ 試著爬爬看其他版的文章"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "作者: JRSmith (垃圾蔡母豬) \t 時間: Mon Jun  7 22:39:15 2021\n",
      "標題: [問卦] 麻辣教師GTO會怎麼演遠距教學？\n",
      "\n",
      "\n",
      "作者: songgood (ilovegoodgame) \t 時間: Mon Jun  7 22:39:32 2021\n",
      "標題: Re: [問卦] 噓的人是不是想當吐槽擔當\n",
      "\n",
      "\n",
      "作者: ezJapan (Japan) \t 時間: Mon Jun  7 22:40:38 2021\n",
      "標題: [問卦] 台灣防疫是不是都慢半拍\n",
      "\n",
      "\n",
      "作者: sfishes (然後之後) \t 時間: Mon Jun  7 22:40:42 2021\n",
      "標題: [問卦] 中國甩鍋和林靜儀甩鍋是不是系出同門\n",
      "\n",
      "\n",
      "作者: show9202 (艾鈉) \t 時間: Mon Jun  7 22:41:32 2021\n",
      "標題: Re: [問卦] 巴哈傳的這個是真的嗎？(日本AZ八卦)\n",
      "\n",
      "\n",
      "作者: teddy98 (泰迪!走吧!) \t 時間: Mon Jun  7 22:41:50 2021\n",
      "標題: [問卦] 自購疫苗到底乾中央屁事?\n",
      "\n",
      "\n",
      "作者: god060119 (ˋ＿ˊ) \t 時間: Mon Jun  7 22:41:52 2021\n",
      "標題: [問卦] 有沒有要睡多久才會有精神的八卦?\n",
      "\n",
      "\n",
      "作者: q347 (我家隔壁北川景子) \t 時間: Mon Jun  7 22:41:58 2021\n",
      "標題: [問卦] 怎麼辦 6/28圓寶生日 可是還在三級\n",
      "\n",
      "\n",
      "作者: bluecsky (我要藍藍淡淡的天空) \t 時間: Mon Jun  7 22:42:18 2021\n",
      "標題: [問卦] 金管會是不是抓到禿鷹啦\n",
      "\n",
      "\n",
      "作者: seansho (Sean) \t 時間: Mon Jun  7 22:42:28 2021\n",
      "標題: Re: [新聞] 國產疫苗疑慮難息 院士：民進黨政府不相\n",
      "\n",
      "\n",
      "作者: kons (kons) \t 時間: Mon Jun  7 22:42:38 2021\n",
      "標題: Re: [問卦] 為什麼FB上一堆平行世界？\n",
      "\n",
      "\n",
      "作者: Godmo (   ) \t 時間: Mon Jun  7 22:43:08 2021\n",
      "標題: Re: [新聞] 新竹疫情｜179瓶AZ疫苗發揮極致　衛生所\n",
      "\n",
      "\n",
      "作者: arsonlolita (蘿莉塔大魔王) \t 時間: Tue May 11 15:03:06 2021\n",
      "標題: [公告] 八卦板板規(2021.05.11)\n",
      "\n",
      "\n",
      "作者: arsonlolita (蘿莉塔大魔王) \t 時間: Thu May 27 17:47:11 2021\n",
      "標題: [協尋] 內湖成美橋行車記錄0522上午11:40-12:00\n",
      "\n",
      "\n",
      "作者: ubcs (覺★青年超冒險蓋被水桶) \t 時間: Sun Jun  6 19:44:29 2021\n",
      "標題: [公告] ★卍☯2021年6月八卦板主徵選投票開始☯\n",
      "\n",
      "\n",
      "作者: girl55665566 (同志友善ID) \t 時間: Wed Jun  2 00:08:33 2021\n",
      "標題: [公告] 團結防疫的六月份置底閒聊區\n",
      "\n",
      "\n",
      "作者: ubcs (覺★青年超冒險蓋) \t 時間: Sat Feb  6 15:09:20 2021\n",
      "標題: [公告] 手機投票教學 附圖 板主投票及活動可用\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Your Code\n",
    "'''\n",
    "#八卦版\n",
    "import requests as rq\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "url2='https://www.ptt.cc/bbs/Gossiping/index.html'\n",
    "\n",
    "#八卦版須加cookies\n",
    "r2=rq.get(url2,cookies={'over18':'1'})\n",
    "\n",
    "soup2=BeautifulSoup(r2.text)\n",
    "\n",
    "for i in soup2.find_all(class_='title'):\n",
    "#     print(i.text.replace('\\t','').replace('\\n',''))\n",
    "    try:\n",
    "        r2 = BeautifulSoup(rq.get('https://www.ptt.cc{}'.format(i.find('a')['href']), cookies={'over18':'1'}).text) #將format()內資料填入 {}中\n",
    "        print('作者: ' + r2.find(class_='article-meta-value').text, \"\\t 時間:\", r2.find_all(class_='article-meta-value')[-1].text)\n",
    "        print('標題: ' + r2.find_all(class_='article-meta-value')[2].text )\n",
    "        print('\\n')\n",
    "\n",
    "\n",
    "    except:\n",
    "        continue\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}