forked from cloudlinux/kernelcare-doc
-
-
Notifications
You must be signed in to change notification settings - Fork 0
/
transfer.log
148 lines (146 loc) · 16.1 KB
/
transfer.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
/Users/prefer/.virtualenvs/scrapy1/bin/python /Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 58816 --file /Users/prefer/src/tmp/cl-doc-crawler/doc/run.py --folder /Users/prefer/src/kernelcare-doc/docs --base-url=https://docs.kernelcare.com --start-url=https://docs.kernelcare.com/kcdocs_content.htm
pydev debugger: process 62585 is connecting
Connected to pydev debugger (build 182.5107.15)
2019-02-14 15:05:12 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: doc)
2019-02-14 15:05:12 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.0b4 (v3.6.0b4:18496abdb3d5, Nov 21 2016, 20:44:47) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i 14 Aug 2018), cryptography 2.3.1, Platform Darwin-18.2.0-x86_64-i386-64bit
2019-02-14 15:05:12 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'doc', 'NEWSPIDER_MODULE': 'doc.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['doc.spiders']}
2019-02-14 15:05:12 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2019-02-14 15:05:12 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-02-14 15:05:12 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-02-14 15:05:12 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-02-14 15:05:12 [scrapy.core.engine] INFO: Spider opened
2019-02-14 15:05:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-02-14 15:05:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-02-14 15:05:13 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://docs.kernelcare.com/robots.txt> (referer: None)
2019-02-14 15:05:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kcdocs_content.htm> (referer: None)
2019-02-14 15:05:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/installation.htm> (referer: https://docs.kernelcare.com/kcdocs_content.htm)
2019-02-14 15:05:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/switching_from_ksplice.htm> (referer: https://docs.kernelcare.com/installation.htm)
2019-02-14 15:05:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/settings.htm> (referer: https://docs.kernelcare.com/switching_from_ksplice.htm)
2019-02-14 15:05:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/command_line.htm> (referer: https://docs.kernelcare.com/settings.htm)
2019-02-14 15:05:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kcarectl.htm> (referer: https://docs.kernelcare.com/command_line.htm)
WARNING: Possible broken link - config_options.htm
WARNING: Possible broken link - sticky_patches.htm
WARNING: Possible broken link - managing_servers.htm#addingextratagfield
2019-02-14 15:05:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kcare-uname.htm> (referer: https://docs.kernelcare.com/kcarectl.htm)
2019-02-14 15:05:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/config_options.htm> (referer: https://docs.kernelcare.com/kcare-uname.htm)
WARNING: Possible broken link - sticky_patches.htm
2019-02-14 15:05:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/disabling_some_patches.htm> (referer: https://docs.kernelcare.com/config_options.htm)
2019-02-14 15:05:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/delayed_feed.htm> (referer: https://docs.kernelcare.com/disabling_some_patches.htm)
2019-02-14 15:05:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/extra_patchset.htm> (referer: https://docs.kernelcare.com/delayed_feed.htm)
2019-02-14 15:05:18 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/sticky_patches.htm> (referer: https://docs.kernelcare.com/extra_patchset.htm)
2019-02-14 15:05:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/nagios_plugin.htm> (referer: https://docs.kernelcare.com/sticky_patches.htm)
2019-02-14 15:05:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/zabbix_template.htm> (referer: https://docs.kernelcare.com/nagios_plugin.htm)
2019-02-14 15:05:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/upgrade.htm> (referer: https://docs.kernelcare.com/zabbix_template.htm)
2019-02-14 15:05:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/uninstall.htm> (referer: https://docs.kernelcare.com/upgrade.htm)
2019-02-14 15:05:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/technology.htm> (referer: https://docs.kernelcare.com/uninstall.htm)
2019-02-14 15:05:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/reseller_partner_ui.htm> (referer: https://docs.kernelcare.com/technology.htm)
2019-02-14 15:05:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kernelcare_enterprise.htm> (referer: https://docs.kernelcare.com/reseller_partner_ui.htm)
2019-02-14 15:05:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/eportal_hardware_requirements.htm> (referer: https://docs.kernelcare.com/kernelcare_enterprise.htm)
2019-02-14 15:05:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/installation2.htm> (referer: https://docs.kernelcare.com/eportal_hardware_requirements.htm)
2019-02-14 15:05:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/managing_users.htm> (referer: https://docs.kernelcare.com/installation2.htm)
2019-02-14 15:05:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/accessing_eportal.htm> (referer: https://docs.kernelcare.com/managing_users.htm)
WARNING: Possible broken link - managing_users.htm
2019-02-14 15:05:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/patchset_deployment.htm> (referer: https://docs.kernelcare.com/accessing_eportal.htm)
2019-02-14 15:05:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/managing_keys.htm> (referer: https://docs.kernelcare.com/patchset_deployment.htm)
WARNING: Possible broken link - managing_servers.htm
2019-02-14 15:05:42 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/managing_servers.htm> (referer: https://docs.kernelcare.com/managing_keys.htm)
WARNING: Possible broken link - managing_keys.htm
WARNING: Possible broken link - 'feed_management' for url='kernelcare_enterprise/managing_servers' and text='Feed Management'. Please do find 'feed_management' and fix in the result files
WARNING: Possible broken link - 'addingextratagfield' for url='kernelcare_enterprise/managing_servers' and text='Adding extra Tag field'. Please do find 'addingextratagfield' and fix in the result files
2019-02-14 15:05:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/deploying_kernelcare.htm> (referer: https://docs.kernelcare.com/managing_servers.htm)
2019-02-14 15:05:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kernelcare_client_config_file.htm> (referer: https://docs.kernelcare.com/deploying_kernelcare.htm)
2019-02-14 15:05:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/changing_eportal_ip.htm> (referer: https://docs.kernelcare.com/kernelcare_client_config_file.htm)
2019-02-14 15:05:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/configuration__locations.htm> (referer: https://docs.kernelcare.com/changing_eportal_ip.htm)
2019-02-14 15:05:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/stopping__starting.htm> (referer: https://docs.kernelcare.com/configuration__locations.htm)
2019-02-14 15:05:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/log_files.htm> (referer: https://docs.kernelcare.com/stopping__starting.htm)
2019-02-14 15:05:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/eportal_api.htm> (referer: https://docs.kernelcare.com/log_files.htm)
2019-02-14 15:05:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/nagios__zabbix_support.htm> (referer: https://docs.kernelcare.com/eportal_api.htm)
WARNING: Possible broken link - nagios_plugin.htm
WARNING: Possible broken link - zabbix_template.htm
2019-02-14 15:05:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kcare-nexpose.htm> (referer: https://docs.kernelcare.com/nagios__zabbix_support.htm)
WARNING: Possible broken link - installation3.htm
WARNING: Possible broken link - how_it_works.htm
WARNING: Possible broken link - how_to_launch.htm
WARNING: Possible broken link - yaml_config_file_description.htm
2019-02-14 15:05:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/installation3.htm> (referer: https://docs.kernelcare.com/kcare-nexpose.htm)
2019-02-14 15:05:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/how_it_works.htm> (referer: https://docs.kernelcare.com/installation3.htm)
WARNING: Possible broken link - yaml_config_file_description.htm
2019-02-14 15:05:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/how_to_launch.htm> (referer: https://docs.kernelcare.com/how_it_works.htm)
2019-02-14 15:05:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/yaml_config_file_description.htm> (referer: https://docs.kernelcare.com/how_to_launch.htm)
2019-02-14 15:05:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kernelcare_whmcs_plugin.htm> (referer: https://docs.kernelcare.com/yaml_config_file_description.htm)
WARNING: Possible broken link - installation_configuration.htm
WARNING: Possible broken link - installation_and_update.htm
WARNING: Possible broken link - link_via_addon_optional_lice.htm
WARNING: Possible broken link - kernelcare_key_licenses.htm
WARNING: Possible broken link - add-on_licenses_list.htm
2019-02-14 15:05:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/overview.htm> (referer: https://docs.kernelcare.com/kernelcare_whmcs_plugin.htm)
2019-02-14 15:05:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/installation_configuration.htm> (referer: https://docs.kernelcare.com/overview.htm)
WARNING: Possible broken link - installation_and_update.htm
2019-02-14 15:05:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/installation_and_update.htm> (referer: https://docs.kernelcare.com/installation_configuration.htm)
2019-02-14 15:05:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/configuration_of_product.htm> (referer: https://docs.kernelcare.com/installation_and_update.htm)
2019-02-14 15:05:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/configuration_of_addon.htm> (referer: https://docs.kernelcare.com/configuration_of_product.htm)
2019-02-14 15:06:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/management.htm> (referer: https://docs.kernelcare.com/configuration_of_addon.htm)
WARNING: Possible broken link - link_via_addon_optional_lice.htm
WARNING: Possible broken link - kernelcare_key_licenses.htm
WARNING: Possible broken link - add-on_licenses_list.htm
2019-02-14 15:06:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/link_via_addon_optional_lice.htm> (referer: https://docs.kernelcare.com/management.htm)
2019-02-14 15:06:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/link_products_directly.htm> (referer: https://docs.kernelcare.com/link_via_addon_optional_lice.htm)
2019-02-14 15:06:08 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/link_via_configurable_options.htm> (referer: https://docs.kernelcare.com/link_products_directly.htm)
2019-02-14 15:06:09 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/link_addons_directly.htm> (referer: https://docs.kernelcare.com/link_via_configurable_options.htm)
2019-02-14 15:06:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/kernelcare_key_licenses.htm> (referer: https://docs.kernelcare.com/link_addons_directly.htm)
2019-02-14 15:06:15 [scrapy.extensions.logstats] INFO: Crawled 52 pages (at 52 pages/min), scraped 0 items (at 0 items/min)
2019-02-14 15:06:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/order.htm> (referer: https://docs.kernelcare.com/kernelcare_key_licenses.htm)
2019-02-14 15:06:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/admin_area.htm> (referer: https://docs.kernelcare.com/order.htm)
2019-02-14 15:06:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/client_area.htm> (referer: https://docs.kernelcare.com/admin_area.htm)
2019-02-14 15:06:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/licenses_list.htm> (referer: https://docs.kernelcare.com/client_area.htm)
2019-02-14 15:06:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/add-on_licenses_list.htm> (referer: https://docs.kernelcare.com/licenses_list.htm)
2019-02-14 15:06:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/common_problems.htm> (referer: https://docs.kernelcare.com/add-on_licenses_list.htm)
2019-02-14 15:06:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/proxy_settings.htm> (referer: https://docs.kernelcare.com/common_problems.htm)
2019-02-14 15:06:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/eol_ubuntu_lts_kernels_support.htm> (referer: https://docs.kernelcare.com/proxy_settings.htm)
2019-02-14 15:06:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://docs.kernelcare.com/downloading_documentation.htm> (referer: https://docs.kernelcare.com/eol_ubuntu_lts_kernels_support.htm)
2019-02-14 15:06:31 [scrapy.core.engine] INFO: Closing spider (finished)
2019-02-14 15:06:31 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 18080,
'downloader/request_count': 61,
'downloader/request_method_count/GET': 61,
'downloader/response_bytes': 555239,
'downloader/response_count': 61,
'downloader/response_status_count/200': 60,
'downloader/response_status_count/404': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 2, 14, 12, 6, 31, 981749),
'log_count/DEBUG': 62,
'log_count/INFO': 8,
'memusage/max': 89899008,
'memusage/startup': 75911168,
'request_depth_max': 59,
'response_received_count': 61,
'scheduler/dequeued': 60,
'scheduler/dequeued/memory': 60,
'scheduler/enqueued': 60,
'scheduler/enqueued/memory': 60,
'start_time': datetime.datetime(2019, 2, 14, 12, 5, 12, 639347)}
2019-02-14 15:06:31 [scrapy.core.engine] INFO: Spider closed (finished)
Process finished with exit code 0