chore: Update pip installation command and requirements for Crawl4AI

README.md

@@ -119,8 +119,8 @@ To install Crawl4AI as a library, follow these steps:
 ```bash
 virtualenv venv
 source venv/bin/activate
+pip install transformers torch chromedriver_autoinstaller
 pip install git+https://github.com/unclecode/crawl4ai.git
-python docs/examples/quickstart.py
 ```
 
 2. Alternatively, you can clone the repository and install the package locally:
@@ -129,6 +129,7 @@ virtualenv venv
 source venv/bin/activate
 git clone https://github.com/unclecode/crawl4ai.git
 cd crawl4ai
+pip install transformers torch chromedriver_autoinstaller
 pip install -e .
 ```
 

crawl4ai/model_loader.py

@@ -43,7 +43,7 @@ def load_spacy_model():
     model_folder = os.path.join(home_folder, name)
 
     # Check if the model directory already exists
-    if True or not (Path(model_folder).exists() and any(Path(model_folder).iterdir())):
+    if not (Path(model_folder).exists() and any(Path(model_folder).iterdir())):
         repo_url = "https://github.com/unclecode/crawl4ai.git"
         # branch = "main"
         branch = MODEL_REPO_BRANCH 

pages/partial/installation.html

@@ -27,7 +27,11 @@ <h1 class="text-2xl font-bold">Installation 💻</h1>
             Install the package from GitHub:
             <pre
                 class="bg-zinc-800 p-4 rounded mt-2 text-zinc-100"
-            ><code>pip install git+https://github.com/unclecode/crawl4ai.git</code></pre>
+            ><code>virtualenv venv
+source venv/bin/activate
+pip install git+https://github.com/unclecode/crawl4ai.git
+pip install transformers torch chromedriver_autoinstaller
+            </code></pre>
         </li>
         <li class="mb-4">
             Alternatively, you can clone the repository and install the package locally:
@@ -37,6 +41,7 @@ <h1 class="text-2xl font-bold">Installation 💻</h1>
 source venv/bin/activate
 git clone https://github.com/unclecode/crawl4ai.git
 cd crawl4ai
+pip install transformers torch chromedriver_autoinstaller
 pip install -e .
 </code></pre>
         </li>