Email Remainder Scripts! (#953)

* feat: basic email-sending template and config

* docs: add docs for email sending + infra

* chore: update email bcc batch sizes based on support ticket answer

* chore: update template + docs: note required wd

* feats: connect to firebase, set up more detailed templates, enhance control flow, create demo targeted template

* feats: connect to firebase, set up more detailed templates, enhance control flow, create demo targeted template

* Revert "feats: connect to firebase, set up more detailed templates, enhance control flow, create demo targeted template"

This reverts commit 7f174f5.

* style: email template v1 done (light mode desktop)

* chore: update email opening line

* chore: update styles for footer links

* style: increase image quality

* fix: map grade level to entrance year + sem

* fix: update images

* style (fix): force logo width

* chore: warn on failed resend email

* chore: comment out currently-unused user name data

* docs: get rid of invalid dns mention

* fix: get rid of bad print statement

* fix: format template

.gitignore

@@ -38,5 +38,11 @@ scripts/out
 # Secret API key
 python/secret_api_keys.py
 
+# Email API key stuff.
+.env.private
+
+# Python compilation files.
+*/**/*.pyc
+
 # Pytest
 python/__pycache__

scripts/email/README.md

@@ -0,0 +1,99 @@
+# CoursePlan Email System
+
+Welcome to the CoursePlan email-sending system. This folder is the home for all logic and templates related to the emails that we send to users, such as onboarding, reminders, notifications about pre-enrollment, etc.
+
+## Infrastructure
+
+This system is built on top of [Resend](https://resend.com)'s email API. We chose Resend over a Google-Cloud based OAuth solution like Nodemailer (used by IDOL) for the following reasons:
+
+1. (**Main reason**): capable of handling high email volumes; Gmail automatically limits the number of emails sent per day and we would go way over that.
+2. Improved deliverability rates.
+3. Simplified API for sending.
+
+Then in comparison to other email services like Sendgrid, Resend stood out for its quality developer documentation and YC backing.
+
+## Getting Started
+
+1. Install dependencies in the `scripts/email` directory:
+   ```bash
+   python3 -m pip install -r requirements.txt
+   ```
+
+2. Set up environment variables in `.env.private` **in the root directory of the project (`courseplan`)**:
+   ```
+   RESEND_API_KEY=your_resend_api_key  # contact Simon or your TPM for access
+   GLOBAL_FROM_NAME=CoursePlan  # what the name of the sender will be
+   [email protected]  # what the email of the sender will be (once DNS records for courseplan.io are configured)
+   [email protected]  # a dummy email address to ensure bcc works
+   ```
+   **Never commit this file!** (Should already be in `.gitignore`.)
+
+3. Create a new template in `scripts/email/templates/` or use the existing `dryrun.py` as a test example with your own email.
+
+4. Update the import in `execute_template.py` to import the template you created in the previous step. As an example, if you created a file called `new_course_reminder.py` in the templates folder, you would update the import in `execute_template.py` to `from .templates.new_course_reminder import *`. For details on how to create a template, see two sections from now.
+
+5. `cd` upwards into the root directory and run the script:
+   ```bash
+   python3 scripts/email/execute_template.py
+   ```
+
+**Important**: Please revert your import change to `execute_template.py` before pushing any changes. By always sticking with `dryrun.py` as the base template, we can avoid accidentally sending emails to thousands of users (surefire way to worsen your chances at getting an A/A+ for the semester 😅).
+
+## How It Works
+
+1. The script loads environment variables and the specified email template.
+2. It chunks the BCC list into groups of 49 recipients to make best use of our 100-email-per-day free tier. (50 is the max number of recipients allowed in a single Resend API call, and this is shared across `to` and `bcc` recipients.)
+3. Emails are sent in batches, with progress updates printed to the console.
+
+## Creating Templates
+
+1. Create a new Python file in `scripts/email/templates/`.
+2. Define `BCC`, `SUBJECT`, and `HTML` variables.
+    - `BCC` should be a list of emails to send to.
+    - `SUBJECT` is the subject of the email.
+    - `HTML` is the body of the email.
+3. **Test your template** by running `python3 scripts/email/execute_template.py` with a simplified BCC list before sending to a large audience.
+
+A couple notes:
+- You can refer to existing templates for best practices and to see how to e.g. have the `BCC` list be dynamically generated from our Firebase users.
+- **Important**: Ensure all HTML styling is inline as we unfortunately cannot use external CSS directives.
+
+## Fetching Users from Firebase
+
+We use Firebase to store user data and retrieve it for our email templates. The process is handled by the `firebase_users_loader.py` helper script under `scripts/email/helpers/`.
+
+1. The script connects to Firebase using a service account key, stored in the root directory of the project as `serviceAccountKey.json`.
+3. It retrieves user data from the `user-onboarding-data` collection.
+2. Then, it fetches all user names from the `user-name` collection.
+4. The data is processed and organized into a dictionary, with keys being tuples of (graduation_semester, graduation_year).
+5. Each user's data includes email, name, colleges, grad programs, majors, minors, and graduation information.
+
+The `USERS` variable in `firebase_users_loader.py` contains this processed data and is imported by individual email templates.
+
+### Using Firebase Data in Templates
+
+Email templates, such as `current_freshman.py`, import the `USERS` data and filter it based on specific criteria. For example:
+
+```python
+from scripts.email.templates.helpers.firebase_users_loader import USERS
+
+BCC = [
+    user["email"]
+    for users in USERS.values()
+    for user in users
+    if (
+        (user["graduation_year"] == "2028" and user["graduation_semester"] == "Spring")
+        or (user["graduation_year"] == "2027" and user["graduation_semester"] == "Fall")
+    )
+]
+```
+
+This code filters the `USERS` data to create a `BCC` list of email addresses for current freshmen (in FA24) based on their expected graduation year and semester.
+
+By using this approach, we can easily create targeted email lists for different groups of students without manually maintaining separate lists.
+
+## Further Notes
+
+- You **must** run the script from the root directory of the project.
+- If you want to have emails land in more than 4,900 inboxes per day, you will need to stagger the emails over several days. Note that these 4,900 inboxes are already "batched" into 100 emails each with 49 bcc recipients.
+- A dummy "to" recipient is required when using BCC for technical reasons. (This does not count towards the 49 bcc recipients.)

scripts/email/execute_template.py

@@ -0,0 +1,74 @@
+import os
+from dotenv import load_dotenv
+import resend
+from typing import List
+
+# Load relevant template to run this script on.
+from templates.versioned_preenroll.sp25.current_freshman import *
+
+# Load environment variables from .env
+load_dotenv(".env.private")
+
+resend.api_key = os.environ["RESEND_API_KEY"]
+
+# Global constants across all sessions — please do not change these.
+FROM = os.environ["GLOBAL_FROM_NAME"] + " <" + os.environ["GLOBAL_FROM_EMAIL"] + ">"
+TO = os.environ["GLOBAL_TO_EMAIL"]
+
+
+def chunk_list(lst: List[str], chunk_size: int) -> List[List[str]]:
+    return [lst[i : i + chunk_size] for i in range(0, len(lst), chunk_size)]
+
+
+def send_emails(bcc_list: List[str]):
+    bcc_chunks = chunk_list(bcc_list, 49)
+    total_chunks = len(bcc_chunks)
+
+    # Add option to list all emails
+    list_emails = (
+        input("Do you want to list all email addresses? (y/N): ").lower().strip()
+    )
+    if list_emails == "y":
+        print("\nList of email addresses:")
+        for email in bcc_list:
+            print(email)
+        print()
+
+    # Add confirmation prompt
+    confirm = (
+        input(
+            f"Do you want to proceed with sending {len(bcc_chunks)} {'email' if len(bcc_chunks) == 1 else 'emails'} to {len(bcc_list)} {'recipient' if len(bcc_list) == 1 else 'recipients'}? (y/N): "
+        )
+        .lower()
+        .strip()
+    )
+
+    if confirm != "y":
+        print("Email sending cancelled. Goodbye!")
+        return
+    else:
+        print(
+            f"Starting email job with {len(bcc_list)} {'recipient' if len(bcc_list) == 1 else 'recipients'} in {total_chunks} batch(es).\n"
+        )
+
+    for i, bcc_chunk in enumerate(bcc_chunks, 1):
+        params: resend.Emails.SendParams = {
+            "from": FROM,
+            "to": TO,  # Must always have a to recipient for bcc to work.
+            "bcc": bcc_chunk,
+            "subject": SUBJECT,
+            "html": HTML,
+        }
+
+        try:
+            resend.Emails.send(params)
+            print(f"Batch {i}/{total_chunks} sent successfully!")
+        except Exception as e:
+            print(f"Batch {i}/{total_chunks} failed with error: {e}")
+
+    print("\nAll email batches sent successfully!")
+    print("Email job completed!")
+
+
+if __name__ == "__main__":
+    send_emails(BCC)

scripts/email/helpers/firebase_users_loader.py

@@ -0,0 +1,77 @@
+import firebase_admin
+from firebase_admin import credentials, firestore
+from collections import defaultdict
+from typing import Dict, List, Tuple
+
+cred = credentials.Certificate("./serviceAccountKey.json")
+firebase_admin.initialize_app(cred)
+
+db = firestore.client()
+
+
+# NOTE: currently not being used for personalization due to BCC logic.
+# If we do end up personalizing sometime, then uncomment this and associated
+# calls later in the file + update the email script.
+# def get_all_user_names() -> Dict[str, Tuple[str, str, str]]:
+#     print("Fetching all user names from Firebase...", end="\n\n")
+#     user_names = {}
+#     docs = db.collection("user-name").get()
+#     for doc in docs:
+#         data = doc.to_dict()
+#         user_names[doc.id] = (
+#             data.get("firstName", None),
+#             data.get("middleName", None),
+#             data.get("lastName", None),
+#         )
+#     return user_names
+
+
+def get_users() -> Dict[Tuple[str, str], List[Dict]]:
+    print("Fetching all user data from Firebase...")
+    # user_names = get_all_user_names()
+    users = db.collection("user-onboarding-data").get()
+    user_map = defaultdict(list)
+
+    for user in users:
+        user_data = user.to_dict()
+        email = user.id
+        # first_name, middle_name, last_name = user_names.get(
+        #     email, (None, None, None)
+        # )  # NOTE: may cause type errors if we don't have a name for this user and try
+        # # to send them an email regardless. (Deliberate decision.)
+
+        # # Deleted accounts or something? Not sure to do with these people.
+        # if first_name is None and middle_name is None and last_name is None:
+        #     print(f"User {email} not found in user-name collection!")
+
+        colleges = [
+            college.get("acronym", "") for college in user_data.get("colleges", [])
+        ]
+        grad_programs = [
+            program.get("acronym", "") for program in user_data.get("gradPrograms", [])
+        ]
+        entrance_sem = user_data.get("entranceSem", None)
+        entrance_year = user_data.get("entranceYear", None)
+        majors = [major.get("acronym", "") for major in user_data.get("majors", [])]
+        minors = [minor.get("acronym", "") for minor in user_data.get("minors", [])]
+
+        key = (entrance_sem, entrance_year)
+        user_map[key].append(
+            {
+                "email": email,
+                # "first_name": first_name,
+                # "middle_name": middle_name,
+                # "last_name": last_name,
+                "colleges": colleges,
+                "grad_programs": grad_programs,
+                "entrance_semester": entrance_sem,
+                "entrance_year": entrance_year,
+                "majors": majors,
+                "minors": minors,
+            }
+        )
+
+    return dict(user_map)
+
+
+USERS = get_users()