Py3 (linkedin#290)

* Py3 migration

* Update to Python 3 for CircleCI

* Fix auth bugs for python 3

Also fix notifier bug to check for active users

* Update notifier exception handling

Ignore role:target lookup failures from Iris, since these don't represent
problems with the underlying system, just that people have inactive users
on-call in the future.

* Add get_id param option (linkedin#246)

* add get_id param option

* removed superfluous select and simplified logic

* Flake8 typo (linkedin#247)

* Hide confusing team settings in an advanced dropdown

* Fix test fixtures

* Add "allow duplicate" scheduler in UI

Already in backend, so enable in FE too

* Add Dockerfile to run oncall in a container

* Move deps into a virtualenv.
Run app not as super user.
Mimick prod setup by using uwsgi

* Fix issue with Dockerfile not having MANIFEST.in and wrong passwords in (linkedin#257)

config

* Update to ubuntu:18.04 and python3 packages and executables

* Open config file as utf8

The default configuration file has utf8 characters, and python3
attempts to open the file as ASCII unless an alternate encoding
is specified

* Switch to the python3 uwsgi plugin

* Update print and os.execv statements for python3

Python3 throws an exception when the first argument to os.execv is empty:
ValueError: execv() arg 2 first element cannot be empty

The module documentation suggests that the first element should be the
name of the executed program:
https://docs.python.org/3.7/library/os.html#os.execv

* Map config.docker.yaml in to the container as a volume

./ops/entrypoint.py has the start of environment variable support
to specify a configuration file, but it is incomplete until we
update ./ops/daemons/uwsgi-docker.yaml or add environment support
to oncall-notifier and oncall-scheduler.

This commit allows users to map a specific configuration file in
to their container and have it used by all oncall programs.

* Convert line endings to match the rest of the project.

* Add mysql port to docker configuration

* Assume localhost mysql for default config.yaml

* Update python-dev package and MySQL root password

* Use password when configuring mysql

The project has started using a password on the mysql instance.
Once password auth is consistently working we can consider extracting
the hardcoded password into an env file that is optionally randomly
generated on initial startup.

* Fix preview for round-robin (linkedin#269)

* linkedin#275 fix for Python3 and Gunicorn load config

* Fixed E303 flake8

* Change encoding & collation + test  unicode name

Co-authored-by: Daniel Wang <[email protected]>
Co-authored-by: ahm3djafri <[email protected]>
Co-authored-by: TK <[email protected]>
Co-authored-by: Tim Freund <[email protected]>
Co-authored-by: Rafał Zawadzki <[email protected]>

.ci/setup_mysql.sh

@@ -1,15 +1,15 @@
 #!/bin/bash
 
 for i in `seq 1 5`; do
-    mysql -h 127.0.0.1 -u root -e 'show databases;' && break
+    mysql -h 127.0.0.1 -u root --password=1234 -e 'show databases;' && break
     echo "[*] Waiting for mysql to start..."
     sleep 5
 done
 
 echo "[*] Loading MySQL schema..."
-mysql -h 127.0.0.1 -u root < ./db/schema.v0.sql
+mysql -h 127.0.0.1 -u root --password=1234 < ./db/schema.v0.sql
 echo "[*] Loading MySQL dummy data..."
-mysql -h 127.0.0.1 -u root -o oncall < ./db/dummy_data.sql
+mysql -h 127.0.0.1 -u root --password=1234 -o oncall < ./db/dummy_data.sql
 
 echo "[*] Tables created for database oncall:"
-mysql -h 127.0.0.1 -u root -o oncall -e 'show tables;'
+mysql -h 127.0.0.1 -u root --password=1234 -o oncall -e 'show tables;'

.circleci/config.yml

@@ -2,10 +2,10 @@ version: 2
 jobs:
   build:
     docker:
-      - image: circleci/python:2.7-stretch-browsers
+      - image: circleci/python:3.7-stretch-browsers
       - image: mysql/mysql-server:5.7
         environment:
-          - MYSQL_ALLOW_EMPTY_PASSWORD=1
+          - MYSQL_ROOT_PASSWORD=1234
           - MYSQL_ROOT_HOST=%
 
     steps:
@@ -14,7 +14,7 @@ jobs:
           name: Install dependencies
           command: |
             sudo apt-get update
-            sudo apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev mysql-server
+            sudo apt-get install libsasl2-dev python3-dev libldap2-dev libssl-dev mysql-server
       - run:
           name: Prepare virtualenv
           command: |

Dockerfile

@@ -0,0 +1,32 @@
+FROM ubuntu:18.04
+
+RUN apt-get update && apt-get -y dist-upgrade \
+    && apt-get -y install libffi-dev libsasl2-dev python3-dev \
+        sudo libldap2-dev libssl-dev python3-pip python3-setuptools python3-venv \
+        mysql-client uwsgi uwsgi-plugin-python3 nginx \
+    && rm -rf /var/cache/apt/archives/*
+
+RUN useradd -m -s /bin/bash oncall
+
+COPY src /home/oncall/source/src
+COPY setup.py /home/oncall/source/setup.py
+COPY MANIFEST.in /home/oncall/source/MANIFEST.in
+
+WORKDIR /home/oncall
+
+RUN chown -R oncall:oncall /home/oncall/source /var/log/nginx /var/lib/nginx \
+    && sudo -Hu oncall mkdir -p /home/oncall/var/log/uwsgi /home/oncall/var/log/nginx /home/oncall/var/run /home/oncall/var/relay \
+    && sudo -Hu oncall python3 -m venv /home/oncall/env \
+    && sudo -Hu oncall /bin/bash -c 'source /home/oncall/env/bin/activate && cd /home/oncall/source && pip install .'
+
+COPY . /home/oncall
+COPY ops/config/systemd /etc/systemd/system
+COPY ops/daemons /home/oncall/daemons
+COPY ops/daemons/uwsgi-docker.yaml /home/oncall/daemons/uwsgi.yaml
+COPY db /home/oncall/db
+COPY configs /home/oncall/config
+COPY ops/entrypoint.py /home/oncall/entrypoint.py
+
+EXPOSE 8080
+
+CMD ["sudo", "-EHu", "oncall", "bash", "-c", "source /home/oncall/env/bin/activate && python -u /home/oncall/entrypoint.py"]

Makefile

@@ -25,3 +25,11 @@ check:
 	make test
 
 .PHONY: test e2e
+
+APP_NAME=oncall
+
+build:
+	docker build --ulimit nofile=1024 -t $(APP_NAME) .
+
+run: build
+	docker run -p 8080:8080 $(APP_NAME)

configs/config.docker.yaml

@@ -10,7 +10,8 @@ db:
       scheme: mysql+pymysql
       user: root
       password: '1234'
-      host: mysql
+      host: oncall-mysql
+      port: 3306
       database: oncall
       charset: utf8
       echo: True

configs/config.yaml

@@ -11,12 +11,13 @@ db:
     kwargs:
       scheme: mysql+pymysql
       user: root
-      password: ""
+      password: "1234"
       host: 127.0.0.1
+      port: 3306
       database: oncall
       charset: utf8
       echo: True
-    str: "%(scheme)s://%(user)s:%(password)s@%(host)s/%(database)s?charset=%(charset)s"
+    str: "%(scheme)s://%(user)s:%(password)s@%(host)s:%(port)s/%(database)s?charset=%(charset)s"
   kwargs:
     pool_recycle: 3600
 healthcheck_path: /tmp/status

db/dummy_data.sql

@@ -5,7 +5,7 @@ LOCK TABLES `user` WRITE;
 INSERT INTO `user` VALUES
     (1,'root',1,'God User',NULL,NULL,1),
     (2,'manager',1,'Team Admin',NULL,NULL,0),
-    (3,'jdoe',1,'John Doe',NULL,NULL,0),
+    (3,'jdoe',1,'Juan Doş',NULL,NULL,0),
     (4,'asmith',1,'Alice Smith',NULL,NULL,0);
 /*!40000 ALTER TABLE `user` ENABLE KEYS */;
 UNLOCK TABLES;

db/schema.v0.sql

@@ -1,4 +1,4 @@
-CREATE DATABASE IF NOT EXISTS `oncall` DEFAULT CHARACTER SET utf8 ;
+CREATE DATABASE IF NOT EXISTS `oncall` DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_bin;
 USE `oncall`;
 
 -- -----------------------------------------------------

docker-compose.yml

@@ -0,0 +1,16 @@
+version: '3'
+
+services:
+  oncall-web:
+    build: .
+    ports:
+      - "8080:8080"
+    environment:
+      - DOCKER_DB_BOOTSTRAP=1
+    volumes:
+      - ./configs/config.docker.yaml:/home/oncall/config/config.yaml
+
+  oncall-mysql:
+    image: mysql:5.7
+    environment:
+      - MYSQL_ROOT_PASSWORD=1234

e2e/conftest.py

@@ -13,11 +13,11 @@
 
 
 @pytest.fixture(scope="session", autouse=True)
-def require_db():
+def require_db(request):
     # Read config based on pytest root directory. Assumes config lives at oncall/configs/config.yaml
-    cfg_path = os.path.join(str(pytest.config.rootdir), 'configs/config.yaml')
+    cfg_path = os.path.join(str(request.config.rootdir), 'configs/config.yaml')
     with open(cfg_path) as f:
-        config = yaml.load(f)
+        config = yaml.safe_load(f)
     db.init(config['db'])
 
 
@@ -69,14 +69,17 @@ class TeamFactory(object):
         def __init__(self, prefix):
             self.prefix = prefix
             self.created = set()
+            self.created_ids = set()
             self.connection = db.connect()
             self.cursor = self.connection.cursor()
 
         def create(self):
             name = '_'.join([self.prefix, 'team', str(len(self.created))])
             re = requests.post(api_v0('teams'), json={'name': name, 'scheduling_timezone': 'utc'})
             assert re.status_code in [201, 422]
+            team_id = requests.get(api_v0('teams/%s' % name)).json()['id']
             self.created.add(name)
+            self.created_ids.add(team_id)
             return name
 
         def mark_for_cleaning(self, team_name):
@@ -85,8 +88,8 @@ def mark_for_cleaning(self, team_name):
         def cleanup(self):
             for team in self.created:
                 requests.delete(api_v0('teams/' + team))
-            if self.created:
-                self.cursor.execute('DELETE FROM team WHERE name IN %s', (self.created,))
+            if self.created_ids:
+                self.cursor.execute('DELETE FROM team WHERE id IN %s', (self.created_ids,))
                 self.connection.commit()
             self.cursor.close()
             self.connection.close()

e2e/test_schedules.py

@@ -4,7 +4,7 @@
 #!/usr/bin/env python
 # -*- coding:utf-8 -*-
 
-import urllib
+import urllib.parse
 import requests
 from testutils import prefix, api_v0
 
@@ -163,5 +163,5 @@ def test_api_v0_schedules_with_spaces_in_roster_name(team):
     assert re.status_code == 201
 
     re = requests.get(api_v0('teams/%s/rosters/%s/schedules' %
-                             (team_name, urllib.quote(roster_name, safe=''))))
+                             (team_name, urllib.parse.quote(roster_name, safe=''))))
     assert re.status_code == 200

e2e/test_users.py

@@ -25,32 +25,32 @@ def clean_up():
     assert re.status_code == 200
     response = json.loads(re.text)
     assert 'contacts' in response
-    assert response['full_name'] != 'John Doe'
+    assert response['full_name'] != 'Juan Doş'
 
     # test updating user
-    re = requests.put(api_v0('users/'+user_name), json={'full_name': 'John Doe', 'time_zone': 'PDT'})
+    re = requests.put(api_v0('users/'+user_name), json={'full_name': 'Juan Doş', 'time_zone': 'PDT'})
     assert re.status_code == 204
 
     # test updating user contacts
-    re = requests.put(api_v0('users/'+user_name), json={'full_name': 'John Doe', 'contacts': {'call': '+1 333-333-3339'}})
+    re = requests.put(api_v0('users/'+user_name), json={'full_name': 'Juan Doş', 'contacts': {'call': '+1 333-333-3339'}})
     assert re.status_code == 204
 
     # make sure update has gone through, test get
     re = requests.get(api_v0('users/'+user_name))
     assert re.status_code == 200
     response = re.json()
-    assert response['full_name'] == 'John Doe'
+    assert response['full_name'] == 'Juan Doş'
 
     user_id = response['id']
     re = requests.get(api_v0('users?id=%s' % user_id))
     assert re.status_code == 200
     response = re.json()
-    assert response[0]['full_name'] == 'John Doe'
+    assert response[0]['full_name'] == 'Juan Doş'
 
     re = requests.get(api_v0('users'), params={'name': user_name, 'fields': ['full_name', 'time_zone', 'contacts']})
     assert re.status_code == 200
     response = json.loads(re.text)
-    assert response[0]['full_name'] == 'John Doe'
+    assert response[0]['full_name'] == 'Juan Doş'
     assert response[0]['time_zone'] == 'PDT'
     assert response[0]['contacts']['call'] == '+1 333-333-3339'
 

ops/daemons/uwsgi-docker.yaml

@@ -1,6 +1,6 @@
 prod:
-    plugins: python,logfile
-    chdir: /home/oncall/source/src
+    plugins: python3,logfile
+    chdir: /home/oncall/source
     socket: /home/oncall/var/run/uwsgi.sock
     chmod-socket: 666
     master: True

ops/entrypoint.py

@@ -14,7 +14,7 @@
 
 
 def load_sqldump(config, sqlfile, one_db=True):
-    print 'Importing %s...' % sqlfile
+    print('Importing %s...' % sqlfile)
     with open(sqlfile) as h:
         cmd = ['/usr/bin/mysql', '-h', config['host'], '-u',
                config['user'], '-p' + config['password']]
@@ -24,18 +24,18 @@ def load_sqldump(config, sqlfile, one_db=True):
         proc.communicate()
 
         if proc.returncode == 0:
-            print 'DB successfully loaded ' + sqlfile
+            print('DB successfully loaded ' + sqlfile)
             return True
         else:
-            print ('Ran into problems during DB bootstrap. '
+            print(('Ran into problems during DB bootstrap. '
                    'oncall will likely not function correctly. '
-                   'mysql exit code: %s for %s') % (proc.returncode, sqlfile)
+                   'mysql exit code: %s for %s') % (proc.returncode, sqlfile))
             return False
 
 
 def wait_for_mysql(config):
-    print 'Checking MySQL liveness on %s...' % config['host']
-    db_address = (config['host'], 3306)
+    print('Checking MySQL liveness on %s...' % config['host'])
+    db_address = (config['host'], config['port'])
     tries = 0
     while True:
         try:
@@ -45,17 +45,17 @@ def wait_for_mysql(config):
             break
         except socket.error:
             if tries > 20:
-                print 'Waited too long for DB to come up. Bailing.'
+                print('Waited too long for DB to come up. Bailing.')
                 sys.exit(1)
 
-            print 'DB not up yet. Waiting a few seconds..'
+            print('DB not up yet. Waiting a few seconds..')
             time.sleep(2)
             tries += 1
             continue
 
 
 def initialize_mysql_schema(config):
-    print 'Initializing oncall database'
+    print('Initializing oncall database')
     # disable one_db to let schema.v0.sql create the database
     re = load_sqldump(config, os.path.join(dbpath, 'schema.v0.sql'), one_db=False)
     if not re:
@@ -71,7 +71,7 @@ def initialize_mysql_schema(config):
         sys.stderr.write('Failed to load dummy data.')
 
     with open(initializedfile, 'w'):
-        print 'Wrote %s so we don\'t bootstrap db again' % initializedfile
+        print('Wrote %s so we don\'t bootstrap db again' % initializedfile)
 
 
 def main():
@@ -88,7 +88,7 @@ def main():
             initialize_mysql_schema(mysql_config)
 
     os.execv('/usr/bin/uwsgi',
-             ['', '--yaml', '/home/oncall/daemons/uwsgi.yaml:prod'])
+             ['/usr/bin/uwsgi', '--yaml', '/home/oncall/daemons/uwsgi.yaml:prod'])
 
 
 if __name__ == '__main__':

setup.py

@@ -14,9 +14,9 @@
     packages=setuptools.find_packages('src'),
     include_package_data=True,
     install_requires=[
-        'falcon==1.1.0',
+        'falcon==1.4.1',
         'falcon-cors',
-        'gevent',
+        'gevent==1.4.0',
         'ujson',
         'sqlalchemy',
         'PyYAML',

src/oncall/api/__init__.py

@@ -1,8 +1,6 @@
 # Copyright (c) LinkedIn Corporation. All rights reserved. Licensed under the BSD-2 Clause license.
 # See LICENSE in the project root for license information.
 
-from __future__ import absolute_import
-
 from falcon import HTTPNotFound
 
 

src/oncall/api/v0/event.py

@@ -12,7 +12,7 @@
 )
 from ...constants import EVENT_DELETED, EVENT_EDITED
 
-from events import columns, all_columns
+from .events import columns, all_columns
 
 update_columns = {
     'start': '`start`=%(start)s',

src/oncall/api/v0/event_swap.py

@@ -76,7 +76,7 @@ def on_post(req, resp):
         events = events_0 + events_1
         # Validation checks
         now = time.time()
-        if any(map(lambda ev: ev['start'] < now - constants.GRACE_PERIOD, events)):
+        if any([ev['start'] < now - constants.GRACE_PERIOD for ev in events]):
             raise HTTPBadRequest('Invalid event swap request',
                                  'Cannot edit events in the past')
         if len(set(ev['team_id'] for ev in events)) > 1:

src/oncall/api/v0/events.py

@@ -166,7 +166,7 @@ def on_get(req, resp):
     cursor = connection.cursor(db.DictCursor)
 
     # Build where clause. If including subscriptions, deal with team parameters later
-    params = req.params.viewkeys() - TEAM_PARAMS if include_sub else req.params
+    params = req.params.keys() - TEAM_PARAMS if include_sub else req.params
     for key in params:
         val = req.get_param(key)
         if key in constraints:
@@ -176,7 +176,7 @@ def on_get(req, resp):
     # Deal with team subscriptions and team parameters
     team_where = []
     subs_vals = []
-    team_params = req.params.viewkeys() & TEAM_PARAMS
+    team_params = req.params.keys() & TEAM_PARAMS
     if include_sub and team_params:
 
         for key in team_params:

src/oncall/api/v0/ical.py

@@ -47,7 +47,7 @@ def events_to_ical(events, identifier):
                       '%s %s shift: %s' % (event['team'], event['role'], full_name))
         cal_event.add('description',
                       '%s\n' % full_name +
-                      '\n'.join(['%s: %s' % (mode, dest) for mode, dest in user['contacts'].iteritems()]))
+                      '\n'.join(['%s: %s' % (mode, dest) for mode, dest in user['contacts'].items()]))
 
         # Attach info about the user oncall
         attendee = vCalAddress('MAILTO:%s' % user['contacts'].get('email'))