First, check out the repository and run dev/setup to install the
application and its dependencies in a Python virtual environment (that
lives inside of dev/venv in case you ever need to access it):
git clone [email protected]:newfs/goorchids-app.git
cd goorchids-app
git submodule init
git submodule update
dev/setup
Next, make sure that you can access your local PostgreSQL server, which
you can confirm with a quick psql -l, and then start up a Solr full
text index server with:
dev/start-solr
At this point the application should at least run, even though most pages will give errors if your database is not set up yet. To start the application, simply run:
dev/django runserver
You should then be able to visit the application at:
http://localhost:8000/
Before importing data into your database, ensure you have your AWS credentials set in your environment variables, which is often accomplished by sourcing a shell script kept outside the repository. This is needed to ensure that image importing will work.
If you are starting fresh and have no database set up yet, or want to
start over because some tables have changed or Sid had released a new
CSV file, then you can rebuild the database with these commands (the
first one will give an error if you do not have a gobotany database
already sitting in your way; in that case, ignore the error):
dropdb gobotany
createdb -E UTF8 gobotany
dev/django syncdb
dev/django migrate goorchids.core
dev/django createsuperuser
At this point you are done installing and should be able to test and develop the application!
You can load initial data by visiting /admin/core and following the
Import Data link. Choose the latest auth export to load user
data, click Import; Then choose the latest core export and
click Import to load the Orchid data.
If you ever need to activate the virtual environment so that Python prompts or scripts run from your shell have access to the Go Orchids application and its dependencies, then enter:
source dev/activate
If you want to rebuild our minified JavaScript in preparation for a deploy to production, run:
dev/jsbuild
Our various tests can be run with three commands:
dev/test-browser
dev/test-js
dev/test-python
Ensure you have Docker (tested with version 20.10.15) installed.
Fetch the repository and Git submodules:
git clone [email protected]:jazkarta/goorchids-app.git
cd goorchids-app
git submodule init
git submodule update
Build with:
docker compose build goorchids
Start with:
docker compose up -d
Run in foreground (usefull to debug with pdb):
docker compose exec goorchids python manage.py 0:8001
Run database migrations (only the first time):
docker compose exec goorchids python manage.py migrate
Create a solr core:
docker compose run --rm solr bash -c "bin/solr start && bin/solr create -c gobotany_solr_core -d /opt/solr/server/solr/configsets/basic_configs/"
Update solr config and schema:
docker compose up -d solr
docker compose run -v goorchids-app_solr_data:/opt/solr/ --rm goorchids python manage.py build_solr_schema --configure-directory=/opt/solr/gobotany_solr_core/conf --reload-core=gobotany_solr_core
Rebuild solr index:
docker compose run --rm goorchids python manage.py rebuild_index --noinput
Start by checking out this "goorchids-app" repository on your machine:
git clone [email protected]:jazkarta/goorchids-app.git
cd goorchids-app
Follow steps 1–3 at http://devcenter.heroku.com/articles/quickstart_
so that you can run the heroku command, then use the following
command to create and provision a new app on Heroku:
heroku create
heroku addons:add heroku-postgresql:hobby-basic
heroku addons:add memcachier:dev
heroku addons:add redistogo:nano
heroku addons:add scheduler:standard
heroku addons:add sendgrid:starter
heroku addons:add websolr:cobalt # Production only
heroku pg:wait
git push heroku master
Once the Postgres database is up and running, note its color (like "RED" or "SILVER"), and promote it to being the "main database" for the app:
heroku pg:promote <color>
Add three configuration variables to your Heroku app, so that Go Orchids will be able to scan its S3 image repository:
heroku config:add AWS_ACCESS_KEY_ID=...
heroku config:add AWS_SECRET_ACCESS_KEY=...
heroku config:add AWS_STORAGE_BUCKET_NAME=newfs
The application will now be up and running. You can find its URL with
the heroku apps:info command. When you visit, you will see an
exception, because the database tables that it needs have not yet been
created. To set up the database, run these commands:
heroku config:add DJANGO_SETTINGS_MODULE=goorchids.settings
heroku run django-admin.py syncdb --noinput
heroku run python -m goorchids.core.importer zipimport
heroku run bin/import-images.sh
heroku run bin/import-dkey.sh
Prepare Solr by first generating your Solr schema:
heroku run django-admin.py build_solr_schema > schema.xml
Once this file exists, you can visit the Heroku web site, navigate to
your app's configuration, select the addon "Websolr", choose the section
"Advanced Configuration", and paste in the contents of schema.xml
that you just created. Once the schema is installed (give it a few
minutes to make sure the change has the chance to propagate to WebSolr's
servers), you can build the Solr index and thereby activate the Go
Orchids site's search field:
heroku run django-admin.py rebuild_index --noinput
To run our Python tests you can either:
dev/test-python # to run all tests
dev/test-python api site # to hand-pick Django apps to test
To run our JavaScript tests, run:
dev/test-js # to run all tests
dev/test-js test/Filter.js # to select which modules to test
Our selenium-powered browser tests are intended to cover things that cannot be tested without a browser and JavaScript. To run them:
dev/test-browser # to run all tests
dev/test-browser.sh FilterFunctionalTests # select which tests
Detailed notes about testing under selenium can be found in:
externals/gobotany-app/gobotany/simplekey/testdir/README-SELENIUM.txt
The Go Orchids search feature uses Haystack and Solr.
Our unit and functional tests aim to ensure various aspects of the search feature including desired ranking.
Ranking relies mostly on Haystack document boost, as seen in several
places in our search_indexes.py. For more fine-grained control where
boost is not enough, some hidden repeated keywords are added to search
indexes such as in the search_text_species.txt template.
To adjust ranking: cycle through running the functional tests, adjusting
the boosts in search_indexes.py, and, if necessary, adjusting the
hidden-keyword sections at the end of search_*.txt templates. The Solr
Admin full interface, which allows examining details including ranking
scores, may also be helpful:
http://localhost:8983/solr/admin/form.jsp