Setting Up COPO Project Locally with Docker
The central instance of COPO runs on a pool of three virtual machines. The following set up instructions are structured in a similar manner using one node. Feel free to make changes for a bigger or smaller pool.
Clone the GitHub COPO project repository.
Visual Studio Code (VSCode) is recommended for running the COPO project after having sat up the Docker environment. You can download VSCode from here for your local machine.
Note
There are a number of parameters in the command below that need to be updated or you may want to change for your local deployment. Please read through carefully.
Hint
A Python virtual environment is not required to run the COPO project application locally since the project is running via Docker containers. However, if you would like to run the project locally without Docker and within a Python virtual environment, please refer to the COPO project setup with PyCharm documentation.
Warning
The ENA_SERVICE environment variable is set to the ENA development server. All submission to this
server will be deleted after 24 hours. To submit to the production ENA server remove dev and set prod.
Install Docker
Follow the official Docker installation documentation for your distribution.
Make changes to your firewall, iptables and security groups to serve a website, use Docker swarm and redis. The port number will depend on your setup and if you choose to use the default ports for each service.
Initialise Docker Swarm
docker swarm init --advertise-addr <the IP of the machine you want to advertise>
e.g. $ docker swarm init 127.0.0.1 where 127.0.0.1 is the IP address of localhost.
This command will return a token. You need this token to make the other VMs join the swarm if you plan to use more than one one node. On the other machines run:
docker-swarm join --token <the token returned by the previous command> <the IP advertised previously>:2377
Check out the Docker documentation if you want to change the default port.
Note
You may need to change the following instructions depending on the server you are deploying to. Please consider which services need to run in the backend or frontend network.
Add Docker Node Labels
Label the node as a web service, nginx service, mongo service, postgres service and backup service. The web service and nginx service will be deployed on the frontend network. The mongo service, postgres service and backup service will be deployed on the backend network.
Adding One Docker Node Label
Hint
$HOSTNAME is a bash variable that returns the hostname of the machine. You can use the hostname of the machine
instead of the variable. In a Linux OS, you can run $ echo $HOSTNAME to get the hostname.
If you are using more than one node, you will need to label the other nodes as well.
docker node update \
--label-add web-service=true \
--label-add nginx-service=true \
--label-add mongo-service=true \
--label-add postgres-service=true \
--label-add backup-service=true \
$HOSTNAME
Adding More than Docker One Node Labels
If you are using more than one Docker node, you can label the other nodes as follows:
Node 1
docker node update \
--label-add web-service=true \
--label-add nginx-service=true \
--label-add mongo-service=false \
--label-add postgres-service=false \
--label-add backup-service=false \
copo-0
Node 2
docker node update \
--label-add web-service=false \
--label-add nginx-service=false \
--label-add mongo-service=false \
--label-add postgres-service=true \
--label-add backup-service=true \
copo-1
Node 3
docker node update \
--label-add web-service=false \
--label-add nginx-service=false \
--label-add mongo-service=true \
--label-add postgres-service=false \
--label-add backup-service=false \
copo-2
Create Docker Volumes
Docker volumes are used to persist data via the plugin, local-persist. This ensures that the data is not lost when the containers are restarted. Volumes are created on the swarm manager.
Substitute the paths in commands before running it.
Hint
You may need to install curl before running the command below. You can install curl by running
$ sudo apt-get install curl.
You may need to install the local-persist plugin to persist volumes before running the command below. You can install
it by running:
$ curl -fsSL https://raw.githubusercontent.com/MatchbookLab/local-persist/master/scripts/install.sh | sudo docker
docker volume create -d local-persist -o mountpoint=/path/to/web-data --name=web-data
docker volume create -d local-persist -o mountpoint=/path/to/static-data --name=static-data
docker volume create -d local-persist -o mountpoint=/path/to/submission-data --name=submission-data
docker volume create -d local-persist -o mountpoint=/path/to/logs-data --name=logs-data
docker volume create -d local-persist -o mountpoint=/path/to/mongo-data --name=mongo-backup
docker volume create -d local-persist -o mountpoint=/path/to/postgres-data --name=postgres-backup
docker volume create mongo-data
docker volume create postgres-data
Create Networks on Docker Swarm Manager
On the swarm manager create two networks - one for the frontend and one for the backend. The front-end network will be used by the web service and the nginx service while the backend network will be used by the database services.
docker network create -d overlay copo-frontend-network
docker network create -d overlay copo-backend-network
docker network ls
Create Secrets on Docker Swarm Manager
All secrets are file based. You will need to create some of the keys with third parties and choose passwords before proceeding with the COPO setup.
docker secret create copo_mongo_initdb_root_password copo_mongo_initdb_root_password
docker secret create copo_mongo_user_password copo_mongo_user_password
docker secret create copo_postgres_user_password copo_postgres_user_password
docker secret create copo_web_secret_key copo_web_secret_key
docker secret create copo_orcid_secret_key copo_orcid_secret_key
docker secret create copo_orcid_client_id copo_orcid_client_id
docker secret create copo_figshare_consumer_secret_key copo_figshare_consumer_secret_key
docker secret create copo_figshare_client_id_key copo_figshare_client_id_key
docker secret create copo_figshare_client_secret_key copo_figshare_client_secret_key
docker secret create copo_google_secret_key copo_google_secret_key
docker secret create copo_twitter_secret_key copo_twitter_secret_key
docker secret create copo_facebook_secret_key copo_facebook_secret_key
docker secret create copo_webin_user copo_webin_user
docker secret create copo_webin_user_password copo_webin_user_password
docker secret create copo-project.crt copo-project.crt
docker secret create copo-project.key copo-project.key
docker secret create copo_nih_api_key copo_nih_api_key
docker secret create copo_public_name_service_api_key copo_public_name_service_api_key
docker secret create copo_mail_password copo_mail_password
docker secret create copo_bioimage_path copo_bioimage_path
docker secret create ecs_secret_key ecs_secret_key
docker secret ls
Build COPO Project Docker Image
Download the Dockerfile Dockerfile_local.
for your local machine.
Place the downloaded Dockerfile in the COPO project root directory.
Alternatively, you can use the Dockerfile present in the root project directory Dockerfile for demonstration environment.
Note
The Dockerfile is configured to build the local_copo_web container image with the tag, v1.0.1. If you have
a different tag and container name, you will need to change the Dockerfile accordingly.
If you are using a Mac OS, download the Dockerfile_mac.
Visit here for more information on how to build an application with a Docker image.
cd <path-to-project-root-directory>/COPO
docker build . -f Dockerfile_local -t local_copo_web:v1.0.1
Deploy Docker Image on Docker Swarm Manager
The redis, postgres and mongo Docker services are created on the swarm manager. Download the
local compose file file to create the services.
Alternatively, you can download compose file for demonstration environment.
Replace the <path-to-project-root-directory> with the absolute path to the COPO project root directory.
Note
The Docker compose file is configured to use the secrets and volumes created above. If you have used different names for the secrets and volumes, you will need to change the compose file accordingly.
If you are using a Mac OS, download the Mac compose file.
The following commands should be run from the root directory of the COPO project.
Warning
The ENA_SERVICE environment variable is set to the ENA development server. All submission to this server will be deleted after 24hours. To submit to the production ENA server remove "dev"
nano local_copo.compose.yaml
Update the tag, save the file then, exit by inputting: CTRL + O then, ENTER then, CTRL + X
docker stack deploy --compose-file '<path-to-file>/local_copo.compose.yaml' copo
Compose file to configure COPO project application locally with Docker
version: "3.8"
services:
web:
image: local_copo_web:v1.0.1
command: >
bash -c "code-server serve-local --disable-telemetry --auth none --host 0.0.0.0 --port 8100"
ports:
- "8000:8000"
- "8100:8100"
volumes:
- <path-to-project-root-directory>/COPO:/copo
environment:
- ENVIRONMENT_TYPE=prod
- ASPERA_PLUGIN_DIRECTORY=aspera_linux_plugin
- SECRET_KEY_FILE=/run/secrets/copo_web_secret_key
- MEDIA_PATH=media/
- DEBUG=true
- REDIS_HOST=copo_redis
- REDIS_PORT=6379
- WEBIN_USER_FILE=/run/secrets/copo_webin_user
- WEBIN_USER_PASSWORD_FILE=/run/secrets/copo_webin_user_password
- ENA_SERVICE=https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/
- MONGO_USER=copo_user
- MONGO_USER_PASSWORD_FILE=/run/secrets/copo_mongo_user_password
- MONGO_DB=copo_mongo
- MONGO_HOST=copo_mongo
- MONGO_PORT=27017
- MONGO_MAX_POOL_SIZE=100
- POSTGRES_DB=copo
- POSTGRES_USER=copo_user
- POSTGRES_PORT=5432
- POSTGRES_SERVICE=copo_postgres
- POSTGRES_PASSWORD_FILE=/run/secrets/copo_postgres_user_password
- ORCID_SECRET_FILE=/run/secrets/copo_orcid_secret_key
- ORCID_CLIENT_FILE=/run/secrets/copo_orcid_client_id
- FIGSHARE_CONSUMER_SECRET_FILE=/run/secrets/copo_figshare_consumer_secret_key
- FIGSHARE_CLIENT_ID_FILE=/run/secrets/copo_figshare_client_id_key
- FIGSHARE_CLIENT_SECRET_FILE=/run/secrets/copo_figshare_client_secret_key
- GOOGLE_SECRET_FILE=/run/secrets/copo_google_secret_key
- TWITTER_SECRET_FILE=/run/secrets/copo_twitter_secret_key
- FACEBOOK_SECRET_FILE=/run/secrets/copo_facebook_secret_key
- NIH_API_KEY_FILE=/run/secrets/copo_nih_api_key
- PUBLIC_NAME_SERVICE_API_KEY_FILE=/run/secrets/copo_public_name_service_api_key
- MAIL_PASSWORD_FILE=/run/secrets/copo_mail_password
- MAIL_PORT=587
- MAIL_ADDRESS=data@earlham.ac.uk
- MAIL_SERVER=outlook.office365.com
- MAIL_USERNAME=eidata@nbi.ac.uk
- ALLOWED_HOSTS=
- PUBLIC_NAME_SERVICE=https://id.tol.sanger.ac.uk/api/v2/
- ENA_ENDPOINT_REPORT=https://wwwdev.ebi.ac.uk/ena/submit/report/samples
- ASPERA_PATH=/root/.aspera/cli
- BIOIMAGE_PATH_FILE=/run/secrets/copo_bioimage_path
- BIOIMAGE_SERVER=bsaspera_w@hx-fasp-1.ebi.ac.uk
- ECS_SECRET_KEY_FILE=/run/secrets/ecs_secret_key
- ECS_ACCESS_KEY_ID=copo@nbi.ac.uk
- ECS_ENDPOINT=http://ei-copo.obj-data.nbi.ac.uk
- ENA_V2_SERVICE_SYNC=https://wwwdev.ebi.ac.uk/ena/submit/webin-v2/submit
- ENA_V2_SERVICE_ASYNC=https://wwwdev.ebi.ac.uk/ena/submit/webin-v2/submit/queue
- B2DROP_PERMITS=/copo/b2drop/permits
depends_on:
- redis
- postgres
- mongo
networks:
- copo-frontend-network
- copo-backend-network
deploy:
replicas: 1
restart_policy:
condition: on-failure
placement:
constraints:
- "node.labels.web-service==true"
resources:
limits:
cpus: '4'
memory: 7GB
reservations:
cpus: '2'
memory: 2GB
update_config:
parallelism: 1
delay: 10s
secrets:
- copo_web_secret_key
- copo_postgres_user_password
- copo_mongo_user_password
- copo_google_secret_key
- copo_facebook_secret_key
- copo_twitter_secret_key
- copo_orcid_secret_key
- copo_orcid_client_id
- copo_figshare_client_id_key
- copo_figshare_client_secret_key
- copo_figshare_consumer_secret_key
- copo_webin_user
- copo_webin_user_password
- copo_nih_api_key
- copo_public_name_service_api_key
- copo_mail_password
- copo_bioimage_path
- ecs_secret_key
redis:
image: copo/copo-redis:redis-v6.2.6.18
networks:
- copo-frontend-network
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
placement:
constraints:
- "node.labels.web-service==true"
update_config:
parallelism: 1
delay: 10s
resources:
limits:
cpus: "0.5"
memory: 1GB
reservations:
cpus: "0.3"
memory: 512mb
postgres:
image: postgres:9.6
networks:
- copo-backend-network
volumes:
- postgres-data:/var/lib/postgresql/data
secrets:
- copo_postgres_user_password
ports:
- "5432:5432"
environment:
- POSTGRES_DB=copo
- POSTGRES_USER=copo_user
- POSTGRES_PASSWORD_FILE=/run/secrets/copo_postgres_user_password
deploy:
replicas: 1
endpoint_mode: dnsrr
restart_policy:
condition: on-failure
placement:
constraints:
- "node.labels.postgres-service==true"
update_config:
parallelism: 1
delay: 10s
resources:
limits:
cpus: "0.5"
memory: 1GB
reservations:
cpus: "0.3"
memory: 512mb
mongo:
image: copo/copo-mongo:20
networks:
- copo-backend-network
volumes:
- mongo-data:/data/db
secrets:
- copo_mongo_initdb_root_password
- copo_mongo_user_password
ports:
- "27017:27017"
environment:
- MONGO_INITDB_ROOT_USERNAME=copo_admin
- MONGO_INITDB_ROOT_PASSWORD_FILE=/run/secrets/copo_mongo_initdb_root_password
- MONGO_USER=copo_user
- MONGO_USER_PASSWORD_FILE=/run/secrets/copo_mongo_user_password
- MONGO_DB=copo_mongo
deploy:
replicas: 1
#endpoint_mode: dnsrr
restart_policy:
condition: on-failure
placement:
constraints:
- "node.labels.mongo-service==true"
update_config:
parallelism: 1
delay: 10s
resources:
limits:
cpus: "0.8"
memory: 2GB
reservations:
cpus: "0.4"
memory: 512mb
networks:
copo-backend-network:
external: true
copo-frontend-network:
external: true
secrets:
copo_web_secret_key:
external: true
copo_postgres_user_password:
external: true
copo_mongo_user_password:
external: true
copo_google_secret_key:
external: true
copo_figshare_client_id_key:
external: true
copo_facebook_secret_key:
external: true
copo_twitter_secret_key:
external: true
copo_orcid_secret_key:
external: true
copo_orcid_client_id:
external: true
copo_figshare_client_secret_key:
external: true
copo_figshare_consumer_secret_key:
external: true
copo_webin_user:
external: true
copo_webin_user_password:
external: true
copo_nih_api_key:
external: true
copo_public_name_service_api_key:
external: true
copo_mail_password:
external: true
copo_mongo_initdb_root_password:
external: true
copo-project.crt:
external: true
copo-project.key:
external: true
copo_bioimage_path:
external: true
ecs_secret_key:
external: true
volumes:
web-data:
external: true
static-data:
external: true
submission-data:
external: true
logs-data:
external: true
mongo-backup:
external: true
postgres-backup:
external: true
mongo-data:
external: true
postgres-data:
external: true
b2drop-permits:
external: true
docker service ls
docker ps
docker ps -a
docker image inspect local_copo_web:v1.0.1
copo_web Docker container (if it is not started)
docker service scale copo_web=1
copo_web Docker container
docker service scale copo_web=0
Set up PostgreSQL database
In the terminal, navigate to the root directory of the COPO project application then, run the following commands.:
Hint
Retrieve the local_copo_web container ID by running the $ docker ps command below in the root project
directory of the COPO project application in the terminal for the local_copo_web:v1.0.1 Docker image.
docker run --name postgresql -e POSTGRES_USER=<postres-username> -e POSTGRES_PASSWORD=<postres-password> -p 5432:5432 -v /data:/var/lib/postgresql/data -d postgres:9.6
Replace <postgres-username> and <postgres-password> with the username and password for PostgreSQL database respectively.
docker exec -it <local_copo_web-container-id> bash
python manage.py makemigrations
python manage.py makemigrations chunked_upload
python manage.py makemigrations allauth
python manage.py migrate
python manage.py setup_groups
python manage.py setup_schemas
python manage.py createcachetable
python manage.py social_accounts
python manage.py makemigrations
python manage.py makemigrations chunked_upload
python manage.py makemigrations allauth
python manage.py migrate
python manage.py setup_groups
python manage.py setup_schemas
python manage.py createcachetable
python manage.py social_accounts
python3 manage.py createsuperuser
Enter the required details to create the Django admin/superuser. The Django admin/superuser can log into the COPO project application from the local Django admin website.
CTRL + P
CTRL + Q
exit
The commands above can be accessed in the 3_db_setup.sh script.
This file is located in the set_up_scripts directory of the COPO project root directory.
In the following steps, we will create the PostgreSQL database for the COPO project application in the root directory of the project.
Hint
Retrieve the PostgreSQL container ID by running the command below in the root project directory of the COPO project
application in the terminal for the postgres:9.6 Docker image:
$ docker ps
docker exec -it <postgres-container-id> bash
psql -h 'localhost' -U $POSTGRES_USER -d 'copo' -c 'DELETE FROM socialaccount_socialapp_sites'
psql -h 'localhost' -U $POSTGRES_USER -d 'copo' -c 'DELETE FROM django_site'
psql -h 'localhost' -U $POSTGRES_USER -d 'copo' -c 'DELETE FROM socialaccount_socialapp'
psql -h 'localhost' -U $POSTGRES_USER -d 'copo' -c "INSERT INTO django_site (id, domain, name) VALUES (1, 'www.copo-project.org', 'www.copo-project.org')"
psql -h 'localhost' -U $POSTGRES_USER -d 'copo' -c "INSERT INTO socialaccount_socialapp (id, provider, name, client_id, secret, key) VALUES (1, 'orcid', 'Orcid', '$ORCID_CLIENT_ID', '$ORCID_SECRET', '')"
psql -h 'localhost' -U $POSTGRES_USER -d 'copo' -c 'INSERT INTO socialaccount_socialapp_sites (id, socialapp_id, site_id) VALUES (1, 1, 1)'
The commands above can be accessed in the 3_db_setup.sh script.
This file is located in the set_up_scripts directory of the COPO project root directory.
Updating COPO Website Service
The COPO project is updated frequently and as such is under active development. To update your instance to a newer
(or the latest) version, download the
local compose file or the compose file for
demonstration environment
on the swarm manager or root directory of the project if you have one node.
Then, run the following commands in the terminal:
Note
The Docker tag below needs to be changed to reflect the most recent version available in DockerHub.
Please check the latest version there. You can safely ignore the *feature tags as they are not stable releases.
For stable releases look for *rc.
Hint
Retrieve the copo-web container ID by running the $
docker pscommand below in the root project directory of the COPO project application in the terminal for the copo/copo-web:v1.0.1 Docker image.To check if the web service is running, run the command below in the root project directory of the COPO project application in the terminal for the copo/copo-web:v1.0.2 Docker image: $
docker logs -f <container-ID-for-updated-copo-web>If you update often we recommend taking care of removing old docker images regularly.
nano local_copo.compose.yaml
Update the Docker tag, save the file then, exit it by inputting: CTRL + O then, ENTER then, CTRL + X
docker stack deploy --compose-file '<path-to-file>/local_copo.compose.yaml' copo
Launch COPO Website
The COPO project application can be accessed locally on port 8100 via the VSCode browser extension.
Within the VSCode IDE browser, add a new configuration by following the steps below:
Navigate to Run -> Add Configuration
Edit the launch.json file that is created with the following file contents:
VSCode configuration file
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Django",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/manage.py",
"env" : {
"PYTHONPATH1" : "${workspaceFolder}/lib:${PYTHONPATH}"
},
"args": [
"runserver",
"0.0.0.0:8000"
],
"django": true,
"justMyCode": false
},
{
"name": "Python: Django makemigration",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/manage.py",
"env" : {
"PYTHONPATH" : "${workspaceFolder}/lib:${PYTHONPATH}"
},
"args": [
"makemigration"
],
"django": true,
"justMyCode": false
},
{
"name": "Python: Celery Workers",
"type": "python",
"request": "launch",
"module": "celery",
"console": "integratedTerminal",
"env" : {
"GEVENT_SUPPORT": "True"
},
"args": ["-A", "web", "worker", "-l", "debug", "-P", "gevent", "--concurrency" , "10", "-Q", "celery", "-E"],
"justMyCode": true
},
{
"name": "Python: Celery Beat",
"type": "python",
"request": "launch",
"module": "celery",
"console": "integratedTerminal",
"args": ["-A", "web", "beat", "-l", "debug"]
}
]
}
The COPO project application can be accessed locally on on port 80000 or on port 81000.
Note
Install required VSCode extensions for the COPO project application by following the steps below:
Navigate to the Extensions tab on the left-hand side of the VSCode IDE
Search for and install the following extensions:
Python (required)
If your local machine is restarted, you will need to start the Docker container again at startup. To do this, run the
following command in the terminal: $ docker start <container-ID-for-copo-web>. You can retrieve the container ID
by running the command below in the root project directory of the COPO project application in the terminal for the
copo/copo-web:v1.0.2 Docker image: $ docker ps
Tips
Enable Manage Unsafe Repositories in Source Control in VSCode browser application to allow VSCode to access the COPO GitHub project repository.
-
Install the following VSCode extensions:
GitHub Copilot
Prettier - Code formatter
Git Extension Pack
git config --global user.name "<GitHub-username>"
git config --global user.email "<GitHub-email-address>"
git tag <tagname>
git push origin <tagname>
git tag -d <tag-name>
docker ps
sudo systemctl start docker
sudo systemctl stop docker
sudo systemctl restart docker
docker start
docker stop
docker restart
docker exec it <container-ID> bash
docker version
docker ps -a
docker volume create <volume-name>