Docker Installation

Initial Setup

Clone

• Clone the repository, and change directory to raw-data-api folder on your computer.

git clone https://github.com/hotosm/raw-data-api.git
cd raw-data-api

Configurations

config.txt approach

• Create config.txt inside / folder. You can use any of the appropriate commands below or you use your familiar methods in your code editor/file explorer. (if you are using docker-compose , you can edit docker-compose-config.txt)

touch config.txt #Linux
wsl touch config.txt #Windows with WSL
echo >> config.txt #Windows without WSL

.env approach

if you prefer configurations as env variables you can put them in .env and pass it to dockerfile or export them

• Database configuration:

• To use the default database(with sample data) , Run docker compsoe and update the docker-compose-config.txt

• To use a local postgres (with postgis enabled) database, you can follow the instruction on how to set it up with raw data here. or export them as system env variables

• OSM Authentication:

• Follow this step to setup OSM OAuth: Setup Oauth Block.

• Update your config.txt with the [OAUTH] block from the step above.

Run Docker

You can either use full composed docker-compose directly or you can build docker containers manually .

Spin up the containers using docker compose

docker-compose up -d --build

OR

Run Docker without docker compose for development

• Build your image

  docker build -t rawdataapi:latest . 
  

• Run API

  docker run --name rawdataapi -p 8000:8000 \
      -v "$(pwd)/API:/home/appuser/API" \
      -v "$(pwd)/src:/home/appuser/src" \
      -v "$(pwd)/config.txt:/home/appuser/config.txt" \
      rawdataapi:latest \
      uvicorn API.main:app --reload --host "0.0.0.0" --port "8000" --no-use-colors --proxy-headers
  

• Run Workers

  docker run --name rawdata-worker \
      -v "$(pwd)/API:/home/appuser/API" \
      -v "$(pwd)/src:/home/appuser/src" \
      -v "$(pwd)/config.txt:/home/appuser/config.txt" \
      rawdataapi:latest \
      celery --app API.api_worker worker --loglevel=INFO --queues="raw_ondemand" -n 'default_worker'
  

• Run Flower on port 5555

  docker run --name rawdata-flower -p 5555:5555 \
      -v "$(pwd)/API:/home/appuser/API" \
      -v "$(pwd)/src:/home/appuser/src" \
      -v "$(pwd)/config.txt:/home/appuser/config.txt" \
      rawdataapi:latest \
      celery --broker=redis://redis:6379// --app API.api_worker flower --port=5555 --queues="raw_daemon,raw_ondemand"
  

Development instruction: If you are running Dockerfile only for the API , Have postgresql redis installed on your machine directly then you should change following :

• Change --broker Host address in flower command (You can use redis if it is docker compsoe container or use redis://host.docker.internal:6379/0 if you want API container to connect to your localhsot , Follow #troubleshoot section for more)
• Change DB Host & Celery broker url accordingly with the same logic

Note:

In above example we have attached our working dir to containers along with config.txt for efficiency in development environment only . It is recommended to use proper docker copy as stated in dockerfile and system environement variables instead of config.txt in Production

Check the servers

By default, Uvicorn would be running on port 8000, Redis on default port(i.e 6379), Celery with a worker and Flower on port 5555.

• API Documentation

Visit the route below to access the API documentation.

http://127.0.0.1:8000/v1/docs

API docs will be displayed like this upon successfull server startup

• Flower dashboard

Vist the route below to access the Flower dashboard

http://127.0.0.1:5555/

Flower dashboard will look like this on successfull installation with a worker online. (Change the port accordingly to your command)

Proceed to the readme for further configurations: here.

Troubleshooting

NOTE: If any of the solutions provided below does not work for you, please don't hesistate to open an issue.

• If you can't connect to your local postgres database from raw-data-api.

Since export-tool-api is running in docker containers, it means if you setup your config.txt file to use a local postgres database on your machine, the database port may not be accessible as localhost from the docker containers. To solve this, docker needs to connect to your local network. Try any of these hacks to troubleshoot the issue:

1. Option one :

   • For windows/ Mac docker user Replace localhost with host.docker.internal – This resolves to the outside host and lets you connect to your machine's localhost through container. For example if postgres is running on your machine on port 5432 , docker container can connect from host.docker.internal:5432
   • For linux user : Linux users can enable host.docker.internal via the --add-host flag for docker run. Start your containers with this flag to expose the host string: docker run -d --add-host host.docker.internal:host-gateway my-container:latest

2. Option two :

   • Find your network ip address (for linux/mac you can use ifconfig -l | xargs -n1 ipconfig getifaddr ) and use your ip address as the database host instead of localhost in config.txt file.
   • If connection still fails : You may need to edit your postgres config file ( ask postgres where it is using this query: show config_file; ) and edit/enable listen_addresses = '*' inside postgresql.conf . Also add host all all 0.0.0.0/0 trust in pg_hba.conf.