Skip to content

Raw Data API

Unit test Build

Documentation: hotosm.github.io/raw-data-api

Source Code: github.com/hotosm/raw-data-api

Raw Data API is a set of high-performant APIs for transforming and exporting OpenStreetMap (OSM) data in different GIS file formats.

Features

  • Fast: Built on top of FastAPI
  • OAUTH 2.0 Authentication with OpenStreetMap(OSM)
  • Multiple GIS formats support via GDAL's ogr2ogr - see table below for currently supported formats. Out of which , GeoJSON Follows Own Raw Data API conversion script
Formats Status
Esri Shapefile ✔
KML ✔
Mbtiles ✔
FlatGeoBuf ✔
CSV ✔
GeoPackage ✔
PGDUMP ✔
GeoJSON ✔

Installation

To setup Backend Follow Backend_Installation

Raw Data API can be installed through docker or locally on your computer.

NOTE: The installation guide below is only tested to work on Ubuntu, we recommend using docker for other operating systems.

Local Installation Requirements.

  • Install GDAL on your computer using the command below:
sudo apt-get update && \
sudo apt-get -y install gdal-bin python3-gdal && \
sudo apt-get -y autoremove && \
sudo apt-get clean
  • Install redis on your computer using the command below:
sudo apt-get -y install redis
sudo apt-get -y install redis-tools # For client
  • Confirm Redis Installation
redis-cli

Type ping it should return pong.

If redis is not running check out its documentation

  • Clone the Raw Data API repository on your computer
git clone https://github.com/hotosm/raw-data-api.git
  • Navigate to the repository directory
cd raw-data-api
  • Install the python dependencies
pip install -r requirements.txt

Additional required configurations for Raw Data API

Setup the necessary configurations for Raw Data API from configurations.

Setup config.txt in project root.

Start the Server

uvicorn API.main:app --reload

Start Celery Worker

You should be able to start celery worker by running following command on different shell

celery --app API.api_worker worker --loglevel=INFO

Start flower for monitoring queue [OPTIONAL]

Raw Data API uses flower for monitoring the Celery distributed queue. Run this command on a different shell , if you are running redis on same machine your broker could be redis://localhost:6379//.

celery --broker=redis://redis:6379// --app API.api_worker flower --port=5000

After sucessfully starting the server, visit http://127.0.0.1:8000/v1/docs on your browser to view the API docs.

http://127.0.0.1:8000/v1/docs

Flower dashboard should be available on port 5000 on your localhost.

http://127.0.0.1:5000/

Basic Usage

  • Confirm that Authetication works

  • Hit the /auth/login/ endpoint

  • Hit the url returned on the response
  • You will get an access_token
  • You can use the access_token in all endpoints that requires authentication.

  • To check token pass token in /auth/me/. It should return your OpenStreetMap (OSM) profile

  • Try extracting some data:

You can use the /raw-data/current-snapshot/ endpoint with the following input to check both authentication, database connection and download the export:

curl -d '{"geometry":{"type":"Polygon","coordinates":[[[83.96919250488281,28.194446860487773],[83.99751663208006,28.194446860487773],   [83.99751663208006,28.214869548073377],[83.96919250488281,28.214869548073377],[83.96919250488281,28.194446860487773]]]}}' -H 'Content-Type: application/json'   http://127.0.0.1:8000/v2/raw-data/current-snapshot/

Tests

  • Raw Data API uses pytest for tests, navigate to the root directory and install package in editable mode:
pip install -e .
  • At this point you should have PostgreSQL + PostGIS extension enabled on your computer, now run Pytest:
py.test -v -s
  • Running individual tests
py.test -k test function name

Contribution & Development

see CONTRIBUTING

Workflow

Understand how Raw Data API works and connected pieces , see Workflow

License

see LICENSE

Authors

Created by HOTOSM and Friends