Schedule Loading of Shapefile into a PostgreSQL Database

The code below automatically loads the shapefile found in a certain directory into a PostgreSQL database within a given interval(in this example 2 minutes).

Downside (in which I'm trying to figure out):
What if the filename of the shapefile changes?

Anyway, this suits of what I need right now.

__author__ = 'Matt'
import os
from django.db import connection
from django.contrib.gis.utils import LayerMapping
from .models import FloodHazard
import datetime
import time
import logging
from apscheduler.scheduler import Scheduler

logging.basicConfig()
# Start the scheduler
sched = Scheduler()
sched.daemonic = False
sched.start()

#ogrinspect evidensapp/data/flood_hazard/FloodHazard.shp FloodHazard --srid=32651 --mapping --multi
# Auto-generated `LayerMapping` dictionary for FloodHazard model
floodhazard_mapping = {
    'hazard': 'Hazard',
    'date_field': 'Date_',
    'geom': 'MULTIPOLYGON'
}

bound_shape = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                           'data/flood_hazard/FloodHazard.shp'))
exist = os.path.isfile(bound_shape)

if exist:
    def run(verbose=True):
        try:
            cursor = connection.cursor()
            cursor.execute("TRUNCATE TABLE evidensapp_floodhazard RESTART IDENTITY")
            lm = LayerMapping(FloodHazard, bound_shape, floodhazard_mapping,
                              transform=False, encoding='iso-8859-1')
            lm.save(strict=True, verbose=verbose)
            print(datetime.datetime.now())
            print(exist)
            time.sleep(60)

        except FloodHazard.DoesNotExist:
            lm = LayerMapping(FloodHazard, bound_shape, floodhazard_mapping,
                              transform=False, encoding='iso-8859-1')
            lm.save(strict=True, verbose=verbose)
            print(datetime.datetime.now())
            time.sleep(60)

    # Schedules job_function to be run every 2 minutes
    sched.add_interval_job(run, minutes=2)

else:
    print(exist)
    sched.shutdown()

Change the model name and path/directory. As you noticed that if the table has an existing rows, it performed a TRUNCATE query to save memory and the system that I'm developing need the latest Flood Hazard Map. And also, it gives an error if you load a shapefile on the table with existing rows (Why?).

Next task to do is, check if the file has been modified otherwise stop the schedule.

This is based on this question and using Advanced Python Scheduler and LayerMapping utility of Django.


UPDATE:  Here is the link of the updated code.

Popular posts from this blog

Auto-calculate your Splinterlands Assets/Income

Creating a Simple Button for Demo

Splinterlands: Water Deck for Beginners