Module API.api_worker

Variables

ALLOW_BIND_ZIP_FILTER

DEFAULT_HARD_TASK_LIMIT

DEFAULT_README_TEXT

DEFAULT_SOFT_TASK_LIMIT

ENABLE_SOZIP

ENABLE_TILES

EXPORT_PATH

HDX_HARD_TASK_LIMIT

HDX_SOFT_TASK_LIMIT

WORKER_PREFETCH_MULTIPLIER

celery

celery_backend

celery_broker_uri

use_s3_to_upload

Functions

create_readme_content

def create_readme_content(
    default_readme,
    polygon_stats
)

remove_file

def remove_file(
    path: str
) -> None

Used for removing temp file dir and its all content after zip file is delivered to user

zip_binding

def zip_binding(
    working_dir,
    exportname_parts,
    geom_dump,
    polygon_stats,
    default_readme
)

Classes

BaseclassTask

class BaseclassTask(
    /,
    *args,
    **kwargs
)

Base class for celery tasks

Args: celery (type): description

Ancestors (in MRO)

• celery.app.task.Task
• celery.app.task.Task

Descendants

• API.api_worker.process_custom_request

Class variables

MaxRetriesExceededError

OperationalError

Request

Strategy

abstract

acks_late

acks_on_failure_or_timeout

app

default_retry_delay

expires

from_config

ignore_result

max_retries

name

priority

rate_limit

reject_on_worker_lost

request_stack

resultrepr_maxsize

send_events

serializer

soft_time_limit

store_errors_even_if_ignored

throws

time_limit

track_started

trail

typing

Static methods

add_around

def add_around(
    attr,
    around
)

annotate

def annotate(

)

bind

def bind(
    app
)

on_bound

def on_bound(
    app
)

Called when the task is bound to an app.

Note: This class method can be defined to do additional actions when the task class is bound to an app.

Instance variables

backend

request

Get current request object.

Methods

AsyncResult

def AsyncResult(
    self,
    task_id,
    **kwargs
)

Get AsyncResult instance for the specified task.

Arguments: task_id (str): Task id to get result for.

add_to_chord

def add_to_chord(
    self,
    sig,
    lazy=False
)

Add signature to the chord the current task is a member of.

.. versionadded:: 4.0

Currently only supported by the Redis result backend.

Arguments: sig (Signature): Signature to extend chord with. lazy (bool): If enabled the new task won't actually be called, and sig.delay() must be called manually.

add_trail

def add_trail(
    self,
    result
)

after_return

def after_return(
    self,
    status,
    retval,
    task_id,
    args,
    kwargs,
    einfo
)

Handler called after the task returns.

Arguments: status (str): Current task state. retval (Any): Task return value/exception. task_id (str): Unique id of the task. args (Tuple): Original arguments for the task. kwargs (Dict): Original keyword arguments for the task. einfo (~billiard.einfo.ExceptionInfo): Exception information.

Returns: None: The return value of this handler is ignored.

apply

def apply(
    self,
    args=None,
    kwargs=None,
    link=None,
    link_error=None,
    task_id=None,
    retries=None,
    throw=None,
    logfile=None,
    loglevel=None,
    headers=None,
    **options
)

Execute this task locally, by blocking until the task returns.

Arguments: args (Tuple): positional arguments passed on to the task. kwargs (Dict): keyword arguments passed on to the task. throw (bool): Re-raise task exceptions. Defaults to the :setting:task_eager_propagates setting.

Returns: celery.result.EagerResult: pre-evaluated result.

apply_async

def apply_async(
    self,
    args=None,
    kwargs=None,
    task_id=None,
    producer=None,
    link=None,
    link_error=None,
    shadow=None,
    **options
)

Apply tasks asynchronously by sending a message.

Arguments: args (Tuple): The positional arguments to pass on to the task.

kwargs (Dict): The keyword arguments to pass on to the task.

countdown (float): Number of seconds into the future that the
    task should execute.  Defaults to immediate execution.

eta (~datetime.datetime): Absolute time and date of when the task
    should be executed.  May not be specified if `countdown`
    is also supplied.

expires (float, ~datetime.datetime): Datetime or
    seconds in the future for the task should expire.
    The task won't be executed after the expiration time.

shadow (str): Override task name used in logs/monitoring.
    Default is retrieved from :meth:`shadow_name`.

connection (kombu.Connection): Re-use existing broker connection
    instead of acquiring one from the connection pool.

retry (bool): If enabled sending of the task message will be
    retried in the event of connection loss or failure.
    Default is taken from the :setting:`task_publish_retry`
    setting.  Note that you need to handle the
    producer/connection manually for this to work.

retry_policy (Mapping): Override the retry policy used.
    See the :setting:`task_publish_retry_policy` setting.

time_limit (int): If set, overrides the default time limit.

soft_time_limit (int): If set, overrides the default soft
    time limit.

queue (str, kombu.Queue): The queue to route the task to.
    This must be a key present in :setting:`task_queues`, or
    :setting:`task_create_missing_queues` must be
    enabled.  See :ref:`guide-routing` for more
    information.

exchange (str, kombu.Exchange): Named custom exchange to send the
    task to.  Usually not used in combination with the ``queue``
    argument.

routing_key (str): Custom routing key used to route the task to a
    worker server.  If in combination with a ``queue`` argument
    only used to specify custom routing keys to topic exchanges.

priority (int): The task priority, a number between 0 and 9.
    Defaults to the :attr:`priority` attribute.

serializer (str): Serialization method to use.
    Can be `pickle`, `json`, `yaml`, `msgpack` or any custom
    serialization method that's been registered
    with :mod:`kombu.serialization.registry`.
    Defaults to the :attr:`serializer` attribute.

compression (str): Optional compression method
    to use.  Can be one of ``zlib``, ``bzip2``,
    or any custom compression methods registered with
    :func:`kombu.compression.register`.
    Defaults to the :setting:`task_compression` setting.

link (Signature): A single, or a list of tasks signatures
    to apply if the task returns successfully.

link_error (Signature): A single, or a list of task signatures
    to apply if an error occurs while executing the task.

producer (kombu.Producer): custom producer to use when publishing
    the task.

add_to_parent (bool): If set to True (default) and the task
    is applied while executing another task, then the result
    will be appended to the parent tasks ``request.children``
    attribute.  Trailing can also be disabled by default using the
    :attr:`trail` attribute

ignore_result (bool): If set to `False` (default) the result
    of a task will be stored in the backend. If set to `True`
    the result will not be stored. This can also be set
    using the :attr:`ignore_result` in the `app.task` decorator.

publisher (kombu.Producer): Deprecated alias to ``producer``.

headers (Dict): Message headers to be included in the message.

Returns: celery.result.AsyncResult: Promise of future evaluation.

Raises: TypeError: If not enough arguments are passed, or too many arguments are passed. Note that signature checks may be disabled by specifying @task(typing=False). kombu.exceptions.OperationalError: If a connection to the transport cannot be made, or if the connection is lost.

Note: Also supports all keyword arguments supported by :meth:kombu.Producer.publish.

before_start

def before_start(
    self,
    task_id,
    args,
    kwargs
)

Handler called before the task starts.

.. versionadded:: 5.2

Arguments: task_id (str): Unique id of the task to execute. args (Tuple): Original arguments for the task to execute. kwargs (Dict): Original keyword arguments for the task to execute.

Returns: None: The return value of this handler is ignored.

chunks

def chunks(
    self,
    it,
    n
)

Create a :class:~celery.canvas.chunks task for this task.

delay

def delay(
    self,
    *args,
    **kwargs
)

Star argument version of :meth:apply_async.

Does not support the extra options enabled by :meth:apply_async.

Arguments: args (Any): Positional arguments passed on to the task. *kwargs (Any): Keyword arguments passed on to the task. Returns: celery.result.AsyncResult: Future promise.

map

def map(
    self,
    it
)

Create a :class:~celery.canvas.xmap task from it.

on_failure

def on_failure(
    self,
    exc,
    task_id,
    args,
    kwargs,
    einfo
)

Logic when task fails

Args: exc (type): description task_id (type): description args (type): description kwargs (type): description einfo (type): description

on_replace

def on_replace(
    self,
    sig
)

Handler called when the task is replaced.

Must return super().on_replace(sig) when overriding to ensure the task replacement is properly handled.

.. versionadded:: 5.3

Arguments: sig (Signature): signature to replace with.

on_retry

def on_retry(
    self,
    exc,
    task_id,
    args,
    kwargs,
    einfo
)

Retry handler.

This is run by the worker when the task is to be retried.

Arguments: exc (Exception): The exception sent to :meth:retry. task_id (str): Unique id of the retried task. args (Tuple): Original arguments for the retried task. kwargs (Dict): Original keyword arguments for the retried task. einfo (~billiard.einfo.ExceptionInfo): Exception information.

Returns: None: The return value of this handler is ignored.

on_success

def on_success(
    self,
    retval,
    task_id,
    args,
    kwargs
)

Success handler.

Run by the worker if the task executes successfully.

Arguments: retval (Any): The return value of the task. task_id (str): Unique id of the executed task. args (Tuple): Original arguments for the executed task. kwargs (Dict): Original keyword arguments for the executed task.

Returns: None: The return value of this handler is ignored.

pop_request

def pop_request(
    self
)

push_request

def push_request(
    self,
    *args,
    **kwargs
)

replace

def replace(
    self,
    sig
)

Replace this task, with a new task inheriting the task id.

Execution of the host task ends immediately and no subsequent statements will be run.

.. versionadded:: 4.0

Arguments: sig (Signature): signature to replace with. visitor (StampingVisitor): Visitor API object.

Raises: ~@Ignore: This is always raised when called in asynchronous context. It is best to always use return self.replace(...) to convey to the reader that the task won't continue after being replaced.

retry

def retry(
    self,
    args=None,
    kwargs=None,
    exc=None,
    throw=True,
    eta=None,
    countdown=None,
    max_retries=None,
    **options
)

Retry the task, adding it to the back of the queue.

Example: >>> from imaginary_twitter_lib import Twitter >>> from proj.celery import app

>>> @app.task(bind=True)
... def tweet(self, auth, message):
...     twitter = Twitter(oauth=auth)
...     try:
...         twitter.post_status_update(message)
...     except twitter.FailWhale as exc:
...         # Retry in 5 minutes.
...         raise self.retry(countdown=60 * 5, exc=exc)

Note: Although the task will never return above as retry raises an exception to notify the worker, we use raise in front of the retry to convey that the rest of the block won't be executed.

Arguments: args (Tuple): Positional arguments to retry with. kwargs (Dict): Keyword arguments to retry with. exc (Exception): Custom exception to report when the max retry limit has been exceeded (default: :exc:~@MaxRetriesExceededError).

    If this argument is set and retry is called while
    an exception was raised (``sys.exc_info()`` is set)
    it will attempt to re-raise the current exception.

    If no exception was raised it will raise the ``exc``
    argument provided.
countdown (float): Time in seconds to delay the retry for.
eta (~datetime.datetime): Explicit time and date to run the
    retry at.
max_retries (int): If set, overrides the default retry limit for
    this execution.  Changes to this parameter don't propagate to
    subsequent task retry attempts.  A value of :const:`None`,
    means "use the default", so if you want infinite retries you'd
    have to set the :attr:`max_retries` attribute of the task to
    :const:`None` first.
time_limit (int): If set, overrides the default time limit.
soft_time_limit (int): If set, overrides the default soft
    time limit.
throw (bool): If this is :const:`False`, don't raise the
    :exc:`~@Retry` exception, that tells the worker to mark
    the task as being retried.  Note that this means the task
    will be marked as failed if the task raises an exception,
    or successful if it returns after the retry call.
**options (Any): Extra options to pass on to :meth:`apply_async`.

Raises:

celery.exceptions.Retry:
    To tell the worker that the task has been re-sent for retry.
    This always happens, unless the `throw` keyword argument
    has been explicitly set to :const:`False`, and is considered
    normal operation.

run

def run(
    self,
    *args,
    **kwargs
)

The body of the task executed by workers.

s

def s(
    self,
    *args,
    **kwargs
)

Create signature.

Shortcut for .s(*a, **k) -> .signature(a, k).

send_event

def send_event(
    self,
    type_,
    retry=True,
    retry_policy=None,
    **fields
)

Send monitoring event message.

This can be used to add custom event types in :pypi:Flower and other monitors.

Arguments: type_ (str): Type of event, e.g. "task-failed".

Keyword Arguments: retry (bool): Retry sending the message if the connection is lost. Default is taken from the :setting:task_publish_retry setting. retry_policy (Mapping): Retry settings. Default is taken from the :setting:task_publish_retry_policy setting. **fields (Any): Map containing information about the event. Must be JSON serializable.

shadow_name

def shadow_name(
    self,
    args,
    kwargs,
    options
)

Override for custom task name in worker logs/monitoring.

Example: .. code-block:: python

    from celery.utils.imports import qualname

    def shadow_name(task, args, kwargs, options):
        return qualname(args[0])

    @app.task(shadow_name=shadow_name, serializer='pickle')
    def apply_function_async(fun, *args, **kwargs):
        return fun(*args, **kwargs)

Arguments: args (Tuple): Task positional arguments. kwargs (Dict): Task keyword arguments. options (Dict): Task execution options.

si

def si(
    self,
    *args,
    **kwargs
)

Create immutable signature.

Shortcut for .si(*a, **k) -> .signature(a, k, immutable=True).

signature

def signature(
    self,
    args=None,
    *starargs,
    **starkwargs
)

Create signature.

Returns: :class:~celery.signature: object for this task, wrapping arguments and execution options for a single task invocation.

signature_from_request

def signature_from_request(
    self,
    request=None,
    args=None,
    kwargs=None,
    queue=None,
    **extra_options
)

starmap

def starmap(
    self,
    it
)

Create a :class:~celery.canvas.xstarmap task from it.

start_strategy

def start_strategy(
    self,
    app,
    consumer,
    **kwargs
)

subtask

def subtask(
    self,
    args=None,
    *starargs,
    **starkwargs
)

Create signature.

Returns: :class:~celery.signature: object for this task, wrapping arguments and execution options for a single task invocation.

subtask_from_request

def subtask_from_request(
    self,
    request=None,
    args=None,
    kwargs=None,
    queue=None,
    **extra_options
)

update_state

def update_state(
    self,
    task_id=None,
    state=None,
    meta=None,
    **kwargs
)

Update task state.

Arguments: task_id (str): Id of the task to update. Defaults to the id of the current task. state (str): New state. meta (Dict): State meta-data.