Creating a new driver¶

A driver is a Python program which allows Onitu to synchronize files with a remote service, such as Dropbox, Google Drive, SSH, FTP or a local hard drive.

Basics¶

The drivers communicate with Onitu via the Plug class, which handles the operations common to all drivers. Each driver implements its specific tasks with the system of handlers. Those handlers will be called by the Plug at certain occasions.

In Onitu the file transfers can be made by chunks. When a new transfer begin, the Plug asks the others drivers for new chunks, and then call the upload_chunk handler. The transfers can also be made via the upload_file handler, which upload the full content of the file. Both protocols can be used together.

Each driver must expose a function called start and an instance of the Plug in their __init__.py file. This start function will be called by Onitu during the initialization of the driver, and should not return until the end of life of the driver (cf Plug.listen()).

When a driver detects an update in a file, it should update the Metadata of the file, specify a Metadata.revision, and call Plug.update_file().

Note

During their startup, the drivers should look for new files or updates on their remote file system. They should also listen to changes during their lifetime. The mechanism used to do that is specific to each driver, and can’t be abstracted by the Plug.

Each driver must have a manifest describing its purpose and its options.

Onitu provide a set of functional tests that you can use to see if your driver respond to every exigence.

Handlers¶

A handler is a function that will be called by the Plug on different occasions, such as getting a chunk from a file or starting a transfer. The drivers can define any handler they need. For example, some driver don’t need to do anything for initiating a transfer, so they might not want to implement the end_upload handler. In order to register a handler, the Plug.handler() decorator is used.

Warning

All the handlers must be thread-safe. The plug uses several threads to handle concurrent requests, and each handler can be called from any of those threads. The Plug itself is fully thread-safe.

At this stage, the list of the handlers that can be defined is the following :

get_chunk(metadata, offset, size)¶

Return a chunk of a given size, starting at the given offset, from a file.

Parameters:	metadata (`Metadata`) – The metadata of the file offset (int) – The offset from which the content should be retrieved size (int) – The maximum size of the chunk that should be returned
Return type:	string

get_file(metadata)¶

Return the full content of a file.

Parameters:	metadata (`Metadata`) – The metadata of the file
Return type:	string

upload_chunk(metadata, offset, chunk)¶

Write a chunk in a file at a given offset.

Parameters:	metadata (`Metadata`) – The metadata of the file offset (int) – The offset from which the content should be written chunk (string) – The content that should be written

upload_file(metadata, content)¶

Write the full content of a file.

Parameters:	metadata (`Metadata`) – The metadata of the file content – The content of the file

set_chunk_size(chunk_size)¶

Allows a driver to force a chunk size by overriding the default, or provided, value. The handler takes the plug chunk size as argument, and if that size is invalid for the driver, it can return a new value. Useful for services that require a minimum size for transfers.

Parameters:	chunk_size (int) – the size the plug is currently using

start_upload(metadata)¶

Initialize a new upload. This handler is called when a new transfer is started.

Parameters:	metadata (`Metadata`) – The metadata of the file transferred

restart_upload(metadata, offset)¶

Restart a failed upload. This handler will be called during the startup if a transfer has been stopped.

Parameters:	metadata (`Metadata`) – The metadata of the file transferred offset (int) – The offset of the last chunk uploaded

end_upload(metadata)¶

Called when a transfer is over.

Parameters:	metadata (`Metadata`) – The metadata of the file transferred

abort_upload(metadata)¶

Called when a transfer is aborted. For example, this could happen if a newer version of the file should be uploaded during the transfer.

Parameters:	metadata (`Metadata`) – The metadata of the file transferred

close()¶: Called when Onitu is closing. This gives a chance to the driver to clean its resources. Note that it is called from a sighandler, so some external functionalities might not work as expected. This handler should not take too long to complete or it could cause perturbations.

The Plug¶

Metadata¶

Exceptions¶

If an error happen in a driver, it should raise an appropriate exception. Two exceptions are handled by the Plug, and should be used accordingly to the situation : DriverError and ServiceError.

Manifest¶

A manifest is a JSON file describing a driver in order to help the users configuring it. It contains several informations, such as the name of the driver, its description, and its available options. Each option must have a name, a description and a type.

The type of the options will be used by Onitu to validate them, and by the interface in order to offer a proper input field. The available types are : Integers, Floats, Booleans, Strings and Enumerates. An enumerate type must add a values field with the list of all the possible values.

An option can have a default field which represents the default value (it can be null). If this field is present, the option is not mandatory. All the options without a default value are mandatory.

Here is an example of what your manifest should look like :

{
  "name": "Your driver's name",
  "description": "The description of your driver.",
  "options": {
    "my_option": {
      "name": "Option's name",
      "description": "The description of the option",
      "type": "string",
    },
    "another_option": {
      "name": "Another option",
      "description": "The description of the option",
      "type": "enumerate",
      "values": ["foo", "bar"],
      "default": "foo"
    },
  }
}

Example¶

Usually, the drivers are created as a set of functions in a single file, with the Plug in a global variable. However, you can use a different style if you want, such as a class.

Here is an example of a simple driver working with the local file system :

import os

from onitu.api import Plug, ServiceError, DriverError

# A dummy library supposed to watch the file system
from fsmonitor import FSWatcher

plug = Plug()


@plug.handler()
def get_chunk(metadata, offset, size):
    try:
        with open(metadata.filename, 'rb') as f:
            f.seek(offset)
            return f.read(size)
    except IOError as e:
        raise ServiceError(
            "Error reading '{}': {}".format(metadata.filename, e)
        )


@plug.handler()
def upload_chunk(metadata, offset, chunk):
    try:
        with open(metadata.filename, 'r+b') as f:
            f.seek(offset)
            f.write(chunk)
    except IOError as e:
        raise ServiceError(
            "Error writting '{}': {}".format(metadata.filename, e)
        )


@plug.handler()
def end_upload(metadata):
    metadata.revision = os.path.getmtime(metadata.filename)
    metadata.write_revision()


class Watcher(FSWatcher):
    def on_update(self, filename):
        """Called each time an update of a file is detected
        """
        metadata = plug.get_metadata(filename)
        metadata.revision = os.path.getmtime(metadata.filename)
        metadata.size = os.path.getsize(metadata.filename)
        plug.update_file(metadata)

    def check_changes(self):
        """Check the changes on the file system since the last launch
        """
        for filename in self.files:
            revision = os.path.getmtime(filename)
            metadata = plug.get_metadata(filename)

            # If the file is more recent
            if revision > metadata.revision:
                metadata.revision = os.path.getmtime(metadata.filename)
                metadata.size = os.path.getsize(metadata.filename)
                plug.update_file(metadata)


def start():
    try:
        root = plug.options['root']
        os.chdir(root)
    except OSError as e:
        raise DriverError("Can't access '{}': {}".format(root, e))

    watcher = Watcher(root)
    watcher.check_changes()
    watcher.start()

    plug.listen()

This is what a driver’s __init__.py file should look like:

from .driver import start, plug

__all__ = ["start", "plug"]