file_caching
============

Introduction
------------

The :mod:`file_caching` package helps when working with slow, possibly remote and unreliable data sources through
semi-transparent on-demand caches configured on a per-system level.

You will mainly use the :mod:`file_caching.cached_path` API. While the caching needs to be explicitly supported by the user code,
it depends on the user’s system setup (namely an environment variable specifying which paths to cache) on whether
caching is used for any particular path or not.

Preconfigured Caching
---------------------

Via the :mod:`file_caching.cached_path` module, preconfigured caching allows you to specify a number of 'to be cached' root paths
and return cached or original paths depending on whether they are located there.

Therefore, one has to configure the environment variable *DirCache_RootPaths*, e.g. like this (Windows example):

.. code-block:: bash

    setx DirCache_RootPaths = "//gaia/functionalTestData;//gaia/imageData"

Now we can just use :meth:`file_caching.cached_path.get_path` in our code:

.. code-block:: python
   :linenos:

    # import the cached_path api
    from file_caching import cached_path

    [...]

    #-------------------------------------------------------------
    # Example 1: Access caching-preconfigured path

    # This may take some time to fill/sync the cache
    path = cached_path.get_path( "//gaia/functionalTestData/MeVisLabModules/General/ML/FastWatershed" )

    # The second call will return almost immediately because the cache is still up-to-date
    path = cached_path.get_path( "//gaia/functionalTestData/MeVisLabModules/General/ML/FastWatershed" )

    # path points to a local cache copy
    print( path ) # --> C:\Users\TESTUS~1\AppData\Local\Temp/__DIRCACHE__/____gaia__functionalTestData/MeVisLabModules/General/ML/FastWatershed


    #-------------------------------------------------------------
    # Example 2: Access non-caching path:

    # Request path for which no caching is confiugred
    path = cached_path.get_path( "//gaia/fme/multimedia" )

    # path just points to the original path
    print( path ) # --> //gaia/fme/multimedia


Explicit Caching
----------------

The :class:`file_caching.dir_cache.DirCache` class handles caching for on particular source directory structure.

For example:

.. code-block:: python
   :linenos:

    # Import the DirCache class
    from file_caching.dir_cache import DirCache

    # Set up a new cache
    cache = DirCache( "//gaia/functionalTestData" )

    # Request data, this may take some time to fill/sync the cache
    cachedPath = cache.get( "//gaia/functionalTestData/MeVisLabModules/General/ML/FastWatershed" )

    # The second call will return almost immediately because the cache is still up-to-date
    cachedPath = cache.get( "//gaia/functionalTestData/MeVisLabModules/General/ML/FastWatershed" )

    # cachedPath points to a cached path (located in a local temp folder by default)
    print( cachedPath ) # --> C:\Users\TESTUS~1\AppData\Local\Temp/__DIRCACHE__/____gaia__functionalTestData/MeVisLabModules/General/ML/FastWatershed

Note that the second `get` call would also 'work' if //gaia would suddenly no longer be accessible.

Documents
---------

.. toctree::
   :maxdepth: 2
   :caption: You will find the following in this document:

   file_caching/cached_path
   file_caching/DirCache


Indices and tables
------------------

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`