diff --git a/docs/source/_static/audio_data/ex1.svg b/docs/source/_static/audio_data/ex1.svg deleted file mode 100644 index 8249d5468..000000000 --- a/docs/source/_static/audio_data/ex1.svg +++ /dev/null @@ -1,124 +0,0 @@ - - - - - - - - - - - - - - - FILE - ITEM - DATA - - - diff --git a/docs/source/_static/audio_data/ex_1.svg b/docs/source/_static/audio_data/ex_1.svg new file mode 100644 index 000000000..1778591c3 --- /dev/null +++ b/docs/source/_static/audio_data/ex_1.svg @@ -0,0 +1,170 @@ + + + + + + + + + + + + + + FILE + + + time + + + + + + + DATA + + + + + ITEM + + + + diff --git a/docs/source/_static/audio_data/ex_2.svg b/docs/source/_static/audio_data/ex_2.svg new file mode 100644 index 000000000..e969751db --- /dev/null +++ b/docs/source/_static/audio_data/ex_2.svg @@ -0,0 +1,406 @@ + + + + + + + + + + + + + + FILE + + + + + ITEM + + + + + FILE + + + + + ITEM + + + + time + + + FILE + + + + + ITEM + + + + + + ITEM + + + + + FILE + + + + + ITEM + + + + + + + + + + + + + + + + + + DATA + + + diff --git a/docs/source/_static/data_structure/data_structure.svg b/docs/source/_static/data_structure/data_structure.svg new file mode 100644 index 000000000..9bf8d2a63 --- /dev/null +++ b/docs/source/_static/data_structure/data_structure.svg @@ -0,0 +1,408 @@ + + + + + + + + + + + + + + FILE + + + + + + DATA + + + + + + FILE + + + + FILE + + + + + + ITEM + + + + + + + + ITEM + + + + + + + + ITEM + + + + + + + + + + + ITEM + + + + + + + + + + + time + + + + + + + + diff --git a/docs/source/_static/data_structure/file_item_data_0.svg b/docs/source/_static/data_structure/file_item_data_0.svg deleted file mode 100644 index f5d4898de..000000000 --- a/docs/source/_static/data_structure/file_item_data_0.svg +++ /dev/null @@ -1,144 +0,0 @@ - - - - - - - - - - - - - FILE - FILE - FILE - DATA - DATA - - diff --git a/docs/source/_static/data_structure/file_item_data_1.svg b/docs/source/_static/data_structure/file_item_data_1.svg deleted file mode 100644 index f925e34cb..000000000 --- a/docs/source/_static/data_structure/file_item_data_1.svg +++ /dev/null @@ -1,164 +0,0 @@ - - - - - - - - - - - - - - - - - - FILE - FILE - FILE - DATA - DATA - - diff --git a/docs/source/_static/data_structure/file_item_data_2.svg b/docs/source/_static/data_structure/file_item_data_2.svg deleted file mode 100644 index e5fcbdf12..000000000 --- a/docs/source/_static/data_structure/file_item_data_2.svg +++ /dev/null @@ -1,244 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - FILE - FILE - FILE - ITEM - ITEM - DATA - DATA - I - I - - diff --git a/docs/source/_static/data_structure/optimized_IO.svg b/docs/source/_static/data_structure/optimized_IO.svg new file mode 100644 index 000000000..1f0197788 --- /dev/null +++ b/docs/source/_static/data_structure/optimized_IO.svg @@ -0,0 +1,426 @@ + + + + + + + + + + + + + + FILE + + + + + FILE + + + + + ITEM + + + + + + ITEM + + + + + + ITEM + + + + + time + + + + + + + + + + + + + + + DATA + + + + + DATA + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/source/_static/ltas/ltas.gif b/docs/source/_static/ltas/ltas.gif deleted file mode 100644 index 6a797ac5c..000000000 Binary files a/docs/source/_static/ltas/ltas.gif and /dev/null differ diff --git a/docs/source/_static/ltas/ltas.svg b/docs/source/_static/ltas/ltas.svg new file mode 100644 index 000000000..815db1948 --- /dev/null +++ b/docs/source/_static/ltas/ltas.svg @@ -0,0 +1,1301 @@ + + + + + + + + + + + + + + + + + + + + 1500700 + + + + + + + + + 3000 + + + + + + + + + + + + + + + + + + + + + + + 4137 + + + + + + + 3000 + + + + + + + + + + + + + + + + + + + + + + + + + 4137 + + + + + + + + + + + + + + + + + + + + + + + + + + + + 4137 + + + + + + + 3000 + + + + 1 + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + + 60 + + + + 1 + + + + + + + + + + + + + 60 + + + + 1 + + + + time + diff --git a/docs/source/coreapi_introduction.rst b/docs/source/coreapi_introduction.rst index eb9a211f1..20f6eede4 100644 --- a/docs/source/coreapi_introduction.rst +++ b/docs/source/coreapi_introduction.rst @@ -29,17 +29,32 @@ nor if it contains empty time periods between files. To do so, **OSEkit** uses an intermediary object between the ``Data`` and the ``File``: an ``Item``, which is a **part** of ``Data`` that is found in a **single** ``File``. +.. note:: + - ``File`` : Object that represents an actual file on disk (e.g. an audio file, a spectrogram matrix, etc.) + - ``Data`` : Object that represents the data contained by file(s) between two timestamps (e.g. a spectrogram plot from a part of an audio file) + - ``Item`` : Part of a ``Data`` that is found in a singular ``File``. If a ``Data`` spans over multiple ``File`` , an ``Item`` is created for each one of them and their concatenation forms the ``Data`` values. + As an example, let's consider 3 files that differ in duration (along the x-axis in the figures here below). The first two files are consecutive, and the third one is separated from the end of the second by a duration for which there is no data. -From these 3 files, the user want to manipulate 2 data objects (for example to plot spectrograms from audio files at these 2 time periods): +From these 3 files, the user want to manipulate a data object (for example to plot a spectrogram from the audio files in a given time period). + +Under the hood, **OSEkit** will create intermediate ``Item`` objects that will be used for the data to be fetched seamlessly, with an optimized I/O workflow. + +An ``Item`` correspond to a single link between a ``Data`` and a ``File``: although a ``Data`` can be linked to multiple ``File`` (and vice versa), + + +.. image:: _static/data_structure/data_structure.svg -.. image:: _static/data_structure/file_item_data_0.svg +Optimized I/O +""""""""""""" -Under the hood, **OSEkit** will create intermediate ``Item`` objects that will be used for the data to be fetched seamlessly, with an optimized I/O workflow: +The ``Data``/``Item``/``File`` structure allow for an optimized handling of the files. -.. image:: _static/data_structure/file_item_data_2.svg +The values from the ``File`` objects transit through the ``Item`` **on demand**, and ``File`` related files are opened/closed only when needed, +that is when another ``Item`` requests values from another ``File`` than the currently opened one: +.. image:: _static/data_structure/optimized_IO.svg Different type of data """""""""""""""""""""" diff --git a/docs/source/coreapi_usage.rst b/docs/source/coreapi_usage.rst index dc3a0ea6e..93addf1c8 100644 --- a/docs/source/coreapi_usage.rst +++ b/docs/source/coreapi_usage.rst @@ -26,7 +26,7 @@ Audio Data Design """""" -The :class:`osekit.core.audio_data.AudioData` class represent a chunk of audio taken between two specific timestamps from one or more `AudioFile` instances. +The :class:`osekit.core.audio_data.AudioData` class represent a chunk of audio taken between two specific timestamps from one or more ``AudioFile`` instances. This class offers means of easily resample the data and access it on-demand (with an optimized I/O workflow to minimize the file openings/closings). @@ -46,7 +46,7 @@ Example for an ``AudioData`` representing the first 2 seconds of the ``foo/7189. This will lead to the following *data* <-> *file* structure (see the :ref:`Data and Files ` section): -.. image:: _static/audio_data/ex1.svg +.. image:: _static/audio_data/ex_1.svg If the ``AudioData`` begin and end timestamps cover multiple ``AudioFile``, the corresponding ``AudioItem``. @@ -73,7 +73,7 @@ Let's create a single ``AudioData`` that covers the total duration of the 4 file This will create one ``AudioItem`` per file, plus one **empty** ``AudioItem`` for the gap: -.. image:: _static/audio_data/ex2.svg +.. image:: _static/audio_data/ex_2.svg We can check that with code: @@ -450,21 +450,33 @@ The resulting figure presents the full-scale spectrogram at the top (from 0 to 7 LTAS Data ^^^^^^^^^ -OSEkit provides the :class:`osekit.core.ltas_data.LTASData` class for computing and plotting LTAS (**L**\ ong-\ **T**\ erm **A**\ verage **S**\ pectrum). +OSEkit provides the :class:`osekit.core.ltas_data.LTASData` class for computing and plotting long-term spectral averages. -LTAS are suitable when a spectrum is computed over a very long time and that the spectrum matrix time dimension reach a really high value. -In that case, time bins can be averaged to form a LTAS, which time resolution is lower than that of the original spectrum. +When computing an STFT over a long audio recording, the resulting spectrum matrix can contain a very large number of time bins. ``LTASData`` reduces this temporal resolution by averaging STFT time bins while preserving the spectral information along the frequency axis. -In OSEkit, LTAS are computed recursively: the user specifies a target number of time bins in the spectrum matrix, noted ``n_bins``. +The target number of time bins is specified with the ``n_bins`` parameter. OSEkit computes the result recursively so that the full STFT matrix never needs to be materialized when it would exceed this target size. -The visualization below depicts the process: the LTAS is computed with a target ``n_bins = 3000``. -Yellow rectangles depict the audio data (the x-axis being the time axis), and the number in the lower right -corner depicts the number of time bins in the spectrum matrix for this audio data. -The audio is recursively split in ``n_bins`` parts (it is split in 3 in the -representation instead of 3000 for clarity purposes) until the number of time bins in the spectrum gets below ``n_bins``. -Then, these spectrum parts are computed (hatched rectangles) and averaged across the time axis (filled rectangles). +The algorithm works as follows: -.. image:: _static/ltas/ltas.gif +1. Compute the number of STFT time bins that would be produced for the current audio segment. +2. If this number is greater than ``n_bins``, split the segment into ``n_bins`` equal-duration sub-segments and process each sub-segment recursively. +3. Otherwise, compute the STFT of the segment and average it along the time axis, producing a single averaged spectrum. +4. Use the averaged spectra returned by the recursive calls as the time bins of the parent segment. + +The illustration below shows the process for ``n_bins = 3000``. Yellow rectangles represent audio segments, and the number in the lower-right corner indicates the number of STFT time bins that would be produced for the corresponding segment. +For readability, the illustration shows three sub-segments instead of the ``3000`` sub-segments used in the example. + +* The complete recording would produce ``1 500 700`` STFT time bins, which exceeds ``n_bins``. +* The recording is therefore split into ``3000`` equal-duration segments. +* The first segment would still produce ``4 137`` STFT time bins, so it is split again. +* One of its sub-segments produces only ``60`` STFT time bins. Its STFT can therefore be computed directly. +* The ``60`` time bins are averaged along the time axis, producing a single spectrum. +* This spectrum is returned to the parent level and becomes one time bin of the resulting LTAS. +* The process continues until all ``3000`` top-level segments have produced their averaged spectra. + +This recursive approach limits the number of STFT time bins held in memory at each computation step while allowing the computation to be distributed across independent audio segments. + +.. image:: _static/ltas/ltas.svg :width: 300px :align: center