3.3.1.1   LSST Observatory Visualization Environement (LOVE)¶

LOVE is a web-based user inteface for the observatory control system. It is currently available at the summit (VPN required) and all other test stands.

The system is developed using the popular React JavaScript framework for the frontend, with Python in the backend. It contains an editing tool that allow users to create and customize views, but this feature relies exclusively on existing widget-like components. Extending the system beyond the vendor-provided “widgets” would require some knowledge of React and JavaScript, which is not common in the project. Nevertheless, LOVE does contain a special widget that allows us to embed other web pages into a particular view. This could be used, for instance, to embed Bokeh Plotting Applications alongside other LOVE components.

3.3.1.2   Nublado (Jupyter Notebook) Interface¶

The summit Nublado interface (VPN required) provides the user with a Jupyter Lab interface and the required libraries/packages to perform standard observatory operations including sending commands and running SAL scripts. It can also be used to query the EFD. This tool is set up to mimic the RSP and test-stand environments to the maximum extent possible, providing all the functionalities of a Jupyter Notebook but with direct access to the control system. Observers are expected to use this tool to perform commands or sequences that are not encapsulated into a SAL script. The Nublado interface is also useful for on-the-fly analysis and/or custom monitoring.

3.3.1.3   Camera Visualization Tool¶

The camera image visualization tool is designed to run on the camera diagnostic cluster and to deliver full-resolution focal plane images to the operator within a few seconds of image acquisition. The front-end is web based and supports natural zooming and panning, including zooming to full pixel resolution, as well as the ability to display pixel coordinates and raw pixel value by moving the mouse over the image. It supports a simple but extensible set of image processing algorithms which have been found to be useful for viewing images during camera testing.

The tool uses the International Image Interchange Framework (IIIF) with the front-end based on the open source OpenSeadragon framework, and the backend based on the open source Cantaloupe product, modified to be able to read images from FITS files and directly from the DAQ. The tool is currently deployed at SLAC for camera construction, and on the summit in Chile for Auxtel and ComCam.

3.3.1.4   Engineering Facilities Database and Large File Annex¶

The Engineering Facilities Database (EFD) records all data sent by components over the DDS control network. This content essentially tracks the observatory state as a function of time and is very useful in diagnosing issues and understanding (both desired and undesired) operational behaviours. The database is best queried using the EFD client from the Nublado (Jupyter Notebook) Interface (or any python-based method/script) when making custom plots. Instructions on how to access the EFD, and specifically the other instances of the data, are found in SQR-034. However, the Chronograf graphical front-end offers a nice solution for building simple plots (dashboards).

The Large File Annex (LFA) contains files over ~1 MB that are accessible both from the summit and the RSP. When a file is published to the LFA its presence (and location of the file) is published via SAL/DDS and therefore the location of LFA files can be found via an EFD query. It is anticipated that artifacts generated from automated on-the-fly analyses will be stored in this area. An example of this would be the Papermill Executed Parameterized Notebooks.

3.3.1.5   Chronograf¶

Chronograf is a tool that provides a user-friendly graphical interface to each deployed instance of the EFD. During operations, observers will use the summit instance (VPN required). It is particularly useful for quick and simple analysis as well as for creating dashboards to show a specific information about the status of the observatory; especially when a similar functionality is not available in LOVE.

Chronograf is not particularly well suited for complex queries and figures, especially those that require some additional logic to be meaningful. For example, if a Controllable SAL Component (CSC) crashes, a view displaying its summary state will continue to show the last value in the EFD, which is likely no longer applicable. To derive relevant information, it is necessary to combine it with some additional data, such as the time since the last published heartbeat. This is not easily accomplished in Chronograf as it is not designed for this purpose. Furthermore, Chronograf views (queries, plots, dashboards, etc) are not easily replicated, and can vary significantly for different users, depending on their local configurations. For these reason (and many others), it is not an appropriate substitute for a true monitoring system, such as what is provided by the LOVE interface.

3.3.1.6   Watcher¶

The Watcher CSC <https://ts-watcher.lsst.io/>_ monitors data from multiple sources and publishes alarm events to DDS depending on predefined conditions. Although it is currently fed mainly with DDS data published by CSCs, it does has the capability to listen to other external data streams. Alarms are composed by a set of “rules” that are defined and added to the application software package. Most alarms are generic enough such that they can be configured to act for multiple CSCs, whereas others are specific to a particular component. For example, the heartbeat rule can be configured for any CSC, and will issue an alarm if it does not receive a heartbeat for less than a configurable time (usually 5s, which is the default).

The Watcher itself is not a display tool nor does it have any display functionality. When a condition defined by an rule is met, an alarm is generated and the observer is alerted via a LOVE screen. The alarm has a series of levels and audible alerts are sent out via LOVE, as well as a visual notification. Once the alarm is acknowledged by the observer the alert is considered to be completed. There is no feedback or interaction for the observer beyond the acknowledgment of the alarm. Alarms will only deactivate (stop re-occuring) if the condition for which they are activated cease to exist, e.g., if a CSC resumes publishing heartbeats after being restored.

3.3.1.7   SAL Scripts¶

SAL scripts are a series of coordinated sequences, often consisting of commands to CSCs, that are executed by the ScriptQueue. The LOVE system contains a graphical ScriptQueue interface to control and monitor the execution of scripts. It is anticipated that most standard operations will utilize scripts. Also possible, although not standard practice, is to manually execute a script from the Nublado (Jupyter Notebook) Interface. From within a SAL script, users can send standard commands to components as well as send data to the OCPS for processing. The script can then either wait for the analysis to complete and continue, with the ability to act based on the result, or launch the process (e.g. image reduction) and continue executing the script. Scripts are not intended to perform intensive data analysis and, in general, are not expected to produce artifacts. Scripts are also unable to display visualizations.

3.3.1.8   OCPS¶

The Observatory Controlled Pipeline Service (OCPS) is a CSC that allows observers (and SAL Scripts) to execute pipeTasks to perform data reductions and analyses. The CSC runs on the summit but the data processing is currently running on the commissioning cluster at the base (although it may be relocated to the summit). The OCPS is not a display tool, but can be used to produce artifacts (such as images, spectra etc) that observers want to display. The current scope of this service is to only provide image-related processing. It cannot currently query the EFD.

At this time, the OCPS is being used to perform the analysis of daily calibrations executed from the scriptQueue.

3.3.1.9   Prompt Processing¶

The Prompt Processing Pipeline is expected to run at the United States Data Facility (USDF). Within 60s, the images taken on-sky get reduced and a series of data products are made available. A small number of these data products are sent back to the summit via the Telemetry Gateway. This service is not yet in place.

It is expected that metrics coming from Prompt Processing (and faro) will be used in on-the-fly displays.

3.3.1.10   faro¶

faro is a framework for automatically and efficiently computing scientific performance metrics on the outputs of the LSST Science Pipelines. It supports multiple units of data of varying granularity, ranging from single-detector to full-survey summary statistics, and persists the results as scalar metric values alongside the input data products in the same butler repo.

In the “first-look analysis” context, it is intended that faro would be run as an afterburner to the OCPS and Prompt Processing (run automatically as part of the same pipeline) and would compute scalar metrics to quantify the performance of individual visits as they are acquired. For example, faro could be used to estimate the effective depth of individual images by computing the flux of a point source that would be measured with signal-to-noise of 10, or to measure variations in effective depth across the focal plane that would be indicative of variable atmosphere transparency. The intent is that these metrics would be available within minutes to the observers to inform nighttime operations. faro is designed to be modular and configurable so that additional metrics can be readily added to support summit operations.

faro is NOT itself a visualization tool, but rather generates scalar metric values that could be used as input to visualization tools.

3.3.1.11   RubinTV¶

The Rubin TV image display service is a collection of auto-generated webpages that displays the output (PNGs) of a series of manually deployed analysis scripts. It was relatively straightfoward to implement and has proved to be a highly useful tool during early AuxTel Commissioning. We expect it to continue to be useful as the framework recommended in this report is developed and the systems are matured. It is not expected to be one of the long-term solutions to the Rubin Visualization challenges because its capabilities will be incorporated into the other toolsets discussed in Deliverable 4: Augmenting Current and Adding New Functionalities.

3.4.2.1   Catcher CSC¶

A series of new functionality, which for the purposes of this document we have grouped into a single “Catcher” Controllable SAL Component (CSC) is required to handle the low-level coordiation of identifying when a specific condition is met, then launching and monitoring an analysis process. It is still being evaluated if it is required to generate a new CSC or if the Watcher CSC can be augmented to handle this new functionality. For the purposes of this document we will assume the functionality is to be captured in a new CSC. The Catcher functionality also requires a LOVE display to show which tasks are running, links to generated reports, and alarms or notifications for observers.

More details on the design and implementation can be found in the Catcher Design Document currently being worked on as a google doc.

At the end of an analysis task, a “report” is generated and produced as a result. The reports from the Catcher can be derived in multiple ways and take on multiple formats. There is no requirement that the analysis results generated by the Catcher-managed tasks be persistent. However, it should be recommended practice as the results can be re-run at a later date. When possible, this committee recommends that analysis tasks produce reports in the form of Papermill Executed Parameterized Notebooks. Once executed, the notebooks and their contents and archived to the LFA, where they can be looked at (and even re-run) either immediately or at a later date. The capabilities of the notebooks is vast; allowing image analysis via sending commands to the OCPS, queries of the EFD, grabbing of SAL events or data from the Butler. The queries to obtain the data to create any plots or other displays are also contained in the notebook. This allows plots to be modified and re-generated as required at a later date. Lastly, the notebooks can be used to create a dataset that can be displayed by a Bokeh Application that is nested inside a LOVE display, which is one of the key use-cases that will be encountered during observations. There are details that remain to be solved, specifically aspects such as how to account for multiple reports that can be generated by re-running the same notebook multiple times. These are presumed to be solvable problems but will require further investigation that goes beyond the scope of this group.

It is worth mentioning that having analysis results persisted in the Butler is not a viable option in many cases. The Butler requires an artifact to be associated with a dataId. For many analyses, there will be no image associated with it, and therefore the Butler cannot be used.

3.4.2.2   Bokeh Plotting Applications¶

Multiple plotting packages were considered before ultimately settling on the Bokeh Visualization Library. The majority of Python users are familiar with matplotlib and therefore finding a system to support this was the preferred option. However, matplotlib is very limited with regards to the creation of interactive and dynamic web-embedded plots, for this reason the packages of vega-lite and Bokeh were considered. Both of these systems could be directly integrated into the LOVE framework with the currently existing functionality. Vega-lite is able to create apps to embed on webpages, however, Vega-Lite does not expose the handles for more sophisticated call-back actions that go beyond the plotting itself. This was the most significant inadequacy of Vega-lite, however, the capabilities of vega-lite are also limited and certain use-cases required the use of Vega, which means ultimately working with javaScript; to which very few people on the project have familiarity. Based on the evaluation of the use-cases Bokeh was the preferred solution to support all of our requirements (see Figure Generation Requirements) Furthermore, it is already familiar to certain people on the project and can be easily integrated into our current system design.

Bokeh Applications are extremely flexible in design and can render data from multiple sources if configured to do so. This includes SAL events, EFD queries, Butler served information, or files from the LFA. The apps can then create dynamic (or static) plots, display images, or even be setup to send commands to move the telescope based on a calculation (e.g. offsetting to a star). One major advantage of Bokeh is that the very high majority of the application can be developed inside a notebook. Once functioning as expected, it can be ported to a python file with minimal intervention required.

As part of the selection criteria, demonstation projects were created and executed to verify certain aspects of the required functionality and usability. Examples of Bokeh apps and their use is found in the Proof-of-concept Demonstrations section. It was possible to build out and deploy the Jitter application in a few hours. The centering application, which can be used for determining offsets, took roughly two hours. Rough notes on how to turn a notebook-based Bokeh plot into an application can be found as part of Simon’s draft. These will be turned into more formal instructions before the close-out of this working group. One caveat is that Bokeh Apps can only be used where they are deployed. Should they wish to be used at the Rubin Science Platform (RSP) for instance, they will need to be deployed there as well (and obviously any SAL commands will not work).

3.4.2.3   Papermill Executed Parameterized Notebooks¶

Papermill introduces the concept of parameterized notebooks and provides tools to execute them in a batch-like mode.

Parameterized notebooks are similar to regular Jupyter notebooks. The process consists of:

• identifing variables in the notebook that one wants to expose as parameters,
• have all these variables defined in a single cell (ideally, with default values),
• tagging the cell with the “parameters” keyword.

More details on how to parameterize a notebook on a particular environment can be found in the papermill usage documentation page.

Notebooks can then be executed using Papermill command line interface or through its Python API.

Benefits of using Papermill includes:

• Simplify the process of creating on-the-fly reports and re-runable notebooks that will store the parameters used for the execution and generation of plots etc.

  • Users would be able to develop their reports directly on notebooks, with practicaly no overhead in integrating them with the system aftewards.
  • The reports can be stored directly as notebooks and/or be expored and rendered as webpages with nbconvert.

• Store notebooks in the LFA.

  Papermill can save the output directly to the s3bucket, which is used as the backend for the LFA.

• Possible to interact with the control system with salobj from the notebooks, the same way we execute tasks with the control system from nublado.

• Can also send information to Bokeh app (if Bokeh app is configured to do so).

Some challenges with using Jupyter Notebooks and Papermill are:

• It is notably difficult to develop unit testing for Jupyter Notebooks.

  Papermill does make it possible to write unit tests for Jupyter Notebooks, but certain tasks (like using mocks to isolate external libraries) can be exceedingly difficult.

• Overhead on loading the environment.

  When executing a notebook through Papermill, there is an overhead associated with loading the environment prior to the execution by the Jupyter lab server. The size of the overhead depends on the environment which the notebook is running.

  For reference, in the ScriptQueue environment the overhead is about 2s. At the same time, some highly customized users environments on nublado take up to 10s to load.