Understand Watchfolders and the Project Store (Tutorial)

At a Glance

Use watchfolders to ingest incoming data into HISE, especially when the data must trigger an automated pipeline or be transferred according to project-specific ingestion rules. Use the Project Store to access, manage, retain, and work with files that belong to a specific project.

When to Use These Features

Both watchfolders and the Project Store support project-specific file workflows in HISE, but the following sections describe some important differences between them. The table at the end of this document summarizes those capabilities.

Watchfolders

A watchfolder is like a pickup spot for data that's ready to be ingested. Data does not persist in a watchfolder. If the data meets specified pipeline requirements, it's claimed by an automated pipeline that picks up and processes the data. For example, when sequencing files are dropped into a watchfolder, the automated sequencing pipeline is triggered. Likewise, Olink files trigger the Olink pipeline, and so on for each file type associated with an automated pipeline. The files then move to a storage bucket for analysis. Users can locate them there using an advanced search. Then they can download the files to their IDE for secondary analysis. 

For file formatting instructions, see Use Watchfolders to Ingest Data (Tutorial).

Project Store

Every project has a project-specific, read-only storage space called a Project Store. You can use the Project Store UI to browse file contents, associate files with their corresponding sample references (if you didn't do so during ingestion), preview PDF and JPG files, move files directly into an IDE, mark files for deletion, or upload derived insights for storage. 

 If a watchfolder has been set up for the Project Store, it is either a designated watchfolder, in which all files go directly to the Project Store, or an "allowed file types" watchfolder, in which a file moves to the Project Store only if its type is consistent with one of the file types specified by the project administrator.

For details, see Ingest Data Into the Project Store (Tutorial).

Access

Watchfolders

Watchfolders are set up by the HISE administrator on request. Not all projects have watchfolders. If your project has no watchfolders or you're not authorized to ingest data, you won't see any watchfolders or be able to create them. Check with your system administrator to see who is authorized to ingest data for your organization.

If a project does have a watchfolder, access is granted to a subset of project users. To see the available watchfolders for your account and projects, click your name in the upper-right corner of the screen. Then click Watchfolders. For watchfolder setup, contact Support.

 Project Store

The Project Store is tied to a specified project. Every project user has access to the Project Store. You might also have access to a watchfolder to ingest data into a Project Store.

Data Ingestion and Storage

Watchfolders

Use watchfolders to ingest incoming files into HISE. A project can have a watchfolder configured for specific data types, such as CBC results, survey results, demographic data, raw data for automated pipelines, or files intended for IDE analysis. Files uploaded to a watchfolder must meet the folder's configured file-type requirements unless it's a wildcard watchfolder. Eligible files are typically picked up by an automated pipeline and routed to an analysis bucket or another configured destination. Each upload creates an ingest receipt that confirms the outcome and reports any errors.

Project Store

Use the Project Store to retain and work with project files after they've been ingested, transferred, or uploaded as derived insights. Each project has its own read-only Project Store that's available to all project users. From the Project Store, users can browse stored files, associate files with sample references, preview supported PDF and JPG files, move files to an IDE for secondary analysis, mark files for deletion, and store derived insights or other project outputs. If a watchfolder is configured to send eligible files to the Project Store, those files become available there after ingestion.

Related Resources

Use Watchfolders to Ingest Data (Tutorial)

Ingest Data into the Project Store (Tutorial)

Add Searchable Metadata to Project Store Files (Tutorial)