Use Watchfolders to Ingest Data (Tutorial)
Last updated 2026-08-28At a Glance
Watchfolders are project-specific spaces used to transfer data to HISE. If the data is associated with an automated pipeline, it can be moved to the Project Store for analysis. (For ingestion of data that's not associated with an automated pipeline, see Use the Project Store to Ingest Data .) Contact immunology-support@alleninstitute.org for watchfolder setup or modification.
File Format
Label your ingestion files with the correct file type, such as `LabResults`, `TestResults`, or `clinical_labs`. If your files are not labeled or the label doesn't match the file type, a dismiss error appears on the ingest receipt.
You can create multiple file types (for example, `SurveyResults`) that contain different content types. The regex should include a filename. See the following table for examples, and refer to the boxed instructions to test your filename.
| csv filename | Regex |
Lab results | `clinical_data` or `clinical_labs` or `labresults` | `(?i)(.*((lab|test)results).*)|(.*((clinical)\_(labs|data)).*)|(.*(clinicaldata).*).csv` |
Survey data | `survey_data`, `survey_results`, or `surveyresults` | `(?i)(.*((survey)results).*)|(.*((survey)\_(survey|data)).*)|(.*(surveydata).*).csv` |
EMR data | `emr_data` or `emrdata` | `(?i)(.*(emrdata).*)|(.*((emr)\_(data)).*).csv |
1. In the upper-right corner of your screen, click your name. 2. Click Watchfolders. 3. Click the file type you plan to upload, such as OctetStream Lab Results (.csv). 4. Paste your proposed filename into the box, and press Enter. If your Regex is properly constructed for the selected file type, a green X appears to the right of text entry field, and a link to the watchfolder appears below it. If the format is not correct, a red X appears. |
Select Files for Ingestion
Use Advanced Search to find the files and data that you want to ingest. Select the appropriate search type: Files, Samples, EMR, or Public Files. The available filters vary by search type. For example, the Files search requires a project and file type, while the other search types provide filters relevant to the records they return.
Files: Search for file records by project, file type, and available advanced filters. Use this search type when you need to locate files associated with a project or modality before ingesting them through a watchfolder.
Samples: Search for sample records and sample-level metadata. Use this search type when the files you’ll ingest through a watchfolder must be associated with specific samples or when you need to identify samples by their available metadata.
EMR: Use EMR to search clinical, laboratory, and survey metadata associated with subjects or samples.
Public Files: Use Public Files to search files that are available publicly. After you identify the files or data you need in Advanced Search, use a watchfolder to ingest the corresponding files. For EMR data, configure the filename regex to recognize the appropriate filenames, such as clinical_data, clinical_labs, labresults, emr_data, or emrdata.
Instructions
For special preparation of files containing human metadata, see the preceding section and consult your organization's data privacy policy or legal representative.
To upload properly prepared files of any kind, with or without human metadata, follow the process outlined below. If your data upload requires an automated process, contact us at immunology-support@alleninstitute.org to discuss the options.
1. To upload data to a watchfolder, navigate to HISE and log in with your organization's email account.
2. In the upper-right corner below your email address, make sure you're in the account you want to work with. If not, use the account favorites menu to move to a different account. For details, see Switch Your Account or Projects (Tutorial).
6. Click your email address and select Watch Folders from the drop-down list.
7. Choose the watchfolder for your account and project.
8. Click UPLOAD FILES, or drag and drop your files into the watchfolder. Note that files in Google Drive, such as csv files created in Google Sheets, can't be uploaded directly to your Project Store or dragged into it. Download the file first, and then upload it from your Downloads folder.
8. To see the status of your uploaded files, from the top navigation menu, choose Data Processing > Pipeline Dashboard > Ingest Receipts.
