What this does

The batch ingest process for complex assets, or paged content, allows you to batch-create DAMS assets that consist of component parts, typically pages of a book/serial issue/archival file/etc. See pages Anatomy of DAMS digital assets and Content models for details on the structure of digital assets.

Assets created with this batch ingest method will consist of a set of page-level assets, a book/publication issue-level asset, and optionally a publication (series)-level asset. You can also use this method to add publication issues to an existing publication series, and you can add pages to existing book-level or issue-level assets.

Typically, a book-level asset and its pages are created using image files that are the result of a scanning process (digitally reformatted content). This batch ingest method also allows to use a PDF document as a source file, and the DAMS software will automatically create an image file for each page in a PDF submitted. For digitally reformatted (scanned) content, using a PDF is strongly discouraged, as the automatically created page images are almost invariably of lower quality than the original scan images. Contact the DAMS managers for a consultation (click here to submit a DAMS service request).

For born-digital content (for instance modern PDF ebooks or PDF documents directly exported from a word processor), other content models and ingest processes will be more appropriate. Contact the DAMS managers for a consultation (click here to submit a DAMS service request).

General information for batch ingest

The batch ingest process runs continuously, looking for newly queued batch jobs approximately every 5 minutes. You can add batch ingest jobs to the queue at any time.

Batch jobs are subject to the following batch job size and file size limitations:

• max. 100GB/batch job
  
• max. 10GB/file

Step 1: Stage files for batch ingest job

Organise files in a batch job folder, using subfolders if appropriate. Refer to the instructions/options listed below for preparing batch jobs.

Staging files for batch ingest of paged content

For batch ingest, your assets MUST be organized within a top level directory which represents the batch job. You will reference the batch job directory in the batch submission form in the DAMS GUI. Each batch job directory MUST contain one or more subdirectories, representing a book/publication issue, a publication/series, or a set of pages that should be added to an existing paged content asset. You can combine multiple books, issues, page addition sets in a batch job, as long as the job does not exceed the file and job size limitations.

Creating a datastreams.txt manifest file

Subdirectories in the batch job folder MUST each contain a manifest file named datastreams.txt. The manifest file specifies the intended structure of the DAMS asset, for instance specifying the order of page images, pointing to the MODS XML containing the metadata for the asset or a thumbnail image to be ingested as the TN datastream. The manifest file can also designate the language of the content, to instruct the DAMS to perform Optical Character Recognition (OCR) on the ingested pages.

Each line of the manifest file contains an argument-value pair in the following format:

<ARGUMENT>==<VALUE> 

Use 2 (two) equal signs to separate arguments and values.

MODS==modsfilename.xml
TN==thumbnailfilename.jpg
PAGE001==filename001.tif
PAGE002==filename002.tif
PAGE003==filename003.tif
LANG==eng

Manifest Arguments

The available manifest arguments are:

PAGE001==filename001.tif
PAGE002==filename002.tif
(etc.)

Folder naming conventions and folder hierarchy

Subdirectories inside of the batch job folder MUST contain one of the following suffixes as part of their name, to denote the content type or type of batch ingest:

• _PUBLICATION
• _ISSUE
• _BOOK
• _PAGES

<foldername>_PUBLICATION

Use to create a publication/series-level asset. These assets are intended to organize publication issues inside the DAMS. The DAMS GUI will show a calendar display to navigate publication issues based on creation/issuance month and year. Publication/series-level assets cannot be published to the Collections Portal and there is currently no calendar display available on the Collections Portal to browse serial issues.

A <foldername>_PUBLICATION folder can contain multiple folders with the suffix _ISSUE, in order to create the publication/series-level asset and corresponding issue-level assets at the same time.

DO NOT nest _ISSUE folders under a _PUBLICATION folder if you want to add issue-level assets to an existing publication/serial-level asset. Refer to the following section for instructions on how to add issue-level assets to an existing publication/serial-level asset.

MODS==modsfile.xml

<foldername>_ISSUE

Use to create an issue-level asset from scanned page images or a PDF.

<foldername>_ISSUE folders can be nested under a folder with the _PUBLICATION suffix, in order to create the publication/series-level asset and corresponding issue-level assets at the same time.

If you want to add an issue-level asset to an existing publication/series-level asset, DO NOT nest _ISSUE folders under a _PUBLICATION folder. Instead, place the _ISSUE folder directly inside the batch job folder and specify the host publication asset in the datastreams.txt manifest like this:
HOSTPUBLICATION==<PID of the publication/series-level asset> 

When you are ingesting PDF files as paged content with the intention to have the document split up into individual pages, prepare your source files as follows:

• Stage only the PDF file and applicable metadata or derivative datastreams. Do not stage page images along with the PDF. The DAMS software will not attempt to split a PDF file into page images if you ingest a PDF document together with page image files.
• Remove spaces from PDF filenames. If you stage a PDF file with a name containing spaces for ingest, the DAMS software will not create page images.

HOSTPUBLICATION==e0026f7d-9a79-4a8d-8a83-efc153c6a449 <-- only needed when adding an issue to an existing publication!
MODS==modsfile.xml
PAGE001==page01.tif
PAGE002==page02.tif
PAGE001_OCR_CUSTOM==page01_custom_ocr.txt
PAGE002_OCR_CUSTOM==page02_custom_ocr.txt
LANG==eng

<foldername>_BOOK

Use to create a book-level asset from scanned page images or a PDF.

When you are ingesting PDF files as paged content with the intention to have the document split up into individual pages, prepare your source files as follows:

• Stage only the PDF file and applicable metadata or derivative datastreams. Do not stage page images along with the PDF. The DAMS software will not attempt to split a PDF file into page images if you ingest a PDF document together with page image files.
• Remove spaces from PDF filenames. If you stage a PDF file with a name containing spaces for ingest, the DAMS software will not create page images.

MODS==modsfile.xml
PAGE001==page01.tif
PAGE002==page02.tif
PAGE001_OCR_CUSTOM==page01_custom_ocr.txt
PAGE002_OCR_CUSTOM==page02_custom_ocr.txt
OCR_CUSTOM==book_level_custom_ocr.txt
PDF==book_level_pdf.pdf

<foldername>_PAGES

Use to add page images and other datastreams to an existing book or issue-level asset.

Place the <foldername>_PAGES folder directly inside the batch job folder and add page images. In the datastreams.txt manifest file, add entries for each page image and specify the page order as part of the PAGE<NUMBER> argument, for example:

PAGE001==filename001.tif
PAGE002==filename002.tif
(etc.)

HOSTBOOK==e0026f7d-9a79-4a8d-8a83-efc153c6a449
PAGE001==page01.tif
PAGE002==page02.tif
PAGE001_OCR_CUSTOM==page01_custom_ocr.txt
PAGE002_OCR_CUSTOM==page02_custom_ocr.txt
OCR_CUSTOM==issue_level_custom_ocr.txt
PDF==issue_level_pdf.pdf

Sample batch folder structure

eid1234_example-batch-submission/ (batch job folder)
├── grapes_of_wrath_BOOK/
│   ├── datastreams.txt
│   ├── modsfile.xml
│	├── book_level_custom_ocr.txt
│	├── book_level_pdf.pdf
│   ├── page01.tif
│   └── page02.tif
├── wall_street_journal_PUBLICATION/
│   ├── datastreams.txt
│   ├── modsfile.xml
│   ├── wsj_jan_2016_ISSUE/
│	│	├── datastreams.txt
│	│	├── modsfile.xml
│	│	├── page01.tif
│	│	└── page02.tif
│   └── wsj_feb_2016_ISSUE/
│		├── datastreams.txt
│		├── modsfile.xml
│		├── page01.tif
│		└── page02.tif
├──	ascii_art_monthly_july_2021_ISSUE/
│   ├── datastreams.txt
│   ├── modsfile.xml
│   ├── page01.tif
│   ├── page01_custom_ocr.txt
│   ├── page02.tif
│   └── page02_custom_ocr.txt
└──	nyt_2020-11-04_PAGES/
	├── datastreams.txt
	├── issue_level_custom_ocr.txt
	├── issue_level_pdf.pdf
    ├── page01.tif
    ├── page01_custom_ocr.txt
    ├── page02.tif
    └── page02_custom_ocr.txt

Step 2: Upload batch job to Jscape

1. Connect to jscape in SFTP client:

   Host: jscape.its.utexas.edu
   port: 22

2. Upload your batch job folder into the appropriate location in Jscape:

   1. TEST corresponds to running batch on dams-t01-rh7.lib.utexas.edu, PROD corresponds to running batch on dams.lib.utexas.edu

   2. Place your batch job folder in the appropriate top-level collection folder within the INGEST folder
      Example: /DAMS/TEST/INGEST/utlmisc/my_batch_job_folder (is what I would do for a batch upload to the miscellaneous collection on the DAMS Test Server).

      Any spaces in folder names must be represented by underscores (e.g. special_collection_1).

      


      We recommend naming your batch folder with your eid, a reference to the destination collection name in the DAMS, or anything else that will help you recognize the batch. In the example <my eid>_<what I am ingesting>, the folder name would be mm63978_EnPatufet1908-1911.

3. Go over to the DAMS interface and submit your batch job to queue it to be run (see steps below).
4. Note: Your batch job folder will be removed from the JScape server after seven days whether or not you have run the batch. Back up your batch requests in box or on your local machine. 

Step 3: Set up collection and submit form in DAMS interface

1. Navigate to or create the target sub-collection to receive batch ingested files in DAMS

2. Locate and copy the target sub-collection PID to clipboard (namespace:UUID, e.g. utlarch:9ebf6ac8-1823-4bf4-8398-654b54090776)

3. Navigate to the Batch Ingest form in the DAMS:
   1. Production system: https://dams.lib.utexas.edu/utdams/batch_queue
   2. Test system: https://dams-t01-rh7.lib.utexas.edu/utdams/batch_queue
4. Select the DAMS Top-Level collection from the dropdown field
5. Paste the PID of the target sub-collection into the form
6. Enter the name of the folder on the Jscape/FTP server that contains the files to be ingested (e.g. mm63978_EnPatufet1908-1911)
7. For batch ingest of simple assets and paged content: select the appropriate ingest type
8. Click submit

The DAMS should indicate at the top of the form that the batch ingest job was queued.

You will get an email notification after your request for a batch ingest has been received and another notice once the batch ingest process has finished.