/
Compliance

Compliance

Jun 21, 2018

CSV file inputs

The CSV file input will have the details of the compliance model.

The script should be able to take the following inputs (the order of listing in this document is inconsequential):

  1. a CSV file containing a list of compliance details

Compliance metadata

Write about the compliance metadata.

Requirements for the Python Script

  1. The script should be executable from the command-line, and should take inputs from the command line.

  2. The documents should be available in the database with series and/or subseries as the key.

Interface

The script will be a command-line utility that will serve the requirements as outlined above.

Invocations of the script would look like this:

python3 compliance.py [OPTION]... -f CSV

CSV refers to the CSV file that can be supplied as argument to process. The ellipsis refers to any switch-specific arguments that might be needed.

OPTION represents a the following set of switches that can be applied to modify the default behavior of the script:

Switch

Argument

Description

-f

Path to CSV file

Path to CSV file for batch processing.

-q

(none)

Quiet mode. Disables all informational prints. All exception and error related prints will still be output.

-h

(none)

The script displays a help document on the screen and exits.

Architecture

  1. Changes in the globalvars.py file to incorporate global variables. Added global variables which is used extensively in the python script.

  2. Build the compliance metadatautils file based on the metadata schema. The attributes of the metadata schema is formed in the compliance file under metadautils folder.

Behavior and Implementation

The script (compliance.py) will perform the following high-level operations:

  1. Parse command line arguments

    1. set variables in accordance with the arguments

    2. inform user about errors in the arguments, print help, and exit

  2. Read CSV file

    1. validate header structure

    2. parse 'arrange' information from header for series, sub-series

    3. parse 'compliance' information from header

  3. Read metadata property names from labels.json

    1. store all labels within a Python object

  4. Read controlled vocabulary from vocab.json

    1. store the vocabulary as a Python object

  5. Create a connection to the database

  6. For each row of series, sub-series combination in the CSV file:

    1. create a compliance record

      1. read compliance information from CSV file

      2. store information from file into an object

      3. create a compliance property using this information

    2. Query database for all documents specified by the series, sub-series

    3. For each document in the list:

        1. read the document, if profile is there, save information in an object

        2. attach compliance property to document

        3. add preservation event for successfully updated documents

          1. metadata event - include new and old fields

Output(s)

Print helpful information in a csv file when errors were encountered. Errors to be reported include errors in command usage, as well as any errors encountered while carrying out property extraction. Error csv name: "compliance_profile_errors_<timestamp>.csv"

Test cases / Validation

  1. No header in the csv file should print an error.

  2. Update an existing record with compliance metadata by passing both series and subseries information.

  3. Update an existing record with compliance metadata by passing only series information.

  4. Update an existing record with compliance metadata by passing only subseries information.

  5. Update an existing record with compliance metadata by not passing both series and subseries information.

  6. Update a record which has already a compliance data.