How to include metadata from external file

Questions, comments, discussions. Over time certain topics might be moved to their own category.
Post Reply
bwakkie
Posts: 16
Joined: Fri Feb 14, 2020 8:28 pm

How to include metadata from external file

Post by bwakkie »

Hi all,

I'm moving 9000 documents from FileMaker to Mayan EDMS.
I have extracted all metadata from FileMaker in tsv format. Per line a number refers to the corresponding pdf file (3456.pdf 500.pdf 9234.pdf etc). The pdf's where stored on a regular filesystem.
How do I import this metadata to the correct documents?

Maybe directly in de db after I recreated all fields as metadata fields?

Does someone have experience with this? Any tip would be nice!

Cheers,

Bastiaan
User avatar
michael
Developer
Developer
Posts: 48
Joined: Sun Apr 19, 2020 6:21 am

Re: How to include metadata from external file

Post by michael »

Howdy, because there are so many way to import documents into Mayan, most import projects are custom for each client. Based on the experience accumulated over many projects we've started to narrow down an import paradigm. We are working on an universal import tool. There is already an MVP and once the code is stable it will become part of the core set of apps of Mayan.

For now, the suggestion is to do an import loop that creates the documents directly using the ."new_document" method from the document_type instance:

Code: Select all

with open(file_to_import, mode='rb') as file_object:
    document = document_type.new_document(
        file_object=file_object
    )
This will run synchronously and block but it will be the fastest and easiest implementation.

Metadata mapping is a whole different problem because the mapping will be unique for each import. For your application I suggest you just hardcode the CVS columns to the metadate type instances.

You can also write an API client to the import and the advantage is that you can write this in any language you are most comfortable with, but it will be a bit slower due to all happening over HTTP.

We expect the import app to be included during the series 3.6 cycle.
bwakkie
Posts: 16
Joined: Fri Feb 14, 2020 8:28 pm

Re: How to include metadata from external file

Post by bwakkie »

Hi Michael,

That's good to hear in import tool will be integrated at one point!
In the end I placed all document in the watch folder (-+50Gb) and all where absorbed as Default Document Type.

I am still very much a noob at workflow setup, but my though was to indeed create the matadata fields that where in the filemaker database and then import them directly into postgresql mayan db. I suppose that will work? Are there any problems with this approach I need to be aware of?

Cheers,
Bastiaan
Post Reply