Leaving files in 'Watch Folder'

When things don't work as they should.
Krzysztof
Posts: 11
Joined: Wed Aug 22, 2018 11:28 am

Leaving files in 'Watch Folder'

Post by Krzysztof »

Roberto,

Mayan does not remove files from 'Watch Folder' and adds the same files over and over again.

Is it possible to get around it or is it the 'Watch Folder' principle?

Regards,
Krzysztof

KevinPawsey
50 Posts
50 Posts
Posts: 85
Joined: Wed Aug 22, 2018 2:52 pm

Re: Leaving files in 'Watch Folder'

Post by KevinPawsey »

I had this issue before, and I managed to pin the issue down to the following:

I was using a Mac to connect to an SMB share... it turns out that there is something in the way that the Mac writes to SMB shares. I used to get 2-3 copies of the same file because the file wasn't unlocked to be able to be deleted, so it remained in the folder to be re-scanned.

Not sure if this is your setup, but what improved the situation for me was to stop sharing the watched folder as SMB, and share it as AppleTalk, and that enabled the file lock to unlock properly and be deleted after the folder scan.

Hope that helps.

Kevin
Running Mayan-EDMS on: OpenMediaVault, (Docker plugin), on x86 dual-core

teqqyde
Posts: 1
Joined: Mon Jan 28, 2019 10:16 am

Re: Leaving files in 'Watch Folder'

Post by teqqyde »

Hi,
I've got the same issue. I created a folder, installed apache2 and configured webdav. Now I'm using my Phone with the ScannerPro App and save the pdf to my webdav folder.
The pdf is recognised by the system, but leave for always there. The first time i recognised it, i got over 1200 documents from one pdf in Mayan.

So, is there any automatic workaround in Mayan to not run in the same issue another time? I cannot create a script to remove all files from this folder, because i can be deleted and not be indexed by mayan if the trigger "goes wrong".

Thx.

User avatar
rosarior
Developer
Developer
Posts: 505
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Leaving files in 'Watch Folder'

Post by rosarior »

When this happens it is usually caused by a missing filesystem permission. Check to see if the Mayan system user account has the filesystem permission to delete files from the watch folder.

KevinPawsey
50 Posts
50 Posts
Posts: 85
Joined: Wed Aug 22, 2018 2:52 pm

Re: Leaving files in 'Watch Folder'

Post by KevinPawsey »

Just to add to this, and in case it is of use to anyone else...

I run Mac OSX and I scan things straight to a watch folder... when a drive is mapped in MacOS it writes a ._[filename] file to the folder, which caused me issues when I was using the staging folder. I was seeing these ._ files as being available for import.

I discovered that this is something that was quite common in OSX, and there is a way of switching this off for network folders.

Run Terminal:
- Command Spacebar, type "terminal" (without quotes) and press enter
At the prompt (assuming you have permissions to run terminal on your machine) type in the following command:

Code: Select all

defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true
and then press enter... it will look like nothing has been done. Then type the following:

Code: Select all

killall Finder
... this will restart Finder (MacOS equivalent of windows explorer).
Voila! ... no more ._ files on network drives.

For the above, where is says "type in" you can also copy/paste of course.

This should help on Watch Folders, whereby these ._ files are created automatically if you have the drive mapped, but can't be imported... keeps things tidy. These files wont show up in Finder on the MacOS, as they are a part of how the OS works... but you can see them by going to a CLI on Linux, changing to the Watch Folder, and then doing a:

Code: Select all

ls -al
This will show all files, and if you want to remove them, then the following command should remove them:

Code: Select all

rm ._*
not tested the above command... use with caution

Anyway, hope that helps someone
Running Mayan-EDMS on: OpenMediaVault, (Docker plugin), on x86 dual-core

qweqweqwe
Posts: 3
Joined: Wed Feb 27, 2019 11:16 pm

Re: Leaving files in 'Watch Folder'

Post by qweqweqwe »

Check permissions. My folder allowed read and execute for mayan user but not delete. Use chmod.

KevinPawsey
50 Posts
50 Posts
Posts: 85
Joined: Wed Aug 22, 2018 2:52 pm

Re: Leaving files in 'Watch Folder'

Post by KevinPawsey »

qweqweqwe wrote:
Wed Feb 27, 2019 11:17 pm
Check permissions. My folder allowed read and execute for mayan user but not delete. Use chmod.
If you are going to use chmod, I believe that the chmod command should be

Code: Select all

chmod 755 -R {folder_name}
Hope that helps.
Running Mayan-EDMS on: OpenMediaVault, (Docker plugin), on x86 dual-core

itsthejb
Posts: 8
Joined: Thu Sep 26, 2019 11:21 am

Re: Leaving files in 'Watch Folder'

Post by itsthejb »

Hi all,

Also having this issue and struggling to solve it. Everything seems ok, but still the files aren't deleted (although they're imported correctly). In my case, the watch folder is a samba share on the server where a scanner dumps files. Example:
drwxrws---+ 3 media share 4096 Nov 17 22:12 .
drwxrwxr-x. 10 media share 4096 Nov 17 10:56 ..
drw-rwS---+ 2 media share 4096 Nov 17 19:58 .deleted
-rwxrwx---+ 1 media share 91938 Nov 17 22:13 'scan_2019-11-17 21.57.35.pdf'
Scanned file at the bottom. Running in Docker, the Mayan container sees:
$ docker exec mayan ls -l /watch
total 96
-rwxrwx---+ 1 mayan mayan 91938 Nov 17 22:13 scan_2019-11-17 21.57.35.pdf
The directory has these ACLs:
# file: .
# owner: media
# group: share
# flags: -s-
user::rwx
group::rwx
other::---
default:user::rwx
default:group::rwx
default:other::---
It seems like it should work fine. It certainly worked before I was using the Samba share, but not sure why this would break it (it's a regular directory on the server that Mayan sees)

Hope someone can help

itsthejb
Posts: 8
Joined: Thu Sep 26, 2019 11:21 am

Re: Leaving files in 'Watch Folder'

Post by itsthejb »

Hi all,

If it helps at all, I solved my problem: the issue wasn't permission at all, it was that files are only deleted when the source is checked by the work based on its interval. I had been using the Check Now button to manual check the directory, and it appears this does not also perform the deletion.

@rosarior is this a feature that could be added?

User avatar
rosarior
Developer
Developer
Posts: 505
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Leaving files in 'Watch Folder'

Post by rosarior »

Hi,

This is by design since you are requesting a test and this avoids deleting production files.
itsthejb wrote:
Sat Dec 07, 2019 4:26 pm
If it helps at all, I solved my problem: the issue wasn't permission at all, it was that files are only deleted when the source is checked by the work based on its interval. I had been using the Check Now button to manual check the directory, and it appears this does not also perform the deletion.
2019-12-07_12-32.png
2019-12-07_12-32.png (20.34 KiB) Viewed 779 times
itsthejb wrote:
Sat Dec 07, 2019 4:26 pm
@rosarior is this a feature that could be added?
Yes, we can add a checkbox to allow users to delete files during tests with it defaulting to unchecked to preserve the current behavior.

Post Reply