General
Meet few important directories:
/images/ - real(!) files organized by date (year.month)
/collection/ - symlinks organized by labeled groups
/home/ - scripts collection: file upload and symlinks service
See scripts decription below;
/imports/ - lists imports by date, represented as symlinks.
upload.py
python3 upload.py [--recursive] [--interactive] source [label_1 label_2 ..]
Non-interactive:
Copy images without individual review.
Additional arguments will create symbolic links to the whole group.
Examples:
Copy new files from SD card mounted '/media/mmcblk0' to 'images/'
do not preserve original directory structure,
create symbolic links in 'collection/mountains' and 'collection/elbrus'
`python3 upload.py -r -d "/media/mmcblk0/DCIM" elbrus mountains`
Interactive mode reference:
Review images in terminal, assign individual image labels.
Groups of similar images, i.e bursts are supposed to be uploaded by just one image review.
Slash (/) is allowed and will create sub-directories.
You're asked to create path if it does not exist, default question answers are highlighted.
Type 'exit' to leave interface.
Examples:
`python3 upload.py -i -r /media/mmcblk0/DCIM`
> [image]
> (labels) .. mountains/elbrus landscape snow
> Unknown keyword 'snow', create? (y/N) .. y
Additional options:
--blind will ignore file duplications and create given symlinks to files.
Files will not be writen twice unless have different names and no metadata.
--recursive will also copy subdirectories preserving the file structure in
non-interactive mode and staging for review all subdirectories in interactive mode.
--dcim will break any subdirectories even if --recusive used non-interactively.
symlinks.py
python3 symlinks.py
Can restore broken symlinks and alerts about removed files.
!never remove image files, but remove symlinks instead.
--analize option will output extended information: number of images tracked,
images untracked, long names, also find images not represented as links and
make symlinks to home/.unlinked
genchecksums.py
python3 genchecksums.py
Generate integrity verification file: checksums.md5
Verify the correctness of the image files with command:
md5sum --check checksums.md5 | grep FAILED
Generate reed-solomon codes for file error recovery
Combine find and par2 commands:
find images -type f -not -name "*.par2" -exec par2create {} \;
To restore corrupted file later:
par2repair /path/to/file
The default amount of restoration data is enough to
back metadata header erased with exiftool.
Then you should probably remove the corrupted file.
Export file group by label
As simple as tar command:
tar -h -czvf label.tar.gz -C collection label
To actually copy files without writing and decompressing archive use pipes:
tar -czhf - -C collection label | mbuffer -m 128k | tar xf - -C destination/
mbuffer is supposed to prevent mem leaks and optimize perfomance.
buffer size can be adjusted, but I found an explanation 128k is fastest.
Actually, my laptop still stopped responding when copying 200GB between HDDs.
That's said to be ok with linux kernel, will search futher solutions.
Note: larger transfers appear to need larger buffer.
