Ist there something like "touch"?

Sun, 14 Aug 2022 11:51:24 -0300

einer@digitalesparadies.de

Context

An addon calls a python script as backend. The backend changes the content of some files under webserver-root/store/nick.

Problem

The file attributes in the database (subsequently in /cloud/ ) are not updated

What I tried?

I looked at the class Code/Storage/File.php

Question 1)

I woud go for...

check wether the python script has finished (this has been accomplished for some other reasons)
check compare the modified time of the file under storage/nick/ with the "edited" timestamp in the database
update "edited" and "filesize" if needed

Is there any better approach?

Question 2)

Might it be usefull to implement a function "touch" in Code/Storage/File.php?

show all 10 comentários

Sun, 14 Aug 2022 13:47:32 -0300

Hans

hans@federate.hopto.org

Answer for 2:

https://www.php.net/manual/en/function.fopen.php

Creating a file is default implemented in php

Sun, 14 Aug 2022 17:11:33 -0300

Einer von Vielen

einer@digitalesparadies.de

Creating a file is default implemented in php

The question is about the database of Streams/Hubzilla and how to set the values there

Sun, 14 Aug 2022 20:28:55 -0300

Einer von Vielen

einer@digitalesparadies.de

OK, the best overall solution seems to update timestamp and filsize in table attach directly from the python backend as soon as it writes content to a file. It reads from the database anyway.

Thanks @Hans (hans@federate.hopto.org) for trying to find a solution. Cheers

Sun, 14 Aug 2022 20:46:08 -0300

Einer von Vielen

einer@digitalesparadies.de

Haha, I think my question was a bit tooo cryptic. Never mind the issue is solved.

Mon, 15 Aug 2022 06:31:12 -0300 última edição: Mon, 15 Aug 2022 06:55:33 -0300

Einer von Vielen

einer@digitalesparadies.de

@mike (mike@unfediverse.com) Many thanks!

I think I was not clear enough what the addon is doing and what is needed.

After sleeping about my question I think a new function "touch" in Code/Storage/File.php would be the most elegant solution.

Why? The PHP code of the addon is the ONLY place to create or delete cloud files. It checks if a csv or pkl file is present in directories that contain images (jpg, png). If not it creates one.

Background
The data about where a face is positioned in an image will go into csv and the face representations (you have to compare to match faces) into the pkl file. This way the cpu and time consuming task of face recognition can be done either on the server or remotely via webDAV what is a big advantage to "unburden" the server. (Example: my notebook connects via webdav to the server. The script has no acces to the cloud files on the server and can do the heavy lifting of detecting faces and creating face representation, later on to match the presentation using so called distance metrics.)The addon will work with or without python running on the server. Without python running on the server the addon is still able to read the data (face position in an image) an show it in the webbrowser. If the users sets a name for a face the addon writes it into the csv. Holding data along with an image in the same directory is a disadvantage for the addon because storing the data in the database of Streams would be much easier to handle. On the other side a script can do the hard work remotly via webdav (what is the primary goal) and it is very easy for other addons/scripts/applications to use the data. This all is implemented and working. I give it more intense testing at the moment.

Back to the topic....
The Python backend will NEVER create or delete files. It just looks if a csv or pkl (binary for storing python objects) is present. If yes it reads the file content. At some point content is written back (to existing files only).

The python script will get the relativ path to webroot/store/nick with a query like this

query = ("SELECT display_path, os_path " "FROM `attach` " "WHERE `uid` = %s AND " "`folder` = %s AND " "`is_photo` = '1' AND " "( `filetype` = 'image/jpeg' OR `filetype` = 'image/png')")

and use os_path to find the file in the filesystem of the webserver.

The filesize and timestamp change if content is written by python.

The "missing part"...
The database is not updated - and the file is not synchronized to clones.

Looking into Code/Storage/File.php the most elegant way would be to let this class check for an updated filesize and timestamp there and synchronize it there if any change happened.

The addon will iterate through the directories anyway to check the presence of a csv/pkl file. If it finds an existing one it could call a function touch in Code/Storage/File.php.

Of course the addon can implement it. This is ok but has the disadvantage of beeing not reusable and might not work over time. It seems more reliable to do this in a function touch in Code/Storage/File.php.

What do you think?

If ok I could come up with a proposal (code) for Code/Storage/File.php. (To be honest I trust you much more than me if it comes to code quality.)

Mon, 15 Aug 2022 06:33:25 -0300

Einer von Vielen

einer@digitalesparadies.de

OMG there are some important typos in the text.... Wait...

Mon, 15 Aug 2022 06:54:58 -0300

Einer von Vielen

einer@digitalesparadies.de

I think I was not clear enough what the addon is doing and what is needed.

After sleeping about my question I think a new function "touch" in Code/Storage/File.php would be the most elegant solution.

Why? The PHP code of the addon is the ONLY place to create or delete cloud files. It checks if a csv or pkl file is present in directories that contain images (jpg, png). If not it creates one.

Background
The name of a face and the data about where a face is positioned in an image will go into csv and the face representations into the pkl file. (The representations are used to match faces.) This way the cpu and time consuming task of face recognition can be done either on the server or remotely via webDAV what is a big advantage to "unburden" the server. (Example: my notebook connects via webdav to the server. A script on my notebook has now acces to the cloud files on the server and can do the heavy lifting of detecting faces and creating face representation. Later it match the face representations using distance metrics.) The addon will work with or without python running on the server. Without python running on the server the addon is still able to read the data (name of a face and its position in an image) and to show it in the webbrowser. If the users sets a name for a face the addon writes it into the csv.

Holding data in files is a disadvantage for the addon because storing the data in the database of Streams would be much easier to handle. On the other side storing the data together with an image in the same directory is an advantage because a script can do the hard work remotly connected via webdav (what is the primary goal) and it is very easy for other addons/scripts/applications to use the data. This all is implemented and working. I give it more intense testing at the moment.

Back to the topic....
The Python backend will NEVER create or delete files. It just looks if a csv or pkl (binary for storing python objects) is present. If yes it reads the file content. At some point content is written back (to existing files only).

The python script will get the relativ path to webroot/store/nick with a query like this

query = ("SELECT display_path, os_path " "FROM `attach` " "WHERE `uid` = %s AND " "`folder` = %s AND " "`is_photo` = '1' AND " "( `filetype` = 'image/jpeg' OR `filetype` = 'image/png')")

and use os_path to find the file in the filesystem of the webserver.

The "missing part"...
The filesize and timestamp of a file change if content is written by python.
BUT the database is not updated - and the file is not synchronized to clones.

Looking into Code/Storage/File.php the most elegant way would be to let this class check for an updated filesize and timestamp. If any change happened it can update the database and synchronize the file to clones.

The addon will iterate through the directories anyway to check the presence of csv and pkl files. If it finds existing ones it could call a function touch in Code/Storage/File.php.

Of course the addon can implement it. This is ok but has the disadvantage of beeing not reusable and might not work over time. It seems more reliable to do this in a function touch in Code/Storage/File.php.

What do you think?

If ok I could come up with a proposal (code) for Code/Storage/File.php. (To be honest I trust you much more than me if it comes to code quality.)

Mon, 15 Aug 2022 08:23:44 -0300

Alexandre Hannud Abdo

aleabdo@hubzilla.com.br

FWIW, you might want to reconsider using CSV and, specially, pickle here. For CSV you can easily use JSON instead (and adding a version number field to the stored object would be nice), it should make it much easier for other add-ons, or improvements to your add-on, to interoperate. And pickle is not a safe data exchange nor long-term storage format. Because you can run arbitrary code when loading a pickle, your use case looks like a security problem. It's also dependent on the version of Python. You should probably serialize/deserialize your objects as JSON (or another safe format) instead of pickling them. In which case you could even use the same file for both data if it makes sense.

Mon, 15 Aug 2022 08:54:06 -0300

Einer von Vielen

einer@digitalesparadies.de

@Alexandre Hannud Abdo (aleabdo@hubzilla.com.br) Good points. Thanks for your thoughts! Nothing is release yet. So the best moment to change something would be NOW. Thinking....

Mon, 15 Aug 2022 16:27:03 -0300

Einer von Vielen

einer@digitalesparadies.de

@Alexandre Hannud Abdo (aleabdo@hubzilla.com.br) Ok, I changed the file formats from csv to json and pkl to parquet in the python scripts. Now a lot to change in PHP and JavaScript ;-) But this is just a matter of spare time.