#112 Mastodon attached images not saved/shown in GS 2.0-dev

Open
opened 5 months ago by colegota · 7 comments

Hi!

One of federation problems I've seen it's that images from Mastodon are not shown at timelines.

Looking between logs and code seems that images and other media came as an [attachment] array that is not managed by GS and not saved with the notice.

Runnig nightly/2.0-dev with Apache and .htaccess sample.

Did increased debug traces so I can document a little more.

When an "object" with a notice from Mastodon came to create_notice() method it includes the array with the attachment.

2020-02-17 13:50:26 LOG_INFO: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] HTTPClient: HTTP GET https://mastodon.social/users/CarmePont/statuses/103674488902743883 - 200 OK
2020-02-17 13:50:26 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [notice] object
Array
(
    [@context] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams
            [1] => Array
                (
                    [ostatus] => http://ostatus.org#
                    [atomUri] => ostatus:atomUri
                    [inReplyToAtomUri] => ostatus:inReplyToAtomUri
                    [conversation] => ostatus:conversation
                    [sensitive] => as:sensitive
                    [toot] => http://joinmastodon.org/ns#
                    [votersCount] => toot:votersCount
                    [blurhash] => toot:blurhash
                    [focalPoint] => Array
                        (
                            [@container] => @list
                            [@id] => toot:focalPoint
                        )

                )

        )

    [id] => https://mastodon.social/users/CarmePont/statuses/103674488902743883
    [type] => Note
    [summary] => 
    [inReplyTo] => 
    [published] => 2020-02-17T13:45:57Z
    [url] => https://mastodon.social/@CarmePont/103674488902743883
    [attributedTo] => https://mastodon.social/users/CarmePont
    [to] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams#Public
        )

    [cc] => Array
        (
            [0] => https://mastodon.social/users/CarmePont/followers
        )

    [sensitive] => 
    [atomUri] => https://mastodon.social/users/CarmePont/statuses/103674488902743883
    [inReplyToAtomUri] => 
    [conversation] => tag:mastodon.social,2020-02-17:objectId=154360488:objectType=Conversation
    [content] => <p>Descerebrat</p>
    [contentMap] => Array
        (
            [ca] => <p>Descerebrat</p>
        )

    [attachment] => Array
        (
            [0] => Array
                (
                    [type] => Document
                    [mediaType] => image/jpeg
                    [url] => https://files.mastodon.social/media_attachments/files/025/269/297/original/1cd32f26e31bf634.jpeg
                    [name] => 
                    [blurhash] => UOHx+-9G_2Rk9G00Rk%M^,ofxut7V@IBofIo
                )

        )

    [tag] => Array
        (
        )

    [replies] => Array
        (
            [id] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies
            [type] => Collection
            [first] => Array
                (
                    [type] => CollectionPage
                    [next] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies?only_other_accounts=true&page=true
                    [partOf] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies
                    [items] => Array
                        (
                        )

                )

        )

)

And it's supposed that in that function, attachment should be collected and added to the notice, but it doesn't.

This is the notice content at the end of create_notice().


2020-02-17 13:50:26 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [notice] note
Notice Object
(
    [__table] => notice
    [id] => 16071
    [profile_id] => 556
    [uri] => https://mastodon.social/users/CarmePont/statuses/103674488902743883
    [content] => Descerebrat
    [rendered] => <p>Descerebrat</p>
    [url] => https://mastodon.social/@CarmePont/103674488902743883
    [created] => 2020-02-17 13:50:26
    [modified] => 
    [reply_to] => 
    [is_local] => 0
    [source] => ActivityPub
    [conversation] => 7439
    [repeat_of] => 
    [verb] => http://activitystrea.ms/schema/1.0/post
    [object_type] => http://activitystrea.ms/schema/1.0/note
    [scope] => 0
    [_profile:protected] => Array
        (
            [556] => Profile Object
                (
                    [__table] => profile
                    [id] => 556
                    [nickname] => CarmePont
                    [fullname] => Carme Pont Bonsfills
                    [profileurl] => https://mastodon.social/users/CarmePont
                    [homepage] => 
                    [bio] => 
                    [location] => 
                    [lat] => 
                    [lon] => 
                    [location_id] => 
                    [location_ns] => 
                    [created] => 2020-01-25 11:20:17
                    [modified] => 2020-01-25 11:20:17
                    [_user:protected] => Array
                        (
                        )

                    [_group:protected] => Array
                        (
                        )

                    [_DB_DataObject_version] => 1.11.3
                    [N] => 1
                    [_database_dsn] => 
                    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
                    [_database] => vps13rr4_gsrsn
                    [_query] => Array
                        (
                            [condition] => 
                            [group_by] => 
                            [order_by] => 
                            [having] => 
                            [useindex] => 
                            [limit_start] => 
                            [limit_count] => 
                            [data_select] => *
                            [unions] => Array
                                (
                                )

                            [derive_table] => 
                            [derive_select] => 
                        )

                    [_DB_resultid] => 
                    [_resultFields] => 
                    [_link_loaded] => 
                    [_join] => 
                    [_lastError] => 
                )

        )

    [_attachments:protected] => Array
        (
        )

    [_attentionids:protected] => Array
        (
        )

    [_replies:protected] => Array
        (
        )

    [_groups:protected] => Array
        (
        )

    [_DB_DataObject_version] => 1.11.3
    [N] => 0
    [_database_dsn] => 
    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
    [_database] => vps13rr4_gsrsn
    [_query] => Array
        (
            [condition] => 
            [group_by] => 
            [order_by] => 
            [having] => 
            [useindex] => 
            [limit_start] => 
            [limit_count] => 
            [data_select] => *
            [unions] => Array
                (
                )

            [derive_table] => 
            [derive_select] => 
        )

    [_DB_resultid] => 22
    [_resultFields] => 
    [_link_loaded] => 
    [_join] => 
    [_lastError] => 
)

And at Notice::saveNew() in Notice.php.

2020-02-17 13:50:27 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [Notice.php] saveNew() notice
Notice Object
(
    [__table] => notice
    [id] => 16072
    [profile_id] => 205
    [uri] => tag:gnusocial.sierranorte.red,2020-02-17:noticeId=16072:objectType=note
    [content] => RT @CarmePont Descerebrat
    [rendered] => RT @CarmePont Descerebrat
    [url] => 
    [created] => 2020-02-17 13:50:27
    [modified] => 
    [reply_to] => 
    [is_local] => 1
    [source] => ActivityPub
    [conversation] => 7439
    [repeat_of] => 16071
    [verb] => http://activitystrea.ms/schema/1.0/share
    [object_type] => http://activitystrea.ms/schema/1.0/note
    [scope] => 0
    [_profile:protected] => Array
        (
            [205] => Profile Object
                (
                    [__table] => profile
                    [id] => 205
                    [nickname] => Edgecontrol
                    [fullname] => 
                    [profileurl] => https://mastodon.social/users/Edgecontrol
                    [homepage] => 
                    [bio] => 
                    [location] => 
                    [lat] => 
                    [lon] => 
                    [location_id] => 
                    [location_ns] => 
                    [created] => 2020-01-22 14:48:04
                    [modified] => 2020-02-17 13:50:26
                    [_user:protected] => Array
                        (
                        )

                    [_group:protected] => Array
                        (
                        )

                    [_DB_DataObject_version] => 1.11.3
                    [N] => 1
                    [_database_dsn] => 
                    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
                    [_database] => vps13rr4_gsrsn
                    [_query] => Array
                        (
                            [condition] => 
                            [group_by] => 
                            [order_by] => 
                            [having] => 
                            [useindex] => 
                            [limit_start] => 
                            [limit_count] => 
                            [data_select] => *
                            [unions] => Array
                                (
                                )

                            [derive_table] => 
                            [derive_select] => 
                        )

                    [_DB_resultid] => 
                    [_resultFields] => 
                    [_link_loaded] => 
                    [_join] => 
                    [_lastError] => 
                    [avatar] => https://files.mastodon.social/accounts/avatars/001/014/154/original/4245b5e5de31937f.jpeg
                )

        )

    [_attachments:protected] => Array
        (
        )

    [_attentionids:protected] => Array
        (
        )

    [_replies:protected] => Array
        (
        )

    [_groups:protected] => Array
        (
        )

    [_DB_DataObject_version] => 1.11.3
    [N] => 0
    [_database_dsn] => 
    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
    [_database] => vps13rr4_gsrsn
    [_query] => Array
        (
            [condition] => 
            [group_by] => 
            [order_by] => 
            [having] => 
            [useindex] => 
            [limit_start] => 
            [limit_count] => 
            [data_select] => *
            [unions] => Array
                (
                )

            [derive_table] => 
            [derive_select] => 
        )

    [_DB_resultid] => 
    [_resultFields] => 
    [_link_loaded] => 
    [_join] => 
    [_lastError] => 
)

Refs:

Original notice at mastodon.social https://mastodon.social/@CarmePont/103674488902743883

Notice at my node https://gnusocial.sierranorte.red/conversation/7439#notice-16071

Hi! One of federation problems I've seen it's that images from Mastodon are not shown at timelines. Looking between logs and code seems that images and other media came as an [attachment] array that is not managed by GS and not saved with the notice. Runnig nightly/2.0-dev with Apache and .htaccess sample. Did increased debug traces so I can document a little more. When an "object" with a notice from Mastodon came to [create_notice() method ](https://notabug.org/diogo/gnu-social/src/nightly/plugins/ActivityPub/lib/models/Activitypub_notice.php#L126) it includes the array with the attachment. ~~~ 2020-02-17 13:50:26 LOG_INFO: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] HTTPClient: HTTP GET https://mastodon.social/users/CarmePont/statuses/103674488902743883 - 200 OK 2020-02-17 13:50:26 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [notice] object Array ( [@context] => Array ( [0] => https://www.w3.org/ns/activitystreams [1] => Array ( [ostatus] => http://ostatus.org# [atomUri] => ostatus:atomUri [inReplyToAtomUri] => ostatus:inReplyToAtomUri [conversation] => ostatus:conversation [sensitive] => as:sensitive [toot] => http://joinmastodon.org/ns# [votersCount] => toot:votersCount [blurhash] => toot:blurhash [focalPoint] => Array ( [@container] => @list [@id] => toot:focalPoint ) ) ) [id] => https://mastodon.social/users/CarmePont/statuses/103674488902743883 [type] => Note [summary] => [inReplyTo] => [published] => 2020-02-17T13:45:57Z [url] => https://mastodon.social/@CarmePont/103674488902743883 [attributedTo] => https://mastodon.social/users/CarmePont [to] => Array ( [0] => https://www.w3.org/ns/activitystreams#Public ) [cc] => Array ( [0] => https://mastodon.social/users/CarmePont/followers ) [sensitive] => [atomUri] => https://mastodon.social/users/CarmePont/statuses/103674488902743883 [inReplyToAtomUri] => [conversation] => tag:mastodon.social,2020-02-17:objectId=154360488:objectType=Conversation [content] => <p>Descerebrat</p> [contentMap] => Array ( [ca] => <p>Descerebrat</p> ) [attachment] => Array ( [0] => Array ( [type] => Document [mediaType] => image/jpeg [url] => https://files.mastodon.social/media_attachments/files/025/269/297/original/1cd32f26e31bf634.jpeg [name] => [blurhash] => UOHx+-9G_2Rk9G00Rk%M^,ofxut7V@IBofIo ) ) [tag] => Array ( ) [replies] => Array ( [id] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies [type] => Collection [first] => Array ( [type] => CollectionPage [next] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies?only_other_accounts=true&page=true [partOf] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies [items] => Array ( ) ) ) ) ~~~ And it's supposed that in that function, attachment should be collected and added to the notice, but it doesn't. This is the notice content at the end of create_notice(). ~~~ 2020-02-17 13:50:26 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [notice] note Notice Object ( [__table] => notice [id] => 16071 [profile_id] => 556 [uri] => https://mastodon.social/users/CarmePont/statuses/103674488902743883 [content] => Descerebrat [rendered] => <p>Descerebrat</p> [url] => https://mastodon.social/@CarmePont/103674488902743883 [created] => 2020-02-17 13:50:26 [modified] => [reply_to] => [is_local] => 0 [source] => ActivityPub [conversation] => 7439 [repeat_of] => [verb] => http://activitystrea.ms/schema/1.0/post [object_type] => http://activitystrea.ms/schema/1.0/note [scope] => 0 [_profile:protected] => Array ( [556] => Profile Object ( [__table] => profile [id] => 556 [nickname] => CarmePont [fullname] => Carme Pont Bonsfills [profileurl] => https://mastodon.social/users/CarmePont [homepage] => [bio] => [location] => [lat] => [lon] => [location_id] => [location_ns] => [created] => 2020-01-25 11:20:17 [modified] => 2020-01-25 11:20:17 [_user:protected] => Array ( ) [_group:protected] => Array ( ) [_DB_DataObject_version] => 1.11.3 [N] => 1 [_database_dsn] => [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560 [_database] => vps13rr4_gsrsn [_query] => Array ( [condition] => [group_by] => [order_by] => [having] => [useindex] => [limit_start] => [limit_count] => [data_select] => * [unions] => Array ( ) [derive_table] => [derive_select] => ) [_DB_resultid] => [_resultFields] => [_link_loaded] => [_join] => [_lastError] => ) ) [_attachments:protected] => Array ( ) [_attentionids:protected] => Array ( ) [_replies:protected] => Array ( ) [_groups:protected] => Array ( ) [_DB_DataObject_version] => 1.11.3 [N] => 0 [_database_dsn] => [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560 [_database] => vps13rr4_gsrsn [_query] => Array ( [condition] => [group_by] => [order_by] => [having] => [useindex] => [limit_start] => [limit_count] => [data_select] => * [unions] => Array ( ) [derive_table] => [derive_select] => ) [_DB_resultid] => 22 [_resultFields] => [_link_loaded] => [_join] => [_lastError] => ) ~~~ And at [Notice::saveNew() in Notice.php](https://notabug.org/diogo/gnu-social/src/nightly/classes/Notice.php#L465). ~~~ 2020-02-17 13:50:27 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [Notice.php] saveNew() notice Notice Object ( [__table] => notice [id] => 16072 [profile_id] => 205 [uri] => tag:gnusocial.sierranorte.red,2020-02-17:noticeId=16072:objectType=note [content] => RT @CarmePont Descerebrat [rendered] => RT @CarmePont Descerebrat [url] => [created] => 2020-02-17 13:50:27 [modified] => [reply_to] => [is_local] => 1 [source] => ActivityPub [conversation] => 7439 [repeat_of] => 16071 [verb] => http://activitystrea.ms/schema/1.0/share [object_type] => http://activitystrea.ms/schema/1.0/note [scope] => 0 [_profile:protected] => Array ( [205] => Profile Object ( [__table] => profile [id] => 205 [nickname] => Edgecontrol [fullname] => [profileurl] => https://mastodon.social/users/Edgecontrol [homepage] => [bio] => [location] => [lat] => [lon] => [location_id] => [location_ns] => [created] => 2020-01-22 14:48:04 [modified] => 2020-02-17 13:50:26 [_user:protected] => Array ( ) [_group:protected] => Array ( ) [_DB_DataObject_version] => 1.11.3 [N] => 1 [_database_dsn] => [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560 [_database] => vps13rr4_gsrsn [_query] => Array ( [condition] => [group_by] => [order_by] => [having] => [useindex] => [limit_start] => [limit_count] => [data_select] => * [unions] => Array ( ) [derive_table] => [derive_select] => ) [_DB_resultid] => [_resultFields] => [_link_loaded] => [_join] => [_lastError] => [avatar] => https://files.mastodon.social/accounts/avatars/001/014/154/original/4245b5e5de31937f.jpeg ) ) [_attachments:protected] => Array ( ) [_attentionids:protected] => Array ( ) [_replies:protected] => Array ( ) [_groups:protected] => Array ( ) [_DB_DataObject_version] => 1.11.3 [N] => 0 [_database_dsn] => [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560 [_database] => vps13rr4_gsrsn [_query] => Array ( [condition] => [group_by] => [order_by] => [having] => [useindex] => [limit_start] => [limit_count] => [data_select] => * [unions] => Array ( ) [derive_table] => [derive_select] => ) [_DB_resultid] => [_resultFields] => [_link_loaded] => [_join] => [_lastError] => ) ~~~ Refs: Original notice at mastodon.social https://mastodon.social/@CarmePont/103674488902743883 Notice at my node https://gnusocial.sierranorte.red/conversation/7439#notice-16071
roytam commented 4 months ago

I think that part is not being handled in plugin(s).

I think that part is not being handled in plugin(s).
Diogo Cordeiro commented 4 months ago
Owner

@roytam is correct.

It's true that we don't check the attachments array of the remote notes in ActivityPub (although our notes are sent with one).

We only retrieve a remote attachment if that comes as part of the note's content (in the form of a link) (due to Embed plugin).

We intend to support fetching attachments from remote notices' attachment array in the future, we just didn't implement yet because notes usually come with links to the remote attachments and we didn't want to deal with the disambiguation of what is in the array as well as in the note's body and what's not.

Ideally these fallback urls wouldn't be included in the notice's body when federated via ActivityPub. This is yet something that requires further study and that we've been postponing as we don't see it as a priority for now...

(@roytam who are you btw?)

@roytam is correct. It's true that we don't check the attachments array of the remote notes in ActivityPub (although our notes are sent with one). We only retrieve a remote attachment if that comes as part of the note's content (in the form of a link) (due to Embed plugin). We intend to support fetching attachments from remote notices' attachment array in the future, we just didn't implement yet because notes usually come with links to the remote attachments and we didn't want to deal with the disambiguation of what is in the array as well as in the note's body and what's not. Ideally these fallback urls wouldn't be included in the notice's body when federated via ActivityPub. This is yet something that requires further study and that we've been postponing as we don't see it as a priority for now... (@roytam who are you btw?)
roytam commented 4 months ago

we just didn't implement yet because notes usually come with links to the remote attachments

IIRC only pleroma does this. mastodon never put attachment link into notice body.

(@roytam who are you btw?)

just a normal user. ;)

> we just didn't implement yet because notes usually come with links to the remote attachments IIRC only pleroma does this. mastodon never put attachment link into notice body. > (@roytam who are you btw?) just a normal user. ;)
roytam commented 4 months ago

We intend to support fetching attachments from remote notices' attachment array in the future

by using StoreRemoteMedia plugin?

> We intend to support fetching attachments from remote notices' attachment array in the future by using StoreRemoteMedia plugin?
Diogo Cordeiro commented 4 months ago
Owner

yes

yes
roytam commented 4 months ago

but I think we really need a workaround(for example, appending link(s) into notice body) for showing attachment(s) from mastodon AP status since mastodon doesn't put attachment link(s) into notice body, or mastodon AP notice's attachment(s) will be lost.

but I think we really need a workaround(for example, appending link(s) into notice body) for showing attachment(s) from mastodon AP status since mastodon doesn't put attachment link(s) into notice body, or mastodon AP notice's attachment(s) will be lost.
Diogo Cordeiro commented 4 months ago
Owner

@roytam: hm.... noted, that's a good idea, we will see the best way having that in mind :)

@roytam: hm.... noted, that's a good idea, we will see the best way having that in mind :)
Sign in to join this conversation.
No Milestone
No assignee
3 Participants
Loading...
Cancel
Save
There is no content yet.