Awesome
<img src="https://raw.github.com/snarfed/granary/main/static/granary_logo_512.png" alt="Granary" width="128" /> granary
The social web translator. Fetches and converts data between social networks, HTML and JSON with microformats2, ActivityStreams/ActivityPub, Atom, JSON Feed, and more.
About
Granary is a library and REST API that fetches and converts between a wide variety of social data sources and formats:
- Facebook, Flickr, GitHub, Instagram, Mastodon, and Twitter native APIs
- Instagram and Facebook scraped HTML
- ActivityStreams 1.0 and 2.0 JSON, including ActivityPub
- HTML and JSON with microformats2
- Atom, RSS 2.0, JSON Feed
- Plain XML
- Bluesky/AT Protocol
- Nostr, with many NIPs
Free yourself from silo API chaff and expose the sweet social data foodstuff inside in standard formats and protocols!
Here's how to get started:
- Granary is available on PyPi. Install with
pip install granary
. - Getting started docs.
- Reference docs.
- REST API and demo app at granary.io.
- Source code on GitHub.
License: This project is placed in the public domain. You may also use it under the CC0 License.
Using
The library and REST API are both based on the OpenSocial Activity Streams service. Let's start with an example. This code using the library:
from granary import twitter
...
tw = twitter.Twitter(ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
tw.get_activities(group_id='@friends')
is equivalent to this HTTP GET
request:
https://granary.io/twitter/@me/@friends/@app/
?access_token_key=ACCESS_TOKEN_KEY&access_token_secret=ACCESS_TOKEN_SECRET
They return the authenticated user's Twitter stream, ie tweets from the people they follow. Here's the JSON output:
{
"itemsPerPage": 10,
"startIndex": 0,
"totalResults": 12,
"items": [{
"verb": "post",
"id": "tag:twitter.com,2013:374272979578150912",
"url": "http://twitter.com/evanpro/status/374272979578150912",
"content": "Getting stuff for barbecue tomorrow. No ribs left! Got some nice tenderloin though. (@ Metro Plus Famille Lemay) http://t.co/b2PLgiLJwP",
"actor": {
"username": "evanpro",
"displayName": "Evan Prodromou",
"description": "Prospector.",
"url": "http://twitter.com/evanpro",
},
"object": {
"tags": [{
"url": "http://4sq.com/1cw5vf6",
"startIndex": 113,
"length": 22,
"objectType": "article"
}, "..."],
},
}, "..."]
"..."
}
The request parameters are the same for both, all optional: USER_ID
is a source-specific id or @me
for the authenticated user. GROUP_ID
may be @all
, @friends
(currently identical to @all
), @self
, @search
, or @blocks
; APP_ID
is currently ignored; best practice is to use @app
as a placeholder.
Paging is supported via the startIndex
and count
parameters. They're self explanatory, and described in detail in the OpenSearch spec and OpenSocial spec.
When using the GROUP_ID
@search
(for platforms that support it — currently Twitter and Instagram), provide a search string via the q
parameter. The API is loosely based on the OpenSearch spec, the OpenSocial Core Container spec, and the OpenSocial Core Gadget spec.
Output data is JSON Activity Streams 1.0 objects wrapped in the OpenSocial envelope, which puts the activities in the top-level items
field as a list and adds the itemsPerPage
, totalCount
, etc. fields.
Most Facebook requests and all Twitter, Instagram, and Flickr requests will need OAuth access tokens. If you're using Python on Google App Engine, oauth-dropins is an easy way to add OAuth client flows for these sites. Otherwise, here are the sites' authentication docs: Facebook, Flickr, Instagram, Twitter.
If you get an access token and pass it along, it will be used to sign and authorize the underlying requests to the sources providers. See the demos on the REST API endpoints above for examples.
Using the REST API
The endpoints above all serve the OpenSocial Activity Streams REST API. Request paths are of the form:
/USER_ID/GROUP_ID/APP_ID/ACTIVITY_ID?startIndex=...&count=...&format=FORMAT&access_token=...
All query parameters are optional. FORMAT
may be as1
(the default), as2
, atom
, html
, jsonfeed
, mf2-json
, rss
, or xml
(the default). atom
supports a boolean reader
query parameter for toggling rendering appropriate to feed readers, e.g. location is rendered in content when reader=true
(the default). The rest of the path elements and query params are described above.
Errors are returned with the appropriate HTTP response code, e.g. 403 for Unauthorized, with details in the response body.
By default, responses are cached and reused for 10m without re-fetching the source data. (Instagram responses are cached for 60m.) You can prevent this by adding the cache=false
query parameter to your request.
Include the shares=false
query parameter to omit shares, eg Twitter retweets, from the results.
To use the REST API in an existing ActivityStreams/ActivityPub client, you'll need to hard-code exceptions for the domains you want to use e.g. facebook.com
, and redirect HTTP requests to the corresponding endpoint above.
Facebook and Instagram are disabled in the REST API entirely, sadly.
Using the library
See the example above for a quick start guide.
Clone or download this repo into a directory named granary
. Each source works the same way. Import the module for the source you want to use, then instantiate its class by passing the HTTP handler object. The handler should have a request
attribute for the current HTTP request.
The useful methods are get_activities()
and get_actor()
, which returns the current authenticated user (if any). See the full reference docs for details. All return values are Python dicts of decoded ActivityStreams 1 JSON.
The microformats2.*_to_html()
functions are also useful for rendering ActivityStreams 1 objects as nicely formatted HTML.
Troubleshooting/FAQ
Check out the oauth-dropins Troubleshooting/FAQ section. It's pretty comprehensive and applies to this project too.
Future work
We'd love to add more sites! Off the top of my head, YouTube, Tumblr, WordPress.com, Sina Weibo, Qzone, and RenRen would be good candidates. If you're looking to get started, implementing a new site is a good place to start. It's pretty self contained and the existing sites are good examples to follow, but it's a decent amount of work, so you'll be familiar with the whole project by the end.
Development
Pull requests are welcome! Feel free to ping me in #indieweb-dev with any questions.
First, fork and clone this repo. Then, install the Google Cloud SDK and run gcloud components install cloud-firestore-emulator
to install the Firestore emulator. Once you have them, set up your environment by running these commands in the repo root directory:
gcloud config set project granary-demo
python3 -m venv local
source local/bin/activate
pip install -r requirements.txt
# needed to serve static files locally
ln -s local/lib/python3*/site-packages/oauth_dropins/static oauth_dropins_static
Now, run the tests to check that everything is set up ok:
gcloud emulators firestore start --host-port=:8089 --database-mode=datastore-mode < /dev/null >& /dev/null &
python3 -m unittest discover
Finally, run the web app locally with flask run
:
GAE_ENV=localdev FLASK_ENV=development flask run -p 8080
Open localhost:8080 and you should see the granary home page!
If you want to work on oauth-dropins at the same time, install it in editable mode with pip install -e <path to oauth-dropins repo>
. You'll also need to update the oauth_dropins_static
symlink, which is needed for serving static file handlers locally: ln -sf <path-to-oauth-dropins-repo>/oauth_dropins/static oauth_dropins_static
.
To deploy to production:
gcloud -q beta app deploy --no-cache granary-demo *.yaml
The docs are built with Sphinx, including apidoc, autodoc, and napoleon. Configuration is in docs/conf.py
To build them, first install Sphinx with pip install sphinx
. (You may want to do this outside your virtualenv; if so, you'll need to reconfigure it to see system packages with virtualenv --system-site-packages local
.) Then, run docs/build.sh
.
Release instructions
Here's how to package, test, and ship a new release. (Note that this is largely duplicated in the oauth-dropins readme too.)
- Run the unit tests.
source local/bin/activate.csh CLOUDSDK_CORE_PROJECT=granary-demo gcloud emulators firestore start --host-port=:8089 --database-mode=datastore-mode < /dev/null >& /dev/null & sleep 5 python -m unittest discover kill %1 deactivate
- Bump the version number in
setup.py
anddocs/conf.py
.git grep
the old version number to make sure it only appears in the changelog. Change the current changelog entry inREADME.md
for this new version from unreleased to the current date. - Bump the
oauth-dropins
version specifier insetup.py
to the most recent version. - Build the docs. If you added any new modules, add them to the appropriate file(s) in
docs/source/
. Then run./docs/build.sh
. Check that the generated HTML looks fine by openingdocs/_build/html/index.html
and looking around. git commit -am 'release vX.Y'
- Upload to test.pypi.org for testing.
python setup.py clean build sdist setenv ver X.Y source local/bin/activate.csh twine upload -r pypitest dist/granary-$ver.tar.gz
- Install from test.pypi.org.
cd /tmp python -m venv local source local/bin/activate.csh pip uninstall granary # make sure we force Pip to use the uploaded version pip install --upgrade pip pip install mf2py==1.1.2 pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple granary==$ver deactivate
- Smoke test that the code trivially loads and runs.
Test code to paste into the interpreter:source local/bin/activate.csh python # run test code below deactivate
import json from granary import github github.__file__ # check that it's in the virtualenv g = github.GitHub('XXX') # insert a GitHub personal OAuth access token a = g.get_activities() print(json.dumps(a, indent=2)) from granary import atom print(atom.activities_to_atom(a, {}))
- Tag the release in git. In the tag message editor, delete the generated comments at bottom, leave the first line blank (to omit the release "title" in github), put
### Notable changes
on the second line, then copy and paste this version's changelog contents below it.git tag -a v$ver --cleanup=verbatim git push && git push --tags
- Click here to draft a new release on GitHub. Enter
vX.Y
in the Tag version box. Leave Release title empty. Copy### Notable changes
and the changelog contents into the description text box. - Upload to pypi.org!
twine upload dist/granary-$ver.tar.gz
- Build the docs on Read the Docs: first choose latest in the drop-down, then click Build Version.
- On the Versions page, check that the new version is active, If it's not, activate it in the Activate a Version section.
Related work
Apache Streams is a similar project that translates between storage systems and database as well as social schemas. It's a Java library, and its design is heavily structured. Here's the list of formats it supports. It's mainly used by People Pattern.
Gnip similarly converts social network data to ActivityStreams and supports many more source networks. Unfortunately, it's commercial, there's no free trial or self-serve signup, and plans start at $500.
DataSift looks like broadly the same thing, except they offer self-serve, pay as you go billing, and they use their own proprietary output format instead of ActivityStreams. They're also aimed more at data mining as opposed to individual user access.
Cliqset's FeedProxy used to do this kind of format translation, but unfortunately it and Cliqset died.
Facebook used to officially support ActivityStreams, but that's also dead.
There are a number of products that download your social network data, normalize it, and let you query and visualize it. SocialSafe is one, although the SSL certificate is currently out of date. ThinkUp was an open source product, but shuttered on 18 July 2016. There's also the lifelogging/lifestream aggregator vein of projects that pull data from multiple source sites. Storytlr is a good example. It doesn't include Facebook, or Instagram, but does include a number of smaller source sites. There are lots of others, e.g. the Lifestream WordPress plugin. Unfortunately, these are generally aimed at end users, not developers, and don't usually expose libraries or REST APIs.
On the open source side, there are many related projects. php-mf2-shim adds microformats2 to Facebook and Twitter's raw HTML. sockethub is a similar "polyglot" approach, but more focused on writing than reading.
Changelog
7.1 - unreleased
-
as1
:- Add new
is_dm
,recipient_if_dm
,get_id
, andis_audience
functions.
- Add new
-
as2
:- Add
sensitive
,indexable
, anddiscoverable
support. - Add new
is_server_actor
function (FEP-d556, discussion). from_as1
:- Always convert images to objects with
type: Image
, never to bare string URLs (bridgy-fed#/1000). - Bug fixes for converting links to facets when the link text is the link URL.
- Always convert images to objects with
to_as1
:- Handle other types of tags better, eg non-standard
Hashtag
and innertag
field for name. - Bug fix for videos,
mimeType
goes in outer object, not instream
. - Bug fix for
to
/cc
with mixed dict and string elements.
- Handle other types of tags better, eg non-standard
- Add
-
atom
:atom_to_activity/ies
: Get URL fromlink
for activities as well as objects. (Thanks @imax9000!)
-
bluesky
:- Translate Bluesky
app.bsky.feed.post#langs
to/from AS1contentMap
(which isn't officially part of AS1; we steal it from AS2). - Translate AS2
sensitive
on posts to Blueskygraphic-media
self label, and many Bluesky self labels back tosensitive
with content warning(s) insummary
. - Translate AS1/2 DMs to/from Bluesky chats.
- Translate video embeds in posts.
create
/previewCreate
:- If
inReplyTo
isn't a Bluesky URL or AT URI, returnCreationResult
instead of raisingValueError
.
- If
from_as1
:- Convert
article
s to external embeds with no post text. - Add new
as_embed
boolean kwarg to do the same thing for any object. - When truncating and adding a link to the original post, use
id
ifurl
is not available (snarfed/bridgy-fed#1155). - If the input object has
inReplyTo
orobject
ortarget
with no recognizable ATProto or Bluesky object, raiseValueError
. - Omit images that aren't in
blobs
. - Bug fix for quote posts with text content that's longer than Bluesky's limit (snarfed/bridgy-fed#1197).
- When a
flag
has multiple objects, use the first one that's an ATProto record. - Handle URLs more carefully, don't add link facets with invalid
uri
s. - Bug fix: handle HTML links with
title
incontent
correctly. - Bug fix: handle attachments with no
id
orurl
.
- Convert
to_as1
:- Extract links from
app.bsky.actor.profile#description
intourl
/urls
fields - Bug fix: first URL (singular) goes in
url
, list of URLs goes inurls
. - Bug fix: handle hashtags with regexp special characters.
- Support string and bytes CIDs in blob
ref
s as well asCID
instances.
- Extract links from
Bluesky.get_activities
: skip unknown record types instead of raisingValueError
.
- Translate Bluesky
-
microformats2
:object_to_json
: Improve handling of items with multiple types by removinginReplyTo
from likes, shares, etc (snarfed/bridgy-fed#941).
-
rss
:- Support image enclosures, both directions.
7.0 - 2024-06-24
Breaking changes:
jsonfeed
:jsonfeed_to_activities
: return AS1 objects, not activities.
Non-breaking changes:
as1
:activity_changed
: adddisplayName
,summary
fields.is_public
: returnFalse
if the object/activity containsto
that's empty or has only unknown aliases.
as2
:- Add support for the
Application
,Block
,Flag
, andLink
types. - Generalize actor logic in
to/from_as1
across all actor types, not justPerson
. - Add new
link_tags
function.
- Add support for the
atom
:activities_to_atom
: handle image attachments withouturl
field.
bluesky
:to_as1
:- Add support for:
app.bsky.embed.record
app.bsky.embed.recordWithMedia
app.bsky.feed.defs#notFoundPost
app.bsky.feed.generator
app.bsky.graph.block
app.bsky.graph.list
app.bsky.graph.listitem
com.atproto.admin.defs#repoRef
com.atproto.moderation.createReport#input
com.atproto.repo.strongRef
- Add hashtag facet support.
- Convert blobs in embeds to
getBlob
image URLs. app.bsky.actor.profile
: add HTML links for URLs insummary
(snarfed/bridgy-fed#1065).- Escape HTML characters (
<
,>
,&
) inapp.bsky.actor.profile
description
field. - Bug fix for
create
/update
activities with bare stringobject
.
- Add support for:
from_as1
:- Add hashtag, mention, block, and flag support. Interpret
tags
with missingobjectType
as hashtags. - Guess missing indices in facets based on content text. Otherwise, if we still don't know a facet's indices, discard it.
- Extract HTML links (<a> tags) from HTML content and convert to link facets (snarfed/bridgy-fed#976).
- If an output string value is longer than its
maxGraphemes
ormaxLength
in its lexicon, truncate it with an…
ellipsis character at the end in order to fit. If this happens to post text, include a link embed pointing to the original post. - If the object has a video, include an external embed pointing to the original post and mark it as
[Video]
(snarfed/bridgy-fed#1078). - If the object has images, add the original post link to the end of the text, since Bluesky doesn't support both image and external embeds in the same post (bluesky-social/atproto#2575, snarfed/bridgy-fed#1106).
- If a
note
hassummary
- often used for content warnings in the fediverse - add it tocontent
as a prefix instead of overridingcontent
(snarfed/bridgy-fed#1001). - Populate
reply.root
properly in reply posts (snarfed/bridgy#1696). - Add new
original_fields_prefix
kwarg to store original data in custom (off-Lexicon)*OriginalDescription
and*OriginalUrl
fields inapp.bsky.actor.profile
and*OriginalText
and*OriginalUrl
fields inapp.bsky.feed.post
(snarfed/bridgy-fed#1092). - Support
lexrpc.Client
as well asBluesky
forclient
kwarg.
- Add hashtag, mention, block, and flag support. Interpret
from_as1_to_strong_ref
:- Add
value
boolean kwarg. - Change
client
kwarg fromBluesky
tolexrpc.Client
.
- Add
microformats2
:- Generalize actor logic across all actor types, not just
person
. json_to_object
:- Strip leading
#
prefix (if present) from hashtagu-category
s. - Bug fix for when
name
property is an object, eg anh-card
.
- Strip leading
object_to_json
:- Convert both
id
andurl
insideinReplyTo
toin-reply-to.
- Convert both
- Generalize actor logic across all actor types, not just
nostr
:- Handle connection closing while sending initial query.
source
:Source.postprocess
: when extracting @-mentions, defer to existing tag if it has the samedisplayName
and hasurl
.
6.2 - 2024-03-15
as1
:get_owner
bug fix forpost
,update
,delete
activities.activity_changed
: add newinReplyTo
kwarg.is_public
: add newunlisted
kwarg.
as2
:to_as1
: bug fix, preserveobjectType: featured
for banner/header images even whenmediaType
is also set.is_public
: add newunlisted
kwarg.from_as1
:- For
icon
field, prefer image types that are allowed by Mastodon. - Bug fix, handle
stop-following
with stringobject
id.
- For
atom
:- Add new
extract_entries
function. activity_to_atom
: default actor/author name to username.atom_to_activities
: support top-levelentry
element as well asfeed
.atom_to_*
:- add
object.author
- default
objectType
toarticle
/note
andverb
topost
- convert
link rel=self
/alternate
tourl
- use
displayName
in objects instead oftitle
- Interpret entry
link
withoutrel
as self link.
- add
- If
entry.author
doesn't have id or url, default them to feed author's.
- Add new
bluesky
:- Implement
create
andpreview
. - Fully support both
record
andobject
types infrom_as1
andto_as1
. Useto_as1
'stype
kwarg andfrom_as1
'sout_type
kwarg to disambiguate. - Implement
Bluesky.post_id
. - Add new
blob_to_url
function. - Delete
as1_to_profile
, switchfrom_as1
to return$type: app.bsky.actor.profile
. - Convert HTML
summary
andcontent
to plain text. - Implement
Bluesky.user_to_actor
,Bluesky.get_actor
. - Don't log in (fetch an access token) eagerly in the constructor; wait until the client makes a call.
- Prefer DID to handle in API calls that accept either.
at_uri_to_web_url
: support lists.web_url_to_at_uri
: convert profile URLs likehttps://bsky.app/profile/snarfed.org
to profile record URIs (at://snarfed.org/app.bsky.actor.profile/self
) instead of repo URIs (at://snarfed.org
).- Add
from_as1_to_strong_ref
. - Allow
:
s in record keys (atproto#2224). to_as1
:- Convert blobs, both new and old style, to PDS
getBlob
URLs. - Add new
uri
kwarg. - Translate
handle
tousername
, add newrepo_handle
kwarg. - Add support for
app.bsky.feed.repost
,app.bsky.graph.defs#listView
,app.bsky.feed.defs#blockedPost
. - Add
actor
/author
based onrepo_did
. - Improve
url
field: include custom handles, only userepo_did/handle
forapp.bsky.actor.profile
. - Handle bad facet indices that point inside Unicode code points (example; discussion).
- Convert
!no-unauthenticated
label on profiles to AS1@unlisted
audience target (bridgy-fed#828).
- Convert blobs, both new and old style, to PDS
from_as1
:- Add
out_type
kwarg to specify desired output type, egapp.bsky.actor.profile
vsapp.bsky.actor.defs#profileViewBasic
vsapp.bsky.actor.defs#profileView
. - Add
blobs
kwarg to provide blob objects to use for image URLs. - Add
client
kwarg to fetch and populate CIDs. - Handle mention tags pointing to bare DIDs.
- Use
parent
asroot
in replies. (Technically wrong in cases where the parent isn't the root, but we don't actually know the root. 🤷) - Bug fix: handle bare string URLs in
image
field. - Bug fix: handle tags without
url
field. - Strip trailing slash from home page URLs in order to remove visible
/
from rel-me verified links on Mastodon etc. - Convert
attributedTo
to singular if it has only one element. - If
name
isn't set, fall back topreferredUsername
or infer Webfinger handle fromid
orurl
. - Prioritize bsky.app profile URL before handle URL in
url
field (bridgy#1640). - Convert
bsky.app
inReplyTo
URLs toat://
URIs. - Tighten up
datetime
conversion to match the ATProto recommended format.
- Add
- Implement
facebook
:- Remove
Facebook.fql_stream_to_post
. Facebook turned down FQL in 2016.
- Remove
github
:- When converting data to AS1, use
displayName
in objects instead oftitle
.
- When converting data to AS1, use
mastodon
:get_activities
bug fix: use query params for/api/v1/notifications
API call, not JSON body.- Convert HTTP 200 responses with
error
JSON field (eg from Sharkey) to 400/401 exceptions. - Prefer
media_attachments.remote_url
when available since it may be more long-lived thanurl
for remote statuses (bridgy#1675).
microformats2
:object_to_json
bug fix: handle singularinReplyTo
.json_to_object
bug fix: handle list-valuedlocation
.
nostr:
get_*
: return partial results when the websocket connection is closed prematurely.to_as1
: handle invalid NIP05 values (eg{}
)
rss
:to_activities
:- Use
objectType: note
iftitle
isn't set or is a prefix (possibly ellipsized) ofcontent
/description
. - Add support for images in
media:content
tags (#674).
- Use
Source
:postprocess_activity/object
: addmentions
kwarg to convert @-mentions in HTML links tomention
tags.
6.1 - 2023-09-16
Highlights: Nostr, Bluesky get_activities
, lots of improvements in as2
and microformats2
, and more!
REST API breaking changes:
Twitter is dead, at least in the REST API.
Non-breaking changes:
- Add new
nostr
module! as1
:- Add
get_owner
,targets
. - Add
accept
,reject
,stop-following
toVERBS_WITH_OBJECT
and removerepost
, it's not an AS1 verb. - Handle
url
field list values (even though it's invalid AS1).
- Add
as2
:to_as1
:- Improve
Video
handling: supportLink
objects inurl
, extract stream URLs and types from linktag
s. - Coerce non-float
latitude
andlongitude
to float, raiseValueError
on failure. - Put image attachments into
image
as well asattachments
(bridgy-fed#429). - Handle Hubzilla's composite object attachment
value
s. - Bug fix for null
mediaType
inattachment
andtags
.
- Improve
- Add new
TYPES_WITH_OBJECT
constant. - Add new
get_urls
,address
functions. - Improve
Content-Type
compatibility withapplication/ld+json; profile="https://www.w3.org/ns/activitystreams"
. - Bug fix for
Undo
activities with bare string idobject
s. - Revise HTML in
PropertyValue
attachments on actors to include full URL in anchro text to be compatible with Mastodon's profile link verification.
atom
:activities_to_atom
etc:- Switch
content
from XHTML to HTML inside CDATA to support non-XHTML input content (bridgy-fed#624. - Bug fix, handle bare string URL
image
values. - Bug fix, emove incorrect
type="application/atom+xml"
fromrel="self"
link
inentry
. - Render
objectType: comment
attachments. - Remove invalid
<a>
element for tags. - Bug fix: avoid encoded
<
and>
characters intitle
(#629).
- Switch
- Bug fixes in
activity_to_atom
/activities_to_atom
for dict-valuedurl
fields. - Render images in article/note attachments.
- Render
objectType: service
attachments, eg Bluesky custom feeds.
bluesky
:- Implement
Bluesky
API class, includingget_activities
. - Drop bundled
app.bsky
/com.atproto
lexicons, use lexrpc's instead. - Convert reposts, quotes, inline links, attached links, and mentions, both directions. Includes Bluesky facet (rich text) support.
- Handle quote posts with attached images, both directions.
- Handle likes, both directions.
- Add new
web_url_to_at_uri
function. from_as1
: handle link tags without start/end indices.to_as1
:- Add new
type
kwarg. - Generate staging.bsky.app profile and post URLs.
- Propagate profile
did
into actorid
. - Add unimplemented stub for custom feeds, eg
app.bsky.feed.defs#generatorView
.
- Add new
- Add
as1_to_profile
. - Bug fix for converting follows, both directions:
subject
inapp.bsky.graph.follow
is followee, not follower. (That field is badly named!)
- Implement
jsonfeed
:activities_to_jsonfeed
:- Bug fix, handle bare string values for
image
andstream
. - Bug fix: handle non-object
author
.
- Bug fix, handle bare string values for
mastodon
:status_to_object
: add/fix alt text handling for images.
microformats2
:json_to_html
:- HTML-escape tag and quote attachment names. Fixes GHSA-4w4f-g49g-3f7j; thank you @janboddez!
json_to_object
:- Improve handling of items with multiple types by using post type discovery more aggressively.
- Normalize ISO-8601 format of
published
andupdated
timestamps.
object_to_json
:- Bug fix, handle bare string URL
image
values. - Normalize ISO-8601 format of
published
andupdated
timestamps. - Handle bare string ids for
replies
andshares
(usually from AS2.)
- Bug fix, handle bare string URL
render_content
:- Bug fix for bare string
author
andactor
values.
- Bug fix for bare string
- Include
objectType: service
attachments, eg Bluesky custom feeds, in JSON and HTML output.
rss
:from_activities
: handle bare string idauthor
.
6.0 - 2023-03-22
Breaking changes:
as2
:- Interpret bare string
object
,inReplyTo
, etc values as ids, convert them to bare strings orid
instead ofurl
.
- Interpret bare string
microformats2
:- Convert simple string
in-reply-to
,repost-of
,like-of
etc values to AS1 bare strings orid
s instead ofurl
s.
- Convert simple string
Non-breaking changes:
- Add new
bluesky
module for Bluesky/AT Protocol! as1
:- Add the
organization
object type andACTOR_TYPES
constant (based on AS2). - Add new
get_ids
,get_object
, andget_objects
functions.
- Add the
activity_changed
: ignoreinReplyTo.author
(snarfed/bridgy#1338)as2
:- Support converting between AS1
stop-following
and AS2Undo
Follow
. - Support AS2
Accept
andReject
for follows as well as event RSVPs. - Add support for the
Question
(ie poll),Organization
, andDelete
object types. - Convert
to
/cc
to/from AS1to
for public and unlisted. - Handle
type: Document
video attachments like Mastodon emits. from_as1
: bug fix for image objects withurl
andvalue
fields (for alt text).from_as1
: bug fix, handle bare string URLimage
values.from_as1
: converturls.displayName
toattachment.name
(bridgy-fed#331).from_as1
: preserveinReplyTo
object values as objects, inline single-element lists down down to just single element.to_as1
: useobjectType: featured
for first image inimage
field.to_as1
: populateactor
intoobject.author
forUpdate
s as well asCreate
s.to_as1
: convert Mastodon profile metadataPropertyValue
attachments tourl
composite objects withdisplayName
.- Preserve
to
andcc
values when converting both directions.
- Support converting between AS1
atom
:- Bug fix for rendering image attachments without
image
field to Atom. - Bug fix for
published
andupdated
in entries with objects, eg likes, reposts, RSVPs, bookmarks. Thanks @gregorlove! (#480) - Bug fix for content
activity/ies_to_atom
whenobject
is present and empty. - Bug fix for objects with elements without
objectType
in theto
field.
- Bug fix for rendering image attachments without
flickr
:get_activities
: add support for thecount
kwarg.
github
:get_activities
: add support for thecount
kwarg.
jsonfeed
:- Switch from
white-space: pre
CSS to converting newlines to<br>
s because some feed readers follow it strictly and don't even line wrap (#456).
- Switch from
mastodon
:- Add compatibility support for Truth Social.
- Handle truncated JSON API responses.
microformats2
:json_to_object
: drop backward compatibility support forlike
andrepost
properties. Background discussion.json_to_object
: add newrel_urls
kwarg to allow attachingdisplayName
s tourls
based on HTML text ortitle
attribute (bridgy-fed#331).- Add new
json_to_activities
function. hcard_to_html
/maybe_linked_name
: whenname
is missing, use pretty URL as visible text.- Support the
h-card
org
property. json_to_object
: handle compositersvp
property value.json_to_object
: bug fix whenfetch_mf2
is True, handle when we run the authorship algorithm and fetch an author URL that has au-photo
withalt
.
rss
:from_activities
: fix item ordering to match input activities.
5.0 - 2022-12-03
Breaking changes:
- Drop Python 3.6 support. Python 3.7 is now the minimum required version.
- Twitter, Instagram, Mastodon:
- Drop
get_activities
cache
kwarg's support for App Engine memcache interface. It's now only used as a plaindict
.get_activities
will now make many small modifications, so if you pass an object that implements those as API calls, you'll probably want to batch those separately.
- Drop
- Twitter, Mastodon, Flickr, GitHub:
create
/preview
: support the AS1favorite
verb as well aslike
. (bridgy#1345)
- Atom:
- Switch to converting AS1
id
(instead ofurl
) to Atomid
.
- Switch to converting AS1
- Reddit:
- Implement
get_actor
.
- Implement
- Mastodon:
create
/preview
: allow non-Mastodon replies, ie activities that includeinReplyTo
URLs even if none of them point to a toot. (bridgy#1321)- Raise
requests.HTTPError
withresponse.status_code
502 instead ofJSONDecodeError
on non-JSON responses. This is synthetic, but more helpful for error handling.
- microformats2:
object_to_json
and related functions: handle all escaped HTML entities, not just&
<
>
.- Unify
microformats2.prefix_image_urls
andprefix_video_urls
into a newas1.prefix_urls
function.
- RSS:
- Remove
itunes:category
. It has to be one of Apple's explicit categories, which we aren't prepared to validate, so don't try.
- Remove
- ActivityStreams 2:
- Translate both
url
andurls
from AS1 into multi-valued AS2url
field.
- Translate both
- Move a number of utility methods from the
Source
class to a newas1
module:object_type
,merge_by_id
,is_public
,add_rsvps_to_event
,get_rsvps_from_event
,activity_changed
,append_in_reply_to
,actor_name
,original_post_discovery
. as1.original_post_discovery
: remove deprecatedcache
kwarg.
Non-breaking changes:
- ActivityStreams 2:
- Fix spec compliance bug:
icon
andimage
are singly valued, not multiply valued. - Add new
is_public
method andPUBLIC_AUDIENCE
constant. - Prefer
"objectType": "featured"
first in theimage
field when converting from AS1, last in theicon
field. This matches the ActivityPub (Mastodon) convention of usingicon
for profile pictures andimage
for header images. - Propagate
url
values into newPropertyValue
attachments onPerson
objects; these end up in Mastodon's "profile metadata" link fields. to_as1
: if an attachment'smediaType
isimage/...
, overrideobjectType
and set it toimage
.
- Fix spec compliance bug:
- Twitter
- Trim alt text in line between post preview and creation
- Correctly trim Twitter alt text
- Facebook
- Scraping: extract post id and owner id from
data-ft
attribute and_ft_
query param more often instead ofstory_fbid
, which is now an opaque token that changes regularly. (facebook-atom#27)
- Scraping: extract post id and owner id from
- Instagram
- Add new
Instagram.scraped_json_to_activities
method.
- Add new
- GitHub
create
andpreview
: convert profile URLs to @-mentions, eghttps://github.com/snarfed
to@snarfed
(bridgy#1090).get_activities
withactivity_id
now supportsfetch_replies
andfetch_likes
.
- Reddit
- Add
cache
support toget_activities
.
- Add
- REST API
- Add new
/scraped
endpoint that acceptsPOST
requests with silo HTML as input. Currently only supports Instagram. Requiressite=instagram
,output=...
(any supported output format), and HTML as either raw request body or MIME multipart encoded file in theinput
parameter.
- Add new
- microformats2
- Add new
extra
andbody_class
kwargs toactivities_to_html
. - When converting
u-featured
images to AS1, add new non-standard"objectType": "featured"
field to distinguish them fromu-photo
. - Convert
p-note
to AS1summary
. - Bug fixes for converting
image
attachments tophoto
.
- Add new
Source.original_post_discovery
: add newmax_redirect_fetches
keyword arg.
4.0 - 2022-03-23
Breaking changes:
- Drop Python 3.5 support. Python 3.6 is now the minimum required version.
Non-breaking changes:
- RSS:
- Add support for RSS input via new
rss.to_activities
function.
- Add support for RSS input via new
- Add new
include_shares
kwarg toget_activities
, implemented for Twitter and Mastodon. Defaults toTrue
. IfFalse
, shares (retweets in Twitter, boosts in Mastodon) will be discarded and not returned. Also add a correspondingshares
query param to the REST API. - Instagram (scraping):
- Handle media items with no
user
object, add new fetch for comments. - Add
Instagram.merge_scraped_comments()
.
- Handle media items with no
- ActivityStreams 2:
- Handle error when
type
isn't a string.
- Handle error when
- Reddit:
- Implement
get_activities()
to fetch posts by the current user or a user specified withuser_id
.
- Implement
- Facebook scraping:
- Skip "Suggested for you" posts.
- Add
log_html
kwarg toget_activities
; defaults to False. - Miscellaneous bug fixes.
- JSONFeed:
- Handle malformed
items.author
element.
- Handle malformed
3.2 - 2021-09-15
Source.original_post_discovery
: add newinclude_reserved_hosts
kwarg, defaults toTrue
.- Instagram:
- Update scraping to handle new
feed_v2
JSON format.
- Update scraping to handle new
- Facebook:
- Scraping: handle pictures, videos, link attachments, and text links in timeline/news feed posts.
- Mastodon:
- Bug fix for
get_activities()
withfetch_mentions=True
: handle notifications withstatus: null
. Maybe happens when a status is deleted? create
/preview_create
: support bookmarks. (Nothing special happens with them; theircontent
is posted as a normal toot.)
- Bug fix for
- microformats2:
- Stop rendering
image.displayName
as visible text in HTML, since it's already in the<img>
'salt
attribute. - Add
bookmark-of
support. - Add
prefix_image_urls()
function. - Handle null
content
in AS1/2 objects. json_to_object
bug fix for compositebookmark-of
properties.
- Stop rendering
- Twitter:
create
/preview
: support large videos via async upload. We now passmedia_category=tweet_video
to the chunked uploadINIT
stage, and then make blockingSTATUS
calls until the video is finished processing. (bridgy#1043)create
/preview
: allow bookmarks. (bridgy#1045)create
/preview
: allow non-Twitter replies, ie activities that includeinReplyTo
URLs even if none of them point to a tweet. (bridgy#1063)get_activities
: support list ids as well as slugs.- Bug fixes for removing t.co links to quoted tweets.
- Bug fix for multiple instances of the same link in tweet text.
get_activities()
: raiseValueError
on invaliduser_id
.
- REST API: ported web framework from webapp2 to Flask. No user-visible behavior change expected.
3.1 - 2021-04-03
- Add Python 3.8 support, drop 3.3 and 3.4. Python 3.5 is now the minimum required version.
- Add Pixelfed! Heavily based on Mastodon.
- Standardize Instagram's and Facebook's scraping into new common
scraped_to_activities()
,scraped_to_activity()
, andmerge_scraped_reactions()
methods. - Atom:
- Add the
summary
element (#157).
- Add the
- REST API:
- Bug fix: URL-encode Unicode characters in
Link
HTTP headers (egrel=self
,rel=header
).
- Bug fix: URL-encode Unicode characters in
- Facebook:
- Scraping now uses mbasic.facebook.com instead of m.facebook.com.
- Flickr:
- Add support for adding tags to existing photos (bridgy#857).
get_comment()
: skip fetching comments from API ifactivity
kwarg is provided and contains the requested comment.
- GitHub:
- Handle HTTP 451 Unavailable for Legal Reasons responses (eg for DMCA takedowns) gracefully.
- Add create/preview support for reactions on pull review request comments (ie URLs with
#discussion_r...
fragments).
- HTML/microformats2:
- Add
aria-hidden="true"
to empty links (bridgy#947). - Bug fix: escape
&
,<
, and>
characters in bare mf2content
properties (aaronpk/XRay#102). json_to_object()
: convertnickname
tousername
.
- Add
- JSON Feed:
- Gracefully handle when
content_html
andcontent_text
are incorrectly lists instead of strings.
- Gracefully handle when
- Instagram:
- Include threaded (ie nested) comments in scraping (bridgy#958).
- Mastodon:
- Bug fix for alt text with image attachments (bridgy#975).
- Omit empty
limit
param for compatibility with Pleroma (bridgy#977).
- Meetup:
create()
: handle API errors and return the error message in theCreationResult
(bridgy#921).
- Twitter:
- Bug fix: URL-encode list names in API calls.
- Bug fix: propagate alt text into AS1
photo.displayName
so that it gets all the way into microformats2 JSON and HTML (#183).
- Reddit:
- Implement
post_id()
. - Cache user data fetched from the API for 5m to avoid repeating user profile API requests (bridgy#1021). when fetching multiple comments or posts from the same author
- Bug fix: use 'displayName' instead of 'name' in AS1 objects for submissions.
- Bug fix: use tag URIs for activity ids.
- Implement
- ActivityStreams 2:
to_as1()
: forCreate
activities, include the activity actor's data in the object's author (snarfed/bridgy-fed#75).to_as1()
: convertpreferredUsername
tousername
.from_as1()
: convertusername
topreferredUsername
.from_as1()
: bug fix, makecontext
kwarg actually work.
3.0 - 2020-04-08
Breaking changes:
- Python 2 is no longer supported! Including the App Engine Standard Python 2 runtime. On the plus side, the Python 3 runtime is now supported! See this list of differences for more details.
Non-breaking changes:
- Migrate demo app and API to the App Engine Standard Python 3 runtime.
- Instagram:
- Scraping: fetch 50 likes instead of 24. (snarfed/bridgy#898)
- Scraping bug fix for
get_actor()
withuser_id
.
- Twitter:
- Add image alt text support to
get_activites()
etc (#183).
- Add image alt text support to
- RSS:
- Add
itunes:image
,itunes:author
, anditunes:category
. - Strip HTML from
title
element (#177). Background. - Always include author in items (#177).
- Bug fix: extract feed image from
hfeed
correctly. - Bug fix: don't crash on
article
ormention
tags in items with enclosures.
- Add
- Atom:
- Bug fix: extract feed image from
hfeed
correctly.
- Bug fix: extract feed image from
- REST API:
- Add HTTP
HEAD
support. - Add support for URL fragments with
input=html
. If a fragment is provided, only that specific element is extracted and converted. (#185)
- Add HTTP
- GitHub:
- Publish: preserve
<code>
tags instead of converting them to `s so that GitHub renders HTML entities like>
inside them instead of leaving them escaped. Background.
- Publish: preserve
- JSON Feed:
- Handle malformed attachments better.
- microformats2:
- Don't crash on string
context
fields. html_to_activities()
: limit toh-entry
,h-event
, andh-cite
items (#192).
- Don't crash on string
- The
cache
kwarg toSource.original_post_discovery()
now has no effect.webutil.util.follow_redirects()
has its own built in caching now. - Added Meetup.com support for publishing RSVPs.
2.2 - 2019-11-02
- Add Mastodon support!
- Add Python 3.7 support, and improve overall Python 3 compatibility.
- Update a number of dependencies.
- Switch from Python's built in
json
module toujson
to speed up JSON parsing and encoding. - Add
duration
andsize
support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON. microformats2 support is still emerging for both. Both integer seconds and ISO 8601 string durations are supported forduration
. Integer bytes is used forsize
everywhere. microformats2 HTML also includes human-readable strings, eg5.1 MB
. (#169) - Twitter:
[preview]_create()
: detect attempts to upload images over 5MB and return an error.
- Facebook:
- Add
get_activities(scrape=True)
for scraping HTML from m.facebook.com. Requiresc_user
andxs
cookies from a logged in session (snarfed/bridgy#886). - Upgrade Graph API version from 2.10 to 4.0.
- Add
- Atom:
- Bug fix for de-duping images in attachments.
- RSS:
- Wrap all
<description>
element contents inCDATA
sections. - Render images in
<description>
with HTML<img>
tags (#175). from_activities()
bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)
- Wrap all
2.1 - 2019-09-04
- Convert AS2
Mention
tags to AS1objectType
mention
(non-standard) and vice versa (snarfed/bridgy-fed#46). - Twitter:
- Bug fix for large block list fetches that get rate limited after a few successful requests.
- Handle HTTP 403 + error code 200 when fetching retweets for a protected or otherwise unavailable tweet (bridgy#688).
- Demote @-mentions from person-tags to mentions. Specifically, this means they'll no longer get rendered with
u-category
mf2.
- Instagram:
- Disabled in the REST API entirely due to Instagram's aggressive rate limiting and blocking (bridgy#655).
- Update scraping to handle replies in new
edge_media_to_parent_comment
field (#164). - Use cookie for all scraping HTTP requests, not just for likes.
- microformats2:
- Revise whitespace handling; use
white-space: pre
CSS in HTML output.
- Revise whitespace handling; use
- Facebook:
- Bug fix: don't interpret
photo.php
as username in post URLs.
- Bug fix: don't interpret
- Atom:
- Switch from
white-space: pre
CSS back to converting newlines to<br>
s because some feed readers (eg NewsBlur) follow it too strictly and don't even line wrap.
- Switch from
- RSS:
- Default title to ellipsized content.
2.0 - 2019-03-01
Breaking change: drop Google+ since it shuts down in March. Notably, this removes the googleplus
module.
1.15 - 2019-02-28
- Add RSS 2.0 output! (#124)
- All silos:
- Switch users' primary URLs from web site to silo profile (#158).
- GitHub:
- Don't enclose bare URLs in
<
/>
(snarfed/bridgy#850).
- Don't enclose bare URLs in
- Atom:
- Bug fix for actors and attachments with multiple image URLs.
- Bug fix for attachment author objects with no properties.
- Google+:
- Drop from web UI and REST API since consumer Google+ is shutting down entirely (more).
- Switch from deprecated global API endpoint to G+ endpoint. Background in snarfed/bridgy#846, Google blog post and docs.
- Instagram:
- Fix individual photo/video link urls for multi-photo/video posts.
- Handle user-provided alt text (#159).
- Twitter:
- Update max video upload size from 5MB to 512MB (#162).
/url
: Return HTTP 400 when fetching the user's URL results in an infinite redirect.
1.14 - 2018-11-12
Add delete()
. Currently includes Twitter and Flickr support.
- Instagram:
- Make extra HTTP fetch (with cookie) to get individual likes (snarfed/bridgy#840).
- Update scraping logic to handle feed HTML changes.
- Link @-mentions in comments as well as photo/video captions.
- GitHub:
create
/preview_create
bug fixes for issues and comments on private repos.- Handle HTTP 410 Gone responses from REST API, eg when a repo has been deleted or issues for the repo disabled.
- Twitter:
- Add
delete()
andpreview_delete()
for deleting tweets.
- Add
- Flickr:
- Add
delete()
andpreview_delete()
for deleting photos.
- Add
- microformats2:
- Atom:
- Encode
&
s in author URL and email address too. (Thanks sebsued!)
- Encode
- AS2:
- Add
Follow
support.
- Add
1.13 - 2018-08-08
- Twitter:
- Support ISO 8601 formatted created_at timestamps, which the archive download uses, as well as RFC 2822 from the API.
create()
andpreview_create()
: support RSVPs. Tweet them as normal tweets with the RSVP content. (snarfed/bridgy#818)create()
andpreview_create()
: support alt text for images, via AS1displayName
. (snarfed/bridgy#756).
- Instagram:
- Add global rate limiting lock for scraping. If a scraping HTTP request gets a 429 or 503 response, we refuse to make more requests for 5m, and instead short circuit and return the same error. This can be overridden with a new
ignore_rate_limit
kwarg toget_activities()
.
- Add global rate limiting lock for scraping. If a scraping HTTP request gets a 429 or 503 response, we refuse to make more requests for 5m, and instead short circuit and return the same error. This can be overridden with a new
- GitHub:
- Add
tag
support tocreate
/preview_create
to add label(s) to existing issues (snarfed/bridgy#811). - Escape HTML characters (
<
,>
, and&
) in content increate()
andpreview_create()
(snarfed/bridgy#810). get_activities()
andget_comment()
now returnValueError
instead ofAssertionError
on malformedactivity_id
andcomment_id
args, respectively.get_activities()
bug fix for issues/PRs with no body text.- Switch from GraphQL to REST API for creating comments and reactions, since GraphQL hits authorization errors on many org repos. (snarfed/bridgy#824)
- Improve GraphQL support for comments and users.
- Add
- Atom:
- Shorten and ellipsize feed title when necessary (#144).
- microformats2:
- Upgrade mf2py to improve a few things like implied p-name detection and whitespace handling (#142, fixes #145, snarfed/bridgy#756, snarfed/bridgy#828).
- Support
alt
attribute in<img>
tags (snarfed/bridgy#756).
1.12 - 2018-03-24
- Add Python 3 support! Granary now requires either Python 2.7+ or Python 3.3+.
- Instagram:
- Fix scraping profile pages.
- Twitter:
- Update character counting to handle Twitter change that now auto-links all ccTLDs. Background.
- GitHub:
- Bug fix for
get_activities()
with deleted issues and repos.
- Bug fix for
- microformats2:
object_to_json()
: convert tags to simple strings in thecategory
property, not full nested objects likeh-card
s (#141).- Special case GitHub issues that are in-reply-to a repo or its
/issues
URL to be objectTypeissue
. - Render simple string categories in HTML output.
This release is intentionally small and limited in scope to contain any impact of the Python 3 migration. It should be a noop for existing Python 2 users, and we've tested thoroughly, but I'm sure there are still bugs. Please file issues if you notice anything broken!
1.11 - 2018-03-09
- Add GitHub!
- Twitter:
- Prefer MP4 and other video/... content types to HLS (.m3u8) etc. Background.
- Prefer HTTPS URLs for media images.
get_activities()
: Support @-prefixed usernames inuser_id
.
- Facebook:
- Support new recurring aka multi-instance events.
create()
andpreview_create()
now only support RSVPs to individual instances of multi-instance events, to match the Facebook API itself. - Try harder to find original (full) sized photo URLs, specifically
_o.jpg
files instead of_s.jpg
. create()
bug fix for photo and image URLs with unicode characters.- Fixed bug where
get_activities(user_id=...)
included the authenticated user's own recent photos, albums, and news publishes.
- Support new recurring aka multi-instance events.
- Instagram:
- Extract more user (
author
) data from scraped profile pages. - Fix home page feed scraping.
- Extract more user (
- microformats2, Atom:
- Add enclosures for image attachments.
- Bug fixes for rendering image, video, and audio attachments inside shares and attachments. De-dupe images.
- microformats2:
- Handle simple string-only author properties.
- Add
fetch_mf2
kwarg tojson_to_object()
for fetching additional pages over HTTP to determine authorship. - Generate explicit blank
p-name
in HTML to prevent old flawed implied p-name handling (#131). - Fix
share
verb handling inactivity_to_json()
andactivities_to_html()
(#134). - Remember which content contains HTML, preserve newlines in it, and don't translate those newlines to
<br>
s (#130).
- Atom:
- Fix timezone bugs in
updated
andpublished
.
- Fix timezone bugs in
- JSON Feed:
- Omit title from items if it's the same as the content. (Often caused by microformats2's implied
p-name
logic.)
- Omit title from items if it's the same as the content. (Often caused by microformats2's implied
1.10 - 2017-12-10
- Moved web site and REST API to granary.io! granary-demo.appspot.com now 301 redirects.
- Twitter:
- Update the publish character limit to 280. Background.
- Fix a bug in preview_create that auto-linked @-mentions inside URLs, e.g. Medium posts.
- Support videos and animated GIFs in
get_activities()
etc.
- Instagram:
- Add cookie query param to REST API to allow scraping that logged in user's feed.
- HTML (including Atom content):
- Render image, video, and audio attachments more often and consistently.
- Include microformats2
u-photo
,u-video
, andu-audio
classes more often and consistently.
- Atom:
- Add
atom_to_activities()
for converting full feed documents. - Add to REST API and web UI.
- Include source URL in
rel=alternate
link as well as actor/author URL (#151).
- Add
- JSON Feed:
- Fix bug that omitted title in some cases (#122).
1.9 - 2017-10-24
- Add ActivityStreams 2.0! New
as2
module includesto_as1()
andfrom_as1()
functions. Currently supported: articles, notes, replies, likes, reposts, events, RSVPs, tags, attachments. - Atom:
- Add new
atom_to_activity()
function for converting Atom to AS1. - Add email field to author, if provided.
- Add new
- JSON Feed:
- Raise ValueError on bad (non-dict) input.
- REST API:
- Add
as2
value forformat
andinput
. Revise existing ActivityStreams and microformats2 value names toas1
,as1-xml
, andmf2-json
. Old valuesactivitystreams
,json
,json-mf2
, andxml
are still accepted, but deprecated.
- Add
1.8 - 2017-08-29
- Add JSON Feed support to both library and REST API.
- Twitter:
- Add
get_blocklist()
. - Bug fix for creating replies, favorites, or retweets of video URLs, e.g. https://twitter.com/name/status/123/video/1 .
- Bug fix for parsing favorites HTML to handle a small change on Twitter's side.
post_id()
now validates ids more strictly before returning them.
- Add
- Facebook:
- Instagram:
- Update scraping to handle new home page (ie news feed) JSON schema, which changed sometime around 2017-02-27. (Profile pages and individual photo/video permalinks still haven't changed yet.)
- microformats2:
- Add
u-featured
to ActivityStreamsimage
. - Improve
h-event
support. - Minor whitespace change (added <p>) when rendering locations as HTML.
post_id()
now validates ids more strictly before returning them.- Fix bugs in converting latitude and longitude between ActivityStreams and mf2.
- Add
- Google+:
- Update HTML scraping to handle changed serialized JSON data format.
- Atom:
- Add new
activity_to_atom()
function that renders a single top-level<entry>
instead of<feed>
. - Add new
reader
query param for toggling rendering decisions that are specific to feed readers. Right now, just affects location: it's rendered in the content whenreader=true
(the default), omitted whenreader=false
. - Include author name when rendering attached articles and notes (e.g. quote tweets).
- Only include AS
activity:object-type
andactivity:verb
elements when they have values. - Render AS image and mf2 u-photo if they're not already in content.
- Render
thr:in-reply-to
fromobject.inReplyTo
as well asactivity.context.inReplyTo
.
- Add new
- REST API:
- Fix bugs in html => json-mf2 and html => html conversions.
- Upgrade brevity to 0.2.14 for a couple bug fixes.
1.7 - 2017-02-27
- microformats2:
- Interpret h-cite and u-quotation-of (experimental) as attachments, e.g. for quote tweets.
- Convert audio and video properties to AS attachments.
- Twitter:
- Linkify @-mentions and hashtags in
preview_create()
. - Support creating quote tweets from attachments with Twitter URLs.
- When converting quote tweets to AS, strip quoted tweet URL from end of text.
- Raise ValueError when
get_activities()
is passedgroup_id='@search'
but notsearch_query
.
- Linkify @-mentions and hashtags in
- Instagram:
- Improve HTML scraping error handling.
- Support multi-photo/video posts.
- Facebook:
- Disable creating "interested" RSVPs, since Facebook's API doesn't allow it.
- Atom:
- Support media enclosures for audio and video attachments.
- Source.get_activities(): start raising ValueError on bad argument values, notably invalid Facebook and Twitter ids and Instagram search queries.
- Fix rendering and linkifying content with Unicode high code points (ie above the 16-bit Basic Multilingual Plane), including some emoji, on "narrow" builds of Python 2 with
--enable-unicode=ucs2
, which is the default on Mac OS X, Windows, and older *nix.
1.6 - 2016-11-26
- Twitter:
- Handle new "extended" tweets with hidden reply-to @-mentions and trailing URLs for media, quote tweets, etc. Background: https://dev.twitter.com/overview/api/upcoming-changes-to-tweets
- Bug fix: ensure like.author.displayName is a plain unicode string so that it can be pickled normally, e.g. by App Engine's memcache.
- Bug fix: handle names with emoji correctly in favorites_html_to_likes().
- Bug fix: handle search queries with unicode characters.
- Atom:
- Render full original quoted tweet in retweets of quote tweets.
- microformats2 HTML:
- Optionally follow and fetch rel="author" links.
- Improve mapping between microformats2 and ActivityStreams 'photo' types. (mf2 'photo' type is a note or article with a photo, but AS 'photo' type is a photo. So, map mf2 photos to underlying type without photo.)
- Support location properties beyond h-card, e.g. h-adr, h-geo, u-geo, and even when properties like latitude and longitude appear at the top level.
- Error handling: return HTTP 502 for non-JSON API responses, 504 for connection failures.
1.5 - 2016-08-25
- REST API:
- Support tag URI for user id, app id, and activity id.
- Twitter:
- Better error message when uploading a photo with an unsupported type.
- Only include original quote tweets, not retweets of them.
- Skip fetching retweets for protected accounts since the API call always 403s.
- Flickr:
- Better username detection. Flickr's API is very inconsistent about username vs real name vs path alias. This specifically detects when a user name is probably actually a real name because it has a space.
- Uploading: detect and handle App Engine's 10MB HTTP request limit.
- Bug fix in create: handle unicode characters in photo/video description, hashtags, and comment text.
- Atom:
- Bug fix: escape &s in attachments' text (e.g. quote tweets).
- Bug fix: handle multiply valued 'object' fields in ActivityStreams 1 activities.
- GitHub:
- Switch creating comments and reactions from GraphQL to REST API (bridgy#824.
1.4.1 - 2016-06-27
- Bump oauth-dropins requirement to 1.4.
1.4.0 - 2016-06-27
- REST API:
- Cache silo requests for 5m by default, 60m for Instagram because they aggressively blocking scraping. You can skip the cache with the new cache=false query param.
- Facebook:
- Upgrade from API v2.2 to v2.6. https://developers.facebook.com/docs/apps/changelog
- Add reaction support.
- De-dupe event RSVPs by user.
- Twitter:
- Switch create() to use brevity for counting characters. https://github.com/kylewm/brevity
- Fix bug in create() that occasionally incorrectly escaped ., +, and - characters.
- Fix text rendering bug when there are multipl photos/videos.
- When replying to yourself, don't add a self @-mention.
- Instagram:
- Fix bugs in scraping.
- Upgrade to requests 2.10.0 and requests-toolbelt 0.60, which support App Engine.
1.3.1 - 2016-04-07
- Update oauth-dropins dependency to >=1.3.
1.3.0 - 2016-04-06
- Support posting videos! Currently in Facebook, Flickr, and Twitter.
- Instagram:
- Add support for scraping, since they're locking down their API and requiring manual approval.
- Linkify @-mentions in photo captions.
- Facebook:
- Fetch Open Graph stories aka
news.publish
actions. - Many bug fixes for photo posts: better privacy detection, fix bug that attached comments to wrong posts.
- Fetch Open Graph stories aka
- Twitter:
- Handle all photos/videos attached to a tweet, not just the first.
- Stop fetching replies to @-mentions.
- Atom:
- Render attachments.
- Add
xml:base
.
- microformats2:
- Load and convert h-card.
- Implement full post type discovery algorithm, using mf2util. https://indiewebcamp.com/post-type-discovery
- Drop support for h-as-* classes, both incoming and outgoing. They're deprecated in favor of post type discovery.
- Drop old deprecated
u-like
andu-repost
properties.
- Misc bug fixes.
- Set up Coveralls.
1.2.0 - 2016-01-11
- Improve original post discovery algorithm. (bridgy #51)
- Flickr tweaks. (bridgy #466)
- Add mf2, activitystreams, atom, and search to interactive UI. (#31, #29)
- Improved post type discovery (using mf2util).
- Extract user web site links from all fields in profile (e.g. description/bio).
- Add fabricated fragments to comment/like permalinks (e.g. #liked-by-user123) so that object urls are always unique (multiple silos).
- Improve formatting/whitespace support in create/preview (multiple silos).
- Google+:
- Add search.
- Facebook:
- Fetch more things in get_activities: photos, events, RSVPs.
- Support person tags in create/preview.
- Prevent facebook from automatically consolidating photo posts by uploading photos to "Timeline Photos" album.
- Include title in create/preview.
- Improve object id parsing/resolving.
- Improve tag handling.
- Bug fix for fetching nested comments.
- Misc improvements, API error/flakiness handling.
- Flickr:
- Create/preview support for photos, comments, favorites, tags, person tags, location.
- Twitter:
- Create/preview support for location, multiple photos.
- Fetch quote tweets.
- Fetching user mentions improvements, bug fixes.
- Fix embeds.
- Misc AS conversion improvements.
- microformats2:
- Improve like and repost rendering.
- Misc bug fixes.
- Set up CircleCI.
1.1.0 - 2015-09-06
- Add Flickr.
- Facebook:
- Fetch multiple id formats, e.g. with and without USERID_ prefix.
- Support threaded comments.
- Switch from /posts API endpoint to /feed.
- Google+:
- Support converting plus.google.com HTML to ActivityStreams.
- Instagram:
- Support location.
- Improve original post discovery algorithm.
- New logo.
1.0.1 - 2015-07-11
- Bug fix for atom template rendering.
- Facebook, Instagram: support access_token parameter.
1.0 - 2015-07-10
- Initial PyPi release.