mattl
/
mediagoblin-website


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594
							<html>
<head>
  <title>State of the Goblin: September 2015</title>
  <meta name="date" contents="2015-09-25 11:10" />
  <meta name="author" contents="Christopher Allan Webber" />
  <meta name="tags" contents="mediagoblin, updates" />
</head>
<body>

<blockquote>
  <i>
    Quick announcement: I'm going to be making two appearances over
    the next week!  First I'll be
    <a href="https://chicagolug.org/meetings/2015-09-30.html">giving
    two talks on September 30th at Red Hat's Chicago office for the
    Chicago GNU/Linux User Group</a> on Guix and Federation (both
    mentioned in this post)!  Second I'll be attending the
    <a href="https://fsf.org/fsf30/celebration">FSF 30th Birthday Party</a>
    in Boston on October 3rd.  If you're able to make it to either,
    do stop by and say hello... I'm expecting both to be a lot of fun!
  </i>
</blockquote>

<p>
Hello everyone!  It's been a while since a comprehensive update of
what's happening in MediaGoblin land.  Despite the quiet, there is a
lot to report, so let's get down to business and start reporting!
</p>


<h2>O'Reilly Award (again!)</h2>

<p class="centered">
  <img src="/blog_images/receiving_oreilly_award.jpg"
       alt="receiving the award" />
  <br />
  <i>
    Photo taken by Brandin Grams,
    <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>,
    originally microblogged by
    <a href="https://twitter.com/o0karen0o/status/624660833805533184">Karen Sandler</a>
  </i>
</p>

<p>
First of all, something fun: I was fortunate enough to receive the
O'Reilly Open Source Award!  (Yes, I know about the terminology
mismatch, we're free software people, but it's still a great honor,
and I was presented the award under the description of my free
software advocacy and GNU MediaGoblin work.)  The
<a href="https://en.wikipedia.org/wiki/O%27Reilly_Open_Source_Award#Award_winners">other recipients of the award</a>
is quite the incredible group of people, and I'm honored to be listed
among them.  But here's what's really cool: you may remember that
<a href="/news/deb-nicholson-oscon-award.html">MediaGoblin co-founder Deb Nicholson won the O'Reilly Award last year</a>.
How's that as a vote of confidence in the things we're working on?
</p>

<p class="centered">
  <a href="http://dustycloud.org/blog/oreilly-award/">
    <img src="/blog_images/oreilly_award-scaled.jpg"
         alt="O'Reilly award, on display" /></a>
</p>

<p>
Anyway, if you want a more personal reflection, I
<a href="http://dustycloud.org/blog/oreilly-award/">wrote
more on my personal blog</a>!
</p>


<h2>Releases</h2>

<p class="centered">
  <a href="/news/mediagoblin-0.7.0-time-travelers-delight.html">
    <img src="/blog_images/0.7.0/time_travelers_delight-scaled.jpg"
         alt="MediaGoblin 0.7.0: Time Traveler's Delight banner" />
  </a>
</p>

<p>
So right, what about shipping software out the door?  Well...
</p>

<p class="centered">
  <a href="/news/mediagoblin-0.8.0-gallery-of-fine-creatures.html">
    <img src="/blog_images/0.8.0/gallery_of_fine_creatures_banner-scaled.png"
         alt="MediaGoblin 0.8.0: A Gallery of Fine Creatures banner" />
  </a>
</p>

<p>
Since the crowdfunding campaign, we've gotten out two major releases,
<a href="/news/mediagoblin-0.7.0-time-travelers-delight.html">0.7.0: Time Traveler's Delight</a>
and
<a href="/news/mediagoblin-0.8.0-gallery-of-fine-creatures.html">0.8.0: A Gallery of Fine Creatures</a>.
I'm extremely proud of both of these releases!  We have a lot more to
do though on the road to 1.0, and we've been directly been putting the
funds from the campaign to work to achieve that goal, so let's talk
about that.
</p>


<h2>Putting your money to good work: Jessica and Federation</h2>

  <!-- What Jessica's doing, how we're spending the funds from the campaign -->

<p class="blog_image">
  <a href="/news/welcome-jessica-tallon.html">
    <img src="/blog_images/jessica_tallon_screenshot-scaled.jpg"
         alt="Dropdown menu for administrative features" />
  </a>
</p>

<p>
You may recall that
<a href="/news/welcome-jessica-tallon.html">we hired on the talented Jessica Tallon</a>
to get federation working in MediaGoblin.  Jessica recently
<a href="/news/state-of-federation.html">gave an update on the state of federation</a>.
Jessica is doing great work, though as expected, converting
MediaGoblin to be a federated project has been no small task (knowing
what a big task it was, hope that we could hire Jessica on to do this
work was my #1 goal in the last campaign, in fact!).  This decision
has turned out to be absolutely the right one.  Some of the best parts
of the last two releases have been adopting the client to server Pump
API.  Federation has been MediaGoblin's goal since day 0, and Jessica
is helping us to actually get there.
</p>

<p>
However (and now I'm going to do a pretty technical deep dive, so you
can skip this paragraph if that isn't your thing), the most
complicated aspect to making MediaGoblin into a federated project has
had to do with updating the database to handle things while preserving
data correctly for existing users.  Why is this so complicated?  A
number of years ago
<a href="/news/mediagoblin-0.2.1-gearing-up.html">we switched MediaGoblin over from using MongoDB to using either PostgreSQL or SQLite</a>
and while I believe this was absolutely the right decision,
adding federation made the relational database system we had in place
substantially trickier.  For the more database-technically inclined,
you can see that the
<a href="http://w3c-social.github.io/activitypump/">ActivityPump API</a> /
<a href="https://github.com/e14n/pump.io/blob/master/API.md">Pump API</a>
require that any ActivityStream type object (in our case, that can be
media or comments or even users) be referenceable by any type of
activity.  Furthermore, our existing comment system simply held that
comments referenced media entries, whereas now comments can reference
simply <i>anything</i> that is an ActivityStreams object.  This means a
large portion of our relations in our relational database needed an
overhaul, and we needed a way to handle generic relations between
tables.  (The solution used is not unlike the "generic foreign key"
implementation in Django.)  There are more technical details on what
has been done, but Jessica has been neck deep in this for months, but
we believe we're finally on the home stretch, in which case Jessica
can finally knock out server to server federation.
</p>

<p>
(I've thought that a whole post on database structure lessons learned
may be a good blogpost of its own.  One thing I'd note is that if
<a href="http://www.postgresql.org/docs/current/static/datatype-json.html">jsonb</a>
had been an option when our current database design was put together,
adopting that would have simplified things greatly, though it would
require being PostgreSQL-only.  But moving to that now would require a
massive overhaul.  If you're starting a new federated project from day 1,
maybe keep that in mind!)
</p>

<p>
So the summary of all the federation stuff is: it's complex, but we're
making good progress through Jessica's hard work.  Expect more on this
soon, and huge strides in the next release!
</p>


<h2>Federation and the W3C Social Working Group</h2>

<p>
So something Jessica and I have both been involved in over the last
year is the
<a href="http://www.w3.org/Social/WG">W3C Social Working Group</a>
working towards official standards for federated web applications.
</p>

<p class="centered">
  <img src="/blog_images/w3c-f2f-2015-03-18-scaled.jpg"
       alt="W3C Social WG, first GMG represented meeting" />
  <br />
  <i>
    W3C Social Working Group
    <a href="http://www.w3.org/wiki/Socialwg/2015-03-17">March
      2015 face to face meeting</a> attendees.  Jessica's holding the laptop
    on the left, and I'm right behind her.
    <br />
    Photo taken by <a href="http://aaronparecki.com/">Aaron Parecki</a>,
    <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0 1.0</a>,
    originally
    <a href="http://www.w3.org/wiki/File:2015-03-18-w3c-f2f.jpg">posted to the W3C wiki</a>.
  </i>
</p>

<!-- Give a caption about Jessica and I attending the second face to face -->

<p>
The federation protocol that MediaGoblin has been working towards
until this point is primarily based on the
<a href="https://github.com/e14n/pump.io/blob/master/API.md">Pump API</a>, but
this is really just a semi-formalization of the interface for the
<a href="http://pump.io/">pump.io</a> API.  In the Social Working Group we are
working towards defining a new standard,
<a href="http://w3c-social.github.io/activitypump/">ActivityPump</a>, which is
based off of the
<a href="http://www.w3.org/TR/activitystreams-core/">ActivityStreams 2.0</a>
standard.  We're very excited with where this standard is going and
feel it's a clean refinement over the Pump API we're already working
with, while still keeping many of those same conventions.
</p>

<!-- Joining the W3C Group, showing picture of Jessica and I as part of it
     Give a caption about Jessica attending the third F2F... I was just remote
     on this one! -->

<p class="centered">
  <img src="/blog_images/w3c-f2f-2015-05-04-scaled.jpg"
       alt="W3C Social WG, second GMG represented meeting" />
  <br />
  <i>
    W3C Social Working Group
    <a href="http://www.w3.org/wiki/Socialwg/2015-05-04">May
      2015 face to face meeting</a> attendees.  I didn't make it to this
    one, but Jessica did, and did a stellar job representing MediaGoblin
    and ActivityPump!
    <br />
    Photo taken by <a href="http://aaronparecki.com/">Aaron Parecki</a>,
    <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0 1.0</a>,
    originally
    <a href="http://www.w3.org/wiki/File:2015-05-04-w3c-f2f.jpg">posted to the W3C wiki</a>.
  </i>
</p>

<p>
This has taken a lot of our time, but I believe it the results are
worth it.  Jessica and I have been attending weekly calls related to
this standardization, and have thus far attended two face to face
meetings at well.  (More accurately, Jessica attended the second
MediaGoblin-represented one without me, giving a kick-ass presentation
on how ActivityPump works to the group!  Go Jessica!)
</p>

<p>
As for my own personal work advancing this, I'll go into this a bit
further on in this post!
</p>


<h2>Google Summer of Code result: Goblinoid!</h2>

<p class="centered">
  <img src="/blog_images/goblinoid_current.png"
       alt="Current screenshot of Goblinoid" /><br />
  <i>Screenshot of Goblinoid, as it looks now!</i>
</p>

<p>
Dylan Jeffers joined us for Google Summer of Code student this year
work on a pretty cool project: a MediaGoblin client for Android
or... really nearly anything... called
<a href="https://notabug.org/SapienTech/MediaGoblinApp/">Goblinoid</a>!
There's two really interesting features about Goblinoid: one, it's
written in <a href="http://kivy.org/">Kivy</a>, a GUI toolkit which emulates
the Android look and feel, but is actually can run nearly anywhere
Python can run... making it quite portable, yet ideal for mobile
computers!
</p>

<p class="centered">
  <img src="/blog_images/goblinoid_mockup.png"
       alt="Mockup of Goblinoid future UI" /><br />
  <i>
    Mockup of what Dylan would like Goblinoid to look like in
    the future!
  </i>
</p>

<p>
So Goblinoid works... it could use more user testing and packaging, if
you're interested in helping with that!  But you can already upload
images on the go via Goblinoid, and we expect more to come.
<a href="https://notabug.org/SapienTech/MediaGoblinApp/">Give it a go!</a>
</p>

<p>
We've long been interested in having a client for MediaGoblin which
makes use of the Pump API implementation we've been working on.
Thank you Dylan for making that happen!
</p>


<h2>Infrastructure challenges</h2>

<p>
This has been a challenging year as in terms of supporting
MediaGoblin's infrastructure.  Spammers attacked both our
<a href="https://wiki.mediagoblin.org/Main_Page">wiki</a> and
<a href="https://issues.mediagoblin.org/">bugtracker</a> hard, at one point
leaving me to take several weeks to fight issues and to try to find
solutions.  (Unfortunately, for the bugtracker, no great solutions
have been found, and we are on a request-an-account basis... not a
great situation to be in.)
</p>

<p>
Additionally, the primary <a href="http://gitorious.org/">Gitorious instance</a>
went down, where MediaGoblin's code was hosted along with many, many
feature branches from contributors.  The MediaGoblin community spent
a while debating what we were going to do. A move to GitHub was
tempting but is not an option; that's exactly the opposite of the type
of world we want to build.  Not everyone in MediaGoblin's community is
comfortable with GitLab, and their primary instance is running a
proprietary version.  There are some other communities hosting things
like <a href="http://notabug.org/">Notabug</a> who seem to be run by great
people, but we run the risk of running into these same problems all
over again... though so we did with most of these other solutions.
We could self-host, but there is very little time for extra server
maintenance burdens right now.  So we
<a href="http://savannah.gnu.org/projects/mediagoblin">moved our git repository over to Savannah</a>
and I'm glad to have hosting there by people I trust, though I do also
feel that contributors may expect more modern hosting facilities, but
we don't have time to run them ourselves.
</p>

<p>
The Gitorious shutdown came around the time of great exhaustion of
already dealing with issues related to the bugtracker and left me
feeling very burnt out.  But it also lead to a great amount of
reflection... who am I to feel frustrated with?  The Gitorious people
graciously hosted our software repositories for some time, and I was
not willing to run an instance of my own.  Why not?  Well, one problem
is that if you run your own instance of a free software web
application that isn't federated, you're working in Yet Another
Semi-Free Micro-Silo (TM).  But let's face it, that's not the real
major problem... looking at the frustrations we've had with Trac, the
answer is obvious: running free network services is a huge
maintenance burden.
</p>

<p>
But wait... yes, you may be catching a whiff of irony here... if I'm
not willing to run a free software web application because I don't
want to take on the maintenance burden of <i>someone else's</i> software,
how can I ever expect MediaGoblin to gain adoption?
</p>

<p>
And here's where I come to a tough, but I think necessary, conclusion:
there's simply no way for MediaGoblin to succeed if the world of
deployment stays where it's at.  Something must be done.  But what?
</p>


<h2>Research into deployment and federation</h2>

<p>
Partly affected by the above, summer came around, and I had a talk
with MediaGoblin contributors in our monthly meetings.  I wanted to
take a sabbatical... a sabbatical where I was on break from "direct"
MediaGoblin things, so I can advance things that affect MediaGoblin
greatly indirectly.  (My spouse and I were also going through a big
move so this was a good time to do it, I figured.)  I wanted to do
research into two things: deployment and federation.
</p>

<p>
Framing the deployment side of things, Deb Nicholson and I gave a talk
that kicked off <a href="http://mediagoblin.org/news/userops.html">"userops"</a>,
a term that I still feel accurately captures what we're trying to do
(and a talk which I still believe accurately summarizes the challenges
we're facing).  Within this context I began exploring options of what
can be done to improve deployability.
</p>

<p>
The directions I've explored, and why I've come to the conclusions I
have, are a series of blogposts of their own (if you're interested, I
suggest subscribing to the
<a href="http://lists.mediagoblin.org/listinfo/userops">userops mailing list</a>,
where I will be posting more as I go).  The short of it is that I laid
out a set of requirements to achieve easy and <i>maintainable</i>
deployments, attempted to explore them with the most popular current
tools (Ansible, Puppet, Salt, Docker, etc), and found that it is not
possible to build both easy and maintainable systems on top of them
based on what I believe users need.  This lead me eventually down the
path towards <a href="https://www.gnu.org/software/guix/">Guix</a>, a package
manager (and with GuixSD, also a distro) and soon to be deployment
system which I am very confident has the potential to solve many of
the challenges which make deploying and maintaining systems too
exhausting a task for the average user.  The software is nowhere near
being "easy" at present (I think it's very telling to say that it's
"the Emacs of package managers / distributions"... one can extrapolate
on that in many ways, most of which are correct, except for the
Emacs-haters kind... lay off, Emacs haters!) but I think has the
potential to become so.  An easy to use web user interface has already
been demonstrated, and I believe the foundations are good for building
a complete and easy to use system for everyday users.  But again, to
go into detail beyond what has already been said is something that
will be explored elsewhere.
</p>

<p>
There is a more pressing need for me to have explored deployment at
present as well... though I want to explore deployability for the sake
of other users, I also need to explore deployability for <i>my own</i>
sake... particularly because we promised that we will provide premium
hosting in the last campaign we ran!  Yes, in case you have been
wondering about it, I have not forgotten about this promise.  How to
fulfill this promise without being crushed by the maintenance burden
of hosting?  I came to realize that there was every risk that I would
spend all my time supporting and maintaining servers that were running
MediaGoblin and I would not have the opportunity to any longer be a
steward of MediaGoblin itself... which could lead to failure.  So
figuring out a better path forward on hosting has become a necessity.
I'll explore what this means further in the next section, but first, I
should say a word on federation and what my sabbatical lead to on this
front.
</p>

<p>
As I have said, Jessica and I have been involved in the
<a href="http://www.w3.org/Social/WG">W3C Social Working Group</a>.
Part of the activities of the group have been the definition of
the <a href="http://www.w3.org/TR/activitystreams-core/">ActivityStreams 2.0</a>
and <a href="https://github.com/e14n/pump.io/blob/master/API.md">Pump API</a>
standards.
</p>

<!--
   Cut text, but hey, if you're reading the html, you can still see this!

   Well, one of the funny things about MediaGoblin's goals and history is
   that when I sat out to start MediaGoblin, I knew from day 0 I wanted
   it to support federation.  At the time, I had no idea *how* I was
   going to support federation, I just knew that StatusNet was doing it,
   that it seemed a critical part of making this decentralized web thing
   work, and that therefore I wanted MediaGoblin to do it.  But how to
   actually do federation I've had to learn along the way.  Just as I
   started to wrap my head around the pile of documents that constituted
   OStatus federation, the Pump API came out, and while it was clear to
   me that it was a much better way forward, I lacked the experience to
   know how to do it.  Thus I consider myself very fortunate that we
   managed to land pulling Jessica Tallon into the project, who has been
   able to actually convert MediaGoblin towards being a federated
   application, partly building on her experience of having built PyPump.

   So I wanted to gain a deeper understanding of how things were pieced
   together, and well, ActivityPump needed someone to actually start an
   implementation, so I decided to start an experimental repository where
   I can play around with these ideas.
-->

<p>
I set out to explore this a bit more deeply under a repository
with the joking and interim name of
<a href="https://github.com/cwebber/activitystuff">activitystuff</a>
(the Social Working Group is using GitHub for its work, so I'm making
an exception there).  Along with some other projects, this has
contributed to a deeper understanding of how federation should work,
which is something useful to take back to MediaGoblin.  There are some
potentially useful things in that repository (including a partially
complete implementation of the
<a href="http://json-ld.org/">JSON-LD</a>
<a href="http://www.w3.org/TR/json-ld-api/">API</a>),
and it may turn into a full implementation of ActivityPump, though its
original and primary goal was for exploration, a purpose it has served
well.
</p>

<p>
One major event relating to federation has just occurred over the last
week and bears note here: a number of <a href="http://pump.io/">Pump.IO</a>
community members and myself are now working with Evan Prodromou (who
has near-certainly contributed more to the federated web space than
anyone else alive) to transition the project from being a project of
primarily stewardship under Evan to one of community stewardship and
governance.  I posted a
<a href="https://identi.ca/cwebber/note/1YADzufsQ_29LCJWTA-K_g">summary of a recent meeting</a>,
and (MediaGoblin contributor!) Laura Arjona put together a
<a href="https://github.com/e14n/pump.io/wiki/Community">community document</a>.
In sum, Pump.IO could use your help if you're interested.
</p>

<p>
So anyway, it's been a busy last couple of months!  But it's time to
return to MediaGoblin-ville, so...
</p>


<h2>What's next?</h2>

<p>
Well, that's a whole lot of text above, so how about a bulleted list
next?  I hear those are easier on the eyes.
</p>

<ul>
  <li>
    I'm swooping back into MediaGoblin territory starting next month.
    However, my initial focus will be on getting MediaGoblin to a
    deployable point for myself, particularly forward-looking towards
    making premium hosting feasible.  Expect more soon!
  </li>
  <li>
    Jessica is working on MediaGoblin federation.  We expect the massive
    database changes she has been working on to tidy up in the next few
    weeks, and if all is well, we'll have server to server federation
    basics in place by the end of the year.
  </li>
  <li>
    Work towards 0.9.0 will proceed, unless 1.0 lands first!
    Expect major infrastructure improvements in 0.9.0,
    and federation to land in 1.0.
  </li>
</ul>

<p>
Those are all great things, and I think we are indeed on track towards
our goals, slowly but surely.
</p>

<p>
There is a more rough side to this: we've allocated nearly all of the
funds from the last campaign towards paying for Jessica's work on
MediaGoblin, and she's devoted enough to the project that she's been
working well, well below market rate.  (And aside from some travel
reimbursements, I have not taken money personally from the last
campaign.)  I'll post a financial transparency report soon, but the
short of it is: the MediaGoblin funds are running low.  Even with
Jessica generously working for such a relatively low amount of money,
we won't be able to pay Jessica to work on MediaGoblin much longer,
given our present finances.
</p>

<p>
But we aren't giving up!  The goals of MediaGoblin and of federating
the web are just too important, and so onward we press, into the
future!  Though MediaGoblin is an ambitious project, I am confident we
can achieve the things we have set out to do, but we are constrained
by limited resources and time.
</p>

<p>
Appreciate what we are doing?  Want to help us in our quest to bring
network freedoms to everyone?  You can help!
</p>

<p class="centered">
  <a href="https://my.fsf.org/civicrm/contribute/transact?reset=1&id=36" class="campaign_donate"><img src="/images/campaign/heart.png" alt="<3" />Donate...</a>
</p>

<ul>
  <li>
    The simplest way to help?
    <a href="https://my.fsf.org/civicrm/contribute/transact?reset=1&id=36">Donate!</a>
    Though the official <a href="http://mediagoblin.org/pages/campaign.html">campaign</a>
    is over, you can still
    <a href="https://my.fsf.org/civicrm/contribute/transact?reset=1&id=36">donate to MediaGoblin through the FSF</a>!
  </li>

  <li>
    Jessica and I have been contracting so we can pay the bills (doing
    this allowed us to "stretch out" the amount of time Jessica could
    spend on the project... useful with all that W3C stuff in
    progress!).  Unfortunately, while the people we have been working
    with are great, the contract we've been working on is coming to an
    end.  Are you looking for contractors or part time workers who are
    capable of developing high quality free software and providing
    community leadership?  Jessica and I are both interested in working
    in such cases... feel free to email me at:
    cwebber AT dustycloud DOT org
  </li>
</ul>

<p>
We're committed to making a better, decentralized web.  I hope this
piece cleared up where we're at and where we're going.  I believe
we've got exciting times ahead... till next time, goblins!
</p>

</body>
</html>