mxm - IT's Mad Science

How to export all redirects from portal_redirection in an older Plone site

2011-11-08T06:53:00.000-08:00

Just add the method below to the RedirectionTool and call it from the browser as:

http://localhost:8080/site/portal_redirection/getAllRedirects

Assuming that the site is running at loaclhost:8080 that is :-S

That will show a list of redirects that can be imported into plone 4.x


   security.declareProtected(View, 'getAllRedirects')
   def getAllRedirects(self):
       "get'm'all"
       result = []
       reference_tool = getToolByName(self, 'reference_catalog')
       for k,uuid in self._redirectionmap.items():
           obj = reference_tool.lookupObject(uuid)
           if obj is None:
               print 'could not find redirect from: %s to %s' % (k, uuid)
           else:
               path = '/'.join(('',)+obj.getPhysicalPath()[2:])
               result.append( '%s,%s' % (k,path) )
       return '\n'.join(result)

Another micro hack

2009-11-06T05:29:00.000-08:00

I often write a pagetemplate that needs to be published later. But page templates has no workflow state.

The problem is when the customer needs to test the page on the live page, but others must not be able to see it yet.

I have a small hack to ensure that only managers can see the page.

I call a low overhead method that is protected by the "Manage portal" permission. Like:

<tal:managerCheck tal:define="managerCheck python: here.portal_migration.getInstanceVersion()" />

When the site is to be published I just remove the line.

Well I told you it was a micro hack!

School of Small Hacks

2009-01-26T08:19:00.000-08:00

Being a web developer it bothers me that I cannot have seperate sessions open in Firefox. You can do it in IE. If you open a new browser you get a new session.

So I (mis)use the hosts file to do something similar in Firefox.


127.0.0.1 localhost
127.0.0.1 localtest # insert this.

Then I can go to http://localhost/ in one tab and http://localtest/ in another. That way the browser sees it as different session.

A small trick for socket timouts in Plone

2009-01-06T12:06:00.000-08:00

A few times I have had the need for a socket to timout in Plone. Eg when calling an external site that might be down.

You can set the timeout in Plone, but that is not a good idea as you sometime need long running processes to end properly, eg for packing etc.

A workaround is spwaning seperate threads and callback routines. But that is a bother to set up.

Suddenly it stroke me today that it would be possible to change the timout temporarily for a single thread:


import socket
import feedparser

def parseUrl(self, url):
    "Fetch the url"
    orig_timout = socket.getdefaulttimeout() # get original timeout
    # give it a 20 second timout
    socket.setdefaulttimeout(20)             # set to 20 seconds
    try:
        result = feedparser.parse(url)['entries']
    except:
        result = []
    socket.setdefaulttimeout(orig_timout)    # set back to original
    endresult = []
    for r in result:
        r2 = {}
        r2.update(r)
        endresult.append(r2)
    # sort for latest first
    decorated = [(r['updated_parsed'], r) for r in endresult]
    decorated.sort()
    decorated.reverse()
    sorted = [d[-1] for d in decorated] 
    return sorted

Sweet and simple ...

Put on the Pyjamas and code

2008-11-28T14:46:00.000-08:00

Javascript is a nice language, but Python is nicer. So how about writing in Python and having it automatically translated to Javascript. I will have to check that out.

http://pyjs.org/

Installing Deliverance

2008-09-02T01:44:00.000-07:00

Deliverance is a cool tool, that makes it possible to give web based software a new skin.
Without changing the original software.

It opens a lot of new options to development, migration and integration.

You can have site running that uses Plone, plain html, mailman, blogger etc. And still looks like a single coherent site.

I had few problems installing it using the manual method, so this is the code from how I actually got it to work.


easy_install virtualenv
virtualenv deliverance_env
cd deliverance_env/
source bin/activate
svn co http://codespeak.net/svn/z3/deliverance/buildout/trunk .
python bootstrap/bootstrap.py
./bin/buildout
./bin/deliverance

It started up fine, but the first page I tried to get threw an exception:


Error - exceptions.ImportError: No module named HTML4

Turned out that the default generated deliverance-proxy.ini had a bug:


serializer = deliverance.serializers.HTML4

Changing it to:


serializer = deliverance.serializers:HTML4

works

json_migrator Alpha release.

2008-06-04T12:18:00.000-07:00

The code for my Plone json migrator has been "sort of functional" for a while now. I will probably not change it before the summer vacation, but that is no reason not to let other look at it.

It is only available in the Collective.

http://svn.plone.org/svn/collective/json_migrator/trunk/

It was successfully used to migrate a 2.0.x site to 3.1.

Export of member data from Plone as a CSV file

2008-05-23T16:11:00.000-07:00

I needed to export member data from Plone for mail merging etc. There was nothing really suitable available so I wrote this small product for it.

The manual is:

Install "portal_memberdata_export" with the quickinstaller.

Export by calling the memberdataExportCSV script, like http://www.example.com/memberdataExportCSV

You need to have the "Manage Portal" permission.

So far it is only available from the collective.

http://svn.plone.org/svn/collective/memberdata_export/trunk/

Note to self. Practical csv import module

2008-05-23T14:09:00.000-07:00

It seems that I have a bunch of data import projects recently. I have made this simple module for converting a csv file to a list of dicts with unicode keys and values. So I just write a note to future self to remember to use it.

Max, it is here: http://www.mxm.dk/papers/csv-import

Tooting my mailhorn - MailHostMonkeyPatch

2008-05-22T16:02:00.000-07:00

I have been writing some simple maillist software the last few days. A simple form that can send an email message to all members on a site.

It made me remember my MailHostMonkeyPatch which patches the zope/plone mailhost so that emails are saved in a directory structure instead of being sent to innocent recipients.

One use is that you don't need an smtp host to test your mail software.

Another is that you can see the emails that users will actually be receiving. In this case it has string substitution of member data in each email. So it is *very* practical to be able to see the mails.

If you are writing Plone software that uses the MailHost, then this is highly recommended for debugging.

http://www.mxm.dk/products/public/mailhost-monky/

A word of warning though. It works on MailHost and SecureMailHost, but I have only tried it up to Plone 2.5. So on 3.0 it might not work.

Upgraded my Gutsy Gibbon notebook to Hardy Heron

2008-05-05T01:24:00.000-07:00

It was almost effortless. I had a few minor issues though.

First it stalled during the upgrade with a package called "wvdial". I googled it, and it turned out to just be a modem dialer. I have not used a modem for 10 years, so I just did a "ps -A" to find it, and a "kill". Then the install continued without any more problems.

After the install everything seemed to run fine, but I noticed something off with the video display. Windows where not scaling smoothly and video clips played where choppy and had lower framerates when they where resized or fullscreen.

After another bit of googling about my ATI graphics adapter I found a tip to remove xserver-glx. I needed xserver-glx under Gibson to make that run smoothly. But apparently not under Heron. So now I have working video and a smooth desktop.

My custom and buildout based Python/Plone stuff continued to work, which was my major worry.

All in all a very pleasant experience for such a major upgrade.

It turned out that I was a bit premature:

Java is broken in Firefox 3. I use it mostly for netbanking. But that is pretty important. So I just spend an hour reinstalling firefox 2 and then had to:


cd /usr/lib/firefox/plugins
sudo ln -s /usr/lib/jvm/java-6-sun-1.6.0.06/jre/plugin/i386/ns7/libjavaplugin_oji.so

Then I could use my netbanking again.

Turned out that Eclipse broke too

I had quickly started and stopped eclipse, so I thought it worked. But it turned out that it crashed every time it tried to "build workspace".

I renamed the workspace in my home dir and created a new one with Eclipse. The new one did not crash. So I can manually add my projects and keep working. Worst part is that at lost my font settings etc.

This article about the Rails comunity makes me happy to be a Zopista ..

2008-05-02T04:52:00.000-07:00

I have never ever experienced anything like this in the Zope, Plone, Python comunity.

http://www.zedshaw.com/rants/rails_is_a_ghetto.html

"After Mongrel I couldn’t get a gang of monkeys to rape me, so forget any jobs. Sure people would contact me for their tiny little start-ups, but I’d eventually catch on that they just want to use me to implement their ideas. Their ideas were horrendously lame. I swear if someone says they’re starting a social network I’m gonna beat them with the heel of my shoe."

This is the first time it has occured to me that the relative difficulty of getting in to Zope/Plone actually makes for a higher quality of the comunity.

We Got Dublin Core MetaData But What About "The Rest Of The Core" DataData?

2008-04-14T03:24:00.000-07:00

Dublin core metadata is very useful. But there is a lot of other attributes that could be defined as well. Why don't we make an interface for standard Zope attributes?

I have just finished the migration a medium sized site. about 8000 content objects of which 4000 ended up on the migrated site.

I used the Json migrator that I have described in previous postings.

The old site consisted of a mixture of CMF and AT based types. There was several custom AT products. The new site was Plone 3 based and consists of standard Plone sites and some Custom products.

Some of the old custom content was migrated to standard Plone types as Plone has been catching up in functionality, and some was migrated to new content types.

All of these content types used different attribute names for basically the same attributes.

I tried partly to develop a reusable migration tool and, partly to finish a concrete project.

One of the ugly things I did to get finished faster was to use the getter's of the old site as keys in the json dicts.


{'getText': obj.getText()}

This results in a typical json file looking like this:


{
    "Publisher": "No publisher", 
    "CreationDate": "2007-11-04 03:39:56", 
    "-*-*-*-datafields-*-*-*-": {
        "text": {
            "mimetype": "text\/html", 
            "name": "text", 
            "fname": "document_3428.json.0.blahblah_2007-11-08.html"
        }
    }, 
    "Language": "da", 
    "Title": "Some Title", 
    "getPhysicalPath": [
        "lokalt", 
        "somewehere", 
        "dokumenter", 
        "blahblah_2007-11-08"
    ], 
    "portal_type": "Document", 
    "Creator": "jrh", 
    "site-encoding": "utf-8", 
    "getId": "blahblah_2007-11-08", 
    "ModificationDate": "2008-03-02 08:55:31", 
    "EffectiveDate": "2007-11-04 03:49:38", 
    "getOwner": "jrh", 
    "Format": "text\/html", 
    "meta_type": "Document", 
    "Date": "2007-11-04 03:49:38", 
    "review_state": "published", 
    "Identifier": "http:\/\/localhost:8081\/plone\/lokalt\/somewehere\/dokumenter\/blahblah_2007-11-08", 
    "Type": " Side"
}

This annoys me and made a long standing problem pop up once again. The afore mentioned lack of uniform naming of attributes in Zope/CMF/Plone/Zope3.

We have had several defacto standards since the plain Zope days, from a mix of often used Zope, CMF and AT content types.

The HTML body of a document has been named text, body, content, data etc. in different content types.

There is no real reason that they could not have the same name. Except that no standard has been defined.

Zope standard attributes

The next step in my json migrator project is to make it work from any Plone version to any Plone version. Well at least to make the framework so that it is possible.

Making Python work from 2.3 to 2.4 was 4 lines of code (I use sets). So that is not the problem. Making it work on even older Python versions would be no biggie either.

Currently it migrates fine from Plone 2.0.x to 3.0.x, but there are a lot of interim versions. and parts of it might break due to different attribute names.

So what I need to move on is a meaningful standard for Zope attributes.

I can then define an interface and write a wrapper for each content type in each version of Plone I want to migrate. Something like:


class IZopeStandardAttributesData:

    """
    Interface for data attributes
    """

    def AttributeName():
        "Data fields attribute name on the content type"

    def Data():
        "The raw data"

    def ContentType():
        "Like 'application/msoffice'"

    def Filename():
        "Like 'Some Text Document.doc'"
 

class IZopeStandardAttributes:

    """
    Standard attributes name for Zope content types. Version 1.0
    """

    def HTML():
        "Main content For document types. Raw HTML."

    def HTMLSafe():
        "Main content For document types. Safe html."

    def StartDate():
        "For calendaring event types"

    def EndDate():
        "For calendaring event types"

    def DataFields():
        "Sequence of data fields"

    def Participants():
        "Sequence of participant for groupware content types"

Naturally there would need to be many more.

I could then use wrappers to convert all content types to Json using these attributes names. Converting from Json data to Plone content objects would be just the same. Having standard attributes that all content types would be converted to and from is probably the only way to make this work in reality.

I believe that this would both help in my current migrations story, and would also make Zope/Plone programming easier for newcomers to grasp. Everyone would not need to make up new attribute names for their own products.

UML tools and the like could even support it so they would suggest these attributes in new types.

The interface should be put in the collective or somewhere else that is suitable.

Is this an idea worth pursuing as a community project, or should I just make it for my migrator and shut up?

What Does A Web Framework Really Need?

2008-04-02T13:54:00.000-07:00

There has been some discussions recently about the role of Plone. Is it a framework or a platform. Should we make a new CMS with Z3/Grok. So I will try to take a fresh approach and look at an absolute minimal system, that would not need a new version for many years.

That got me to think about what is really needed in a web framework to make a coherent user experience. Notice that I mean end-user experience. Not the programmers experience.

Plone has modules for many things

They are all coupled together in a tightly knit system. This is what makes Plone so usable out of the box. It is also the reason that Plone is so hard to learn.


access control, authentication.
security, roles and permissions, Workflow.
data storage, ZODB, ZEO, SQLAdapters.
traversing, ZPublisher, VirtualHostMonster.
content types, OFS types (Old Zope), CMS Types, AT Types, Calendaring. Content Type Translations (html -> txt etc.), Version Control.
form handling
css structure, resource registry.
Templating, zpt, dtml, python scripts, portal_skins.
Editing content, stx, rest, html, kupu, ajax.
Data Querying, ZCatalog, SQLAdapters.
Language support, PlacelessTranslationService, PloneLanguageTool.
Mail Handling
Product Installation

But if we are talking about a framework, most of these things are not necessary. you only need 3-4 technologies to make a stable and usable web framework. The most radical idea I have is a completely new way to think about the templating system.

In the following a "page" is any unique url in the site. AT is Archetypes.

CSS and design guidelines

If you are the user of a website, all you want is for the site to have a strong internal logic. The most important part of that is that the entire site should look the same.

That makes a common css for a site the most important part of a web framework ... go figure.

If all pages developed for a website shares the css, then you can work around the rest. Pages can be made in html, zope2, zope3, php etc. It will not matter as long as they follow the same design guidelines.

Templating

The second most important feature for the user to get a coherent feel for a site, is a system that makes as many of the pages follow a common page structure as possible.

In Plone there is the famous o-ring design. A Page Template called main_template that defines the look of the entire site. This model works well in most of the cases, but it hinders in many other.

If you want to create a new content type in Plone using AT, the O-ring design and shared css is a *big* help.

This was the main difference between plain old Zope 2 and Plone. You could add a product and it would look and feel the same as any other product.

AT got the look and feel integration locked down even tighter, AT content types would share the edit forms machinery.

But that was also where it started to get difficult to work in anything but AT. Because Plone and AT got so tightly integrated.

The main problem with the shared templates being integrated into the system, is that it is hard to use non framework software. Mailman is a great mail list system. Integrating it into Plone? Good luck! There are a lot of BBS systems written in php or other inferior languages ;-) Integrating them into Plone? Good luck!

Imagine if all that was required from the templating system was that the content would generate a well formed XML file with a defined DTD. Then the templating system would just convert that XML file into the actual file served by the http server. It would layer the o-ring, css etc. on top.

Outside applications would just need to be rewritten to output its content as XML and Python developers could use their templating system of choice.

I believe that the site theming / templating system currently is in the wrong layer. I guess it is the same thing that Paul Everit keeps talking about with his XML/XSLT stuff. This would be like WSGI in steroids. A kind of rendering layer between WSGI and the web apps.

Another way to look at it would be as a proxy that fetches content from several urls and rerenders them to fit a common template and site structure.

There could even be a visual tool for changing the layout, that all web apps could share.

Authentication

The two first two features are for anonymous users. If you want your users to be able to log in to the site you must have a common form of authentication. Separate pages requiring separate logins is just not considered cool by users.

Currently it is done in Plone. But it could just as easily be done by a separate authentication server that could run on another port or server:

http://localhost:8081/authenticate?user=admin&email=some@example.com&password=admin

That would make it pretty easy to share authentication.

Username and passwords could also be handed down from a traverser to each page, to automatically log on to 3rd part software.

Traverser

I am not to sure about this one. I don't know much about how the ZPublisher works. It seems like a job for WSGI, or a configuration system to make sure that a certain branch of pages is handled by (proxied from) a specific app server.

Conclusion

So that's it. I think you could make a very good web framework with just those few items. The rest of the things from the Plone stack does not really need to be in a core framework.

That is especially clear with something like content types or templating where many solutions have been used. We cannot get rid of any of them as it will destroy backwards compatibility. Even refactoring will destroy backwards compatibility.

If it was made as I described above, you could use Plone from some parts of a site and Zope 3 for other parts. A specific framework could be used for what it was especially good for. Like Plone for Content management and you could change a discussion board from PloneBoard to PHPBoard in a part of the site, while maintaining the look and feel.

Comments? Would a minimal web app like that be useful? A sort of meta framework.

Python Unicode lessons from the school of hard knocks

2008-03-30T13:34:00.000-07:00

Unicode is becomming increasingly popular, but is often misunderstood. That is a pitty, as it is really not that difficult. Especially since unicode is a really powerfull tool to know, and it becomes the standard string type in Python 3K.

There are many unicode articles around, but they are often very "theoretical" and computer science like. So I have collected af few examples I have learned in the school of hard knocks, that are practically oriented.

I will use "unicode" as shorthand for a unicode string, and "string" as a shorthand for a plain Python string. I live in Europe and am used to using latin-1 as default encoding. Whenever I write latin-1 just substitute that with your own language default encoding. Should work just fine.

I remember when I tried to understand unicode that I could not get my head around when to use encode and decode. In these examples I practically only go from unicode to string. So I use the encode() method for most examples. I believe this makes it easier to understand and remember.

If there is popular demand I might write an article using only the decode() method.

Working with unicode in code

First examples from a python console:


>>> 'this is a python string'
'this is a python string'

>>> u'this is a python unicode string'
u'this is a python unicode string'

Not much difference there. That is because they both contain only ascii characters. When I try to insert a danish character it changes:


>>> 'this is a pythøn string'
'this is a pyth\xc3\xb8n string'

>>> u'this is a pythøn unicode string'
u'this is a pyth\xf8n unicode string'

The string example shows something interresting as it shows the ø as '\xc3\xb8'. Whenever you see international characters showing up as two encoded characters/bytes like this, it is usually a sign that you are seeing a utf-8 encoded string.

It is by no means the law, but it is a good rule of thumb.

In this example the string is encoded to utf-8 because that is the default I use in the console window that runs the examples.

Thinking about unicode

A good way to think about the difference of unicode and string is as a text and a binary file.

Unicode is the text and the string is the binary file format. So when you want to save your text somewhere you save it in a file format. Different file formats you can use are latin-1 (iso-8859-1), ascii, utf-8 etc.

You convert to the correct file format by using the 'encode()' method. Like this:


>>> u'this is a pythøn unicode string'.encode('latin-1')
'this is a pyth\xf8n unicode string'

>>> u'this is a pythøn unicode string'.encode('utf-8')
'this is a pyth\xc3\xb8n unicode string'

>>> u'this is a pythøn unicode string'.encode('ascii')
Traceback (most recent call last):
  File "", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 14: ordinal not in range(128)

Whoops. ascii is an illegal file format for text with international characters. That is an important lesson.

You can get from unicode to every other encoding without any other information about the unicode string. If the encoding supports the characters your unicode use. So unicode is the singular type from and to which all the others text formats can be converted.

unicode to anything

String encodings are archeological. Well sometimes you even get the feeling that they are geological. Especially when writing something to support an RFC. Each geological layer is there for backwards compatibility.

First there was ascii, then iso-8859-1 (latin-1), iso-8859-15 (adds the € sign and more) and finally utf-8. They are mostly supported in that order too. The older the encoding, the better support it has in software out there in the wild.

Theoretically you can just use utf-8 for it all end be done with it. But I have had to rewrite almost any email/maillist application I have written to support latin-1 or other encodings. Many mail clients supports utf-8 correctly these days. But many web based clients do not. Neither hotmail nor gmail does actually.

Most likely you will be writing software that works perfectly, and passes every test and mail client you and your customer have in your organisations. But when the brand new maillist goes live, customers complains about unreadable characters.

So it can make good sense to try and encode unicode into a gradually more "modern" encoding. Trying the older encodings first, and if that fails then try newer ones. This function does that::


# -*- coding: utf-8 -*-
>>> def optimal_encode(st):
>>>     for encoding in ['ascii','iso-8859-1','iso-8859-15','utf-8']:
>>>         try:
>>>             return (encoding, st.encode(encoding))            
>>>         except UnicodeEncodeError:
>>>             pass
>>>     raise UnicodeError, 'Could not find encoding'
>>> 

>>> st = u'this is a pythøn unicod€ string'
>>> print optimal_encode(st)

('iso-8859-15', 'this is a pyth\xf8n unicod\xa4 string')

The € sign makes it return 'iso-8859-15'. If that encoding was not in the list, it would return::


('utf-8', 'this is a pyth\xc3\xb8n unicod\xe2\x82\xac string')

Working with unicode in your editor

Any modern text editor can handle utf-8 encoding. So preferably you should use that in your Python files. You tell Python that your files are utf-8 encoded by adding an encoding declaration to the top of you file.

# -*- coding: utf-8 -*-

When this is done, you can write international characters directly in your source code and every string in your file is utf-8 encoded. This makes it true that:


u'pythøn'.encode('utf-8') == 'pythøn'

Saving unicode in files

Unicode only exists in memory. You cannot write it to a text file unless you write it as pickled data. But nobody else would then be able to read it, and you cannot look at it in an ordinary text editor.

To put your data into a file, you must encode it first.


>>> st
u'this is a pyth\xf8n unicode string'
>>> f = open('unicodetest.txt', 'w')
>>> f.write(st)
Traceback (most recent call last):
  File "", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 14: ordinal not in range(128)
>>> st_utf8 = st.encode('utf-8')
>>> f.write(st_utf8)
>>> f.close()

When you try to write the unicode string to a file, it tries to convert it to a string and fails. But when you encode it first there is no problem.

There is a special 2 byte code that can be inserted into the beginning of a text file to mark it as some kind of unicode encoded string. It is called a Byte Order Mark (BOM)

There are a few of those for the different encodings, but I have only ever had to use utf-8


>>> import codecs
>>> codecs.BOM_UTF8
'\xef\xbb\xbf'
>>> f.write(codecs.BOM_UTF8 + st_utf8)

The Byte Order Mark (BOM) is used especially on Windows. Frankly it is a bit of a mess in Python until 2.5. But many text editors can recognize it and then knows automatically that the file is utf-8 encoded.

As far as I can figure it out, the Page Template skin system in Zope does recognize the BOM. So you can use international characters in Page Templates.

At least I have tried editing files that where utf-8 encoded, but the international characters displayed wrong. They looked like: 'this is a pyth\xc3\xb8n string' So Zopes Page Template system got them wrong. Other times I have written them and they looked like 'this is a pythøn string'.

I asume that is due to the difference in having a correct BOM or not that Zope can recognize.

Generally though I work on many different Zope/OS system combos. So when I see these kind of problems in html, I generally take the lazy way out and just use html entities so it looks like: 'this is a pythøn string' :-s

Unicode in html

In html you can represent international characters as both html entities (eg ø) and as encoded strings. Normally you will use utf-8 for encoded strings.

If you choose the encoded strings you must tell what the encoding is. You do this by setting it as meta data in the head:


<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

Cou can then use normal procedures for converting unicode html to utf-8 strings.

Encoding strings

Unicode is not the only python class to have an encode() method. The string object also have one. That can be handy in many cases.

If you need to use a url, or any other string with special characters, as a filename it will cause you problems. Any string can be encoded in a simpler format that can be used as a filename or even as part of a path.

The hex encoding is the simplest, but base64 is the most space efficient.


>>> 'http://mxm-mad-science.blogspot.com/'.encode('hex')
'687474703a2f2f6d786d2d6d61642d736369656e63652e626c6f6773706f742e636f6d2f'

>>> 'http://mxm-mad-science.blogspot.com/'.encode('base64')
'aHR0cDovL214bS1tYWQtc2NpZW5jZS5ibG9nc3BvdC5jb20v\n'

A caveat here is that base64 always have a newline character in the end, and it might have a '=' as padding. So you need to modify it a bit to remove those:


>>> 'http://mxm-mad-science.blogspot.com/'.encode('base64')[:-1].strip('=')
'aHR0cDovL214bS1tYWQtc2NpZW5jZS5ibG9nc3BvdC5jb20v'

And when you want to decode it again you must always append a padding character. Otherwise it will fail about 50% of the time ...:


>>> ('aHR0cDovL214bS1tYWQtc2NpZW5jZS5ibG9nc3BvdC5jb20v' + '=').decode('base64')
'http://mxm-mad-science.blogspot.com/'

I once made some mailinglist software in Zope, where The best solution was to save the subscribers with their emails as the id of the subscriber object. But Zope does not accept @ in a path. All I then had to do to make it work was:


>>> id = 'maxm@mxm.dk'.encode('hex')
>>> id
'6d61786d406d786d2e646b'

What have this got to do with unicode you might ask? Well you can use this method to use unicode string as filenames without needing to remove special characters.

First make the unicode string, then convert it to utf-8 and then convert that to hex or base64.


>>> st = u'this is a pythøn unicod€ string/text'
>>> st_utf8 = st.encode('utf-8')
>>> st_utf8
'this is a pyth\xc3\xb8n unicod\xe2\x82\xac string/text'
>>> st_hex = st_utf8.encode('hex')
>>> st_hex
'7468697320697320612070797468c3b86e20756e69636f64e282ac20737472696e672f74657874'
>>> st == st_hex.decode('hex').decode('utf-8')
True

Unicode in emails

Once upon a time when Bill Gates invented email he thought "7 bits is more than enough for every character." Well ok. Maybe it was not Bill Gates. But someone must have thought it. As that is the foundation that email is built upon.

Body Text

Actully, email messages can contain 8 bit characters. But SMTP can only transmit 7 bit messages. And as Dolly Parton once said: You cannot put 10 punds of potatoes into a five pound sack. So if your are sending your email over SMTP. like everyone is, you must convert your code to 7 bits. This is called content transfer encoding.

First you make the body text in unicode.


>>> st = u'this is a pythøn unicode string'

Then you must convert it to the string encoding you want to send it as. Like latin-1, utf-8 etc.


>>> st_latin1 = st.encode('iso-8859-1')

Python recognise 'latin-1' as an encoding. Mail systems does not, so it is safer to use 'iso-8859-1'. Python will translate it automatically in the email module, but I have been bitten when composing emails wihtout it, so I have made it a habbit to always use the long form.

Latin-1 is still an 8 bit string. So we must content transfer encode that to a 7 bit ascii string.

A simple email message is made like this. Note that the set_payload() method does not do any encoding. You merely tell it what encoding your string is already in:


>>> st = u'this is a pythøn unicode string'
>>> st_latin1 = st.encode('iso-8859-1')
>>> from email.Message import Message
>>> msg = Message()
>>> msg.set_payload(st_latin1, 'iso-8859-1')
>>> str(msg)
'From nobody Sun Mar 30 15:13:31 2008\nMIME-Version: 1.0\nContent-Type: text/plain; charset="iso-8859-1"\nContent-Transfer-Encoding: quoted-printable\n\nthis is a pyth=F8n unicode string'

The email module can also encode as base64, but that is mostly used for 8 bit binary content as it makes the files smaller than quoted-printable would.

You could use base64 for all email content types, but keeping text messages human readable is simply more practical.

Sending HTML

Sending HTML works exactly like plain text from a unicode point of view. HTML files can also be made as unicode and then encoded as utf-8 or latin-1 etc. Just set the content type:


>>> del msg['Content-Type']
>>> msg.add_header('Content-Type', 'text/html', charset='iso-8859-1')
>>> str(msg)
'From nobody Sun Mar 30 15:23:14 2008\nMIME-Version: 1.0\nContent-Transfer-Encoding: quoted-printable\nContent-Transfer-Encoding: base64\nContent-Type: text/html; charset="iso-8859-1"\n\ndGhpcyBpcyBhIHB5dGg9RjhuIHVuaWNvZGUgc3RyaW5n'

Based on the conent type the email module chooses to convert html to base64 for you.

Email Headers

Email headers are special, and especially ugly. You press send on your new maillist software that has worked correctly during testing. Then all of a sudden there is one of them nasty international characters in the subject.

You did set the charset to iso-8859-1 in the add_header() method. So why does it fail?

Well because each header field needs to be set individually. The content of the headers has nothing to do with the message content of the text (payload).

you can set it like this:


>>> from email.Header import Header
>>> from email.Message import Message
>>> msg = Message()
>>> subject = u'This is a pythøn subject'
>>> subject_latin1 = subject.encode('latin-1')
>>> h = Header(subject_latin1, 'iso-8859-1')
>>> msg['Subject'] = h
>>> msg.as_string()
'Subject: =?iso-8859-1?q?This_is_a_pyth=F8n_subject?=\n\n'

The email headers like To, From etc. are a little bit tricky.


>>> from email.Utils import formataddr
>>> email = u'maxm@mxm.dk'.encode('latin-1')
>>> name = u'max møller'.encode('latin-1')
>>> From = formataddr( (name, email) )
>>> From
'max m\xf8ller '
>>> from email.Header import Header
>>> h = Header(From, 'iso-8859-1')
>>> str(h)
'=?iso-8859-1?q?max_m=F8ller_=3Cmaxm=40mxm=2Edk=3E?='
>>> msg = Message()
>>> msg['From'] = From
>>> msg.as_string()
'From: max m\xf8ller \n\n'

UTF-7 and IMAP

If you are writing an email client in Python you will most likely need to support Imap. Imap has folders, and their names use a special encoding not seen anywhere else. It is an encoding based on utf-7. It is not a common problem so I will not use much space on it here. But I have written an encoder for it that you can get here:

http://svn.plone.org/svn/collective/mxmImapClient/trunk/imapUTF7.py


>>> from imapUTF7 import imapUTF7Encode
>>> st_utf7_imap = imapUTF7Encode(st)

More about the issue can be found here:

5.1.3. Mailbox International Naming Convention
http://www.faqs.org/rfcs/rfc2060.html

Migrating Plone Site via JSON files #2

2008-02-15T13:38:00.000-08:00

I sometimes wonder if I am particularly stupid, or if everybody goes through the same process as I do. Every programming problem I try to solve I basically approach with an idea of how I want to do it. Then I make a first version. No more planning than that.

If I have made something similar before, the solution is very close to my initial idea. If it is a new area of expertice the solution often ends up very differently from what I had visualized. I find a problem, i fix it and refactor the code. I find a new problem i fix it and refactor the code.

It works for me, and I find that it usually produces quality code in the end. Especially since I am pretty rutheless with the refactoring.

But I keep wondering if I would code faster if I did more upfront design.

Export is done

I think that I am finally finished with the export part of this json migrator. Well ok. This is a customer project and I am focusing on a specific subset of the content types. So it is not a general tool, but that is ok for now. I keep refactoring and using general methods to solve the problems.

I can export folders, files, document and images. Events should be easy too. I just don't need them yet. I can also export some custom types. Each custom type need about 15 lines of code.

The major thing I have not looked at yet is discussions/talkbacks.

Recap of the data structure

The (simplified) content object now looks like this.


class Document:
   getId='my-document'
   meta_type='Document'
   Title = 'My Title'
   Description = 'An example content object'
   content_type = 'text/html'
   text = 'The text in the content object'

A file like this


class Image:
   getId='this-is-an-image.jpg'
   meta_type='Portal Image'
   Title = 'This is an image'
   Description = 'An example image object'
   content_type = 'image/jpg'
   data = 'dagfhh3gh3gf3hgf' # mysterious data that is a jpeg image :-s

The document type was easy to save as json. I just mapped every attribute to a key/val in a dict.


document = {
   u'getId':u'my-document',
   u'meta_type':u'Document',
   u'Title':u'My Title',
   u'Description':u'An example content object',
   u'getRawText':u'The text in the content object',
   u'content_type':u'text/html',
}

The paths chosen, with many backtracks

A minor problem was that the large amount of html, in some of the documents, made it harder to read the json files in the text editor. Just like xml files with cdata sections.

A bigger problem was the image/file content type. I first tried to save the "data" attribute as a value in the dict. But the json writer did not like that. It turned out that there was some chars that it could not escape. It probably needs to be utf-8 or something like that. I did not digg to deep into the reason, but chose to encode the data.

The data is a string, so it is pretty simple to just use base64:


>>> 'this will be encoded'.encode('base64')
'dGhpcyB3aWxsIGJlIGVuY29kZWQ=\n'

But large image files did not exactly improve the visual signal to noise ration in the json files either. I tried for about 15 minutes to find a library that would linebreak the encoded data to at least improve the readability a bit, but finally realised that I had fallen in love with the idea of the json purity and neede to let go.

It is like that sometimes. You get an idea and have some resistance to letting it go. Authors have an expression called 'kill your darlings'. Same thing.

Base64 encoded data is also about 30% larger, and when the first 19 MByte (.bmp) files turned up in my export dir I was sure I was on the wrong track again.

The huge .bmp image file was just a portait of some random person. there was absolutely no reason for it to take up more than 1/2 MB as a reasonably compressed jpeg file.

There is one thing I have learned through the years. Users do not do what they should. They do what they can. And if they can upload a 12 MB bmp image, they will. So I expect this to be a normal use case on many sites.

A 19 MBytes file would need to be loaded into ram, and then converted back into the binary file, that would be about 12 MBytes. Meaning that I would use 30 MBytes of ram just for this single json file.

With a Plone site of several thousand objecs it can get ugly real fast.

Besides that an object made with AT can easily have several image fields each with a multi MB image.

My idea of using just json for export did not look to good.

Then I played with the idea of keeping the data in the data field and then swap it out with a filename when I wrote it out to disc. So this structure:


document = {
   ...
   u'data':u'dagfhh3gh3gf3hgf' # jpeg image
}

Would be changed to this when saved:

document = {
   ...
   u'data':u'file.1.json.1.bin' # jpeg image file name
}

And it would be saved as two seperate files on the filesystem.


file.1.json         # the json file
file.1.json.1.bin   # the data file

That was pretty clean, and I liked that it was easy to see which data file belonged to which json file. The json file was kind of a meta data file for the data file.

But now that the data was saved seperately it annoyed me that I could not see what kind of file it was. If I want to view the file "file.1.json.1.bin" i have to look in "file.1.json", see that it was a jpeg file and then rename it to "file.1.json.1.jpg". And then I could still not guess the content from the name.

Sometimes unskilled labor is a better solution than an overpriced programmer. If you only have a few files that you care about in your old CMS, it can be practical to just export them and then do a manual import.

If you can see all the files from your site in one directory it can also give a good overview of the work you need to do.

For these reasons I found that I had to rewrite once more and give the data files some sort of meaningfull names.

This turned out to be surpricingly difficult. As usual the devil is in the details.

More in the next article

Migrating Plone Site via JSON files #1

2008-02-14T03:04:00.000-08:00

I am currently migratin an old Plone 2.0.1 site to 3.0. Normally you would try portal_migrations, and then upgrade one minor Plone version at a time.

I have tried this several times in the past. It sucks!

On every step one of your custom products, and quite a few of the built ins, will break. Making every step slow and frustrating. I have had 0% succes rate with making this method work without glitches.

Going from a 2.x to a 3.x site is especially fraught with danger as the old site contains a mixture of CMF and AT based content.

A New Approach

So this time I wanted to try a different approach. Export the content object as data on the disc from the old site, and then importing it to the new site from those files.

I also want the exported files to be human readable, as it makes debugging a whole lot easier. I suck at reading pickled data. Besides that, the old site is running Python 2.3 and another is running 2.4 so pickles might break.

The method also has the advantage that json is cross platform, so when the data is exported it can even be imported into completely different CMS systems.

Remapping the Contents Type and Location

When the content is imported to the new site, it also needs to be remapped a lot. Some older content objects are no longer neede, and the data should be put into Documents or custom content types. Some of the folders are also moved around.

I could convert custom content types to Documents before upgrading, but then I would be unable to migrate special stuff like custom views, or other things that made us drop the custom types to begin with.

I can also convert the types ind the end, but then I would need to make the custom types compatible with every step in the migration.

Doing this will take som custom scripting anyay. I might as well do it in one swoop while migrating. It would be difficult with the old method.

Goal

I expect to end up with a migration script that can be run first on the old instance, and then on the new. Which will move and "massage" the content while both instances are up and running.

I also aim to make a general framework for migration via json.

I write this as I try out things, so it might not succed in the end, but it would be a real long posting if I wait until the end.

Here we go then

A (simplified) content object like the one below cannot be easily saved to disc:


class Document:
    portal_type='Document'
    Title = 'My Title'
    Description = 'An example content object'
    getRawText = 'The text in the content object'

I could save it as a xml:


<content>
    <portaltype>Document</portaltype>
    <title>My Title</title>
    <description>An example content object</description>
    <getrawtext>The text in the content object</getrawtext>
</content>

but quite frankly I find xml parsing a bore. Even with the brilliant lxml library. I would also have to do xml generation, which can be tricky.

So I decided to try out json as the format. It is best described as a data format using only primitive JavaScript data types. Luckilly these are almost exactly like primitive Python data types.

The trick here is that I can just generate primitive Python data structures, and the use one of the many json python libraries to convert to json that can be stored on disc. My simple data structure looks like this:


document = {
    u'portal_type':u'Document',
    u'Title':u'My Title',
    u'Description':u'An example content object',
    u'getRawText':u'The text in the content object',
}

And this is how easy it is to save this structure:


import simplejson
f = open('mydocument.json', 'wb')
simplejson.dump(document, f)
f.close()

Opening and getting it back to a primitive Python data structure is just as simple.

More in next article.

Running ArgoUML with Java Webstart on Ubuntu

2008-02-12T05:19:00.000-08:00

I had no idea what Java Web Start was, but it turns out to be ok. Well I have not used it long enough to see the pitfalls, but initial impressions are good.

I have relatively recently changed from Windows to Ubuntu on my development maschine. It was not that big a deal as I have been using Linux and BSD on servers for ages. But I am generally satisfied.

In that process I am trying to keep up with the Joneses in Plone development. Buildout is great, and so is Paster and ZopeSkel. So I tried to make a new content type using the new local command for Paster. And it went sort of bad. I have not seen it documented anywhere, but it only works for Plone 3 templates.

I really needed to start up a new content type on a Plone 2.5 site. So it was back to ArgoUML and AGX. Just like the good old days ...

On the argouml website there was a link to something called Java Webstart. I clicked on it, and ArgoUML started up. Just like that. No install options no nothing. So I closed it down.

Now I wanted to start it again without clicking. It was not in my ubuntu menu. I had not used synaptic so that was unsurprising. It downloaded a pretty big file, so I knew I had a local copy somewhere. But I could not find it. I tried a few commands that I googled. To no avail. In the end it turned out to be very simple:
javaws http://argouml-downloads.tigris.org/jws/argouml-latest-stable.jnlpRunning software directly from an url. Sweet.

Next step was trying to get it to work with the agx_argouml_profile, so that I could just click on the tagged values etc. I could not figure out how to do that. But I really did not dig that deep. I tried passing it the option as I would have done with plain java. That did not work:


javaws -Dargo.defaultModel==agx_argouml_profile http://argouml-downloads.tigris.org/jws/argouml-latest-stable.jnlp

If any of you have tried it with succes, please let me know.

Python eggs - a Simple Introduction

2008-02-06T04:09:00.000-08:00

Python eggs used to be the wave of the future. But for Zope and Plone developers this has evolved into a true tsunami. They are everywhere now.

Yet there is a lot of confusion of what they are and how to use them.

To understand them, you need to understand Pythons way of organizing code files.

Module

The basic unit of code reusability in Python: a block of code imported by some other code. It is most often a module written in Python and contained in a single .py file. Also called a script.


hello.py

Let us say that this hello.py contains a function:


def helloworld():
    print 'Hello World'

Then it is possible to import that function like this:


from hello import helloworld

Package

A module that contains other modules; typically contained in a directory in the filesystem and distinguished from other directories by the presence of a file __init__.py.

A step up from a script is a module, which is a library with an __init__.py file in it.


hello/
    __init__.py

You can then put the helloworld function into the __init__.py script, and import it like you did before:


from hello import helloworld

You could also keep it in the hello.py file from before.


hello/
    __init__.py
    hello.py

But then you must import it like this:


from hello.hello import helloworld

Unless you import it into the module namespace. You do this in the __init__.py script:


from hello import helloworld

Then you will once more be able to write:


from hello import helloworld

This ensures the you can reorganize your code and still remain backwards compatibility.

You can have modules inside modules. A python library is just a module, or a structure of modules.

A structure of modules is called a package.

Distutils

So far it has all been about writing and organizing Python code. But the next step is distribution af said code. First step in this direction is distutils.

Distutils was written to have a single unified way to install Python modules and packages. Basically you just cd to the directory of the module and write:


python setup.py install

Then the module will automagically install itself in the python it was enwoked with.

Distutils defines a directory/file structure outside your module, that has nothing to do with the module per se, but is used distribute the module.

If you want to make a distribution of the hello module you must put it inside a directory that also contains a setup.py file.


somedir/
    setup.py
    hello/
        __init__.py
        hello.py

The setup.py could contains this code, that runs the setup function:


from distutils.core import setup

setup(name='hello',
    version='1.0',
    packages=['hello',],
)

You then run the code like this:

python setup.py sdist

And it will create a new directory structure like this:


somedir/
    setup.py
    hello/
        __init__.py
        hello.py
    dist/
        hello-1.0.tar.gz

The hello-1.0.tar.gz then contains the package distribution. It has this structure when unpacked:


hello-1.0/
    PKG-INFO
    setup.py
    hello/
        __init__.py
        hello.py

The hello package is inside it. It is just a copy of your own package with no changes.

setup.py is there too. It is also just a copy of the one you wrote to create the package with. The clever thing about distutils is that it can use the same script to create the distribution as it use to install the package.

PKG-INFO is a new file and it just contains some metadata for the package. Those can be set in the setup.py.

Setuptools

Setuptools is built on top of distutils. It makes it possible to save modules in pypi, or somewhere else. It uses eggs for distribution.

eggs

An egg is created very much like a distutil package. You just have to change a line in your setup.py


from setuptools import setup # this is new

setup(name='hello',
    version='1.0',
    packages=['hello',],
)

Then you call it with:


python setup.py bdist_egg

And you get a new file in your dist directory:


dist/
    hello-1.0-py2.4.egg

This is the egg that you can put on your website, or even better, publish to pypi. you can get an account on pypi, and then you will be able to add your eggs via the command line like:


setup.py bdist_egg upload

Easy Install

When you have uploaded your egg, all the world is able to use it by installing it with easy_install:


easy_install hello

Easy install will then find the egg on pypi, download it, compile if necessary and add it to your sys.path so that Python will find it.

Buildout

Buildout is a configuration based system for making complicated but repeatable setups for large systems.

Phew. That sounds complicated. Well buildout can be. But what is interresting from an eggs based point of view is that you configure what eggs are to be installed in your system.

Inside your buildout.cfg you can have a line like:


eggs =
    hello

Then buildout will automatically download and install the hello package in your system.

Buildouts can themself be distributed as eggs, and you can extend a buildout to add new packages. This is how you can install a Plone buildout and then add your own packages to it. Basically creating your own custom Plone distributions.

Resources

Python modules and packages

http://www.python.org/doc/2.4.4/tut/node8.html

Distutils

http://docs.python.org/lib/module-distutils.html

Setuptools

http://peak.telecommunity.com/DevCenter/setuptools

Eggs

http://peak.telecommunity.com/DevCenter/PythonEggs

Buildout

http://pypi.python.org/pypi/zc.buildout

Oh glory days... buildout has changed the game

2008-02-04T02:50:00.000-08:00

I have worked on many projects with developers scattered all around the globe. In that process I have spend a lot of time making setup scripts for Plone sites. Basically a product that sets up the site without content.

These days it is called a policy product.

They made it possible to have a local development version that was in sync with the global development server. At least it was possible to install a subset of the solution on your local dev machine that would make it possible to do your own part of the solution.

The problem arised as soon as the site went live. The setup script would be run one final time, and then never touched again, and it would never be updated any more.

The dependencies would be too large on a big live site that anybody would actually like to install a complete copy of the production site back on the dev machine.

Finding and downloading every little version of dependencies was a big enough problem to try and avoid it.

For that reason I have always had a love / hate relationship with those policy products. A lot of necessary work that would never be used again as soon as the site went live.

Then a year or two later when you need to do a modification to the site it would be a real mess to get a dev site up and running.

Buildout has changed the game dramatically. The automatic installation and fetching of dependencies actually makes it feasible to run a perfect copy of the production server.

This is big!

We are going to save so many hours not doing manual installs of dev sites.

At some time we are going to see problems when products and libraries dissapearing from the net. But I am pretty sure that most companies will have local proxies for downloads soon.

Installing ZMySQLDA in Zope using buildout on Ubuntu and perhaps Debian in general

2008-01-31T05:06:00.000-08:00

Another note to future self. This is how I did it.

libmysql++-dev

To make Zope work with Mysql I needed to compile the database adapter. For this I needed the mysql dev libraries. On Ubuntu that is done like this:

aptitude install libmysql++-dev

MySQL-Python

For ZMySQLDA to work I needed the plain Python MySQL-Python library installed and compiled.

Luckily the MySQL-Python package is distributed as an egg, so it can just be added to buildout.cfg

eggs =
... other packages
MySQL-Python

Buildout will then download, compile and install the package automatically.

ZMySQLDA

ZMySQLDA is old. Latest release 2.0.8 is from 2001. But that is just because it is stable. No worry.

When I tried to install it with by adding it in the buildout.cfg:

urls = http://www.zope.org/Members/adustman/Products/ZMySQLDA/2.0.8/ZMySQLDA-2.0.8.tar.gz

It made a small tarbomb and put itself inside a directory structure like:

lib/python/Products/ZMySQLDA

And Zope was not be able to find it.

Buildout to the rescue again. Radim Novotny has made a buildout recipe for doing it right:

http://pypi.python.org/pypi/cns.recipe.zmysqlda/

So all I needed was a few additions to the buildout.cfg

[buildout]
parts =
...
zmysqlda

and

[zmysqlda]
recipe = cns.recipe.zmysqlda
target = ${buildout:directory}/products

And then I was good to go.

Installing Python, Zope and Plone on Ubuntu 7.10 - Gutsy Gibbon

2008-01-30T07:03:00.000-08:00

Last time I installed plone on a Ubuntu system, I wrote a log of what I did. It might be that I am getting old, but I cannot seem to remember how I do it from time to time. So this is as much a note to future me as a help to others.

It is a newer version of this article on my main site.

http://www.mxm.dk/papers/installing-plone

With a little luck you should be able to run it as a script on your own server, or at least copy/paste the lines into your terminal, and the see where it fails.

####################################
# System installs, libraries etc.

# Installing python and plone on ubuntu server 7.10 Gutsy Gibbon
# much of it can probably be used on any debian system.

sudo bash
# as root: install system packages and libraries for compiling python
aptitude install gcc g++ make

# readline is needed for an interactive python prompt
aptitude install libreadline5 libreadline5-dev

# zlib is needed for zope & PIL
aptitude install zlib1g zlib1g-dev

# libjpeg is neded for PIL
aptitude install libjpeg62 libjpeg62-dev

# libssl is needed by buildout for downloading with the https protocol
aptitude install libssl0.9.8 libssl-dev

# create a user called zope.
adduser zope

####################################
# zope, plone specific install

# Do the rest as this user
su zope
# from the users home dir /home/zope
cd ~
# create the directory structure
mkdir downloads
mkdir pythons
mkdir instances

# download and install python in the zope users home dirs pythons directory
cd ~/downloads
wget http://www.python.org/ftp/python/2.4.4/Python-2.4.4.tgz
tar xzf Python-2.4.4.tgz
cd Python-2.4.4
./configure --prefix=/home/zope/pythons/python-2.4.4
make
make altinstall

# make a symbolic link for this python. This is optional. But you avoid having
# to type "python2.4" instead of "python"
ln -s /home/zope/pythons/python-2.4.4/bin/python2.4 /home/zope/pythons/python-2.4.4/bin/python
# make this the default python to use by pushing it to the front of the path with this command
PATH=/home/zope/pythons/python-2.4.4/bin:$PATH
# you can add the command to the zope users ~/.bashrc
# to automatically use that python at login. I normally do this.

# download and install PIL
cd ~/downloads
wget http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz
tar xfz Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
python setup.py build
python setup.py install

# download and install easy install
cd ~/downloads
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

# create a default buildout config file. This way your eggs and downloads will
# automatically end up in the same dir.
mkdir ~/.buildout
vi ~/.buildout/default.cfg

# insert this text (without the "## ") and save
## [buildout]
## executable = /home/zope/pythons/python-2.4.4/bin/python2.4
## eggs-directory = /home/zope/.buildout/eggs
## download-directory = /home/zope/.buildout/downloads

# then install ZopeSkel with easy install
easy_install -U ZopeSkel

# you can see what buildouts can be created with this command
paster create --list-templates

# as I write this, the list looks like this.
#Available templates:
# archetype: A Plone project that uses Archetypes
# basic_namespace: A project with a namespace package
# basic_package: A basic setuptools-enabled package
# basic_zope: A Zope project
# nested_namespace: A project with two nested namespaces.
# paste_deploy: A web application deployed through paste.deploy
# plone: A Plone project
# plone2.5_buildout: A buildout for Plone 2.5 projects
# plone2.5_theme: A Theme for Plone 2.5
# plone2_theme: A Theme Product for Plone 2.1 & Plone 2.5
# plone3_buildout: A buildout for Plone 3 projects
# plone3_portlet: A Plone 3 portlet
# plone3_theme: A Theme for Plone 3.0
# plone_app: A Plone App project
# plone_hosting: Plone hosting: buildout with ZEO and any Plone version
# recipe: A recipe project for zc.buildout

# so to create a plone 3 server you write

cd ~/instances
paster create -t plone3_buildout p3test

# It asks a few questions. If you are in doubt just press enter for default values.
# now run the bootstrap script

cd p3test
python bootstrap.py

# at this point you should check out builout documentation for your options,
# if you want to run anything but a plane plone site.
# you can edit you buildout.cfg file in p3test. There is a good intro at:
# http://plone.org/documentation/tutorial/buildout/understanding-buildout.cfg

# now let buildout install and compile Plone. Zope will be compiled automatically.

./bin/buildout -v

# then start up and test the site with
./bin/instance fg

Good luck!

Small lesson about extending buildout configs

2008-01-30T03:58:00.000-08:00

It turns out that it is very easy to have a develop.cfg that only changes minor stuff in the buildout.cfg.

I had a production server running on 8082, and I prefer my development instances to run on 8080. So I tested out how little extra configuration I could get away with. I was pleasantly surprised.

development.cfg

[buildout]
extends =
buildout.cfg
parts =
instance

[instance]
http-address = 8080
debug-mode = on
verbose-security = on

So the production server is build with:

./bin/buildout

And the development server with:

./bin/buildout -c development.cfg

This is just a simpler version of Martin Aspelis tutorial found here...

But it is so nice when software actually makes your life easier, and buildout is a tremendous time savior in group development situations.

Death to deprecations!

2008-01-29T11:58:00.000-08:00

Sometimes I love how easy it is to get start a project in Plone. At other times it makes me really furious.

When I started programming (omg like 25 years ago) I was really fascinated about the idea of only having to do a boring job once, and then build on top of it.

For the rest of my life the machine would do my boring job for me. Yeah right! In this respect Plone has failed miserably!

About a year ago I had a relatively complex task of writing a code for importing adresses from xml files. Not that the task was that difficult in itself. But the details made it take about a week.

It ran on a very old Plone version on Python 2.3.

I knew that I would later have to reuse the code in an upcomming Plone version, so I wrote it very conservatively. I made a simple Zope 2 product as a tool, and saved tha data on that tool in OOBTrees, OOSets etc.

The last week I had to rewrite it for Plone 3. I basically had to change the transaction semantics from get_transaction().commit() to transaction.commit(). It was the easiest rewrite I have done in years. Oh yeah and the 20K adresse ran so fast that it shook me to my AT bones.

This for me is the core of the problem with Plone. It is layer upon layer of backwards incompatible stuff. For every new Plone release my products have been broken. Everytime! I have spent far more time rewriting my products to new Plone versions than I have ever spent writing them in the first place.

And I have made a lot of complicated products like webmail, workgroups, calendaring, contact databases with ics support, page aggregators etc.

Currently they are not being maintained anymore unless a customer specifically needs them and pay for the upgrade. It is simply to much of a bother to do it for free, as it will easily take several days per project.

The problem with every new layer that has been added to Plone, is that it doesn't really change the programming efficiency very much, but stops old code from working. And what worse is, is that you still need to understand every goddarn layer to code efficiently in Plone.

Plones style of 'sort of' backwards compatibility is a real serious problem for the project. A new layer of code should generally never break something that allready works, and new layers should be independent of the old layers.

I think that the best approach for Plone and Zope 3 would be to just develop them in parallel, so they can run on the same instance, on the same zodb, and then make shure that they do not break each other.