Something in advance. It takes GREAT courage and openness to tell in your blog this story. I think is really a great attitude about it.

Saying this, I must say that the whole story a recipe for disaster. Lots of steps make my spider-sense to tingle. Strongly.

Doing a script on the python interactive shell to update your production data, while in a convention, between presentations… Well, it’s not the right moment to do ANYTHING that could change your data. A quick look a monitoring tool, that’s grand. But anything more complex that that is highly risky. And specially using the interactive shell.

You have to RESPECT your production environment and data. Ideally, every change in production should be automated and tested before in advance. That means everything but the most extreme cases, like bugs that are blocking the whole application. Sometimes, in extraordinary cases, could necessary to take extraordinary measures. But it should always be treated with the proper caution.

You have to set all your attention each time you have to change anything on production and have a clear view in advance of what are you trying to do. Think really carefully what are you going to do. And double check everything you type. Every step that has not been previously tested on a staging environment is a possible disaster for your application.

Anyway, stories like that only make me remember how much attention should I put into changing production data and keep a healthy fear of what could happen. Treat your production environment with proper R-E-S-P-E-C-T or it can bite. Hard.

Bonus: Really, really, REALLY the best way of testing that something works is SAVING everything AGAIN????????

This code give me nightmares…

from mongoengine import connect
from models import Family
connect('the-production-database')
for family in Family.objects:
   family.save()

So, here are the slides

UPDATE: In case anyone is interested, here is the talk, courtesy of PyCon Ireland

There are currently two available positions in the development team, a lead Flash developer and a senior Python developer. The links are the official job descriptions, but I would like to tell a bit more about the tools we use everyday and the company, and about my experience working here.

First of all, basic info about the company. Jolt Online Gaming is a games company based in Dublin, Ireland. Specifically, online free-to-play games, with social elements on them. Our latest releases are Playboy Party (in Facebook) and Utopia Kingdoms (available in Facebook and Kongregate). We are just about to release a new football management game, Championship Manager on Facebook.

On the technical side of things, we try to find the right tool for each problem, so a lot of innovation is not only expected, but encouraged. We build our systems thinking of millions of players, which is always challenging. Scalability and performance are always important when making choices. Learning new things and being up-to-date is important.

Some of the cool technologies and tools that we currently are using are:

• Languages: Recently, the focus is mainly Python for backend and Flash (ActionScript) for frontend, but there are also develops in Ruby, JavaScript and other languages.
• Frameworks: Several Python ones, including Pylons, CherryPy, Django and web.py. In the past we have used also Ruby on Rails.
• Databases: NoSQL databases like MongoDB and Cassandra are being more and more important, but we also make good use of MySQL when fits.
• Amazon services: Our infrastructure is based on Linux servers living in Amazon cloud services, making extensive use of Ec2, S3 and others.
• DVCS: We use git in all our projects.

This is a translation of a post by Ricardo Galli about some of the lessons he has learned on Menéame, a social news website in Spanish similar to Digg. I wanted to share some of the concepts with my co-workers, but I thought that it could be interesting to translate the complete work and share it with the whole world Any English errors are my own. I will also like to thank David Brodigan for help me reviewing the English version.

Bored of having to wait more than 5 seconds to display a blog’s page? Annoyed with those sites with dozens of widgets, gadgets, AJAX effects and mashups that take hours to load? Troubled because you have developed a very efficient program for the last hot framework and “it’s slow”? Me too, and that worries me a lot, These sites are incredibly crap pieces of work that don’t take into account the basics about usability and human interface: Response time perceived by the user.

You can criticize everything else about Menéame, except its speed or that we have not taken into account this very important aspect, that’s why I’m sharing the few rules we have been following very strictly. We already knew some, but we have also learnt many more during these past five years of development .

In July 2001 I wrote an article at Bulma [in Spanish] where I explained, according to measurements made during the development of the first sites of Diari de Balears and Última Hora (during the years 1997-1998), the fundamental technical parameters to measure and take into account: response time, return time, download time and “display time”. That last parameter, display, is the one that has the most impact for the user. The user expects a quick response, and that’s mostly perceived as the time that takes for the page to start to display on the browser.

In that article I commented, giving examples, of how to achieve a shorter display time that the total time needed to render and download the whole HTML using optimized generation. So, a rendered page without taking those parameters in mind, takes about six seconds to display.

Those benchmarks were made during the reign of HTML tables, when CSS and JavaScript where not even used yet. Now the situation has gotten worse, even with the spectacular increase in bandwidth (the best available in 2001 was RDSI at 64 kbps and a few 128 or 256 kbps connections)*, better browsers and increased processor power. Today, there are a lot of popular sites that take more than six seconds to display.

* [of course, he is talking about the situation in Spain at the time, in other countries maybe the connection was better at the time]

CSS and the majority of JavaScript files are included at the top of HTML and they are blocking, meaning they make the browser to stop the rendering process until the downloading is completed. Including a lot of CSS and JavaScript files on a website should be carefully planned. Most of the time, it’s just annoying the users. A good rule of thumb is to keep the number of those files down to a minimum, don’t include unused libraries and reduce the number of images inside CSS, as they generate additional connections.

It’s very common for blogs and websites to include every kind of external JavaScript (widgets) without taking into account the penalty on display time. Those scripts usually block the render, especially if they weren’t carefully developed to avoid it. A good script usually is just a few lines of code that defines a fixed-sized iframe and then afterwards loads the rest of the code (using the proper requests). An example of good code in this respect are the Google AdSense scripts.

So, if external JavaScript code has to be included, a closer look at how it works should be taken, and those that block the rendering while making their queries to external servers should be discarded. If there is no choice but to use those kind of slow, blocking scripts, some strategies can be used to circumvent the problem. It’s what Menéame does on the two advertising blocks it has.

That advertising is generated from JavaScript libraries from Social Media S.L. server. That server first verifies if there are any specific advertising for the site from their own and third party databases. If not, it redirects to load AdSense. This process is slow and the server API is not optimized. The solution that Menéame uses can be adapted for use on any site with similar problems.

The JavaScript is not loaded directly. Instead, a fixed-size iframe is defined which loads an independent HTML. That code is the one loading the advertising scripts. The load of the iframe content does not block the rendering of the page and has a lower priority. That is the reason that, when arriving at Menéame, advertising is the last to be displayed, after rendering of the complete page.

So, if dealing with those slow widgets is required, the users will appreciate if they can be loading, avoiding those annoying waiting times. But even more important, when developing widgets or plugins the same techniques can be applied. Here you have an example of how we do it for the Menéame “votes counter” check_url.js or this similar but more elaborate example.

As mentioned previously, the number of script files included at the top of the page should be reduced to a minimum. When additional files are loaded, the best approach is to include them at the end of HTML, just before the </body> tag.

That sometimes is not easy, as the code can be needed to render or display content on the same page. That can be solved, like in the Menéame case of news with geolocation, when the Google Maps map is displayed on the side bar. The main idea is to delay as much as possible the load of the additional JavaScript.

The code can be seen at the end of the html, just before the </body> tag (the first script includes Google API, the second Menéame’s own functions):

<script src="<a href="><!--mce:0--></script>

The moment the browser interprets the CSS of those three divs, it can calculate the needed sizes to quickly render: the width of the visible part, the height of the top header, the height of the left menu and the width of the right column (as well as the width of the central main column) The part that takes most of the CPU time in Menéame page is the link or comments requests. To reduce the display time even more, the first block sent to the browser is the content of the right column (the #sidebar). As the width is already specified, it can be rendered even when the browser hasn’t received the HTML of the main column. That’s why on slow connections (or slow computers), the first element to be displayed is the right column.

Browsers (at least most of them) parallelise downloads from different sites, so it is better to download static files from a different domain.

...

...
<script src="<a href="><!--mce:1--></script>
...

...
<img src="<a href=" alt="" />http://aws.mnmstatic.net/img/common/search-left-04.png" alt="" width="6" height="22" />
...
<img src="<a href=" alt="" />http://aws.mnmstatic.net/cache/00/01/1-1280021590-20.jpg" alt="gallir" width="20" height="20" />

This point could have been included as part of the previous one, but it has been stated as a different one,  as it is a nice trick for complex sites (with use of cookies) and a lot of static files or images.  As seen in the previous point, a subdomain meneame.net (i.e. static.meneame.net) is not used to serve static content. Instead, a completely different one is used instead. The objective is to avoid the generation of aditional traffic due the use of cookies. Each defined cookie implies sending the cookie each connection for that domain. This happens even when downloading small images of just a few bytes. Also, upload bandwidth is usually smaller than download bandwith. This situation is even worse with cookies defined by statistical tools like Google Analytics, which tend to be big. The way of avoiding all that useless traffic is to use a completely different domain. According to some benchmarks after implementing this solution on Menéame, about 14 KB was saved on each request for a browser with an empty cache.

To avoid the browser connecting each time to verify that a static file was modified, define a timeout. That way the browser won’t come back to check the file until that time has expired. In Menéame the timeout is set to 10 days, in nginx:

location ~* \.(gif|jpg|png|js|css|html|ico)$ {
  expires   10d;
  ...
}

Cache-Control: max-age=864000
Expires: Mon, 06 Sep 2010 16:12:01 GMT

gzip  on;
gzip_comp_level 4;
gzip_proxied any;
gzip_min_length 512;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
gzip_http_version 1.0;
gzip_vary on;
gzip_types text/css text/xml application/x-javascript application/atom+xml text/plain application/json application/xml+rss text/javascript;

There are more rules and tricks to minimize times, like set all the CSS images in just one file and treat them as sprites [*], but using just these seven rules, the speed improvement in any site will be noticeable.

[*] I don’t particularly like this option for sites where the design is dynamic like Menéame. Each time an icon is changed or one icon is added a new image should be generated, which has to be coordinate-compatible with the previous one, or it has to be redefined on the CSS. It’s a lot of “admin” work and quite error prone. But I think it’s a good strategy for plugins of widgets that include icons and are going to be used in lots of places (so stability is greater). I  hold the same opinion about minimizing the JavaScript code.

This section is geekier and is the one that motivates me to write the whole article. But for different reasons it goes last and is smaller that I would like. The trend in web programming is to use more and more complex and sophisticated templates. Some typical examples are to generate all the queries to the DB and then generate the HTML results inside a FOR loop inside a template. More sophisticated examples are to include a template inside other templates or use inheritance (blocks). Something similar happens in simpler cases, like using the PHP output buffer. Informatics Mythology says that the better way to use TCP/IP is to lessen the number of sent packages. Probably it is better for simple sites which can be generated very fast. But the truth is that current sites generate pages of several tenths of a KB (the main HTML in Menéame has an average of 50-60KB and some news with lots of comments can easily reach 200KB) Some weeks ago we released the code of Haanga, the Menéame template system. Even though it allows inheritance we don’t use that feature. Nor the PHP output buffer. We also don’t use big FOR loops, but the content is progressively generated for each news,  comment or note. This progressive generation of HTML is used for all the parts of the page, header, footer, etc. That way, the response and return time is lower.

QQQQQQQQQQHHHHHNNAAANNNNNNNNNNNNNNNNNN

Of course, that’s only an approximation, if more complex templates are used, the HTML render could take more time than the logic and queries, or the opposite if the HTML is simple but queries are complex. In the example, the total time is 38 ticks, and the user can start to see something after 20 ticks. If the web is very slow, most programmers will decide that the best approach is to upgrade the database server or web server to a faster one, lowering the CPU load, and reducing query and HTML generation time. But that is not easy, specially because the access to the DB server involves (at least on big websites) network connections. Even if the processor speed is increased, the latency of the network is the same.

Even so, let’s assume that is possible to change the servers and network for faster ones and reduce to the half the time[*] for querying and rendering the HTML

QQQQQHHHHHNNAAANNNNNNNNNNNNNNNNNN

After that investment to achieve such an inprovement, plus all the extra work in the migration to the new infrastructure, the time will be reduced to 31 ticks (improvement of 18%) and the display time to 13 ticks(improvement of 35%) [*] Usually web programming is single threaded, so getting more CPUs or cores doesn’t speed up the process. The core clock can also be increased,  but doubling the MHz is not going to double the speed (not at all) and the monetary cost can be very high. But the programmer is forgetting something important that he learned over his studies and that is fundamental in IT, operating systems and multiprogramming in general: intercalation.

NNQQQQQQQQQQHHHHH
  AAANNNNNNNNNNNNNNNNNN

As it would be a lot of work to redo programs and templates, I made some benchmark enabling and disabling buffering and compression. The next table shows the total average time (in seconds, taking into account the time to create a new TCP/IP connection) that takes to load portrait HTML (about 14KB compressed, 57KB uncompressed):

Conclusions

Output buffer shouldn’t be used unless you’re not absolutely sure about what you’re doing and/or have measured the impact. In general it will be better without and will consume less server resources (basically RAM memory). If an application with templates is complex and the time of processing and querying is relatively big, generating a big, final template should be avoided. That will save RAM and CPU time.

If serializing the generation of each object can’t be achieved, at least an independent template with the HTML header can be generated and sent as fast as possible, so the browser can download in parallel the CSS and JavaScript files.

A lot of access comes from cell phones and iPads. The users should be happy to access with their gadgets when they browse through your site, that’s why they spend a lot of money on them! There are some easy tricks, like not showing secondary content, not loading heavy API and use the @media selector on CSS to adjust sizes, margins and not display complex sections.

Menéame has had its own optimized mobile site for a long time. Very few users knew this, so an automatic redirection was implemented to the mobile version if it detected a mobile browser (we had over 10,000 daily visits). A lot of people complained about it, so the access was restricted to direct access from external sites, or if a short link like http://m.menea.me/m5za is used (for example, it’s used on automated tweets)

To make the browsing on the main site more lightweight for those users, some differences with the non-mobile version have been implemented

/* Definitions for mobile, iPad and TVs */
@media print,tv,handheld,all and (max-device-width: 780px) {
body {
font-size: medium;
}

.sneaker {
font-size: small;
}

#wrap {
min-width: 320px;
}

#sidebar {
display:none;
}

#singlewrap, #newswrap {
margin: 5px 5px 0px 5px;
}

#footwrap {
margin: 5em 5px 0 5px;
clear: both;
}

.banner-top {
width: 470px;
}
}

The presentation can be found at Prezi, and there is even a video, if someone wants to make funny comments on my exotic accent A problem with the projector doesn’t allow me to display the slides, so I felt a little weird taking the laptop and pointing at the screen, but the people making the video has make their homework and shows the proper slides on place. Nice!

Presentation

Video

The original idea was to show the same simple application (a simple posting service) make with the three frameworks, but not being able to display on the projector really ruined it. Anyway, the code can be downloaded here, if you want to take a look.

Let me know what do you think!

Algo que se destaca constantemente en webs relacionadas con el empleo y la carrera profesional es la necesidad de tener un plan. Y un plan profesional es algo que casi siempre se pone en las oferta de trabajo (posibilidades de carrera), pero que, lamentablemente, tenemos que crear nosotros mismos. Es lo que Andrés Pérez lleva mucho tiempo sintetizando en “o eliges tú, o alguien va a elegir por tí”

No es nada sencillo, porque lo primero que viene a la cabeza son  los típicos planes detallados al milimetro, como si la vida de uno fuese “Ocean’s 11″ y todo funciona como un reloj. Pero eso es imposible, porque la vida no es una película de Hollywood y hay que improvisar constantemente. Y proponerte objetivos imposibles sólo va a frustrarte.

Lo más importante es tener claros tus objetivos estratégicos. No necesariamente tienen que ser objetivos muy concretos, sino sensaciones generales sobre los que un conocido llamaba “qué quiero ser de mayor”. Añadir algún tema concreto está bien, de todos modos, pero esas preguntas de “¿dónde te ves dentro de 5 años?“, al menos a mí, me resultan terriblemente complicadas de contestar. Cualquier cosa que hubiese podido decir hace 5 años no hubiese tenido nada que ver con lo que hago ahora, posiblemente…

Para poder hacerlo, aunque parezca una perogrullada, necesitamos conocernos bien a nosotros mismos. Pero no lo es. Cuesta mucho realmente darse cuenta de lo que es importante en nuestra vida y no seguir lo que nos imponen los demás, bien sean nuestros padres, nuestros amigos, la sociedad en general, etc… Tenemos que tener claro qué deseos son nuestros y cuales son “lo que se supone que tiene que ser”. Eso supone pensar y meditar sobre uno mismo.

Y, por supuesto, cada cierto tiempo hay que revisarlas y, ¿por qué no?, cambiarlas.

Por ejemplo, algunas de mis ideas actuales:

• En primer lugar, creo que el trabajo de cada uno debe ser algo que llene, que vaya más allá que el mero hecho de cobrar una nómina. Más que lo creo, es que QUIERO que mi trabajo sea algo gratificante y que me importe. Bastantes horas al día se tira uno trabajando como para que luego no merezca la pena más que en la cuenta corriente.

A pesar de que muchas veces se plantea la dicotomía trabajar en lo que te gusta-ganar dinero, a partir de mi experiencia personal, creo que trabajar en lo que a uno le gusta funciona mucho mejor como medio de ganar dinero mas que “sacrificarse estoicamente”.

• Quiero también que mi carrera profesional sea interesante y variada. Me gusta mucho aprender cosas nuevas, aunque eso inevitablemente sea algo complicado y difícil. Require esfuerzo, y todos tendemos a ser vaguetes y a apalancarnos.

A pesar de ser desarrollador de software, que es algo que implica reinventare casi casi cada día, me gusta también probar otras cosas, y trabajar en equipos interdisciplinares. También meto aquí lo interesante que creo que está siendo para mí la experiencia de estar trabajando en Irlanda y en una industria como la de los videojuegos, en las que cada semana parece un año de lo rápido que va.

• Creo que uno debe siempre aspirar a más. Ser ambicioso. Eso no quiere decir ambición desmedida, pero sí tener un cierto sentido de progreso, entendido de manera personal, y, aunque sea secundaria, por qué no en lo económico. También incluye el reconocimiento.

• Valoro mi independencia y no soporto el micromanagement. Quiero ser capaz de proponer objetivos y no hacer las cosas porque las diga el jefe, sino porque es necesario/mejora el proyecto. Quiero estar en un entorno cooperativo y no ordeno-mando. Es muy frustante tener que luchar contra imposiciones.

• Cada vez creo más en ser transparente, en comunicar lo que haces y menos en guardarse secretos. Eso incluye hablar de lo que hago de manera pública (como en este blog), pero también aceptar errores, admitir inseguidades y ser sincero. Y mira que cuesta horrores.

Con estas coas, ya puedes ir sacando diferentes “derivadas”, Por ejemplo, muchas cosas que son importantes para mí (y, destaco, lo mismo no lo son para nadie más) son mucho mas factibles en pequeñas empresas o startups que en grandes empresas (y esa es una buena manera de clasificar qué buscar), o, incluso, llegado el caso, con autoempleo.

Eso implica que, probablemente, tenga que vivir con más riesgo implícito que estando 10 años seguidos en la misma compañía. Bueno, por el momento lo asumo (aunque cada día piense más que trabajar en una gran empresa es muy arriesgado, en cualquier momento prescinden de tí en las “altas esferas” y no hay manera de librarse). Pero creo que es importante entender que no hay nada gratis, para todo se paga un precio y es cuestión de prioridades.  Es parte del plan, decidir si es más importante esto o lo otro e ir a por ello.

Así que, de tanto en tanto, me planteo, ¿lo que estoy haciendo sirve para mis objetivos estratégicos? ¿Hay otro camino para cumplirlos mejor? Y, si lo hay, pues uno se puede poner manos a la obra con un objetivo en la cabeza…

So, here are the slides, so feel free to make comments, ask questions and even criticize them!

You can also download the presentation on PDF here.

PD: When I talk about football game, I’m referring to soccer.

That doesn’t mean that you’ll have to comment everything. Or that adding a comment it’s an excuse to not be clear directly on the code, or the comment should be repeat what is on the code. You’ll have to keep a balance, and I agree that it’s something difficult and everyone can have their opinion about when to comment and when not.

Also, each language has it’s own “comment flow”, and definitively you’ll make more comments on low level languages like C than in a higher level language like Python, as the language it’s more descriptive and readable. Ohhh, you have to comment so many things in C if you want to be able to understand what a function does it in less that a couple of days… (the declaration of variables, for example) #

As everyone has their own style when it comes to commenting, I’m going to describe some of my personal habits commenting the code to open the discussion and compare with your opinions (and some example Python code):

• I put comments summarizing code blocks. That way, when I have to localize a specific section of the code, I can go faster reading the comments and ignoring the code until getting to the relevant part. I also tend to mark those blocks with newlines.

# Obtain the list of elements from the DB
.... [some lines of code]

# Filter and aggregate the list to obtain the statistics
...  [some lines of code]

UPDATED: Some clarification here, as I think that probably I have choose the wrong example. Of course, if blocks of code gets more than a few lines and/or are used in more than one place, will need a function (and a function should ALWAYS get a docstring/comment/whatever) . But some times, I think that a function is not needed, but a clarification is good to know quickly what that code is about. The original example will remain to show my disgrace, but maybe this other example (I have copy-paste some code I am working right now and change a couple of things)
It’s probably not the most clean code in the world, and that’s why I have to comment it. Latter on, maybe I will refactor it (or not, depending on the time).

               # Some code obtaining elements from a web request ....

                # Delete existing layers and requisites
                update = Update.all().filter(Update.item == update).one()
                UpdateLayer.all().filter(UpdateLayer.update_id == update.item_id).delete()
                ItemRequisite.all().filter(ItemRequisite.item == update).delete()

                # Create the new ones
                for key, value in request.params.items():
                    if key == 'layers':
                        slayer = Layer.all().filter(Layer.layer_number == int(value)).one()
                        new_up_lay = UpdateLayer(update=update, layer=slayer)
                        new_up_lay.save()
                    if key == 'requisites':
                        req = ShopItem.all().filter(ShopItem.internal_name == value).one()
                        new_req = ShopItemRequisite(item=update, requisite=req)
                        new_req.save()

# Store found primes to increase performance through memoization
# Also, store first primes
found_primes = [2,3]

def prime(number):
    ''' Find recursively if the number is a prime. Returns True or False'''

    # Check on memoized results
    if number in found_primes:
        return True

    # By definition, 1 is not prime
    if number == 1:
        return False

    # Any even number is not prime (except 2, checked before)
    if number % 2 == 0:
        return False

    # Divide the number between all their lower prime numbers (excluding 2)
    # Use this function recursively
    lower_primes = (i for i in xrange(3,number,2) if prime(i))
    if any(p for p in lower_primes if number % p == 0) :
        return False

    # The number is not divisible, it's a prime number
    # Store to memoize
    found_primes.append(number)
    return True

(Dealing with prime numbers is something that deserves lots of comments!) EDIT: As stated by Álvaro, 1 is not prime. Code updated.

# TODO: Change the hardcoded IP with a dynamic import from the config file on production.
...
# TODO: The decision about which one to use is based only on getting the shorter one. Maybe a more complex algorithm has to be implemented?
...
# Careful here! We are assuming that the DB is MySQL. If not, this code will probably not work.
...

UPDATE: That is probably also related to the tools I use. S.Lott talks about Sphinx notations, which is even better. I use Eclipse to evelop, which takes automatically any “TODO” on the code and make a list with them. I find myself more and more using “ack-grep” for that, curiously… 

• I try to comment structures as soon as they have more than a couple of elements. For example, in Python I make extensive use of lists/dictionaries to initialize static parameters in table-like format, so use a comment as header to describe the elements.

# Init params in format: param_name, value
init_params = (('origin_ip','123.123.123.123'),
               ('destiny_ip','456.456.456.456'),
               ('timeout',5000),
              )
for param_name, value in init_params:
    store_param(param_name, value)

• Size of the comment is important, it should be short, but clearness goes first. So, I try to avoid shorting words or using acronyms (unless widely used). Multiline comments are welcome, but I try to avoid them as much as possible.
• Finally, when in doubt, comment. If at any point I have the slightest suspicious that I’m going to spend more than 30 seconds understanding a piece of code, I put a comment. I can always remove it later the next time I read that code and see that is clear enough (which I do a lot of times). Being both bad, I prefer one non-necessary comment than lacking one necessary one.
• I think I tend to comment sightly more than other fellow programmers. That’s just a particular, completely unmeasured impression.

What are your ideas about the use of comments?

UPDATE: Wow, I have a reference on S.Lott blog, a REALLY good blog that every developer should follow. That’s an honor, even if he disagrees with me on half the post

On one of my first projects on C, we follow a quality standard that requires us that 30% of the code lines (not blank ones) should be comments.

One of my recent leaking abstractions has come using a ORM, in particular mongoengine, but I think it will happen probably on every ORM. On a web application I am developing at the moment, we need to launch a thread to perform some operations, in a timed manner. A request comes to the server, launches a thread, and then that thread stores its status on the database. Then the user can check the status from the database (and do more operations, like pause, etc, but that I will leave that). While performing some tests on the application, I made the following code:

def testing():
    user = User.objects.get(TEST_USER)
    user.launch_thread()
    time.sleep(TIME)
    assert user.status == END

Inside the thread, the code looks similar to this

def thread(user_id):
    thread_user = User.objects.get(user_id)
    # Do things that take a while, but less than TIME
    thread_user.status = END
    thread_user.save()

Ok, so we’re getting an object from the database, the object launches a thread that changes its state to END and saves it after a while. Well, not really… Obviously it’s not working (or I wouldn’t be writing this). But we all already know that threads are the root of all evil, and always have nasty hidden surprises.

The error I was making was assuming (and that’s the abstraction in my mind) that the ORM maps the database into memory, and that the copy is unique. After all, that’s why you have a database. But it’s not true. What it’s happening here it’s that we are creating two different objects in memory. I have (now) used two different names, user and thread_user. In my code I used the same name (user) which probably adds to the confusion. Each one reflects the status of the database when you read the database, but after that, you are not updating the object with the real information on the DB. So, the user object has still the starting status, the first one, as we haven’t refreshed it with the new and changed information that another, rogue thread, has changed while we naively thought that was under our control.

Usually, on a web application (at least the ones developed with high-level tools) the usual situation is having a request, read the data from the DB using a ORM, change something, and then save. We don’t have rogue threads interrupting that operations and requests can be processed fast enough. And even user data is different so two users probably don’t need to write any related information. BUT definitively another request (faster one) could interrupt the process and make the data to not be coherent. It’s going to be (extremely) rare in most applications, but in case of long, threaded operations, could be important to be aware of this and try not to relay on the ORM as a virtual copy of the DB, but to read and write in short operations. Or lock the database.

Just one more thing. It’s possible to use only one object in memory, and pass it to the thread, and avoiding this problems. But that could generate others, like not storing (and loading) any intermediate steps of the process. So, in case the thread is stopped (for example, a server restart), the process is totally lost. Any operation that takes time to execute will ideally have some “resume” process, so that will include storing the partial state, as well as a resume, which will need to read from the DB. Also, in this particular case, there are more than one thread working the same process, communicating through the DB.

But wait! There is still a little more unexpected and funny behavior!

To reload the user object, my first idea was to generate a refresh method, this way:

class User(mongoengine.Document):
     ...

     def refresh(self):
          ''' Refresh the object '''
          self = User.objects.get(self.id)

And again, it’s not working…
Again, the problem is an abstraction. self it’s not always the object, not outside the method. It’s just a label (or pointer, if you know C) to the object. Yes, we have created a new object called selfwhich has the new (and correct) object. BUT the label user is still pointing to the not-updated object we have since the beginning.

So… no shortcuts, we will have to reload the object after the sleep to check that the object on the DB it’s behaving properly

def testing():
    user = User.objects.get(TEST_USER)
    user.launch_thread()
    time.sleep(TIME)
    user = User.objects.get(TEST_USER)
    assert user.status == END

A pesar de que me gustaba mucho la idea de ese mapa entre Memphis y Nashville, que creo que daba mucho juego, este estilo es mucho más limpito, claro, y creo que “pega” también muy bien con la idea de tener un blog más orientado a temas técnicos. Además, he cambiado el idioma por defecto para que los enlaces, etc aparezcan en inglés…

De todas formas, estoy abierto a sugerencias si alguien quiere proponerme otros temas, o si este le parece horrible y propone volver al tema anterior…