Do not spawn processes on users requests

I’ve been playing recently an online game that has recently launched, that uses the following idea.

When a user starts a match, it spawns a process in the server that acts as the opponent, generating the actions against the user.

The game had a rough launch, with a lot of problems due it being played by a lot of people. And, IMHO, a lot of the problems can be traced to that idea.

I see it’s a seductive one. If a user generates an interaction with the service that takes time (for example, a match for this game), spawn a process/thread in the server that generates the responses in “real time“. The user then will be notified through polling or pushing the information, and can react to it. The process will receive the new information from the user and adjust the responses.

I know is seductive because I had it once, and I was very lucky to have someone around with more experience that show me how it will break under pressure. It’s not a sane architecture to scale.

Some bad ideas:

  • No limit on processes, meaning the servers can be overflown by context switching. Once you have several thousand processes  running on a server, you are in a bad place.
harry_potter_replicating_cups.gif
Replication out of control

  • The very definition of state on the server. You need to keep track of processes started on different servers (so no two servers perform the same job). High Availability is impossible, as losing one server will mean destroying the state on all those processes. For scalability, always look at stateless servers: read all the data, store the resulting data.
  • Start up times. Each time a process starts, there’s some time to boot. This can be a problem if processes are always being started and stopped, adding overhead to the system. Even starting a thread is not free (and will require probably starting internal work like connect to the DB, read from cache, etc)
  • Connections explosion. If each process needs to connect to other parts of the infrastructure (DB, logging, cache, etc) you can have a problem in number of connections.
  • Process monitoring. What if a process gets stuck? A request can be cancelled easily by a web server (if a request takes more than X, kill it), but an individual process or thread can be more complicated and require specific tooling.

Alternative: Pool of workers

Generate a defined number of processes that can perform the individual actions that generates a match. Each process will get an action from a queue, execute it, and store the resulting state. Any process can produce an action for any user.

vlcsnap-2012-09-24-19h37m32s14.png
A group of workers can be very efficient

For example, if a match is a set of 20 actions, each one happening every  minute, the start match request will introduce 20 actions in a queue, to be extracted at the proper time, introducing the proper delay on each action. Note that the queue needs to have a way of deliver delayed messages, not every messaging queue can (in particular, RabbitMQ doesn’t have a good support). Beanstalkd or Amazon SQS supports it.

Or, alternatively, a single action, that will end inserting the next step in the queue with the adequate delay. The action can be as simple as checking if it should change something and, if not, end.

The processes will be extracting the next action from the queue, and executing them. Note that here you minimise the time the worker is waiting for a new task to do. Each worker is active as much as possible, while any user has a pending task, ready to be executed.

The number of processes are limited, so you won’t have an explosion. You can test the system and have a good idea on the limit, when your throughput is not good enough to execute the actions within a reasonable delay, so you can stop the users from starting a new match. This is a better fallback option than allowing everyone to start one and then not giving a good experience.

A priority queue can be put in place, in that case, to inform the user: “You will be able to start your match in ~3 minutes

Or you can add more processes/servers to increase the throughput in a predictable manner.

Alternative: Whole match pregeneration

Another alternative is actually generating a set of actions and returning them in the first go, and display them at the proper times in the client side. If any adjustment is required due the actions of the user, redo all the results from that time on.

Emperor1.jpg
This match is proceeding as I have foreseen it

For example, a match starts, and returns the 20 server actions to the client, which shows them to the user one each minute. In the 3rd minute, the user performs an action, which makes the server to recalculate the  remainder of the match and return another 17 actions. This is a good strategy if generating actions in advance is possible and few interactions from the user are expected.

The bottom line

The main word here is stateless. It is a basic component of an scalable system, and it’s always worth it to keep in mind when designing a system to be used to more than a couple of users.

Compendium of Wondrous Links vol IX

wondrous_links

Welcome back to this totally non-regular compilation of interesting reads. Enjoy!

1427381663-20150326

 

 

Do you want to see the whole series?

Compendium of Wondrous Links vol VII

wondrous_links

Here we go again… This time I’m loosely grouping them, it has been a while and there are so many things!

  • An extremely though-provoking article about the possible positive things can have a negative effect. How Perks Can Divide Us. Corporate culture is something extremely complex.
  • Mora about culture and biases on Mirrorcracy.
  • I do not like the idea that people have a “level”, ignoring all the dynamics (I talked about this here). There are no B players.
  • A set of articles talking about management skills. Is a great compilation of the different aspect of management. Great read even if you’re not “an official manager”, as is capital to understand what are the things that a (good) manager should do and challenges that the role presents
  • Some ideas about productivity. I like this quote a lot: “There is no one secret to becoming more productive, but there are hundreds of tactics you can use to get more done”
  • We tend to idealise the work involved in some stuff we really like. In particular in creating video games. It’s okay not to follow your dreams, but I think it could apply to a lot of other aspects

Dragon's Lair

 

 

Future as a developer and the ever changing picture

A few weeks ago I came by a couple of articles my Marco Arment that share the theme of the current status of accelerated change within the development community as a way of stressing up, and being difficult to be up to date. After all, one gets tired of learning a new framework or language every size months. It gets to a point where is not funny or interesting anymore.

It seems like two different options are presented, that are available for developers after some time:

  • Keep up, meaning that you adopt rapidly each new technology
  • Move to other areas, typically management

Both are totally valid options, as I already said in this blog that I don’t like when good developers move to different areas (to me it’s sort of a surgeon deciding she had enough after a few years and move to manage the hospital). Though, obviously each person has absolutely every right to choose their career path.

But I think that it’s all mostly based in an biased and incorrect view of the field of technology and the real pace of changes.

In the last years, there has been an explosion of technologies, in particular for web. Ruby on Rails almost feels introduced at the same time as COBOL. NodeJS seemed to be in fashion for a while. The same with MongoDB or jQuery.

We all know that being stressed is not a great way of learn
We all know that being stressed is not a great way of learn

In the last 6 or 7 years there has been an incredible explosion in terms of open source fragmentation. Probably because GitHub (and other online repos) and the increase in communication through the Internet, the bar to create a web framework and offer it to the world has been lowered so much, that a lot of projects that would’ve been not exposed previously, has gotten more exposure. As a general effect, is positive, but it came with the negative effect that every year there is a revolution in terms of technologies, which forces everyone to catch up and learn the brand new tool that is the best for the current development, increasing the churning of buzz words.

But all this is nothing but an illusion. We developers tend to laugh at the common “minimum 3+ years of experience in Swift”, but we still get the notion that we should be experts in a particular language, DB or framework since day one. Of course, of the one on demand today, or we are just outdated, dinosaurs that should retire.

Software development is a young field, full of young people. That’s great in a lot of aspects, but we need to appreciate experience, even if it comes from using a different technology. It doesn’t look like it, but there’s still a lot of projects done in “not-so-fancy” technologies. That includes really old stuff like Fortran or COBOL, but also C++, Java, Perl, PHP or Ruby.

Technologies gets established by a combination of features, maturity, community and a little luck.  But once they are established, they’re quite resilient and don’t go away easily.  They are useful for quite a long time. Right now it’s not that difficult to pick a tool that is almost guaranteed to be around in the next 10-15 years. Also, most of the real important stuff is totally technology agnostic, things like write clean code, structure, debug ability, communication, team work, transform abstract ideas into concrete implementations, etc… That simply does not go away.

Think about this. iOS development started in 2008. Smartphones are radically different beasts than the ones available 6 years ago, probably the environment that has changed more. The basics are the same, though. And even if Swift has been introduced this year, it’s based in the same principles. Every year there has been tweaks, changing APIs, new functionalities. But the basic ideas are still the same. Today a new web development using LAMP is totally viable. Video games still relay on C++ and OpenGL. Java is still heavily used. I use all the time ideas mainly developed in the 70s like UNIX command line or Vim.

Just because every day we get tons of news about new startups setting up applications on new paradigms, that doesn’t mean that they don’t coexist with “older” technologies.

Of course, there are new tricks to learn, but it’s a day by day additive effort. Real revolution and change of paradigm is rare, and normally not a good sign. Changing from MySQL to PostgreSQL shouldn’t be considered a major change in career. Searching certain stability in the tools you use should be seen as good move.

We developers love to stress the part of learning everyday something new and constantly challenge ourselves, but that should be taken also in perspective with allowing time to breathe. We’ve created a lot of pressure on ourselves in terms of having to be constantly pushing with new ideas, investigating in side projects and devoting ourselves 100% of the time to software. That’s not only not realistic. It’s not good.

You only have to breathe.  And just worry on doing a good work and enjoy learning.

The amazing forgiveness of software

One of the things I like most about developing software is the fact that you can recover from most mistakes with very few long term impact.

Bugs are unavoidable, and most of the people involved on programming deeply understands that is something we all live with.  So,  there’s no hard feelings, once you find a bug, you fix it and immediately move on. Not only no one thinks that you’re a bad developer because you write bugs, but typically the impact of a bug is not that problematic.

Yes, there are some bugs that are just terrible. And there’s always the risk of losing data or do some catastrophic operation on production. But those are comparatively rare, and with the proper practices, the risk and damage can be reduced. Most on the day to day operation involves mistakes that have a much limited effect. Software development is more about being bold and move fast fixing your mess, than it is to play safe (within limits, of course).

This can affect production data! Show warning sign!
This can affect production data! Display warning sign!

Because the greatness of software is that you can break it on purpose and watch it explode, and then fix that problem. In a controlled environment. Without having to worry about permanent effects or high costs. And a good test is the one that ambushes the code and try to viciously stab it with a poisonous dagger. The one that can hurt. So you know that your system is strong enough against that attack. And then iterate. Quickly. It’s like having a permanent second chance to try again.

Not every aspect of live is that forgiving. I guess that a lot of doctors would love to be able to do the same.

Do you mind if we start the trial again and the jury forgets everything, your honor?
Do you mind if the jury forgets everything and we start the trial again, your honor?

Respect Driven Development

I think that one of the most overlooked components on any sane company culture is Respect. That’s probably true also for any relationship, also outside work environment, but I think is usually forgotten when nice places to work are described.

When I look back about the things that bothered me the most, most of them are related to disrespect, even in relatively minor form. It can be personal disrespect or not respecting the work itself or even the customers. Probably because is something engraved, it’s easy to take for granted when it exists, and to identify more problems deviating from the lack of it when is not present. We typically talk about how great cultures are innovative, open, communicative, fun, collaborative, etc. but one of the prerequisites that makes these values worthwhile is Respect, both to your coworkers and to the work itself.

R-E-S-P-E-C-T  Find out what it means to me
R-E-S-P-E-C-T
Find out what it means to me

Without Respect, ideas are accepted mostly depending on who present them, and need to be imposed. Even when there are explicit request for ideas, they take the shape of “suggestion boxes” where no one really looks into them. So, in practice, being proactive is discouraged unless you’re in a power position.

When there is Respect, ideas can be freely exchanged without fear of not being talking seriously. They are also welcomed from any source, not only through the “chosen channels”. There can be hard scrutiny, but it will be fair, and rejections will be reasonably based in facts.

Without Respect, a “funny, relaxed atmosphere” can be easily transformed into harassment and abuse. Jokes will actually hurt. Closed groups,   extremely aggressive with everyone external with them, will be formed. That can include groups outside the company, like mocking customers or partners. Some groups will be appointed as intrinsically “better” (engineers, executives…) as others (secretaries, workers…) and generate asymmetrical relationships, with one part dominating the other.

When there is Respect, jokes are played just for the laugh, and are taken up to the correct limit for everyone, as there are people with thicker skin than others. If those limits happen to be crossed, the problem will be arisen and people will sincerely apologise and correct their behaviours in the future, without external influence. Occasionally the customers or partners can be make fun of, but the quality of the delivered software will be took extremely seriously (the highest form of Respect for customers) and their requests or suggestions will be taken into account when making new features.

Without Respect within the company and the different groups, no particular measures will be enforced to protect anyone or anything. Therefore, it will be easy for someone to take advantage of that, ranging from lower the quality of the work to be a moron and degrade the working environment. Code will devolve into an unreadable mess, and technical debt will grow uncontrollably. Hiring standards will get lower, and not-that-great people will be part of the team (technically, but also in a more personal sense). Also, the expectations will be to work overtime regularly, without any contingency plans or treating it as a bad sign.

When there is Respect, the organisation truly cares about the people, and not just as an empty statement. This includes understanding when overtime is unavoidable evil and work as a team to avoid it as much as possible. And when it happens, everyone do as much as they can to make it as short and enjoyable as possible. There will be understanding when someone wants to leave because they have a genuine different interest, leaving the door open if things don’t turn out for the good. Learning and personal growth will be encouraged with actions, not only with words.

Trust, a extremely important value, can only arise if there is Respect. Without Respect, fear and uncertainty will replace real trust. Being honest needs trust and confidence in the other part, as real honesty can be, and sometimes should be, uncomfortable to hear. Formality and defensiveness take control over honest feedback and team work when respect is not present. Any long-term relationship also needs Respect to stay healthy.

What have I ever done to make you treat me so disrespectfully?
What have I ever done to make you treat me so disrespectfully?

Being imperfect human beings, we cannot probably achieve perfect respectful relationships  at all times. But we should try to be as respectful  as possible, identifying our mistakes and the ones of the organisation, and move up towards the Respect ladder. That makes a much healthier (and happier) environment for all. We should recognise the Real Respect, as the word is often abused.

It is great to aim for having a great organisation or startup, with a thrilling culture. But, in order to get to establish a funny, exciting, learning, diverse and passionate place to work, we should lay strong foundations with Respect. Identify it, and not tolerate the lack of it.

Compendium of Wondrous Links vol I

wondrous_linksAs a way of collecting interesting reads across the Internet, I plan to keep a relatively regular posts with some articles and posts that I’ve read, mostly related to development, software and tech world in general.

Here it goes the first edition.

Hope you enjoy it!