2015 m. lapkričio 2 d., pirmadienis

Event Sourcing - events de-duplication and order of events


One of the challenges teams has to tackle while implementing event sourcing is events de-duplication and processing order of events while updating read model. Once events are being stored in event store we usually publish those so that all interested parties could update their models, including our own projections. Now if you are using service like AWS SQS/SNS or any other queue that does not provide FIFO and one-time delivery features you will have to implement a solution by yourself.

To illustrate the problem let's assume we have six different events coming out of our domain model and consumed by our event handlers concurrently in order like this:

1) EventOne {"id":"aggregate-id-1", "fieldA":"value-a6", "version":6}
2) EventTwo {"id":"aggregate-id-1", "fieldA":"value-a3", "fieldB":"value-b3", "version":3}
3) EventThree {"id":"aggregate-id-1", "fieldB":"value-b5", "version":5}
4) EventFour {"id":"aggregate-id-1", "fieldC":"value-c2", "version":2}
5) EventFive {"id":"aggregate-id-1", "fieldC":"value-c4", "fieldD":"value-d4", "version":4}
6) EventSix {"id":"aggregate-id-1", "fieldD":"value-d1", "version":1}
7) EventOne {"id":"aggregate-id-1", "fieldA":"value-x7", "version":7}

"version" field in each event tells us the sequence events has been created by domain, but once we put those events in queue order of messages is lost and on the other end we get them in mixed order.

Now let's say we want to build a projection that would have all 4 fields in it:

- "fieldA"
- "fieldB"
- "fieldC"
- "fieldD"

Problem is that if events will arrive in order specified above "EventTwo" will override "fieldA" value, "EventSix" will override "fieldD" and so on. If you will say that we can use "version" to reject update, well then you will loose "fieldB" value from "EventTwo". Perhaps "EventTwo" overrides "fieldA", but it does not overrides "fieldB" so it should be added.

If we will add to all this that same message can arrive multiple times this gets even more messy.


The most simple solution I have come up with for this problem was to add version field for each projection field. Basically in our case for those 4 fields i would add another 4 fields. Then all my event handlers would build this generic update query for required storage engine (MySQL, AWS RDS, DynamoDB it does not matter)

"INSERT fields from consumed events with this unique aggregate id, IF id exists, then UPDATE record with this unique id and SET every field to value from consumed event IF version of event is higher the the one stored in DB, otherwise reject change"

Basically we apply versioning and we use "version" value for each field. I do agree that having 2x times fields is not so "clean" but this is a solution that allows you to have idempotent event handlers which can apply same "UPDATE" statement multiple times and result will be the same all the time. Also it does not care about order of events. You can process them from the beginning, middle or start, it has no difference as only the latest version for every field will be set.

Here is a sqlfiddle to illustrate problem and solution.

2014 m. rugsėjo 9 d., antradienis

"Bob design pattern" :)

I am pretty sure that some of you have been in situation when you where drawing some sort of diagram in order to understand how components interact with each other, what are the flows, which component is responsible for what and so on. And particularly in startups or projects where some legacy systems are involved you pretty much always get to the point where you need to draw a component which I have named as Bob.

So who is Bob ? :)

Well, Bob is a great guy, everyone loves him and the ones who does not - can't do a 'shoot' about it! Why? Because if someone will piss off Bob and Bob will decide not to come to the work anymore, well guess what - our system will loose one of the core components and everything will just collapse. It will collapse like a house of cards because this guy is a piece of software that usually integrates and couples few other important components and business flows.

When do you use "Bob design pattern" ?

Usually it is used when someone decides that there is no time to automate some processes and in most cases it goes with a phrase like:
"For now,  Bob will perform these tasks manually and once we will have time - we will automate it".  
For time being it works like a charm. Bob manually creates orders and puts those in database by executing SQL queries, runs few scripts here and there, does some reporting, generates email and few other things. Challenges starts to get in your face when you see that one instance of Bob is not enough anymore and it would be cool to spawn few more. Poor Bob - it does not scale that well. Usually that's when you start thinking that time has come for Bob to go and exactly then this drawing happens where we smile while we draw the almighty Bob.

Today we had a great laugh with guys about this, so I thought I will share it.
Smile next time when you will draw Bob ;)

2014 m. rugpjūčio 30 d., šeštadienis

Does mocking breaks encapsulation ?

Recently I had a discussion regarding unit testing and specifically about how mocking dependencies violates encapsulation. In other words i've been told that your tests should not fail if implementation of unit changes. You just test if given input results to expected output and what dependencies are used there and how, should not be included in your unit tests.

I do not agree with this. I believe unit tests should explain how unit will behave in certain conditions. These conditions can be created only by mocking dependencies. Your tests are like documentation which describes the logic of the unit and what were requirements at the time it was created. Now if your unit implementation will change and dependencies will be used differently or different dependencies will be used, but output of the unit will still be the same, you will have your unit tests that describes dependencies usage failing and only test which asserts output will still be passing. Is this ok ? Does this breaks encapsulation ? Should you be fixing your tests when implementation changes ? I'd say it does not break it and it is ok to change unit test along with refactored unit.

As I understand it, encapsulation allows you to hide the internals of the object and prevent code that is using that object to set it in some invalid or inconsistent state. Over exposed public methods we control access to our object and in this way we protect it from outside world. But does this means that unit tests should not know HOW that public method is working ? What are possible execution paths ? What is the logic of it ?

For me, unit test is a piece of code which is interested in internals of method and by writing tests we make sure that logic inside that method works as we expect. Unit test code is not the same as production code which uses that public method. Production code does depend only on the provided contract, but it does not care about internals and implementation. Besides with unit test we are not changing how our unit is working, all we do is just we change how dependencies behave and we do assert how and with what data those are being called.

Of course if you do not use inversion of control and your dependencies are created in private methods then you will be using reflection to mock these and that is changing internals of your unit. But I would still go for it and would mock it, because I want to be sure that my method works as expected in given conditions. And afterwords I would just refactor it so that dependencies could be injected.

My conclusion on this would be that mocking dependencies to test possible execution paths and tie up your tests to implementation details is ok and no encapsulation is violated here (with exception mentioned above). Because changing implementation is like changing documentation of your method and that should be mirrored on your unit tests.

2014 m. rugpjūčio 17 d., sekmadienis

AwareUnit CmsInn tutorial - the one about labels, languages and images


CmsInn is a Meteor package that allows you to add CMS functionality for your project. This CMS is different from others in a sense that it allows you to pick which bits of your application can be edited over this CMS package. It's not like you will have to use CmsInn to create your application but instead you will dedicate to CmsInn to handle some of CMS functionality. For example perhaps you just want your app to have multiple languages and be able to translate texts. So you can just hook in CmsInn translations plugin in your app and do the rest by yourself. Same principle applies to the rest of the CmsInn features.
Main purpose of CmsInn is to allow developer to choose what he wants to use from offered functionality and what he will do by himself.
This tutorial will focus on how to create a fresh Meteor project and integrate basic features of CmsInn package.

Getting started

I assume that you have Meteor and Meteorite installed already. If not please read their docs on how to do that.
Now first of all let's create a new meteor project and add CmsInn package.
mrt create cms-test-prj
mrt add au-cmsinn
mrt remove insecure
mrt remove autopublish
Once project is created delete all files in root directory except 'smart.json'. Then create `init.js` file in your projects root folder with this content:
if (Meteor.isClient) {
        layoutTemplate: 'layout'
if (Meteor.isServer) {
        plugins: {
                insecure : true

We call configure once on client and once on server. On client we configure which layout should be used and call on server sets 'insecure' to true and that makes CmsInn to publish all content. Bit later we will configure roles which are used for authorization. For now we will enable `insecure` mode which allows us to skip roles configuration.
Reason why we need to set layout is that CmsInn comes with Iron-Router and configures it on this call. Iron-router is used to handle routing to pages that you will create dynamically. It will be explained in next post.
Now let's create `index.html` file in projects root directory with this content:
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
    <meta name="apple-mobile-web-app-status-bar-style" content="black" />
    <meta name="description" content="">
    <meta name="author" content="">
<template name="layout">
    {{> cmsinn_controls_left}}
Run your meteor project and if you will check your page you will see that control panel of CMS has appeared. This control panel is your dashboard and it is used to manage your app content. Each button has it's own purpose. We will go through those in details now.

Content editor

First and the most important feature i'd say is actual content editing. With CmsInn in no time you can enable this for selected bits of your application.
To do that we have to define in our templates which parts of our website should be editable. So in `index.html` file edit your layout template like this:
<template name="layout">
    {{> cmsinn_controls_left}}
    <div class="section">
        <div class="container">
            <div class="row">
                <div class="col-lg-4 col-md-4">
                    <h3 data-au-label="brand">{{c 'brand'}}</h3>
Now after page refreshes if you will click on first button in the control panel you will notice that '<h3>' element becomes editable. And this applies to all elements in your page that has attribute 'data-au-label'. All these elements will have a popup with input field on mouseover and you will be able to change the content and then to output it with a help of '{{c <ID_OF_LABEL>}}' helper.

What is cool about it is that you edit content in selected language. So essentially it is also a translations mode because it allows you to translate content in all languages that you will create. We will see bit later how easy it is to add new language.

So in general whenever you need to make content editable you just add an attribute on element and the rest is handled by CmsInn.

Let's add label which will bring rich text editor. Update your template as follows:
<template name="layout">
   <h3 data-au-label="brand">{{c 'brand'}}</h3>
   <p data-type="wysihtml5" data-au-label="brand_description">{{{c 'brand_description'}}}</p>
You will see that rich text editor appears instead of simple input. This happens because we have changed the type of editor. We have added new 'data-type' attribute to a '<p>' element which allows us to provide different type of editor. CmsInn uses x-editable so idea is that you will be able to define any type that x-editable supports.
One thing to notice is that for rich text you have to use `{{{}}}` triple brackets so that output would not be escaped.

Once you have added new element after page refreshes click on 'Translations edit mode' (same first button) and you will see editor popping up. Change content and save it. After refresh you will see your content loaded.

Now let's add 'date' type field. Update your template as follows:
<template name="layout">
   <h3 data-au-label="brand">{{c 'brand'}}</h3>
   <p data-type="wysihtml5" data-au-label="brand_description">{{{c 'brand_description'}}}</p>
   <span data-type="date" data-au-label="brand_created">{{c 'brand_created'}}</span>
After page refresh you will get date selector on mouseover.
Now when you know how to make your content editable let's move on to add couple of languages.


Adding new language is pretty easy. Somewhere in your template you will have a language selector HTML element. So what you have to do is add 'data-au-locale' attribute on those elements. Then CmsInn will bring in a locale selector and you will be able to attach language to it.
For example update your layout template as follows:
<template name="layout">
    <div class="section">
        <div class="container">
            <div class="row">
                <div class="col-lg-4 col-md-4">
                <a data-au-locale="english" href="#" data-au-label="english">{{c 'english'}}</a> |
                <a data-au-locale="lithuanian" href="#" data-au-label="lithuanian">{{c 'lithuanian'}}</a>
When you will toggle on "Languages edit mode" locale selector UI will appear. Click 5th button from top and you will see a blue dash border around those two elements. Move mouse over it and select locale you want to be switched on when this element is clicked.
Once you have attached locale refresh the page and you will notice that content changes when you switch between languages. Try to edit content in both locales and you will see that content for labels will be loaded depending on selected locale.
Once you switch to language that does not have translations you will notice that labels has content from other language and that in '()' it will show in which language you are currently in.
You can add as many languages as you want. Workflow to do that is the same.


Now let's see how we can make images on our website editable. We have few dependencies here to install first.

For image handling this package uses gm package. So first download and install GraphicsMagick or ImageMagick. In Mac OS X, you can simply use Homebrew and do:
brew install imagemagick
brew install graphicsmagick
Once that is done update your layout HTML like this:
<template name="layout">
    <img data-au-image="logo" src="{{loadImg 'logo' '400x400'}}" />
Now go to website and toggle on 'Image mode' which is fourth button from top. Image element you have just added will get a blue border now. Just drag and drop some image from your desktop in that marked area. In a moment your image will be displayed in a size you have requested it. In this case '400x400'. You can change the size in template and you will see that image will be automatically resized next time you will refresh the page.


This post covers just the basic part of CmsInn. My goal was to show the idea and how it supposed to be integrated in fresh project. In next post I will cover the remaining features - "Navigation", "Records", "Sorting", "Deleting", "Publishing" that has not been explained here. There is some documentation on project's GitHb page.

2014 m. liepos 3 d., ketvirtadienis

AwareUnit - Save some time

What ?

AwareUnit is a platform which automates project setup process, makes it much easier to maintain, customise, extend multiple created projects and embraces continuous delivery.
AwareUnit saves time for developer which then can be dedicate to focus on applications domain, in other words focus on custom features.

Why ?

I have started to work on AwareUnit because I wanted to streamline the process of how I will manage my freelance projects. I needed an efficient solution that would allow me to deliver as much projects as possible without compromising quality and still offer competitive prices.

Meteor framework by itself is a huge step towards goal set above. Combined with available tools and prebuilt packages you can almost have all of that. But still I think there was some space for improvement and AwareUnit is an attempt to fill that gap and improve efficiency.

How ?

To achieve this I have set the following goals as initial requirements for such platform:
  • I want to automate everything what makes sense and could be automated 
  • I want to allow my clients be able to do as much as possible without my intervention 
  • I want that designer could work totally independent from me, be able to work directly with the client and that produced work could be integrated in project without my intervention 
  • I want to be able to maintain multiple projects so that changes would be made only once and then would just cascade down to the rest of the projects 
  • I want to be able to host projects on any hosting provider 
  • I want that code changes would be automatically deployed but only if selected so (Continuously delivery) 
  • I want to be able to make changes without setting up the whole development environment 
In plain english I was looking forward to the workflow where I would be just working on the code that is custom per client and everything else is either automated, either can be done by client or designer.

Result - real world scenario

Couple of users visits awareunit.com website. Each of them needs a different kind of project. User One wants a website where he could present his company. User Two needs a portfolio website. User Three needs a blog. But they all have one common need and that is they all have a custom requirement for their application. May it be a small tweak or a big feature it does not matter, because they know that once project is created they will be able to request me to implement that feature.

So every user by using a simple UI on website selects a blueprint that fits their need and creates a project based on it. In 5 minutes they have a fully working project ready to be used and accessible on url like this http://project_name.somedomain.com.

If user is happy with initial project and does not need any custom thing. I consider project is done. User will use project's CMS to add content and everything else is handled by the platform.

If user needs customisation, then there are two ways it can be done. First one is that client uses his own resources to customise project. Clients has full access to code base and can do whatever they want with it. All changes that are pushed are delivered automatically to live app. If user wants, he can take the project and deploy somewhere else.

Another option is that client uses a dashboard on awareunit.com to request me to do some work. Then we make a contact, clear requirements and I do the work. Client is billed only for the time I spent on implementing that custom feature.

If client needs a custom theme, template for his project, he can go through the same process. Just this time designer will be the one who will do the work. In other post I will explain the solution behind the hood but thing is that all work he does can be integrated in to the project without my intervention. So I waste no time here.

At the end of the day client pays only for the customisation and not for the whole project.

Who should be interested in AwareUnit

If you have an idea that is related to web and you will be building a prototype for it, then this is something that will save you time. You will have a solid base to start with and Meteor is one of the best frameworks when it comes to prototyping. So startups should check it out.

If you have no technical knowledge but you need to create a web application, then this is a place to do that. In 3 simple steps you will have your application up and running.

If you are Meteor developer and you want to save some time, then this will do the trick.

If you are web designer who knows HTML, CSS and a little of Handlebars template engine, this is a place where you can not only distribute your HTML templates but also allow users apply those templates straight on their project.


Next step is to make this platform open for developers and designers. Reason being that then we could transform AwareUnit in to marketplace where developers and designers could distribute their work. This would be alternative to marketplaces like "ThemeForest". Difference being that here users would purchase a whole working solution code wise and templates would be applicable straight away. This would be just so much easier.

Another cool feature is that once user creates a project based on your blueprint, you, as developer, have a lead that potentially will convert to some development time. AwareUnit has integrated dashboard that will let you communicate with clients. So essentially AwareUnit could be named as an alternative for projects like "Elance".


There are still a lot of things remaining to be done, in order to achieve all set goals and open it for public use. But as I have a working prototype, that is a prove that it is doable. You can play with it here at http://www.awareunit.com . Any feedback is more than appreciated.

2014 m. balandžio 23 d., trečiadienis

No idea validation = no code

When I have started all this entrepreneurship venture I thought I really knew enough. And that was true, except one thing. With all my ideas i stuck in so called "hallucinations" phase. In this phase "everything makes sense to you", everything is logical about your idea, your friends "loves" your idea and the ones who says "i don't understand" are just ignored by your brain. You just work your ass off to implement your idea, then you do refactor it COUPLE of times, you empty and prepare a separate room to be filled with trillions. And then once you are done and you start looking for that big red "Go to market" button, something hit's the fan ...  And that something smashes really hard, straight in to your face ... Yes - no one needs your product! Problem you were trying to solve is not that painful that people would even think about spending cash on it.

After this cold shower I have asked myself - "how the heck it happened ?". I knew everything about lean startup, i knew that "i had to get the heck out of the building", i knew i have to collect feedback, i knew i had to build MVP, i have been trying to reuse what i could, but still I did that god damn rookie mistake - "started to build stuff, that no one needs"!

At the beginning i could not admit to myself that i was just blind and stubborn. So I have started to work on another idea. This time it took me a month to get to the point were i have bumped in to same wall ... "What's wrong with you dude!" aren't you learning from your mistakes ?

I am learning! Just it seems, this is just the way I am. I have to experience by myself how it feels to be in some situation in order to draw a conclusion.

After those two attempts my third attempt to build something useful is totally different story. I am not writing a single line of code until I have a validation that someone needs it. What I do now is I lookup for the market in the first place. Then I do try to identify what problem I can solve, so I do write down an assumption and then I call. I do cold call and pitch my solution and if that is not something they need I ask what they need so bad that they would even pay for it. For example i am doing this right now with real estate brokers and you would be surprised how gladly they accept invitation to meet-up and discuss their pains.

Today I can admit that when I did those mistakes in the beginning, I was just not ready to go out of my comfort zone and actually start talking to people. I was convincing myself that I am the smartest person on this matter and I know that it will work. But that was not the case, that was just an excuse.

So conclusion - go out guys! You are not smarter than all your customers collectively together. Validate your idea as soon as you can and pivot if need asap. I really hope that this post will save time to someone, because then at least in that way I could justify those mistakes :)

2013 m. vasario 10 d., sekmadienis

Symfony 2 Bundles usage - Project Easy

Couple of weeks ago I have started to work on a small project which purpose is to suggest people how they could structure their Sf2 bundles that would make them more flexible, easier to maintain and what is very important would allow to create business logic practically framework independent. 

When you think about it, what does really matter in web applications is business logic, right ? Now we have Sf2, we have ZF2, CakePHP and lot of other frameworks and god knows how many will be created. All of them has their own pros and cons but at the end of the day - does the same thing, hosts yours applications business logic. So me, as developer, was concerned how could I create code that could be moved around from one host to another, from one framework to another. I do accept the fact that migration will cost me time and there will be code changes, but if I could isolate my business logic - that would be cool.    

And well, Sf2 bundles concept, composer and a few structures in place makes it possible and project "Easy" is a prove of it.

At the moment I am working on documentation and after words code samples will be coming over, so stay tuned :).

Here is a link to "Easy" GitHub repository.