I had a few thoughts today on a direction that I think that software will be taking and that I think is important to document.  This idea ties into my post earlier today on generic programming and domain specific languages.  Take the followig three concepts as a set:

• Generic Programming
• Domain Specific Languages (DSLs)
• Intentional and Automatic Programming

Generic programming, as mentioned earlier, is programming that invokes a metaprogramming facility from within a hosting language (this is a part of the wikipedia writeup here, and I like it a lot).  It provides a means of extending a programming language to model higher level constructs, that which are much closer to the domain problem being addressed.

What generic programming provides is a means of being able to create extensions to languages that allow new syntaxes to be created that can be used by those working in the domain to be able to solve problems (instead of computer scientists) because they can utilize the computer to solve the problem, not write a program to solve the problem.

Now, if you are following the movement of the people touting DSLs, you might say that what I just described is the exact definition of domain specific languages.  And I agree.  And that is exactly why generic programming is important.

But whither to use generic programmingx or a DSL?

That is the question.  Here is my answer.  Generic programming remains in the domain of the computer programmer, ie the computer scientist worried about syntax and bit twiddling.  DSLs fit in with those who are trying to solve business problems.  I'm not going to go as far as to say that this is a business analyst, but it is potentially (and preferably) a person more atune to the problem domain then to the other problem domain of knowing how to program computers.

My point is, programmers will need to work to create DSLs out of existing languages (be they C++, C#, Ruby, ...), and Domain Specialists will use those DSLs to solve the higher level problems.

If you've read other posts of mine, you will find that I think that programming is too difficult.  It's not that its necessarily hard to do, but that it is tedious to do.  Hence my other ramblings on things like P#, a pattern based language that gets rid of a ton of boiler plate coding that I just hate to type everytime I code something.

Now, lets move up one level, or many levels.  I really believe that intentional and automatic programming (IP and AP respectively) are requirements for the future, and that DSLs are an important component in facilitating both of these into realization.

I personally see DSLs fitting into a role in IP/AP where the problem domain is modeled, and then an entity can automatically integrate and deploy software.  DSLs would in a way function as the implementation of well defined activities that solve business problems.  The DSL code could perhaps be generated out of a software factory where the domain specialist can check a few items off of a list and get an activity that solves the problem, and that is also checked against requirements to solve that problem.

Then, a similar process could be used that can integrate thse activities into higher level processes.  Note that each individual activity could be implemented via a different DSL (see my earlier posts about systems most likely being implemented in multiple DSLs), and the activities glued together via a software factory just as the individual activities generated.  And checked against requirements to work.  And automatically deployed and operated as the computer generated the code, and knows how to deploy and run it through the building of the DSL through generic programming.

Perhaps the eventual goal of fully automatic programming is not obtainable, after all at some point the computer must be told somethign about what to do, but surely this would be a great step forward into being able to quickly and reliably build software, and to business relying of IT to become agile.

A few days ago I was doing some searching around the net to try and find if there was any information I hadn't previously found on doing force directed graph layout.  This came up as someone had noticed here on the blog that I had a control that did that.  Unfortunately I don't consider that code to be very robust, so I was looking around to see if I could find a library or some sample code to do it better.

Well, what I came across was a C++ library called Boost (you can find it here).  It's a generic programming library, and consists of quite a few components, but one of them is a graph library that does have classes for force directed graph layout.  So, I downloaded it with hopes of being able to learn some more about how to generate these types of graphs.

Now, I used to be quite the C++ programmer, but have done almost 100% C# since 2000, but I figured I could just jump right back into it.  Well, apparantly not.  Seems the world of C++ has really moved forward into a concept known as generic programming through the use of C++ templates.  This technique goes well beyond what I used to use templates for in C++, and currently use Generics for in C#.  Generic programming as being currently used utilizes a metaprogramming facility from within the host language to generate new semantics into the language.  In a way, this kind of aligns with research I've been doing lately into domain specific languages (specifically DSLs in Ruby).

Don't get me wrong here, as I really love C# (particulary the stuff in 3.0/3.5), and think it has been pushing the model for programming ahead better than anything else, but I had been feeling that it wasn't moving fast enough.  Dynamic languages is one place where I was seeing usefulness (Ruby/IronRuby specifically), also with Domain Specific Lanaguages (both with Ruby, as well as with the Microsoft DSL tools), and now with generic programming, and most importantly with C++.

It really seems to me that C++ is a toolset that must be put back into any programming gods toolset.  I've always missed several features in the C++ language in C#, particularly the ability to pattern match through automatically promote object creation through a chain of constructors (a technique I used a lot to reduce code).  Now with generic programming, combined with libraries such as Boost and the STL, this really may be one of the best ways of getting particular tasks completed the most effectively.

BTW, on the concept of the STL, I've never understood Microsoft's position on container classes.  I was one of the first adopters of using the STL when it came out, but Microsoft's support for it in their C++ compiler was always buggy, so it was always difficult to use in that environment.  But the design of the classes really are the best.  I mean, every time I still code a dictionary in C#, I still mentally think of the much more flexible stl::map class.  C# containers always just felt incomplete compared to the STL.

One thing that I was really surprised at was that the Boost library actually compiled and ran without problems in VS.NET 2008.  Previous experience has shown that this would not be the case (as with the STL).  Kudos to the C++ team for keeping up.

Now this C++ generic programming thing seems to be quite the adventure to learn.  I can tell that there is something very important there, but its quite a mental shift to pick up.  I'm looking into some resources about it and I'll write about my experiences.

Additionally, the integration of C++ in to C#, through one weekend of effort, has been quite challenging (as well as interesting).  It's not possible to use a library like Boost directly in your C# apps (which I have to keep doing), but it is usable in unmanaged C++ within managed C++ (an interesting topic itself), which is then used as components in a C# application.  This involves several interop techniques, but appears to work quite well.  I'll drop some notes on that too as I move forward on this adventure.

I've always wanted to define a canonical list of software patterns.  There are more type patterns than most people know.  Most programmers I cross think purely in terms of the GoF (gang of four) patterns, but there are many more types to know. 

Here is my current list of all types of software patterns:

• The classic GoF design patterns
• Patterns of Enterprise Application Architecture
• Enterprise Integration Patterns
• Business Process and Workflow Patterns
• Data Access Patterns
• Remoting Patterns
• Threading Patterns
• Real-time patterns
• Testing (TDD) Patterns
• Parallel Programming Patterns
• Ajax Design Patterns
• SOA Design Patterns
• Implementation Patterns
• Cloud Computing Patterns

I was sitting around and thinking about my experiences with distributed agile software development.  I've run a number of projects like this and have had a few challenges to deal with, and thought they were worthwhile writing down.

First, perhaps a definition for distributed agile development.  One of the principles of agile development is to have high bandwidth communications between members of the team.  This isn't necessarily bandwidth in the sense of Internet speed, but in having direct face to fact communications.  Once this direct communications is removed, you then have a distributed agile project.  This can be that some people are on another floor, another building, or state, or country; just that they are not instantly available for face to face interaction.  And this includes all participants in the process.

When distributing agile development, and therefore reducing the bandwidth of communications, you must then realize through communications theory (Shannon's law I believe) that the amount of processing to understand a message must increase.  In a sense, you must either start compressing the content of the communication, and / or increasing the effort to understand the communications.

The issue that then begs to be addressed is that how does the processing increases to handle the restriction in bandwidth?

To control this, you must get up front (as soon as possible):

• High trust / agreed SOW.
• Understanding of where the business value is.
• Agreement to collaborate frequently
• Set up means of collaboration
• Understanding that it is still an iterative process that assumes no specs but frequent changes
• Having the ability to access to subject matter experts

A lot of this can be solved by a kick off meeting that addresses these issues through:

• Discuss the scope
• business outcomes
• Get the team together
• Agree on how to break down communications barriers

This type of get together can range from a few days to a couple of months.  It's great to get everyone together as they will be able to begin to understand each other personally, and from that they can understand who they are dealing with when having communications challenges.

This can be very important in facilitating as remote communications will now feel more like they are face to face, making the communications seem much more personal.

It is also useful to rotate people through the sites.  This helps to facilitate keeping personal bonds tied, as well as in helping all members understand the culture of the other members (since this is often global in nature).

Another important thing is to be able to empower the team.  The team as a whole must be able to make their own decisions.  This will tie into having a vision for the work.  Often this can be set by a CEO explaining the vision (or other visionary).  I've found this to be very useful as it has allowed the team to understand where it is that needs to go.  By having the CEO of a remote partnership get buy in to the process, and selling that to his employees that are a remote part of your team, they will respond much better.

I've also found it useful to still hold daily stand up meetings to integrate the distributed members of the team.  I'd often have a daily stand up in the morning with my immediate team members, but also conference call in members of a remote team (say in Europe).  But, I'd also have a standup late in the day with the team, but with other members of the team in India (which late in the US day is their early morning).

The purpose of these meetings is just like that of every other standup: keep the teams tracking on the immediate goals of the project, and also work to identify any communications or technical issues while going around the horn.  And also make sure that you keep this meeting rolling.  Identify issues in the meeting, don't try to solve them.  Take the things identified offline into other meetings between the specific people needed to solve them.

Also, as a project manager (or Scrum Master, ...), use these times to also facilitate more interaction between the members of the distributed teams.  Basically, this comes down to being able to coach the team members to being up issues, and then to lead them into offline communications.  This is also particularly facilitate if you have co-located team members, as a member of the remote team being local can facilitate your understanding of what is being said from afar.

I've only scratched the surface so far.  I'll surely write some more on this inthe future.

This weeks Hanselminutes was a discussion with Tom and Mary Poppendieck on Lean Software Development and what is "software success".  Much of this discussion struck a chord with me and that I figured it would be good for me to drop a few thoughts onto my blog.

The initial part of the discussion started out with talk on what is the definition of “success” of a software project.  Scott apparently was chastised by Mary on a panel for mentioning that he thought it was to “be on time, in scope, and on budget.”  Mary stated that this is the definition of success according to “The Standish Group”’s chaos report, which measures software success on these criteria, and more specifically, failure of projects by not achieving these. 

She then said that she feels that this is not an adequate (or even correct) measure of success, and that for a software project to be a success it must be a business success, not a technical success.  This means that the software that is being implemented must create something that does what the business needs, such as achieving market share or being profitable, not simply accomplishing some type of task or goal.

I consider this to be a very important statement.  I’ve been on many projects that have been completed on time, on budget and in scope.  But they were never used.  Or used briefly, and then forgotten.  These, in essence, wasted all of the money put into the implementation.  These types of solutions have always been in my mind to not be successful software projects even though many call them successful.  Needless to say I don’t really like these types of projects, as in the long run I haven’t contributed to bettering the bottom line.

This leads to the idea that developers should not be measured by being done “by a certain date”, or on a certain “budget”.  Mary stated that when she was at 3M that the teams that she was on had to show P&L for the projects, marketing plans, …, to be responsible for all that, and in short showing that profit would be made from the project.  Or else, the project would not even be approved.  I agree with this whole heartedly, and more organizations should really learn to do more effective forecasting of the ROI / IRR on software projects before jumping in to that pool and building away.  Additionally, these measurements should also be used to guide the project alongs its path.

Mary then got onto a point that I think is very important, that her teams were held accountable to be responsible at a higher level than just time and budget, which are the typical things for which most software teams and project managers are held accountable.  These teams should be held accountable for creating things that address a tangible business need.  Their success must be measured on the actual problem solution, the successful capture of a financial opportunity by the organization.

One thing that comes to my mind, and that I feel she wants to say but never really gets around to it (probably due to time constraints), but I’ll say it (and I’ve said it to other before), is that it is my opinion that software is just a tool of implementation towards a completion of a solution / product.  Software development is a tool in building things that allow businesses to exploit opportunities, and the focus of software development must be to effectively address increasing profit in an organization, otherwise the criteria for success of the project is incorrect. 

Software development to me is not different than, say, using plastics in injection molding of car parts, or the assembly of those parts into a car, or any type of widget.  If the widget is not something that anyone will buy (that is, it will make money, and money above the cost of building it), then it doesn’t matter if it is plastic or metal machinery, or software development, that put it together – it just shouldn’t have been done in the first place.

Where this discussion in the podcast was going, but never got to, was into discussion of product development, and what I would refer to as software product line engineering (SPLE).  These higher level constructs for measurement that were mentioned by Mary, are in my opinion, components of proper product platform management, that uses software as the tool set for building the product, and where measuring success of products is in making profit (yes, I sound like a ferengi).

I'm going to follow up this post with another on my thoughts about SPLE as I think it is really important, but I need to address one more issue first.  During this discussion, Scott made a mention about the agile technologies, and that don’t these solve all of these problems?  Well, Mary had an interesting response which I totally agree with, and I’ll state it from my perspective:

Agile software technologies and tools as described today are an implementation detail of proper product management.  They are the means by which you create the parts of a singular product made of software, not how you define what should be in your product, or as the parts used to build a suite of products.  Measuring for success on using agile tools and methodologies alone is therefore an invalid measurement of the success of the products built with them.

I get a lot of flack from people I mention this to, especially at things like agile user groups.  I don’t care if you use XP, Scrum, TDD, MDA, Nunit, Mocks, FitNess, … all that stuff … if you are not building the right thing, then all these just help you build the wrong thing better.  Yes, they are important (and I do personally consider them to be very important), but they miss the overall target because the people working with them have myopic views of the overall organization since they are only in the trenches of software development.  It's not their fault, but it is a situation that does need to be fixed.

Also, and perhaps more importantly, even if you are building the correct thing, but you are not building it in a way that can lead to greater agility in the future, then what is the purpose?  This is something that touched on very briefly by Tom in the podcast, but could have used more elaboration as I think it is the real key to software success.  Software must be built in a manner where it is used to accelerate future product development, so that the organization can become faster to market with products down the line.  This is the holy grail of software reuse.  Not using software necessarily to build software faster, but to use product components built with software to build products faster than competitors.

This last note leads again back into software product line engineering.  I'll follow up with more on that topic in some future posts.

During this weeks Hanselminutes, Tom Poppendieck mentioned four types of value that a consultant can provide.  I thought they were good and were worth a note here:

1. Capacity, basically providing more labor.
2. Outsiders can bring in ideas that they have seen elsewhere.
3. Can see your situation in a different light and advise your on your problems
4. Acellerate acquisition of new skills to get better faster

Point of these are that you still need to know your companies core competency, and consultants often can't provide that.