Agility In The Enterprise

CI & Dynamic Languages

I was reading this post and it got me thinking... Do we get a lot of benefit from a continuous build server when using dynamic languages like ruby/python/php? While initially skeptical, I quickly came around. I can think of two big benefits to still using a continuous integration build server.

1. Runs the unit tests on each check-in. This helps maintain the code and does some minimal regression testing.
2. Deploys to an integration test environment. This step is not always done w/ CI servers, but it has a big benefit when that extra effort it taken in that it provides an always up to date environment where you can look @ the latest source in action. It's good for feature demos & it's good for fix verification.

Tracking memory usage - Part 2

This is part 2 of the my earlier post on tracking memory usage.

I also wanted to see what Windows applications were running and how their resource consumption looked; Both in isolation and in comparison to the java application. Here are my thoughts for monitoring that.

Option #1
Use Task Manager to manually check the "VM Size" for the windows applications.
Pros:

• Easy and straightforward.

Cons:

• No detailed memory tracking. Just simple data points.
• Not easily scriptable & therefore prone to error.

Option #2
Use Perfmon to monitor memory of windows & java processes.
Pros:

• Gives a good overal view of memory usage of the application.
• Can be run continuously or scheduled to start at a certain time. It seems that automation of this would be do-able.
• Excel could be used to develop straightforward reports from the files.

Cons:

• You'd have to extract and save the files in order to report on them.

Option #3
Write a custom application to capture process information using .NET APIs.
Pros:

• Simple code.
  
• Easy to save in a custom format. (& integrate w/ other management software.)
  
• Re-usable tool if configurable & reasonably flexible.

Cons:

• Custom, therefore requiring development & maintenance time.
• will probably be less flexible & feature rich than a commercial tool. (i.e. The custom-ness better be worth it)

My next post will conclude w/ my final recommendations for this particular effort.

Tracking memory usage - Part 1

Over the last few weeks I've been putting some thought into how to monitor and track the memory footprint of a java application over time. The point would be to keep an eye on things and provide some historical information that may help when deciding when & how to address growing memory requirements as well as provide some tools to help investigate memory usage when and if it becomes a concern. Seems simple enough, right?

For the most part, commercial packages were either too expensive or too complicated to be usable for this germ of an effort. But I can imagine that many larger enterprise clients would already have a HP Openview, CA Unicenter, or other tool setup and it would make complete sense to just hook that up to the servers you wanted to monitor. We didn't have anything like that in this case and wanted to keep the system simple & cheap.

Here are some of the options I considered:

Option #1
Include a component in the application that tracks JVM memory information over time. (Timer task?) Using either the simple java.lang.Runtime memory methods or a hook into the new JMX mbeans that provide memory information.
Pros:
    - Easy and straightforward.
Cons:
    - No detailed memory tracking. Just simple data points.
    - Requires an update to the production application to include.
    - Doesn't give us any generic tools that could be used elsewhere.

Option #2
Hook up a profiler to the application & take a snapshot every release. (e.g. Netbeans Profiler)
Pros:

• Much better memory information.
• There would be the potential to investigate memory footprint changes @ a very detailed level.

Cons:

• The Netbeans profiler doesn't provide any scripting options that I could find. Therefore you'd have to save the snapshots manually.

Option #3
Use the jstat tool that comes w/ the JDK to capture memory information.
Pros:

• No modifications to the application code would be required.
• Simple & easy to setup and run.

Cons:

• Less detail than a memory snapshot would have.
• Extra work would be required to save this information somewhere where you could report on it.

Option #4
Use the Oracle/BEA Jrockit JVM and their Mission Control tool.
Pros:

• Has command line tools that would make scheduled tasks easy to write & automate.
• Has nice reporting tools.
• JRockit JVM is supposed to be faster than the normal Java JVM.

Cons:

• May require license from BEA to use.
• Not the standard Java JVM. Potential for weird bugs that only Oracle/BEA customers see.

My next post will talk about simple monitoring of (non-java) windows applications. I'll then follow up w/ the chosen approach and some final thoughts.

Maintainability is important

Developers sometimes seem to be rewarded based primarily on the speed at which they can deliver features and secondarily based on the bug count of those new features. Usually totally unrewarded, except maybe through additional peer recognition, is maintainability of the system. I've run into this a lot lately where systems have evolved over time and each new person to look at it has made whatever changes were required in the absolute easiest way to get it working and out the door. In one case the team stuck a bunch of code for one system in another systems batch process because they didn't want to go the trouble of creating another scheduled job. No consideration that I could find was given to the fact that any new maintainers would have no idea where to find that stuff.

I think one of the causes of this situation is that development velocity and the bug counts are pretty obviously forced out of the process, but increased maintenance costs are really hard to quantity and pretty subjective. There should be some way to tailor the development environment in a way that encourages a longer term view and rewards outstanding efforts in areas other than how quickly the code gets out the door and how few bugs are found. If anybody has any suggestions, I'd love to hear them!

Aligning IT groups

For all of us that have worked in a large corporate IT department, this should come as no surprise. Typically these groups are broken up by specialty.  The DBAs in one group, the application developers in another, the infrastructure guys in third. The reason for this is fairly obvious. You get nice leverage by managing like skill sets as a group. The people in those groups learn more and share their knowledge. And the manager gains flexibility by having a set of people who can back each other up. 

Usually you will build project teams with representatives from each of these groups. So called matrixed organizations where you have dual reporting relationships. 1 to the project lead, and 1 to the skill set lead. One of the downsides to this organizational structure is that it can lead to 'he said, she said' blame games and people looking out for their specific domains without focusing on the larger project or business goals. But typically, if managed well it can work out fine.

I read an article a while back about how Amazon (even with their massive IT organization) keeps the teams completely responsible for their own applications. They build it, they deploy it, they maintain it. If users hate it or they aren't able to meet their deadlines, they have no one to blame but themselves. I really love this approach. Setup teams with all the right folks, the appropriate resources, make them accountable, and watch things work. It can be considered 'extreme', but it clearly works for them.

Application developers seem to often escape the misalignment problem (not focused on delivering business goals) because they don't have a lot of non-project work. But other skill set teams can have full time jobs doing non-project work. For example, DBAs in a large, complex environment can easily become bandwidth constrained just maintaining the infrastructure. Same goes for sysadmins.  When this happens and you don't get them on the project team and bought into the project goals, you can have trouble moving things forward. Big organizations have the money to develop standard processes which are often optimized for reliability rather than speed.  In order to avoid getting stuck waiting for things, you need a project with executive sponsorship and advanced coordination with these groups. Often it's as easy as a quick face to face meeting early on in the project explaining what you're working on. It just greases the wheels for when you need some help later on.

Doing the advanced planning, either by getting people on your project team from these groups or by coordinated part time help and getting buy-in on the project goals is often critical to project success. Everybody hits snags. How fast you recover is closely related to how well you anticipated and planned ahead.

Ruby on Rails tools part 2: Capistrano

In just the 2 weeks since I posted my comments about the Rails migration tool, I've seen a bunch of other bloggers talking about it. It's great to see it widely recognized as a great tool independent of the excellent rails web framework.

In sort of an Innovator's Dilemma situation, the simple and highly effective tools seem to originate from projects outside the technology platform mainstream. As opposed to the db migration tool I wrote about previously, Capistrano definitely has competitors. There seem to be two main deployment tool styles. The first is fundamentally just a few extra tasks embedded in your build script; sort of 'ant' or 'msbuild' on steroids. The second is a more feature rich dedicated tool that often has illusions of grandeur. Sometimes these are called "application lifecycle management" tools and are often complicated and "enterprisey" in the worst sense of the word. But none are as flexible, straightforward and simple to understand as the Capistrano tool used for Rails deployments.

In the same way that Rake is a DSL for builds, Capistrano is a little like a DSL for deployments. It knows how to copy files across the network, execute commands on remote servers, interact with various version control systems and most interestingly how to roll back to a working release after a failed deployment. All of this comes with a system based on the "convention over configuration" idea so prevalent in Rails. You only have to configure what is non-standard or cannot be guessed. (e.g. server names) And the most interesting thing about Capistrano for me, being a multi-language, multi-platform consultant, is that you can use it for any kind of deployment. It does have a slight predisposition to unix based servers for paths and due to its reliance on ssh. But for non-Rails servers, it's easy to configure other app servers by specifying custom start & stop tasks and changing the deployment path. (if necessary)

The thing I like about Capistrano is that it seems to hit the right balance of easy for the simple case and able to handle the complex case. It is definitely worth a look no matter which technology you're using for your application.

db migration w/ rails

I attended The Pragmatic Programmer's Ruby on Rails Studio a few weeks ago and was expecting to enjoy digging into Rails. What I didn't expect is to get really psyched about some of the supporting tools provided by the Rails framework. There were two particular tools that I really liked. I'll talk about the first today. (The formulaic multi-part post thing. :-))

It's the tool for handling database migrations. For those of you who haven't tried it out, Rails provides a generator that creates a file and class for use in applying changes to a database. The class is derived from ActiveRecord::Migration and includes an 'up' method and a 'down' method. The idea is that running the 'up' method applies any necessary deltas to the database (upgrades it) and the 'down' method reverses that same delta (downgrades it). When you ask the generator to create a new migration script, it creates a file w/ a descriptive name you provide as well as a unique version number (the most recent version +1). After you fill in the up & down methods appropriately, rake's db:migrate target is used to apply those changes.

When the db:migrate target first runs, it creates a table in the database that tracks the current db state in terms of a version number. The 'db:migrate' build target can then move you forward or backward from that version number by applying the appropriate migrations found in the source tree. For example, if the database is at version 3, and you want to upgrade to the latest version, you just call db:migrate with no version and it will run all the available migrations from 3 forward. If you decide that's not what you wanted to do, you just call 'db:migrate' again with a target version of 3 and it will run each 'down' method in reverse order until the database is back to its previous state.  With just a target version number if can do it all for you. I love it.

We've all used our own home grown methods of tracking changes that need to happen to the database as part of a release. Most often it seems to be a set of SQL scripts checked into the source tree somewhere that someone knows need to be run when the release is deployed. The average project, from my experience, is living in the Stone Age in regards to how they manage their database deployments. The RoR approach, in contrast, has number of cool things going for it.

• The database knows its current version number. That means when you look @ a developers machine, the system test machine, or even production you can easily determine the state of the database.
• The migration is automatic and integrated with the build system.
• It handles reference data changes just as easily as data model changes.
• It forces you to be disciplined about writing migration scripts for all db changes that you wish to propagate. (and makes common tasks (e.g. adding a new table) easy)
• It includes support for backing out a failed migration.
• It's a tool. Whether you like this specific tool or not, having something with a standard way to doing things and a bunch of automation to support that standard way is a good thing.

I actually liked it so much that I copied it and used it for a non-rails project to handle the migrations. It was painless to setup and worked just as well outside as it does inside a Rails application.

organizational learning

I've been writing a decent amount of ruby lately. Mostly because I'm interested in learning it, so whenever a task pops up that is best handled by a script or a small-ish application, I try to do it in ruby. And after a few of these over a couple of months, I'm starting to get the hang of it.

The interesting thing about learning a new a language or platform is that the tutorials almost never do them justice. They're fine for a 'hello world' app or to get you excited but you need to go through a regular (ie. real) project before things start to gel. Things like (1) how to build it consistently and automatically, (2) how to test, (3) how to deploy, and (4) what the best tools are. Many of which are rubber hitting the road kinds of questions.

As I get enough experience w/ ruby to hit some of these issues, it dawned on me that the same kind of learning curve holds true for organizations experimenting with and/or adopting new platforms. They go through the same kind of cycle.

1. Syntax & basic usage <-> training class + proof of concept
2. How to run <-> demo / proof of concept
3. How to debug and test <-> initial development project
4. How to build <-> initial development
5. How to deploy <-> first qa/production deployment cycle.
6. How to monitor <-> Initial roll out and burn in period.
7. Optimizing <-> idioms, best practices, and best of breed tools.

Until the first project (or 2) goes live, there is a lot of experimenting and learning and nothing is very stable or proven. But a few projects later and suddenly the organization starts to understand how to manage this new platform and the developers start to feel good that they are doing things right. It's fun to get to that stage.

Virtualizing development environments

This post is sort of a follow up on my earlier predictions post.  I think virtualization is one of the hot technologies for the next few years. I imagine that upwards of 75% of servers in corporate data centers are less than 50% utilized. That means massive wasted capacity that could be tapped with virtual servers running on that same hardware. It's the best of all worlds. You can roll-out new applications on completely siloed and unique stacks without worrying about what stacks other applications or systems in the data center are using. You can also move the virtual stack around to other servers as capacity needs change. I love it as a corporate IT solution.

But what got me excited a while back was using it as a consulting solution. I could install one or more virtual servers per client and have the ability to store and backup complete development environments irregardless of the physical machine I'm currently using. Upgrading the laptop would be easier, and adding new (potentially conflicting) stacks would change from painful/nearly impossible to a breeze. The downside is that it's pretty painful to setup. You have to setup the virtual environment, install the OS, and install all your development tools before you can even get going. Given all that time, I'm not sure it makes sense from a cost benefit perspective in this type of scenario. Especially if the stacks you're dealing with often do not conflict. I still like the idea, but it's not a clear slam dunk. Perhaps I'll use it in a few specific cases where there are conflicts and a larger investment would pay off, but I think in most cases I'll stick w/ a non-virtual environment. In the corporate IT world, the stacks would last years and you'd have a lot of them so you'd easily recoup even a significant startup cost. But for me where they may only last a month or two it's much harder to justify.

2006 Year in Review

As part of an end of year review, I'm going to do a two part post where I review trends of the past year and predict technology moves coming in 2007.


First off, things that we saw in 2006.


Rising Tech

• AJAX

Last year we saw widespread adoption of AJAX on all manner of websites. And for the most part it has made those sites much nicer to use. If you don't already consider Ajax a mainstream web development technology, it will be in next year. 

• Ruby/Rails

Last year saw continued interest in Ruby and Rails. It actually fueled a lot of other supporting technology development. I'm thinking of Python web frameworks, Grails, Capistrano, etc... I expect interest and use of Rails to continue in 2007.

• Agile Methodologies

Agile is now pretty mainstream as a development methodology. I'm not sure about current number, but I bet Agile methodologies will account for more than 70% of new projects next year. (ignoring projects where no methodology is used)

• OSGi

In java-land, OSGi got some important traction in 2006. I attribute that to Eclipses successful use of it as well as Spring's recommendation and supporting technology.


Stagnate or Declining Tech

• Web Services

WS-* services have sunk into disrepute due to complexity. I expect plain old XML over HTTP and REST services to fill in the gap where required.

• C#

Clearly C# and the .NET framework are the recommended development tools for the Windows platform. But I don't see any moves from non-windows developers onto .NET and C# and I don't expect that to change.

• Java

Java's situation is similar to C# in that it's not attracting new developers, but it's a bit worse off due to the size and complexity of the platform. (Part of that is just maturity.)

• EJB

EJB 3.0 didn't reverse the large scale abandonment of J2EE by the development community. It's decent technology, but to little and to late. Minimal web/app servers running Spring hosted services are the new model.