Rigidity in Data Design

Silverlight Logo

To many developers this may seem odd. I talk with many staunch ALT.NET guys, and the DDD philosophy seems to be that data is a top-down or at worse, bottom up design problem. The issue here is that there is an assumption that just simply not true that data design is part of most software development projects. The reality based on my experience as well as the experience of talking with developers in the community is that many projects (though its hard to exactly quantify what percentage) begin with existing data. This is especially true in the enterprise where data exists in many forms from new databases, legacy servers (e.g. mainframes) or even flat files and XML. It is the rare project that is new code against all new data.

For new projects, creating a new database schema that at least in some part mimics the design of a project makes sense. The problem is that over time the data must evolve. I think the reality is that depending on the organization this can be very different. There are some opinions that I think impact the maturation of a data model:

  • "Well if they want access to the data outside the application, I will expose that as a service interface." - I think this is coming from the world of a consultant not a enterprise developer. The reality is that the data maturation happens over time with the help of DBAs, developers and IT people. Its a complicated situation and isolating the database to the non-development stake holders is something that just doesn't happen in the long run.
  • "Transactional schemas are not a good solution for reporting." - While I don't disagree with this idea, it doesn't happen in the real work either. In most large organizations, they have need for data coming out of systems and cannot wait for data warehouses, data marts or reporting servers to be created and maintained.  If they are created as part of the application development, they usually get the short straw of maintenance.  Reporting happens against transactional schemas...period...get over it.
  • "If the application is designed right, the DBA should never need to get involved...we can solve perf problems with caching and smarter data access." - This assumes that the data is isolated from the rest of the organization.  This doesn't happen either. In addition, data changes over time and that's where DBA's can be exceptionally helpful. In my opinion, the form of the data can be improved over time as real data is used in a system. What works for a 1GB database rarely works for a 1 TB database. Lastly, there are often many more readers of data than you might expect. This is related to the reporting story, but also ad hoc attaching of data (damn you Excel and Access) happens. You can't control the gateway to the data as the guys that sign the checks sometimes make dumb decisions that impact the organization...that won't change.

Now I am not suggesting that we need to dumb down designs to deal with the nature of data, but understand that the IT infrastructure of deployed products is a lot more fluid than developers would like. For me it is, "Hope for the best and plan for the worst." This means understand that pushing a design philosophy that can help developers get their job done in a more efficient way as well as create better software is what we all are trying to do, but there are no black and white lines drawn here. No *rule* applies in all situations. I laud the ALT.NET guys for trying to inject better skills and tools into the process, I just rail against the ferocity of zealotry that comes in some of the message. When enterprise developers hear this, they just tune it out instead of taking what helps them and leaving the rest.

That's just my opinion...

Comments:

Excellent perspective, Shawn. I wish more people would keep this in mind to be better grounded in reality.

I agree with 2 out of 3 of the counterarguments, but I don't think you quite understand "Well if they want access to the data outside the application, I will expose that as a service interface." I think you're battling the SOA crowd on this one, not the Alt.NET/DDD guys in particular.

A few somewhat related links:

Shared Database Integration (check out the downsides)
http://www.informit.com/articles/article.aspx?p=169483&seqNum=3

Data on the Outside vs. Data on the Inside, by Pat Helland
http://msdn.microsoft.com/en-us/library/ms954587.aspx

You're right, I didn't mean to shoe-horn the ALT.NET guys...or anyone in particular...

For me the reality is more based on what the realities and often the requirements in many corporate environments. My problem isn't whether doing things like reporting on the transactional schemas, but more that its going to happen and pretending that saying you *can't* do it will make a difference is ignoring a large segment of the community.

Thanks for the links...

Well put, Shawn; as a database developer, I too often feel stuck in the middle between the app developers and the DBA's. The app developers often joke about how DBA's will be unnecessary (as if the insight into data modeling and business requirements are unnecessary), and the DBA's are often dismissive of the app developers need for better access to the data that drives their applications.

The truth is that both roles need to evolve, but do so in a symbiotic fashion.

Stu

Hi Shawn,

I am not too sure I understand the problem from a DDD the data store and the model can be two separate things (through the use of repositories).

I tend to work on the assumption that DBA's know how to store and manipulate data best at their level - the db, and as a service developer I know how I want to work with the data, perhaps by DDD concepts - or perhaps not. This has worked well (for me at least in new and brownfield applications)

I don't think either top-down or bottom-up design works well. For instance I don't want my DB schema to affect my service design, likewise I know I don't want my schema to exactly my service layer.

You know, I have re-read your post a couple of times now, and I agree with you. It reflects the experiences that I have faced. There is one other facet from the database pov, there is a critical mass of data in the db where it is no longer feasible to refactor at the schema level and I suspect this is reached pretty quickly in the majority of enterprise apps.

Cheers,
Paul.

Twitter: PaulKinlan.


 



 
Save Cancel