In the past few years we've seen things like the apple app store and ios take off... with runaway sales figures and numbers most of us lowly software engineers could only dream of.
How does one dive in and build applications that target these new revenue streams? I aim to change the focus of the blog to cater to programming concepts targeted to mobile and HTML5.
I hope to bring to the table examples of functional mobile to server architectures, as well as building elegant user interfaces. What I wont be going into is the many code generators build "lowest common denominator" solutions for cross platform mobile apps. I hope time will permit and I can document building a full blown mobile app from the ground up for the up and coming android platform.
So.. if you were ever interested in how to achieve x in Android, WebOS, iOS, or how to take all these gold "nuggets" and put them together to build a cohesive mobile app architecture, stay tuned!
Tuesday, September 21, 2010
Friday, April 9, 2010
Data Layers: Do they make sense in their current form?
First of all, let me start off by saying this wasn't intended to be a mindless rant, with no merit or experience or thought behind comments and opinions made here.
To make things better we must first understand the problems at hand. Let us start off with the most basic of problems and follow the natural progress of software evolution to understand how things got to where they are today.
A list of very valid concerns
Consider the following very simple topology (a very typical database schema for two domain objects… a user and an orders table representing exactly those things).
Code above would just be in a "GetUserById(int id);" call.
Conceptually with persist, you provide a connection string in the web config, and mark objects up with attributes. One would argue that if you go that route you are introducing code into your business objects when they should be agnostic, which I agree with... however all the other mechanisms require at least a recompile. We also support an xml mapping provider, however feel the best balance is attribute mapping.
User.cs
Order.cs
Because objects, or factories are typically representing exactly one business objects type, we now have a restriction or a sort of dilemma on how exactly those objects and their factories relate to one another.
Typically if you wanted an user from the database, you'd create an instance of a user factory or call a static userfactory class. And likewise orders would be the same, with of course the use of the orderfactory. In this case how do we expect the relations between the two to work. Theoretically the factory shouldnt REALLY know about the existence of another factory and how to get instances out of the db.
Going against the grain, I'd say the mechanisms of separate factories may reflect the real world, but introduce unnecessary complexity and reduced ability to optimize HOW you are pulling data out of database. You dont necessarily need to go get a user, then go get each order. A well built ORM could know the ultimate goal is an object graph consisting of a user object and a list of that users orders and fetch them in one big query or batch.
Of course this is another gripe about data today and leads us into the next problem.
The point here being the data in the result set thats highlighted in gray... this is redundant data... and there is absolutely no need to return it more than once. Simply an artifact of how RDBMS's handle returning multiple related objects worth of data. This may be minor issue in most ORM's handling of data, but in large applications this has the potential to add a decent amount of wire time.
Say you wanted to take some search results and distinct the results. Ordering by distinct prompts you to specify the column you want to order by, in the select criteria (which in turn negates the distinct or at least potentially does).
This functionality makes sense in the case of aggregate queries but are often the "best" solution for some of the current day problems.
I think its important to stand back and look at problems in the bigger picture and try to engineer new solutions.
Any input or comments would be appreciated!
To make things better we must first understand the problems at hand. Let us start off with the most basic of problems and follow the natural progress of software evolution to understand how things got to where they are today.
A list of very valid concerns
- Performance - Data layer should be able to be as efficient as possible within reason
- Scalable - Will our system handle tables that are in the hundreds of rows to the billions of rows and be performant? Will the act of re-animating objects happen in a timely manner?
- DB Calls - While a single query is usually really quite performant, how many queries does it take to return an object graph
- Maintainable - Will our data layer solution be maintainable, and will new development or changes be something that can happen easily without too much work on our part?
- Portable - Will we be able to use different underlying storage mechanisms?
Consider the following very simple topology (a very typical database schema for two domain objects… a user and an orders table representing exactly those things).
Inline Queries
Probably the first thing a person learns to do regarding databases in code is to make very specific raw inline queries to a db and take the response and build objects from the resulting table. Said code in c# with sql server would like this:string sql=@"Select username, first,last,age from users where id = @id";
using(SqlConnection conn = new SqlConnection(connectionString))
using(SqlCommand command = new SqlCommand(sql,conn))
{
command.Parameters.AddWithValue("@userId",id);
conn.Open();
SqlDataReader reader = command.ExecuteReader();
User returnUser = null;
if(reader.Read())
{
//for simplicity sake we'll leave out necessary checking and dbnull handling etc.
returnUser = new User(reader["username,…..);
}
}
return returnUser;
For re-animating objects from the database this is actually very efficient code, and worked very well. Code was very specific to the objects you were dealing with, and in turn was not re-usable at all.- GOOD: very fast, custom sql was inherently supported, datatypes didn't matter whatsoever as every method was a one off.
- BAD: very bad for reuse, the methods were often copied/pasted and likewise had codebloat and duplication. Very hard to make global changes. Very slow for rapid prototyping and maintenance. Bad for loading object graphs.
Abstracted Methods
So from here, things go a little more abstract... people decided that since most queries across the app were for the same things, to abstract some of the querying into what you might consider a formal data layer. Note I dont call this abstract method.Code above would just be in a "GetUserById(int id);" call.
- GOOD: We're moving in the right direction with getting things abstract, starting to conform to DRY (don't repeat yourself) , and our calls are pretty centralized
- BAD: we are still repeating a lot of code and sql queries are very close with the exception of types you want back and how those objects are populated. Data providers are still very much tied to the database engine.
Modern Practices
This is where you'll find a million opinions on how you should move forward from here. I'd like to break ORM (or object relation mapping) down into a couple categories- Code generation
Code generators, DAL generators, etc are basically separate utilities that run and analyze domain objects and/or the database itself, take information it deduces and generate a ton of class files "UserFactory", "OrderFactory". Certainly calling UserFactory.GetOrderById(id) is a lot more pleasing than writing out full blown queries by hand.
- GOOD: potentially more efficient than true ORM, very quick to implement, if built right will not introduce limitations
- BAD: generating copies of boilerplate code, maintainability, duplication of code, often have data layer related methods, properties, events specific to data layer. Often these objects also require DTO objects to use in the real world.
- True ORM (simple take data storage and business objects and map two together.)
In theory a true orm should not generate code or classes to fetch objects, but rather opt for generic methods that fetch exactly the type of objects you want.
- GOOD: typically among the simplest of ORM's to implement and use, and can be a maintence dream
- BAD: If authored by inexperienced devs, and using reflection, reflected types will need to be cached. Reflection is pretty quick, but you certainly dont want to go and reflect a type each time you fetch.
So what's the right solution?
Well if I had to choose, I'd choose a "real orm". I've had minor participation in a lightweight orm that I believe has it right.Conceptually with persist, you provide a connection string in the web config, and mark objects up with attributes. One would argue that if you go that route you are introducing code into your business objects when they should be agnostic, which I agree with... however all the other mechanisms require at least a recompile. We also support an xml mapping provider, however feel the best balance is attribute mapping.
User.cs
[Persistent]
public class User
{
[PrimaryKey]
public int Id{get;set;}
[Field]
public string Username{get;set;}
[Field]
public string First{get;set;}
[Field]
public string Last{get;set;}
[OneToMany]
public List Orders{get;set;}
}
Order.cs
[Persistent]
public class Order
{
[PrimaryKey]
public int Id{get;set;}
[Field]
public double TotalPrice{get;set;}
[Field]
public OrderType OrderType{get;set;}
}
If you wanted to add a field at a later date, simply add a new field on the class and mark it up with field attribute and rebuild and you are done. If you use xml mapping and happen to have a field that was on business object but not mapped, you could immediately start using the new prop.So is that all?
Not exactly! This is kind of the whole point. We have these solutions that are far from perfect to solve a problem that shouldnt have existed in the first place. Because were mapping object or object graphs into a database, we've got a ton of inherent problems with any mechanism we should choose. Because of these fundamental problems, we've come to accept certain solutions as "best practices", when in-fact they are far reaching and affect the architecture of the app at a low level.Relations between two tables
Because objects, or factories are typically representing exactly one business objects type, we now have a restriction or a sort of dilemma on how exactly those objects and their factories relate to one another.
Typically if you wanted an user from the database, you'd create an instance of a user factory or call a static userfactory class. And likewise orders would be the same, with of course the use of the orderfactory. In this case how do we expect the relations between the two to work. Theoretically the factory shouldnt REALLY know about the existence of another factory and how to get instances out of the db.
Going against the grain, I'd say the mechanisms of separate factories may reflect the real world, but introduce unnecessary complexity and reduced ability to optimize HOW you are pulling data out of database. You dont necessarily need to go get a user, then go get each order. A well built ORM could know the ultimate goal is an object graph consisting of a user object and a list of that users orders and fetch them in one big query or batch.
Of course this is another gripe about data today and leads us into the next problem.
Object Graphs and their SQL Result equivalent
The point here being the data in the result set thats highlighted in gray... this is redundant data... and there is absolutely no need to return it more than once. Simply an artifact of how RDBMS's handle returning multiple related objects worth of data. This may be minor issue in most ORM's handling of data, but in large applications this has the potential to add a decent amount of wire time.
Aggregate Methods as a hack to every day problems
Many many things we do on an every day basis, are a result of working around shortcomings of database engines as they exist today. Just a few really little issues are outlined below...Say you wanted to take some search results and distinct the results. Ordering by distinct prompts you to specify the column you want to order by, in the select criteria (which in turn negates the distinct or at least potentially does).
This functionality makes sense in the case of aggregate queries but are often the "best" solution for some of the current day problems.
Conclusion
The current state of data storage and the data layers that sit on top of them... leave a lot to be desired. ORM's solve problems that probably shouldn't exist in the first place.I think its important to stand back and look at problems in the bigger picture and try to engineer new solutions.
Any input or comments would be appreciated!
Monday, March 15, 2010
A note on exceptions...
I've worked in a plethora of different work environments related to software engineering and it seems like everyone tends to have their own practices for diff things. Exceptions and exception handling is no exception.
What does that mean? Exceptional behavior is behavior that is not the normal. Eg, say part of your applications functionality connects to a website, downloads a file, disconnects and then process the file. What should happen when halfway through the download, the webserver is reset, or the connection is severed in some other manner?
Consider the following...
Depending on the expected functionality at a higher level, you need to decide how to handle it. If you want a failure here to result into a failure failure... dont bother catching... leave it as is. If you'd like to do something with the exception (like perhaps log), catch it, perform extra functionality, or perform extra functionality and then rethrow. If for example you wanted a failed get to result in a log entry, and setting the downloaded file to empty you might do something like.
In the catch statement you could have caught the dirty error, and replaced with an exception that exposed less, or was more friendly. It could have been a subclassed exception with yet even more information that the built in exceptions.
Say you wanted to just log and throw...
I've actually seen, and worked on many projects that actually used exceptions as validation components... which is improper use of Exceptions and in at least .net, aren't exactly the lightest objects to generate.
As you might guess, validation isn't exactly exceptional behavior. Generating a new exception here may save you have processing any further, and bubbling up validation failures... however its a decent tax in a high usage application. Generating an exception, also generates a stack trace among other things... and uses reflection for some of it.
You don't need to sit there and question usage of every last little thing in code, while in a relatively low usage application you'd probably never notice the overhead of throwing exceptions for validation, a little precaution here could definitely help down the road when you start seeing problems. Simpler design USUALLY ends up with cleaner, easier to maintain code.
In the event something does happen, you use finally to clean up resources.. but where it makes sense, a using statement will ensure the resources get cleaned up before moving on. This works out very well for things like drawing and database connections... so:
In the event you caught the exception, the using statement would ensure items that implement IDisposable are disposed properly. Not really specific to exceptions, but helpful to prevent memory leaks, as well as just make things like disconnecting connections pretty common sense.
Thats all for now.
Exceptions are designed to handle exceptional behavior
What does that mean? Exceptional behavior is behavior that is not the normal. Eg, say part of your applications functionality connects to a website, downloads a file, disconnects and then process the file. What should happen when halfway through the download, the webserver is reset, or the connection is severed in some other manner?
Consider the following...
string html = string.Empty;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com/");
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
html = readStream.ReadToEnd();
response.Close();
readStream.Close();
return html;
Depending on the expected functionality at a higher level, you need to decide how to handle it. If you want a failure here to result into a failure failure... dont bother catching... leave it as is. If you'd like to do something with the exception (like perhaps log), catch it, perform extra functionality, or perform extra functionality and then rethrow. If for example you wanted a failed get to result in a log entry, and setting the downloaded file to empty you might do something like.
string html = string.Empty;
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com/");
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
html = readStream.ReadToEnd();
response.Close();
readStream.Close();
}catch(Exception e)
{
//
}finally
{
//
}
//this may very well be empty if the url is bad, or has changed.
return html;
In the catch statement you could have caught the dirty error, and replaced with an exception that exposed less, or was more friendly. It could have been a subclassed exception with yet even more information that the built in exceptions.
Say you wanted to just log and throw...
string html = string.Empty;
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com/");
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
html = readStream.ReadToEnd();
response.Close();
readStream.Close();
}catch(Exception e)
{
//
throw;
}
//this may very well be empty if the url is bad, or has changed.
return html;
Exceptions aren't for validation!
I've actually seen, and worked on many projects that actually used exceptions as validation components... which is improper use of Exceptions and in at least .net, aren't exactly the lightest objects to generate.
if(name.Length >10)
throw new InvalidLengthException("Name must be at least 3 characters long");
As you might guess, validation isn't exactly exceptional behavior. Generating a new exception here may save you have processing any further, and bubbling up validation failures... however its a decent tax in a high usage application. Generating an exception, also generates a stack trace among other things... and uses reflection for some of it.
You don't need to sit there and question usage of every last little thing in code, while in a relatively low usage application you'd probably never notice the overhead of throwing exceptions for validation, a little precaution here could definitely help down the road when you start seeing problems. Simpler design USUALLY ends up with cleaner, easier to maintain code.
An additional note on disposables
In the event something does happen, you use finally to clean up resources.. but where it makes sense, a using statement will ensure the resources get cleaned up before moving on. This works out very well for things like drawing and database connections... so:
try{
using(SqlConnection conn = new SqlConnection(connectionString))
using(SqlCommand command = new SqlCommand(query,conn))
{
conn.Open();
command.ExecuteNonQuery();
conn.Close();
}
}catch(Exception e)
{
//do something
}
In the event you caught the exception, the using statement would ensure items that implement IDisposable are disposed properly. Not really specific to exceptions, but helpful to prevent memory leaks, as well as just make things like disconnecting connections pretty common sense.
Thats all for now.
Wednesday, February 24, 2010
Why bother with Object Oriented Programming: Part 1
I think it goes without saying that object oriented programming has become probably THE choice people tend to go with, as far as building software goes. Why? Why do people choose to go the OO route?
With object oriented programming you have it easier in planning phase in the sense its fairly easy to model real world objects and real world actions, and makes understanding user requirements, and test cases and how that relates to the actual software a little easier.
In addition to that, building software is enhanced with better reliability, extensibility, maintainability, reusability, and testability. It also helps break projects up into smaller components, to distribute development tasks.
So, for most things, OO software makes a whole lot of sense, and this is reflected in the fact that nearly every programming language supports it at the foundational level, or in some cases as a sort of extension to the base programming language.
OO is widely taught, widely suggested, widely supported, and has been widely adopted, and yet I still see so many cases in every day experience, where understanding of object oriented programming falls short.
When should we use an abstract class vs a simple base class vs an interface and its implementors? When should objects or members be marked as private/protected/public/internal etc? What is polymorphism and why is it important?
These are exactly the sort of questions I hope to share *MY* opinions about in this series with real world, understandable projects.
Until next time... ;-)
With object oriented programming you have it easier in planning phase in the sense its fairly easy to model real world objects and real world actions, and makes understanding user requirements, and test cases and how that relates to the actual software a little easier.
In addition to that, building software is enhanced with better reliability, extensibility, maintainability, reusability, and testability. It also helps break projects up into smaller components, to distribute development tasks.
So, for most things, OO software makes a whole lot of sense, and this is reflected in the fact that nearly every programming language supports it at the foundational level, or in some cases as a sort of extension to the base programming language.
OO is widely taught, widely suggested, widely supported, and has been widely adopted, and yet I still see so many cases in every day experience, where understanding of object oriented programming falls short.
When should we use an abstract class vs a simple base class vs an interface and its implementors? When should objects or members be marked as private/protected/public/internal etc? What is polymorphism and why is it important?
These are exactly the sort of questions I hope to share *MY* opinions about in this series with real world, understandable projects.
Until next time... ;-)
Balance in software design
Firstly... please bare with me, I'm an engineer at heart, and sometimes have difficulty composing my ideas and opinions. I'll get better as I go I promise =)
One thing I've come to realize in at least the engineering positions I've held so far, is that it seems like most employers suffered from one of the following two conditions:
Is it going to be ok to "fly by the seat of your pants" for x feature? Do we need to implement the provider model for every chunk of functionality?
I don't mean to make light of the topic of data abstraction or data providers, but I've seen all too many companies commit to building out a suite of providers for the data layers and then in the entire 10 yr lifespan of the application... it never changes. In addition, a lot of times the implementations of one servers client functionality is so different that implementing truly pluggable providers is something that will require a significant effort.
My point isn't any one specific aspect of software design, or that you should choose to lean one direction or another, but rather to suggest that software design is an art, a learned skill... and should never be a hard, predefined path.
One thing I've come to realize in at least the engineering positions I've held so far, is that it seems like most employers suffered from one of the following two conditions:
- Under engineering and under planning - solutions tend to start out as ideas that were never fully fleshed out, proper planning did not take place, and likewise the implementation of the idea never appears to end up more than a glorified proof of concept.
- Over engineering and over planning - solutions undergo huge discovery, planning efforts, use case modeling, weeks of meetings etc. Typically the project consists of a good plan, and good intentions. Ultimate results depend, however frequently are over due, over budget, and tend to put robustness over performance.
Is it going to be ok to "fly by the seat of your pants" for x feature? Do we need to implement the provider model for every chunk of functionality?
I don't mean to make light of the topic of data abstraction or data providers, but I've seen all too many companies commit to building out a suite of providers for the data layers and then in the entire 10 yr lifespan of the application... it never changes. In addition, a lot of times the implementations of one servers client functionality is so different that implementing truly pluggable providers is something that will require a significant effort.
My point isn't any one specific aspect of software design, or that you should choose to lean one direction or another, but rather to suggest that software design is an art, a learned skill... and should never be a hard, predefined path.
Thursday, February 18, 2010
Back in Business!
Its been quite a long time since I thought blogging was important enough to take time out of my day, but its becoming more and more apparent that for my sanity, I need to vent... even if its to 0 readers =)
Expect to hear more soon.
Expect to hear more soon.
Saturday, September 22, 2007
Central Coast Code Camp
I will speaking on the subject of Enterprise Data Layer design for the '07 Central Coast Code camp in San Luis Obispo, CA.
I will be speaking in session 2, at 2:30 time slot on Sunday 9/23/2007.
My session on Data Layers will be the starting point of a series of articles and Visual Studio projects that demonstrate Enterprise Data Layers etc.
The code samples will need a bit of polish before I am ready to just put them out there, however you should be able to expect them within a few days. I will also be publishing detailed articles on http://www.codeproject.com/ so keep your eyes opened =)
I will be speaking in session 2, at 2:30 time slot on Sunday 9/23/2007.
My session on Data Layers will be the starting point of a series of articles and Visual Studio projects that demonstrate Enterprise Data Layers etc.
The code samples will need a bit of polish before I am ready to just put them out there, however you should be able to expect them within a few days. I will also be publishing detailed articles on http://www.codeproject.com/ so keep your eyes opened =)