Running totals in SQL Server

One of the most asked question of the age is the "running total problem" of SQL. When you have to generate a running total of a SQL Server table.


The Problem:
The problem which arrives is this that the user wants to calculate the running totals for a particular column in the database to be either used in a report or for reference to be stored in a column and used later on while querying the database.


Let us suppose the user has this kind of table....

ID	Date	Net Payment Due	Running Total
1	2011-08-19 12:23	10.17
2	2011-08-19 12:32	45.78
3	2011-09-14 15:03	6.80
4	2011-09-14 15:07	6.77
5	2011-09-17 11:14	0.00
6	2011-09-17 13:51	6.75
7	2011-09-19 10:27	4.25

The fourth column "RunningTotal" is what the user wants. He wants a running total of the corresponding payments defined over a period of time, so that he can easily see the Running Total of the Payments over a period of time.


The Solution:
There could be many solutions for this issue.

• Modern day reporting tools give us the flexibility to calculate the running totals within the reports. This way the user won't have to calculate the running totals in the database.
• Keep running totals in a Computed Column in a database and simply query that column whenever required.
• Calculate the Running Totals in the same query whenever required.

First and the third scenarios are used when the user (in this case, the developer) only wants the running total to be calculated for a very few times and he would like to calculate the running totals only when is needed.

One should use the second case only when there is frequent need of the Running Total. The running total, thus would be stored in the database and the developer could access the running totals through a simple query of the column. As we all know, that computed columns have their drawbacks. I would strongly recommend to use the third solution.

The Options:
There are different options available for us to use when calculating running totals using SQL Server....

1. Nested Sub Query
2. Self Join
3. Cursor

Let us see those options one by one...

1. Using Nested Sub Query:

We will use a sub query to calculate sub totals for a result set while fetching the result set in parallel. The query would be like this...

SELECT  ID ,
DataEntryDateTime ,
NetPaymentDue ,
NetPaymentDue + COALESCE((SELECT    SUM(NetPaymentDue)
 FROM      dbo.Payment b
 WHERE     b.id < a.ID
), 0) AS RunningTotal
FROM    dbo.Payment a

Here, the key to the solution is the little 'WHERE' clause, which is restricting only the previous rows to be fetched from the database. 


Note: If your ID is not the primary key, You need to add an 'ORDER BY' at the end of the outer query to sort out the results as well.


2. Using Self Join:
In this method we will use a 'JOIN' statement to join to the same table and will produce the same results. Here is how...

SELECT  a.ID ,
a.NetPaymentDue ,
SUM(b.NetPaymentDue)
FROM    dbo.Payment a
INNER JOIN dbo.Payment b ON (b.ID <= a.ID)
GROUP BY a.ID , a.NetPaymentDue
ORDER BY a.ID , a.NetPaymentDue

If you want the 'Date' column to appear in the select list as well, you will have to put the above query in the sub-query and join to the same table one more time to do it...


3. Using Cursor:
The third option is to use the cursor, which (to me) seems to be the best possible solution for this kind of problem. This also proves the point that cursors are not always slow... :-)

DECLARE @TempTbl TABLE
(
ID SMALLINT ,
NetPaymentDue MONEY ,
RunningTotal MONEY
)
 
DECLARE @ID SMALLINT ,
@NetPaymentDue MONEY ,
@RunningTotal MONEY
 
SET @RunningTotal = 0
 
DECLARE rt_cursor CURSOR
FOR
SELECT ID, NetPaymentDue
FROM dbo.Payment
ORDER BY ID
 
OPEN rt_cursor
 
FETCH NEXT FROM rt_cursor INTO @ID, @NetPaymentDue
 
WHILE @@FETCH_STATUS = 0 
BEGIN
SET @RunningTotal = @RunningTotal + @NetPaymentDue
INSERT  @TempTbl
VALUES  (@ID, @NetPaymentDue, @RunningTotal)
FETCH NEXT FROM rt_cursor INTO @ID, @NetPaymentDue
END
 
CLOSE rt_cursor
DEALLOCATE rt_cursor
 
SELECT  *
FROM    @TempTbl

Like the post.... Give it a Kick......

Major reasons for software failure

There are often the cases, when the software is not shipped on time or even if it is shipped, it is not shipped with the quality that is expected of it, resulting in an eventual failure of the project.


A research conducted by market research firm Info-Tech Research Group says 95 per cent of information technology groups "are not delivering some number of projects on time or to the full satisfaction of the business executive."


While there may be many reasons for this, the major reasons involve the following:


Poorly defined scope:
In most of the cases, the reason is not the clarification of the requirements.

• It may be because of an unreasonable deadline from sales/management
• It may be due to overlooking the customer's expectations or it could have been preconceived notions from the start
• It could have been a developer promising a particular UI widget (The "gee whiz" factor) or it could have been a developer not challenging the inclusion of a particular UI widget (which, the customer, wants desperately)
• It could be that the projected savings were wrong or it could be that an important scope change was delayed until the next version

In the end, it all boils down to the user/customer expectations regarding a software. If you have met the customer's requirements, you have a successful software.

My friend, Hassan, has a very good post here, regarding the requirements gathering of a software project which can be a good reading source.

Unmotivated Developers:

In some of the cases, the software failure arrives because of a lack of interest or respect towards the software from some or whole of the development team. It may be due to the following reasons...

• Development team is not motivated enough for the project
• The hype against the project is not created before the arrival of the project to keep the team thinking about the project
• Lack of interest from the authorities/management in the project sometimes result in the lack of interest from the development team

Lack of Domain Knowledge:

Sometimes, it is the case when developers are assigned tasks and they have not enough knowledge about the project or about the organization for which they are developing the project.

• One of the reason could be that the development team has not been given much time to do R & D for the project
• The development team hired for the project has not enough background experience for related developments
• The development team hired for the project is incompetent for producing the goods on time

In the end, I think it all concludes to the fact that the customer satisfaction is the first and fore-most thing and all the above factors contribute to the satisfaction (or dissatisfaction) of the customer. 

As a project manager, we must look into these factors before going ahead for the development of the project.

We can minimize the risks mentioned above by doing the following simple tasks:

1. Focus on features not implementation tasks.
2. Work in iterations (like weekly or biweekly).
3. Release working features to your staging environment in order of priority.
4. Unit test your code as you go, so you're not slowed down by a bug-list that increases geometrically as you approach the release date.
5. Be prepared to cut scope from the less important features. Stuff always takes longer than you think it will.
6. Make sure you sketch out the UI in advance (if there is a UI), and show it to potential users.
7. Test, test, and test some more. This seems counter-intuitive, but it saves more time than takes.

Deployment Windows 2003 IIS Problem 404 Error

This is a normal problem with Deployments, whenever people deploy their ASP.Net application on windows 2003 server. They face this problem. A 404 (Page Not Found) Error is shown.

This error normally arise when the sequence of installation is wrong.

Following steps must be taken to prevent the error to arise:

1. make sure that the asp.net is registered with the IIS. You can register the asp.net with the IIS using aspnet_regiis.exe using '-i' switch. The aspnet_regiis.exe is located at <%WIN%>\Microsoft.Net\Framework\ folder
2. It is often seen that registering the asp.net with IIS has no effect on the error mentioned above. In that case, you must make sure that the corresponding version in the Web Service Extensions in the IIS Manager is marked as 'Allowed'.

You can achieve this by doing the following steps:

• Point to Start -> Run
• Type inetmgr and then press enter. This will open the IIS Console on the server
• Expand the server node and then click the 'Web Service Extensions' folder
• Make sure that the corresponding ASP.Net version in the right pane is marked as 'Allowed'.
• If not, select the corresponding version of asp.net in the right pane and click 'Allow' button.

Now try browsing the website again. It should run as a charm.

Creating a Facebook like Add Contacts type ahead (auto complete) control in ASP.Net

It once came out to me that I wanted to create a facebook like add contacts control where you can add contacts in the messages section of the face book by using .net framework's auto complete feature.

Just wanted to share this, so that anyone working on .Net and wanted to have a control like that would work on anything like this.

I created a User Control in the application and modified it according to my needs. You may modify any functionality you want for this user control.

Main Theme:

Main Theme is that we need to create a panel. Inside the panel, there should be one textbox with autocomplete extender and one Label for the text box and one repeater control to add the elements, that have been suggested by the user control....

Source code of the control would be like this:

ASCX Page:

<%@ Control Language="C#" AutoEventWireup="true" CodeFile="MultiAutoComplete.ascx.cs"
Inherits="UC_MultiAutoComplete" %>
<table id="tblMain" runat="server" cellpadding="0" cellspacing="0" border="0" align="center">
<tr class="dataEntryTR">
<td>
<asp:Label ID="lblLabel" runat="server"></asp:Label>
</td>
<td>
<asp:Panel ID="pnlMultiAutoComplete" runat="server">
<table cellpadding="0" cellspacing="0" border="0">
<tr>
<td class="MultiAutoComplete" onfocus="document.getElementById('<%= txtAC.ClientID %>').focus();">
<asp:Repeater ID="rptDB" runat="server" OnItemDataBound="rptDB_ItemDataBound">
<ItemTemplate>
<div class="MultiACdiv" style="width: <%# (DataBinder.Eval(Container.DataItem, "Name").ToString().Length * 10) %>px;">
<%# DataBinder.Eval(Container.DataItem, "Name") %>&nbsp;<asp:ImageButton ID="ibtnCross"
runat="server" AlternateText="del" ImageAlign="absmiddle" ImageUrl="~/images/Buttons/cross_button.jpg"
BorderWidth="0" ToolTip='<%# DataBinder.Eval(Container.DataItem, "MainID") %>'
EnableTheming="false" OnClick="ibtnCross_Click" CausesValidation="false" />
</div>
</ItemTemplate>
</asp:Repeater>
<ajaxToolkit:AutoCompleteExtender ID="AutoCompleteExtender1" runat="server" TargetControlID="txtAC"
MinimumPrefixLength="2" CompletionInterval="1000" CompletionListCssClass="autocomplete_completionListElement"
CompletionListItemCssClass="autocomplete_listItem" CompletionListHighlightedItemCssClass="autocomplete_highlightedListItem"
EnableCaching="true" DelimiterCharacters=";">
</ajaxToolkit:AutoCompleteExtender>
<input type="hidden" id="txtID" runat="server" value='<%# _ControlID %>' />
<asp:TextBox ID="txtAC" runat="server" OnTextChanged="txtAC_TextChanged" AutoPostBack="true"
SkinID="multiACtxt"></asp:TextBox>
</td>
</tr>
</table>
</asp:Panel>
</td>
</tr>
</table>

CS Page:

using System;
using System.Data;
using System.Web.UI.WebControls;
using System.ComponentModel;

public partial class UC_MultiAutoComplete : System.Web.UI.UserControl
{
#region Private Variables
private DataTable _DataSource;
private string _DeleteConfirmText;
private string _AutoCompleteServiceMethod;
protected string _ControlID;
#endregion

#region Properties
public string width
{
get { return tblMain.Style["width"].ToString(); }
set { tblMain.Style.Add("width", value); }
}
public string height
{
get { return tblMain.Style["height"].ToString(); }
set { tblMain.Style.Add("height", value); }
}
public string align
{
get { return tblMain.Attributes["align"].ToString(); }
set { tblMain.Attributes.Add("align", value); }
}
public string LabelText
{
get { return lblLabel.Text; }
set { lblLabel.Text = value; }
}
///

/// Gets or Sets the Datasource of the repeater.
/// Must be a DataTable object with one column of name "Name"
///

public DataTable DataSource
{
get { return _DataSource; }
set { _DataSource = value; }
}
public string DeleteConfirmText
{
get { return _DeleteConfirmText; }
set { _DeleteConfirmText = value; }
}
public string AutoCompleteServiceMethod
{
get { return _AutoCompleteServiceMethod; }
set { _AutoCompleteServiceMethod = value; }
}
public string ControlMainID
{
get { return _ControlID; }
set { _ControlID = value; }
}
#endregion

public delegate void MultiAutoCompleteEventHandler(object sender, MultiAutoCompleteEventArgs e);
public event MultiAutoCompleteEventHandler MAC_TextChanged;
protected virtual void MAC_OnTextChanged(MultiAutoCompleteEventArgs e)
{
MAC_TextChanged(this, e);
}

public event MultiAutoCompleteEventHandler MAC_Delete;
protected virtual void MAC_OnDelete(MultiAutoCompleteEventArgs e)
{
MAC_Delete(this, e);
}

protected void Page_Init(object sender, EventArgs e)
{
AutoCompleteExtender1.ServiceMethod = _AutoCompleteServiceMethod;
}

protected void Page_Load(object sender, EventArgs e)
{
txtAC.Focus();
}

public override void DataBind()
{
base.DataBind();
if (_DataSource == null)
{
//MessageBox.Show("Data Source is not assigned...", Page);
return;
}
rptDB.DataSource = _DataSource;
rptDB.DataBind();
txtAC.Text = "";
}

protected void ibtnCross_Click(object sender, EventArgs e)
{
ImageButton ibtn = (ImageButton)sender;
MAC_OnDelete(new MultiAutoCompleteEventArgs(txtID.Value, ibtn.ToolTip, txtAC.Text));
}

protected void rptDB_ItemDataBound(object sender, RepeaterItemEventArgs e)
{
if (e.Item.ItemType != ListItemType.Header && e.Item.ItemType != ListItemType.Footer)
{
ImageButton ibtn = (ImageButton)e.Item.FindControl("ibtnCross");
ibtn.Attributes.Add("onclick", "return window.confirm('" + _DeleteConfirmText + "');");
}
}

protected virtual void txtAC_TextChanged(object sender, EventArgs e)
{
MAC_OnTextChanged(new MultiAutoCompleteEventArgs(txtID.Value, txtAC.Text));
}
}

#region class MultiAutoCompleteEventArgs
public class MultiAutoCompleteEventArgs : EventArgs
{
#region Private Variables
private string _ID;
private string _ItemID;
private string _Text;
#endregion

#region Properties
public string ID
{
get { return _ID; }
}
public string ItemID
{
get { return _ItemID; }
}
public string Text
{
get { return _Text; }
}
#endregion

public MultiAutoCompleteEventArgs(string __ID, string __Text) { _ID = __ID; _Text = __Text; }
public MultiAutoCompleteEventArgs(string __ID, string __ItemID, string __Text) { _ID = __ID; _ItemID = __ItemID; _Text = __Text; }
}
#endregion

Garbage Collector Internal Mechanism

In Dot Net Programming, developers don't often do garbage collection on their own, because they rely on the features provided by dot net for garbage collection. A common usage of garbage collection in dot net is when a programmer explicitly invokes GC.Collect() method. The presence of Garbage Collector frees the programmer of any worries about dangling data.

Let us start with the cause which led us to the effect of garbage collection in Dot Net. We know necessity is the mother of all inventions. So what was the necessity to invent garbage collection?

Why Garbage Collector?

Those people who have been writing code in unmanaged environment had to explicitly release the memory when they had created a new reference to any variable, they had to free it up again in their own code. Failure in doing so resulted in un expected behavior of system (bugs) or memory leakages (the data that has been allocated by a program and has not been de-allocated later in the program -- This is what we call "Garbage Data"). The bugs may be due to you are trying to access some part of memory that has already been freed by your program earlier.

What is Garbage Collector?

Garbage Collector normally runs in a low-priority thread and checks for unreferenced dynamically allocated memory space. If it finds some data that is no longer being referenced by any variable/reference, it re-claims it and returns the occupied memory back to the operating system, so that it can be used by any other program. The garbage collector is optimized to perform the memory free-up at the best time based upon the allocations being made.

How Garbage Collector Works

Every application has a set of roots. Roots identify storage locations, which refer to objects on the managed heap or to objects that are set to null.

For example:

• All the global and static object pointers in an application.
• Any local variable/parameter object pointers on a thread's stack.
• Any CPU registers containing pointers to objects in the managed heap.
• Pointers to the objects from Freachable queue

The list of active roots is maintained by the just-in-time (JIT) compiler and common language runtime (CLR), and is made accessible to the garbage collector's algorithm..

Garbage Collection in dot net is done using tracing collection and specifically the CLR implements the mark/compact collector. This is done in two phases:

In the first phase, When the garbage collector starts running, it makes the assumption that all objects in the heap are garbage. the garbage collector then checks for all the roots and live references in the application and then it starts walking the roots and start making a graph of all the references it finds in the application, avoiding any circular references found in the application. Thus, identifying all the live objects/references in the application. All the objects except the objects in this graph are identified as dead objects and are therefore considered as garbage data and are thus MARKED.

In the second phase, the GC moves all of the live objects/references down the heap, and re assigns the pointers so that the pointers now (again) point to the correct locations. This is the phase which is called COMPACTING the data. After all garbage data has been MARKED and all live references have been COMPACTED, the garbage collector now point just after the last object in the managed heap, so that if the program requires any new memory, it would allocate the memory on this position.

Finalization:

The garbage collector offers an additional feature that you may want to take advantage of: finalization. Finalization allows a resource to gracefully clean up after itself when it is being collected. By default the garbage collector frees up only the memory allocated for managed resources, not the memory allocated for unmanaged resources like if, someone has used a file in his/her program then he/she will have to deallocate the memory explicitly. For this purpose, one has to override the Finalize() method of the class. The Finalize method of the class in dot net is not anyway like the destructor in C++. One need to keep this in mind that the destructor in dot net translates to Finalize() when the code is compiled into IL. Thus, the behavior of destructor is never achievable in dot net.

In such a scenario, what will the GC do? Once the Garbage Collector has identified that the object is garbage, it would execute its Finallize() method and would promote it to next generation. (I will discuss the Generations and Promotions in another article here), for now it would be enough to keep in mind a brief detail about generations. Briefly, a generation is a mechanism implemented by the garbage collector in order to improve performance. The idea is that newly created objects are part of a young generation, and objects created early in the application's lifecycle are in an old generation. Separating objects into generations can allow the garbage collector to collect specific generations instead of collecting all objects in the managed heap.

When designing objects it is best not to use the Finalize method. There are several reasons for this:

• Finalizable objects get promoted to older generations, which increases memory pressure and prevents the object's memory from being collected when the garbage collector determines the object is garbage. In addition, all objects referred to directly or indirectly by this object get promoted as well.
• Finalizable objects take longer to allocate, because when we allocate an object that is finalizable as well then the object is supposed to not only allocate memory for itself but also make an entry in the Finalization Queue, from which GC checks whether the object is finalizable or not.
  
• Forcing the garbage collector to execute a Finalize method can significantly hurt performance. Remember, each object is finalized. So if I have an array of 10,000 objects, each object must have its Finalize method called.
• Finalizable objects may refer to other (non-finalizable) objects, prolonging their lifetime unnecessarily. In fact, you might want to consider breaking a type into two different types: a lightweight type with a Finalize method that doesn't refer to any other objects, and a separate type without a Finalize method that does refer to other objects.
• You have no control over when the Finalize method will execute. The object may hold on to resources until the next time the garbage collector runs.
• When an application terminates, some objects are still reachable and will not have their Finalize method called. This can happen if background threads are using the objects or if objects are created during application shutdown or AppDomain unloading. In addition, by default, Finalize methods are not called for unreachable objects when an application exits so that the application may terminate quickly. Of course, all operating system resources will be reclaimed, but any objects in the managed heap are not able to clean up gracefully. You can change this default behavior by calling the System.GC type's RequestFinalizeOnShutdown method. However, you should use this method with care since calling it means that your type is controlling a policy for the entire application.
• The runtime doesn't make any guarantees as to the order in which Finalize methods are called. For example, let's say there is an object that contains a pointer to an inner object. The garbage collector has detected that both objects are garbage. Furthermore, say that the inner object's Finalize method gets called first. Now, the outer object's Finalize method is allowed to access the inner object and call methods on it, but the inner object has been finalized and the results may be unpredictable. For this reason, it is strongly recommended that Finalize methods not access any inner, member objects.

If you have no other escape and you want to implement the Finalize method in your class then make sure that your Finalize method executes quickly and does not acquire any more resources and it should not do any action that would block Finalize method, like any thread synchronization problem. Also, if you let any exception escape the Finalize method the system would assume that Finalize method has returned without any problem and continue Finalization of other objects.

Important Questions:

What is Finalization Queue?
When an application creates a new object, the new operator allocates the memory from the heap. If the object's type contains a Finalize method, then a pointer to the object is placed on the finalization queue. The finalization queue is an internal data structure controlled by the garbage collector. Each entry in the queue points to an object that should have its Finalize method called before the object's memory can be reclaimed.

What is FReachable Queue?
When a GC occurs, objects B, E, G, H, I, and J are determined to be garbage. The garbage collector scans the finalization queue looking for pointers to these objects. When a pointer is found, the pointer is removed from the finalization queue and appended to the freachable queue (pronounced "F-reachable"). The freachable queue is another internal data structure controlled by the garbage collector. Each pointer in the freachable queue identifies an object that is ready to have its Finalize method called.

Who is responsible for calling Finalize Method?
There is a special runtime thread dedicated to calling Finalize methods. When the freachable queue is empty (which is usually the case), this thread sleeps. But when entries appear, this thread wakes, removes each entry from the queue, and calls each object's Finalize method. Because of this, you should not execute any code in a Finalize method that makes any assumption about the thread that's executing the code. For example, avoid accessing thread local storage in the Finalize method.

What is Resurrection?
When an application is no longer accessing a live object, the garbage collector considers the object to be dead. However, if the object requires finalization, the object is considered live again until it is actually finalized, and then it is permanently dead. In other words, an object requiring finalization dies, lives, and then dies again. This is a very interesting phenomenon called resurrection. Resurrection, as its name implies, allows an object to come back from the dead.