The process cannot access the file because it is being used by another process Task Parallel Library

In one of the projects I am converting to .Net 4.0, I started getting the following error 'The process cannot access the file because it is being used by another process'. The program is processing a bunch of files in a set of folders. I am using .Net 4.0 Task Parallel Library features to handle these folders across different threads on the server. The server has 16 GB RAM and 24 cores.

The code that was causing the error is below

Parallel.ForEach(rangePartitioner,new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount}, (range,loopState) =>
{
     try
     {

         using (FileStream oFileStream = new FileStream(oFileInfo.FullName, FileMode.Open))

         using (GZipStream oGzipStream = new GZipStream(oFileStream, CompressionMode.Decompress))

         using (StreamReader oStreamReader = new StreamReader(oGzipStream, Encoding.ASCII))

         {

                   String sRawDataLine = oStreamReader.ReadLine();

                   while (sRawDataLine != null)

                   {

                           //Do Something();

                            sRawDataLine = oStreamReader.ReadLine();

                    }

           }

     }

     catch (Exception ex)

     {

       oLog.WriteLine("Error processing file name = {0} Exception {1}.", oFileInfo.FullName, ex.ToString());

     }

});

The "key" for the "locks" was to add the following options highlighted in the below code to the FileStream object so the file access is for read operation and that the file could be shared while reading.

Parallel.ForEach(rangePartitioner,new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount}, (range,loopState) =>
{
     try
     {
         using (FileStream oFileStream = new FileStream(oFileInfo.FullName, FileMode.Open,FileAccess.Read, FileShare.Read)))
         using (GZipStream oGzipStream = new GZipStream(oFileStream, CompressionMode.Decompress))
         using (StreamReader oStreamReader = new StreamReader(oGzipStream, Encoding.ASCII))
         {
                   String sRawDataLine = oStreamReader.ReadLine();
                   while (sRawDataLine != null)
                   {
                           //Do Something();
                            sRawDataLine = oStreamReader.ReadLine();
                    }
           }
     }
     catch (Exception ex)
     {
       oLog.WriteLine("Error processing file name = {0} Exception {1}.", oFileInfo.FullName, ex.ToString());
     }
});

I have not seen the error since adding these two options.The process no longer fails and executes on all the 24 cores on the server without any issues.

c# - Remove duplicate lines

A lot of the programs I have written are related to parsing files and converting them to tab delimited files for the data warehouse.The files range from few megabytes to gigabytes in size.In one of the processes I noticed that the output files had duplicate lines.I wanted to remove the duplicate lines from these files.The file sizes ranged from 2 GB to 6 GB.

Below is the c# program I wrote to remove the duplicates.Fortunately I have Windows 2008 server with 16 GB RAM so that helps :)

using System;
using System.Collections.Generic;
using System.Configuration;
using System.IO;
using System.Linq;

namespace DeDupeTextFiles
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                string sInputFilePath = ConfigurationManager.AppSettings["INPUT_FILEPATH"];
                HashSet<string> oDistinctLines = new HashSet<string>();
                using (StreamReader oReader = new StreamReader(sInputFilePath))
                {
                    string sLine = oReader.ReadLine();
                    while (sLine != null)
                    {
                        if (!oDistinctLines.Contains(sLine))
                            oDistinctLines.Add(sLine);

                        sLine = oReader.ReadLine();
                    }
                }
                            File.WriteAllLines(sInputFilePath.Replace(Path.GetFileName(sInputFilePath), Path.GetFileNameWithoutExtension(sInputFilePath) + "_Distinct.txt"), oDistinctLines.ToArray<string>());
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                Console.WriteLine(ex.StackTrace);
            }
            Console.Read();
        }
    }
}

App.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="INPUT_FILEPATH" value="E:\test_data\myfolder\file\lookups\201401\0.txt"/>
  </appSettings>
</configuration>

There are other ways to achieve the same using DOS batch commands or load files to database and distinct it out.The program took less than a minute to remove duplicates from 3 GB file.The final file was reduced to 180 MB !

java - BitConverter

I am currently converting one of the C# .Net program into a pig script that uses a java UDF.The program processes a million+ gz files into tab delimited text files. 
One of the functions in the C# .Net program is to compute the MD5 hash of a few fields in the gz file and convert the hash to a long.I did not find an equivalent function for BitConverter in Java.

Below is the C# code.

 private static long ComputeHash(string p)
 {
            MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();
            byte[] hash = md5.ComputeHash(Encoding.ASCII.GetBytes(p));
            return BitConverter.ToInt64(hash, 0);
 }

The Java equivalent is below

private long ComputeHash(String sHashString) 
{

MessageDigest md5 = MessageDigest.getInstance("MD5");

byte [] hash = md5.digest(sHashString.getBytes());

ByteBuffer buffer = ByteBuffer.wrap(hash);

buffer.order(ByteOrder.LITTLE_ENDIAN);

return buffer.getLong();

}

Below are the test results of Java program and C# .Net.The input is a column from the gz file and output is the string and its equivalent MD5 hash converted to long.Note that results on the left are from the PIG script that is using the Java UDF and the results on the right are from C# .Net program.

The remote server returned an error: (400) Bad Request.

In one of the projects that I was working on I started getting this error as soon as the site was hosted on the test server.
To give a brief background on the requirement, I had a site that was making a HttpWebRequest to the home page of another site to get the cookies.These cookies were then reused in subsequent requests.The sample code which was working on my dev machine is below

HttpWebRequest oRequest = (HttpWebRequest)WebRequest.Create(new Uri(sSiteHomeUrl));
oRequest.Method = "GET";
oRequest.ContentType = @"application/x-www-form-urlencoded";
oRequest.CookieContainer = oCookies;
oRequest.Accept = @"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
oRequest.UserAgent = @"CHROME browser version: 33.0 user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36";
oRequest.Headers["Accept-Encoding"] = @"gzip,deflate,sdch";
oRequest.Headers["Accept-Language"] = @"en-US,en;q=0.8,de;q=0.6,es;q=0.4";

string sResponseData = string.Empty;          
using(WebResponse oWebResponse = oRequest.GetResponse())
using (Stream data = oWebResponse.GetResponseStream())
using (StreamReader reader = new StreamReader(data))
{
      sResponseData = reader.ReadToEnd();
}

Once I tested it on my dev machine I moved the code and hosted the application on the test server.However on the test server I started getting this error.After a day of struggling and going through various forums I was able to fix the issue.

The problem was that both my site that was making a call to another site were on the same server under the same IIS. When creating the URL I was using the server name in the url.By replacing the servername with the Ip address I was able to successfully make the request from my site and get the cookies for subsequent requests.

Old call
http://servername:12348/Home.aspx

New call
http://ipaddressofserver:12348/Home.aspx

c# - Get all attribute values that exceeds int32 max value in an xml

I maintain an application that collects millions of xml files.I have parsers that then run to extract the information from these xml files.Recently we noticed that there was a drop in the number of xml files we were receiving.
We started seeing this error in the logs that caused the drop in the xml posts

System.OverflowException: Value was either too large or too small for an Int32.

So we grabbed one of the xml files that was failing and looked for any attribute that was causing the xml post to fail.The xml file size varies on the data and it was hard to figure out which attribute was causing the failure.
In order to figure this out I wrote a small program to list all the attributes that had a value which exceeded the Int32 value.Using this we were able to identify the data limits and fix it. This code can be tweaked to check for boolean,long,date,string size etc. Below is the code

using System;
using System.Xml.Linq;

namespace GetInvalidXmlIntElement
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                var oXmlDoc = XDocument.Load(@"c:\Data\Sample.xml");
                var oElements = oXmlDoc.Descendants();
                var iElementsCounter = 0;
                var iAttributeCounter = 0;
                var iInvalidValueCounter = 0;
                foreach (var oElement in oElements.Elements())
                {
                    iElementsCounter++;
                    foreach (var oAttribute in oElement.Attributes())
                    {
                        iAttributeCounter++;
                        try
                        {
                            if (Convert.ToInt64(oAttribute.Value) > Int32.MaxValue)
                                Console.WriteLine(" {0}) Element {1} Attribute {2} Int32 Max {3} Value {4} ",++iInvalidValueCounter,oElement.Name, oAttribute.Name, Int32.MaxValue, Convert.ToInt64(oAttribute.Value));
                        }
                        catch { }
                    }
                }
                Console.WriteLine();
                Console.WriteLine("Total Number of Elements in the Xml is:{0}", iElementsCounter);
                Console.WriteLine("Total Number of Attributes in the Xml is:{0}", iAttributeCounter);

            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
            Console.Read();
        }
    }
}

Task Scheduler 0x1

In one of the projects I am working on,I had to create a few scheduled tasks in the Task Scheduler.The server is running on Windows Server 2008 OS.I created the tasks but when it ran I saw the following 0x1 status in the last run result column of the task scheduler.

Neither the event viewer showed any error nor the history tab for the tasks.The solution was that I had not set the working directory for the task.After setting the working directory the tasks ran without any issues.

See below screenshots

Hope that helps

c# - Delete browser cookies,cache

 Directory.Delete(Environment.GetFolderPath(Environment.SpecialFolder.InternetCache),true);

UPDATE failed because the following SET options have incorrect settings: 'ARITHABORT'

While testing one of the web applications I built, I started getting this error on one of the stored procedures.The application was using 3 databases to manage data and the users would make changes to a local database and after reviewing the changes would move the changes to the staging and then an admin would move all the changes by the users to the final database.So,In my original test application setup,I had 3 databases i.e. local,staging,final on the same server and the stored procedure to update the staging from local database was working.

However when we moved the test database to a different server and modified the stored procedure to point to the new test server we received this error.

UPDATE failed because the following SET options have incorrect settings: 'ARITHABORT'. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.

The fix was to reconfigure the user options on the database that was moved to the new server

sp_configure 'user options', 64
reconfigure

More about the user options here http://technet.microsoft.com/en-us/library/ms190763.aspx

asp.net - Disable ASP.NET_sessionId

In one of the applications I developed we did not want to store any cookies for legal issues.Hence we had to disable all the cookies.While testing the application on the test servers I noticed the following Asp.Net_sessionId cookie being set.

More about it here on msdn (http://msdn.microsoft.com/en-us/library/ms178194.ASPX)

We still didn't want to use it even though it was non persistent since we were not allowed to create any cookies.One workaround is to disable the session state on IIS for the site.

Note:You will not be able to use Session if you do this.

 

SqlDependency must be called for each server that is being executed

Recently we rewrote our web applications in VS 2012 and .Net 4.0 and hosted it on IIS 7.For caching content I used the Sql Dependency and SQL service Service Broker.On test environment with load balancing we had two servers configured and everything was working as expected.

However when I moved the applications to production I started getting this error.

The only difference was the connectionstring which specified a failover server.

<add name="CONTENTDSN" connectionString="Data Source=Server1;Failover Partner=Server2;Initial Catalog=Database_Name;uid=UserId;pwd=password;"/>

The error was due to the failover server not having the database objects.Now when you use SQL dependency make sure that the failover server i.e Server2, has all the necessary database objects as the main server i.e. Server1.