Sunday, December 19, 2010

FormatBytes() - How big is that?

A handy function for formatting bytes, such as disk or file sizes.

1,024 bytes = 1 KB
1,048,576 bytes = 1 MB
1,073,741,824 bytes = 1 GB
1,099,511,627,776 bytes = 1 TB
1,125,899,906,842,620 bytes = 1 PB
1,152,921,504,606,850,000 bytes = 1 EB
1,180,591,620,717,410,000,000 bytes = 1 ZB

Kilobyte
Megabyte
Gigabyte
Terabyte
Petabyte
Exabyte
Zetabyte

To format a byte count as a string, here’s a quick little routine to do the trick.

   1:      Public Function FormatBytes(ByVal nBytes As Double, Optional ByVal szFormatString As String = "###,###,###,##0") As String
   2:   
   3:          Dim POSTFIXES() As String = {"Bytes", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"}
   4:          For n As Integer = POSTFIXES.Length - 1 To 0 Step -1
   5:              Dim nPow As Double = Math.Pow(1024, n)
   6:              If nBytes >= nPow Then
   7:                  nBytes /= nPow
   8:                  Return nBytes.ToString(szFormatString) & " " & POSTFIXES(n)
   9:              End If
  10:          Next
  11:          Return nBytes.ToString(szFormatString) & " Bytes"
  12:   
  13:      End Function

Wednesday, December 15, 2010

Using Array.Copy to shift array elements

I wrote a network communications class that will perform socket reads into a pre-allocated Byte buffer.  This buffer is always the target of socket reads, so that I can avoid doing (perhaps many) unnecessary memory allocations.  For this byte buffer, I need to keep track of where “active” data in the buffer lives.  Initially there is no active data, so both START and END equal zero.  Later on, however, START and END may not be pointing to the beginning of the buffer and I need to SHIFT the active bytes to the front of the buffer and adjust the START and END pointers accordingly.  This way the next socket read can read “BUFFER_SIZE – END” bytes.

But how do we shift the bytes in the array down?  Some might create a new array and use Array.Copy to copy the active bytes over.  But we really want to use the same buffer, so we just want to shift the active bytes down to element 0.  Will Array.Copy do this correctly?  Doing a “shift left” (moving elements towards the beginning of the array) requires that the bytes are shifted in ascending array-index order.  If you go the other way, descending, you’ll clobber your data if there is any overlap.  Similarly, you need to shift in descending order when you are doing a “shift right” and moving elements up the array.

In order for Array.Copy to work correctly in both cases, it needs to detect that when there is an overlap between the source and target range and then move the elements in the proper order.  Does it do this?  Well.. YES!

In the example below, I take an array of bytes with 13 elements and shift the middle 9 bytes left; then I do it again, but shift right.

Here’s the output:

image

Here’s the code:

 
Imports System.Text
 
Module Module1
 
    Sub Main()
 
        Console.WriteLine("Testing Array.Copy on Byte array to shift contents up and down:" & vbCrLf)
 
        Dim A() As Byte = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} 
        Console.WriteLine("FROM A(" & (A.Length() - 1).ToString() & ") = " & BitConverter.ToString(A))
        Console.WriteLine()
        Console.WriteLine("Array.Copy(A, 2, A, 0, 9)")
        Console.WriteLine()
        Array.Copy(A, 2, A, 0, 9)
        Console.WriteLine("   = A(" & (A.Length() - 1).ToString() & ") = " & BitConverter.ToString(A))
 
        A = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
        Console.WriteLine()
        Console.WriteLine("FROM A(" & (A.Length() - 1).ToString() & ") = " & BitConverter.ToString(A))
        Console.WriteLine()
        Console.WriteLine("Array.Copy(A, 2, A, 4, 9)")
        Console.WriteLine()
        Array.Copy(A, 2, A, 4, 9)
        Console.WriteLine("   = A(" & (A.Length() - 1).ToString() & ") = " & BitConverter.ToString(A))
 
    End Sub
 
 
End Module

Sunday, December 5, 2010

Tracing Library/DLL calls on Win32

I have searched for a program to verify all the sectors of a hard drive, to detect errors.  Obviously, formatting the drive is one method, but I actually found that the format did not pick up bad sectors that I knew were on the hard drive.  After some searching I found a program called “HD Tune” which has an option to perform an “Error Scan” on a disk.  Using this utility, it successfully detected the error spots on the disk.

image

It would be nice to write such a utility myself.  But how?  Ideally I would be able to detect all the mounted disks on the system, determine the size and geometry, determine the number of sectors on the disk and perform a Read operation on each sector- looking for errors.  I would imagine the KERNEL32 function DeviceIoControl() will play a major part of it.

A quick investigation did not turn up anything promising.  So how does HD Tune do it?  Good question.  I never really played around with any sort of process spying utilities, at least not in a long time, so I fired up Spy++.  It’s good at spying on window messages, but what I want to know is what external Win32 DLL functions was it calling.  Spy++ won’t let you see that.  So another Google hunt for an answer.

Before getting into that, another question of interest is determining DLL dependencies.  What DLL’s is a processes using?  To answer that, one good utility is Dependency Walker, which can be found at http://www.dependencywalker.com/

image

Now I can see all the KERNEL32.DLL functions being called by HD Tune.  The next step is to be able to see the actual calls being made, with parameters and return values.  Can we find a utility to do this?  Let’s find out!

One method, and arguably the most elegent, of viewing calls made by a process is via a technique called Import Address Table (IAT) patching.  Executable programs on windows and DLL’s are built on the Portable Execution (PE) file format.  One section in this format is called .idata contains a table (yeah, you got it, the Import Address Table) with names of imported functions.  An interesting article on this is "API Spying Techniques for Windows 9x, NT and 2000" by Yariv Kaplan at http://www.internals.com/articles/apispy/apispy.htm.  Not only does the table list the functions called, but they act as a jump table.  That is, when a function is called, it is done through an indirect JMP of the function in the IAT.  That turns out to be a perfect spot to intercept the call.

Quickly searching around, we can find a few candidates in the field:

  • APISPY32 – An older program originally written for Windows 95.  Indication is that it works under Windows 2000 and XP also.  Available as a ZIP download at http://www.wheaty.net/ – direct download link is Updated APISPY32.  At this time, I’m not going to look at this one; instead I’ll focus on the following two.
  • Detours – Binary Interception of Win32 Functions (Microsoft Research).  Detours is a library for instrumenting arbitrary Win32 functions on x86, x64, and IA64 machines. Detours intercepts Win32 functions by re-writing the in-memory code for target functions. The Detours package also contains utilities to attach arbitrary DLLs and data segments (called payloads) to any Win32 binary.  http://research.microsoft.com/en-us/projects/detours/  A free “Express” version is available.
  • STraceNT: A System Call Tracer for Windows – Uses IAT patching to trace function calls.  This is the most promising and I’ll try this one first.  http://www.intellectualheaven.com/default.asp?BH=projects&H=strace.htm

STraceNT

image

Unfortunately, under Windows 7 (at least), I was not able to get STraceNT to run HDTune.EXE or even a simple CRC32.exe program that I wrote.  It could be a Windows 7 issue.  So I’ll fire up Windows XP on a Virtual PC and see if it will run on there.  (5 minutes later!)  Yes, it seems to be working fine under Windows XP, so it seems to be an issue with running under Windows 7.

Detours

Detours, from Microsoft Research, is not an application; it’s a library.  With it, you can develop your own utility much like STraceNT.  In fact, that’s just one of the things you can do with it.  One of the samples (C++ code) is an utility like STraceNT that shows how to intercept and log some 1400+ Win32 API function calls.  At least one person has used Detours to make such a utility (no source code)- see http://dev.depeuter.org/xptruss.php