Click-testing or Automatic Regression-testing

Does your company test every feature of your desktop application before every release? How much time and manpower does this take?
My team develops a WPF application, that we release a new version of more or less every week. To find out more about our application, see http://www.delficert.com/. We are 3 developers, plus about a quarter of a job post for manual testing, but we still get more or less all features of the application tested. The way we manage to do this is by having an old recycled computer on which every build is deployed, and automatically regression-tested.
We have written a DSL/framework that encapsulates MS UI Automation and allows us to write readable testscripts. We are now releaseing this DSL to GitHub, so that other people can enjoy the same level of automatic testing that we now have. Check out this link: https://github.com/delfidata/clicktest
A goal of the DSL was to allow us to write tests that are hopefully just a little more detailed than the manual testscripts used by maual testers. This makes the tests more accesible to testers, who often are not developers, and it improves readability. Here is a sample of our testcode with a couple of images for context:

    ComboBox("cmbFieldList").SelectItem("PoNo");

    TextBox("txtSrcValue").Type("klikktest PoNo");

    PressEnter();

    DataGrid("SearchResult").SelectRow(0);

    Button("Edit Certificate").Click();

    // in the edit tab

    Button("Edit").Click();

    WaitWhileBusy();

    SleepIfOnTestMachine(2000);

    // actual test

    Image("ActualCertPage").RightClickMouse(-500);

    Image("ActualCertPage").LeftClickMouse(50);

    WaitWhileBusy();

This test was actually a test-driven bugfix of a bug we discovered in the gui-code, where right-clicking the mouse outside the expected points set the application in an error-state which made it crash on the next left-click inside the image boundary.
This test assumes that the test-framework/DSL opens the application, logs on, and opens the Search-tab, before running. After the test is run, the framework will verify that there is no error-dialogs showing, and that the application is not in an error-state. Last, it will kill the application, so earlier test-runs never affect the current. If it finds any errors, it will take a screenshot that we can use to pin-point the error, in conjunction with the application logs.
A nice feature is that we can start the application in Visual Studio, start the test-run, and get the exception in the debugger.
The code required to click a button in MS UI Automation looks like this:

void ClickButton(AutomationElement window, string name){
  var searchConditions = new List {
    new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Button),
    new PropertyCondition(AutomationElement.NameProperty, name),
  };
  var searchCond = new AndCondition(searchConditions.ToArray());
  AutomationElement result = element.FindFirst(TreeScope.Descendants, searchCond);
  if (result == null)
      throw new AutomationElementNotFoundException("Could not find element: ", searchConditions);
  var invoker = result.GetPattern<InvokePattern>(InvokePattern.Pattern);
  invoker.Invoke();
}

As of writing this post, we have 130 of these tests. The tests run in about 2 hours, doing them manually, we estimate, would take at least 2 full days. With the tests being run after every build, and a release schedule of once a week, this really saves us a lot of time.
Our tests test features like:

• searching for certificates and deliveries
• storing and editing all objects we handle in the application, and then verifying that they have been correctly stored in the database
• compiling zip-packages of certificates and verifying that the zip-file is not empty
• scanning certificates and attachments with a flatbed-scanner
• drag-n-drop of files from Explorer to the application
• typing strings in numeric fields
• information labels show what they are supposed to show

The main huge advantage of this is that all the repetetive tests are run by a computer, and that frees our manual tester to do more constructive work. He still has to do a rough acceptance test, because we don't have a complete set automatic tests yet, and we think it is good to have some human eyes on the application before releasing it. He also does a manual test of all new features, so that all features are always tested by someone other than the developers.
Other things he is freed up to do are:

• Test functionality that is not possible to test automatically, yet, like:
• Visual confirmation that images are shown correctly
• Exploratory testing
• Find test-cases which are not automatically tested yet
• Test and develop usability, workflow etc

Before anyone asks, of course we have unit tests for our code. The unit tests test the domain-layers, the web-service-layer and everywhere we can manage to use them. The tests, which this blog post is about, cover the view-layer, and through that, they also work as integration-tests for our entire system.

This was maybe a bit long for an introductory post, but I will be writing a few more blog posts about this framework and the tests we have, in the near future.

WPF, Notifications and Parallellism

I'm working on getting our GUI responsive, which can be quite a challenge. We fetch a pdf or tiff over a wcf-service from the test-server running on a vm, rendering the first page and showing this to the user. We also have a strict policy on separating as much as possible into the domain-layer, and the domain-layer can not have any gui-classes. Fetching and rendering is done in the domain-layer, and the GUI is notified through the INotifiyPropertyChanged-mechanisms.
I've had several problems with the ownership of wpf-objects when a task in the domain-level sets a property which the gui subscribes to, and the gui then tries to redraw some metainformation in an overlay canvas. I've solved several situations by using TPL's ContinueWith-mechanism where the new task is run on the GUI-thread.
After working on a problem today, where images are fetched every time the user changes the selected row in a table, I started thinking if there is a better way to do this. If I could only get all notifications to run on the GUI-thread, without having to use TPL's ContinueWith...
So I've now created a small test-project that has some classes, a window showing info, and some extension methods that run all the notifications on the GUI-thread.
This is the essence of the extension methods:

public static TaskScheduler GuiContext;

public static void Notify(this PropertyChangedEventHandler self, INotifyPropertyChanged sender, string propertyName) {
if (self == null) return;
Task.Factory.StartNew(() => self(sender, new PropertyChangedEventArgs(propertyName)), CancellationToken.None, TaskCreationOptions.None, GuiContext);
}

The GuiContext field needs to be set on startup from the MainWindow, e.g. in the Loaded-event. It appears as though the extension-methods works like a charm.
I made the test-app start several threads that updates the objects in the controller several times, and as long as I allow the gui some cpu-time(put in some small sleeps in the threads), the gui is responsive and it updates the textboxes with values from the different threads.
I'm aware that just updating the same values from several threads is not a good idea, but for showing data in the gui I'm really just interested in showing the latest values, and I have control over cancellelation on the background tasks in the real application. So for the purpose of the test-app I don't think this is important.

I'm writing this post in the hopes that someone will take a poke at my project and tell me if they find any flaws or large weaknesses that I've overlooked. Here is a link to the project file.

Memory leaks while converting from bitmap to bitmapsource

One project I'm working on is using TwainDotNet. It has been some work to get to work "exactly" like we want it to, but it's overall been a nice library to play with.
But today, I've been working on a memory leak... These bugs are really annoying to work on, but anyhow here is what I found out:
The problem is that the sample code and samples I've found on the internet for converting Bitmap to BitmapSource. The code usually looks something like this:

    image1.Source = Imaging.CreateBitmapSourceFromHBitmap(
        new System.Drawing.Bitmap(args.Image).GetHbitmap(),
        IntPtr.Zero,
        Int32Rect.Empty,
        BitmapSizeOptions.FromEmptyOptions());

This works fine, but it also creates a HBitmap in unmanaged memory that it doesn't free.
A better way to solve this is to store the HBitmap in a variable and call DeleteObject on it. DeleteObject is a function from gdi32.dll:

    [System.Runtime.InteropServices.DllImport("gdi32.dll")]
    public static extern bool DeleteObject(IntPtr hObject);

Here is the code I ended up with in our production code:

    private void TwainOnTransferImage(Object sender, TransferImageEventArgs e) {
        if (e.Image != null) {
            var bmp = new System.Drawing.Bitmap(e.Image);
            IntPtr hBitmap = bmp.GetHbitmap();
            image1.BeginInit();
            image1.Source = Imaging.CreateBitmapSourceFromHBitmap(
                hBitmap,
                IntPtr.Zero,
                Int32Rect.Empty,
                BitmapSizeOptions.FromEmptyOptions());
            image1.EndInit();
            image1.Source.Freeze();
            DeleteObject(hBitmap);
            bmp.Dispose();
            e.Image.Dispose();
        }
    }

Spotify Playlist Folders

Eller kanskje egentlig: koffor har eg tømt Swing-spelelista mi på Spotify.
Spotify kom med ein ny funksjon for litt siden, "Playlist folders", tanken her er at du lager ein folder der du legger inn andre spelelister og får slik hierarkiske spelelister. Så istedetfor å ha ei speleliste som inneholder all swingmusikken min har eg nå fleire, og desse ligger i ein folder. Og vil eg høre på all swingmusikken så velger eg folderen som nåværende speleliste og får slik ein grei variasjon i ka swingmusikk som speles. I tillegg kan eg då ha spelelister som eg kan bruke når eg ska undervise, ei med sakte swing musikk, ei med rask, osb. :) I tillegg har eg endra navn på den gamle lista mi te å nå vere "Swing forslag" i håp om at nogen vil legge inn sangforslag der til meg, som eg igjen kan flytte over i dei spelelistene der eg synes dei passer. ;)

Oppdelingen eg nå har gjort blei for det meste gjort på bussen(offline lister) til Karmøy, og eg hørte for d meste bare kjapt på sangene før eg flytta dei, så eg rekner med at det blir litt meir rydding etterkvart. ;)

Her er linker til spelelistene mine slik eg har laga dei pr.nå:
Lister eg kommer te å bruke på kurs:
Swing Sakte,  Swing Mellomtempo, Swing Rask, Swing Diverse, Hip-Hop-Swing, Blues
Andre lister:
For rask, Sakte ikkje kurs, Mellomtempo ikkje kurs, Rask ikkje kurs, Lindy Sakte, Lindy Mellomtempo, Lindy Rask, Charleston

Og eg har som sagt ei liste der eg håper folk vil legge inn forslag til meg: Swing forslag ;)

ConcurrentQueueWithWait

For the time beeing, I'm working on a rather heavy computing problem at work. A normal case scenario has a computing time of about 1 hour, and a not unlikely, but bad scenario, has a computing time of about 7 hours. One "problem" is that I'm already using a pretty powerful computer:

• Core i7 930 @ 2.80GHz (WEP 7.5)
• 2 striped Intel 160GB X25M 2.gen Disks (WEP 7.9)
• 12GB RAM (Operations WEP 7.5)

I've created several producers and consumers, and though I've done a lot of code optimizing I've got more to go on. My main clog on performance was the SQLite connection. SQLite is an embedded single-connection database, and I have some constraints on the interpretation of the data that forces me to write data more or less sequentially. This limits me too using one thread to execute the SQLiteCommands, and using the phx-lib, the SQLiteCommands is prepared only when execution is called. Preparing the commands is quite time consuming, so I've limited the queue of commands to avoid using more time than necessary on preparations. When a command is prepared I can change the values of the parameters without having to do the complete preparation step over again. I use the ObjectPool to generate and store the SQLiteCommands.
Have you ever worked with the Producer/Consumer pattern on several threads? If you have your code might haved looked something like this (pseudoocode):

Producer 1 and 2 (2 separate threads that I create several of)

while (work to do){

  Do some calculations on obj

  Lock (FirstLevelQueue(s)) { // one queue for both or 1 queue for each

    FirstLevelQueue(s).Enqueue(obj)

  }

}

Producer 3 and 4, that Consumes 1 and 2's queue(s)(2 separate threads)

while (producer 1 and 2 not done){

  dequeued = true

  while (dequeued) {

    Lock (FirstLevelQueue(s)) { dequeued = FirstLevelQueue(s).Dequeue(obj) }

    if (dequeued) {

      Do some work on obj

      while (SecondLevelQueue(s).Count > sizeLimit) Thread.Sleep(100)

      Lock (SecondLevelQueue(s)) { SecondLevelQueue(s).Enqueue(obj) }

    } else if (some complex criterion) {
      Thread.Sleep(100) 
    }

  }

}

Consumer (5. thread)

while (producers 3 and 4 not done) {

  dequeued = true

  while (dequeued) {

    Lock (SecondLevelQueue(s)) { dequeued = SecondLevelQueue(s).Dequeue(obj) }

    if (dequeued) {

      Do some work

    } else if (some complex criterion) {
      Thread.Sleep(100) 
    }

  }

}

I feel this code has a few weak points. First of all, locking is newer good since it takes time and it hinders consumers to dequeue for as long as anyone is queueing. Also in this code there is going to be a lot locks since all access uses this mechanism. Locking can also cause huge problems if you forget one or end up locking for too long, e.g. while sleeping. Second, the sleep statements are going to take a lot of time. In the code, on which I am working, I discovered that even a sleep as short as 10ms was way too long. Furthermore, since the sleep is so short the threads are going to wake up and consume quite a bit of resources while checking if there is any more work to do. Using .Net 4.0 and TPL(Task Parallell Library) you get a somewhat better and nicer code by using the ConcurrentQueue, which is thread-safe:

Producer 1 and 2 (2 separate tasks that I create several of)

while (work to do){

  Do some calculations on obj

  FirstLevelConcurrentQueue(s).Enqueue(obj) // one queue for both or 1 queue for each

}

Producer 3 and 4, that Consumes 1 and 2's queue(s)(2 separate tasks)

while (producer 1 and 2 not done){

  dequeued = true

  while (dequeued) {

    dequeued = FirstLevelConcurrentQueue(s).TryDequeue(obj)

    if (dequeued) {

      Do some work on obj

      while (SecondLevelConcurrentQueue(s).Count > sizeLimit) Thread.Sleep(100)

      SecondLevelConcurrentQueue(s).Enqueue(obj)

    } else if (some complex criterion) {
      Thread.Sleep(100) 
    }

  }

}

Consumer(5. task)

while (producers 3 and 4 not done) {

  dequeued = true

  while (dequeued) {

    dequeued = SecondLevelConcurrentQueue(s).TryDequeue(obj)

    if (dequeued) {

      Do some work

    } else if (some complex criterion) {      Thread.Sleep(100) 
    }

  }

}

Now, this code looks a lot better since I'm rid of the locks. This reduces some common error points and the consumers do not stop the producers from enqueueing. But there is still a lot of sleep statements, and the threads often wake up to check if there is more work to do, and as stated, for my part, even a 10ms sleep were to long.
I decided that I really wanted code that looked somewhat more like this:

Producer 1 and 2 (2 separate tasks that I create several of)
while (work to do){
  Do some calculations on obj
  FirstLevelConcurrentQueue(s).EnqueueWithEvent(obj) // one queue for both or 1 queue for each
}

Producer 3 and 4, that Consumes 1 and 2's queue(s)(2 separate tasks)
while (producer 1 and 2 not done){
  FirstLevelConcurrentQueue(s).TryWaitUntilDequeued(obj)
  Do some work on obj
  SecondLevelConcurrentQueue(s).EnqueueUpToLimit(obj)
}

Consumer(5. task)
while (producers 3 and 4 not done) {
  SecondLevelConcurrentQueue(s).TryWaitUntilDequeued(obj)
  Do some work
}

In my mind, this code is shorter and way more readable. In addition, the threads do not wake up all the time to check if there is some work to do. This actually makes the code more cpu efficient and gives you more cpu resources to do your other computing.

It didn't take me much googling to discover the AutoResetEvent, an easy-to-use, thread-safe event mechanism. Using the AutoResetEvent and the ConcurrentQueue I implemented a class I called ConcurrentQueueWithWait. I included this in the dll for ParallellExtensionsExtras, which we use at work, and after some tweaking and testing I ended up with this class:

public class ConcurrentQueueWithWait : ConcurrentQueue {
  private AutoResetEvent _enqueuedEvent = new AutoResetEvent(false);
  private AutoResetEvent _dequeuedEvent = new AutoResetEvent(false);

  /// The size limit of the ConcurrentQueue, if you use EnqueueUpToLimit for enqueueing
  public int QueueLimit { get; set; }

  /// The timeout for Dequeue-events, and EnqueueUpToLimit-events
  public int TimeOut { get { return _timeOut; } set { _timeOut = value; } }
  private int _timeOut = 5000;

  /// Sets both the Enqueue and Dequeue events twice to try to avoid any deadlocks
  public void ProducerFinished() {
    _enqueuedEvent.Set();
    _dequeuedEvent.Set();
    Thread.Sleep(10);
    _enqueuedEvent.Set();
    _dequeuedEvent.Set();
  }

  /// Enqueues the item if the queue limit is not reached. If limit is reached it will wait until an item is 

  /// dequeued, and then enqueue.
  /// Note: on a timeout it will enqueue wether or not the limit has been reached.
  public void EnqueueUpToLimit(T item) {
    if (QueueLimit > 0 && base.Count > QueueLimit)
      _dequeuedEvent.WaitOne(TimeOut);
    base.Enqueue(item);
    _enqueuedEvent.Set();
  }

  /// Adds an object to the end of the ConcurrentQueue, and sends an event to anyone waiting to dequeue.
  public void EnqueueWithEvent(T item) {
    base.Enqueue(item);
    _enqueuedEvent.Set();
  }

  /// Attempts to remove and return the object at the beginning of the queue. If none is found it will wait for 

  /// an enqueued signal, or timeout.
  /// Should be used as a regular TryDequeue, but without the while(...) Thread.Sleep(xx)
  /// Returns wether or not a dequeue was successful
  public bool TryWaitUntilDequeued(out T result) {
    bool res = true;
    if (base.IsEmpty)
      res = _enqueuedEvent.WaitOne(TimeOut);
    if (!res) {
      result = default(T);
      return false;
    }
    res = TryDequeue(out result);
    _dequeuedEvent.Set(); //I do not need to check result here, since a false should equal an empty queue 

                                        //and therefore any pending enqueues should be fired
    return res;
  }

  /// Attempts to return the object at the beginning of the queue. If none is found it will wait for an 

  /// enqueued signal, or timeout.
  /// Should be used as a regular TryPeek, but without the while(...) Thread.Sleep(xx)
  /// Returns wether or not a peek was successful
  public bool TryWaitUntilPeeked(out T result) {
    bool res = true;
    if (base.IsEmpty)
      res = _enqueuedEvent.WaitOne(TimeOut);
    if (!res) {
      result = default(T);
      return false;
    }
    res = TryPeek(out result);
    return res;
  }
}

I had to put in a timeout to avoid deadlocks both on enqueue and dequeue. In tests, which I've run on a small test-set(2 seconds run-time), I get the extra 5 seconds in about 1 out of 6 runthroughs, and I figure that 5 seconds extra don't matter much with run-times normally from 20 minutes and up.

This class meant that I ended up with this implementation of my tasks:

Producer 1 and 2 (2 separate tasks that I create several of)

while (work to do){

  Do some calculations on obj

  FirstLevelConcurrentQueue(s).EnqueueWithEvent(obj) // one queue for both or 1 queue for each

}

Producer 3 and 4, that Consumes 1 and 2's queue(s)(2 separate tasks)

while (producer 1 and 2 not done){

  while (FirstLevelConcurrentQueue(s).TryWaitUntilDequeued(obj)) {

    Do some work on obj

    SecondLevelConcurrentQueue(s).EnqueueUpToLimit(obj)

  }

}

Consumer(5th task)

while (producers 3 and 4 not done) {

  while (SecondLevelConcurrentQueue(s).TryWaitUntilDequeued(obj)) {

    Do some work

  }

}

Not quite the easy and nice code I was hoping for, but really close enough to make me happy. My threads now only come out of sleep every 5 seconds or when there actually is some work to do.
Some minor problems with this implementation:

• If I get a timeout on EnqueueUpToLimit I've set it to just enqueue anyway; so the limit I set on the queue is not absolute. My tests show that this is not a problem for my program, so I'm pleased with it.
• I do get the timeout as an extra runtime in about 1 out of 6 times.
• I still have to use the try-pattern.
• If a bug finishes the 5th task before the 3rd and 4th, my program gets an extreme runtime since the 3rd and 4th constantly waits for the enqueue-timeout. This appeared to be a Heisenbug when I encountered it.
• There is still some plumbing code needed, but way less than before.

Herlig påske og "ministoltzen"

Då nærmer ein ganske så herlig påske seg slutten... Eg har fått spelt 1,5 dag med Army of two 40th day i lag m venner, eit par-tre dager med Dante's Inferno, eg fullførte siste. Og eg har til og med klart å få gjort ein del fornuftig i tillegg:
Rydda i ein haug m papirer, deriblant kikka gjennom sjølvangivelsen.
Gjort unna litt småting i leiligheten.
Fått lagt ut nogen ting me ikkje trenger lenger på Finn.no og faktisk fått solgt 1 ting allerede første dagen.
Rydda begge bodene, d e virkelig mye bedre når bodene e halvfulle i bredden ut frå alle veggene, istedetfor halvfulle i høyden. Før og etter bilder frå boden oppe:

I dag var me ute og gjekk tur, noge som me har fått gjort eit par-tre-fire gonger eller noge sånt i påsken, og i dag oppdaga me ei ny favorittrute. "Ministoltzen", ei rute som går ganske bra oppøve rett frå over veien. Me må gå opp på asfaltert vei te rett ovenfor Frode og Marianne, og videre derifrå kommer me over på ein natursti som e ganske bratt te å begynne med. Me gjekk eit stykke oppøve og bestemte oss for at der skulle me ha endepunktet på d som blir vår "vanlige" trimtur nå. Turen blir noge kortare og mindre bratt enn Stoltzen, men eg satser på at me kommer oss opp den løypa mye oftare enn Stoltzen, siden turen opp Stoltzen innebærer ein bortimot 7km sykkeltur kvar vei.
Utsikt frå Ministoltzen:

Helvetesøvelsen(Dans)

Eg fekk ein mail frå Per for altfor lenge siden(3 veker) med sps om eg kunne sende han ein beskrivelse av helvetesøvelsen. Eg hadde bare ein kortvariant liggande men her ska eg prøve å skrive ein skikkelig beskrivelse av den.

Helvetesøvelse

I denne øvelsen skal G føre J i kryssteg/crossovers. G vil føre stega i forskjellig tempo, lengde, framover/bakover og føre spesielle figurar som bryter opp desse stega.
Dette er ein føringsøvelse som trener hovudsaklig ramme. Til å begynne med vil den ende opp med å trene opp noget stive armer, men etter kvart som ein blir meir avansert skal ein stå mest mulig i egenbalanse, og slappe mest mulig av mellom punktene der sjølve føringa skjer. Slik at ein trener kor lang tid det tar følger å reagere når fører begynner å føre. I tillegg gjer det fører bevisst på at dei faktisk må følgje med på kor følger er i sine bevegelsar når dei fører.
Ein kraftig forenkling av denne er at J lener seg litt mot G. Siden dei då automagisk må spenne opp armane for å ikkje falle på G.
Delen som går på å bryte ramme trener reaksjonsevnen, samt at den prøver å få folk frå å bruke denne øvelsen til å trene seg inn i fastlåste mønstre.

Variasjoner som kan gjeres i lukka

    * Crossovers tempo og lengde
    * Lukka med spinn
    * Til musikken – kan taes samtidig i åpen og lukka
    * Legge inn i dansen(veldig korte og kjappe steg) – kan taes samtidig i åpen og lukka
    * Halve steg
    * Bakover, husk halve òg her – mye d samme om denne taes i åpen eller lukka

Variasjoner som kan gjeres i åpen

    * Crossovers tempo og lengde
    * Til musikken – kan taes samtidig i åpen og lukka
    * Legge inn i dansen(veldig korte og kjappe steg) – kan taes samtidig i åpen og lukka
    * Halve steg
    * Bakover, husk halve òg her – mye d samme om denne taes i åpen eller lukka

Variasjoner som bryter ramme

    * Lukka: Få foten te å ”sveve” i lufta
    * Åpen/Lukka: Få til J til å ”sveve” og krysse bak G kne. Kan i lukka brukes som inngang til Sweep the floor
    * Lukka: Bakover til J ned
    * Lukka: Ned til sveip(J har foten i lufta bakover, G går rundt)
    * Åpen: Ned til J foten ut direkte bakover. Føring m tilbake og ut. Kan òg taes videre til sveip.