Do not select/focus listviewitem when clicking on image (WinForm)

I'm working in VB.NET on a WinForm app which contain several ListViews. Each listview has a Label and an Image (In a SmallImageList), the checkboxes are disabled.

By using the HitTest method I am able to determine if I clicked the Label or the Image:

Dim result As ListViewHitTestInfo = CheckedListSolutions.HitTest(e.Location)

Basically when I click on an Image I want the current selection to remain unchanged, but when I click on a Label I still want the normal behavior. I did some research and only obtain concretes answer for WPF app unfortunately.

I think I've found that the selection is changed just after the MouseDown event of the Listview. So I tried to reset the selection back to the previous item is the MouseUp event but what happens is I can see the selection changing to the item I clicked and then change back again. I don't want to see that.