Be Lazy - use XPATH to search for element in xml

I worked in application that the input to the application was XML files that was placed in a specific folder.
The XML file represent an order. The application load the XML file from this directory and process the order. The application process the order only if the order was valid . Validation was done by checking some fields like sequence, process date ...
The QA or the developer that debug the application create a new input to the application by taking old XML ,open it in notepad ,look for these fields and fixed them (like increment the sequence) in order to create new valid XML. In some cases that system failed to generated valid XML i saw developers open big XML in notepad and by CTRL+F look item by item to check which item has invalid description.

To all hard workers developers i am saying - Be Lazy do it by XPATH.

Like all other "Be lazy" cases there is a small learning curve but after you got used to it you will see how XPATH is saving time. Keep your favorite XPATH tricks and expression in your knowledge management tool, so in O(1) you will find your favorite XPATH.

The following link is the best single link that i know (you are welcome to share your favorite one). You will find there a simple table with lots of practical XPATH examples.

*Partial snapshot of the XPATH table.

Here are some additional ones:
//Seq[Property/@InternalName="GroupId" and Property="20"]

//PersonList[count(Person)=0] - Empty PersonList with no Person node

count(//PersonList[count(Person)=0])

//title[not(text())] - titles with no content

//title[not(node())] - titles with no child

../ - One level up

//item[ name( following-sibling::*[1] ) != 'description' ]- This XPath matches ALL which are not followed by directly.

The great part that you don't need to open these XML any more and you can do all through PowerShell. 
Some examples:

$xml = ' title="foo"/>'
$xml | Select-Xml '//book'
Node    Path          Pattern
----    ----          -------
book    InputStream   //book

Remove Nodes:

PS> $xml =[xml]' title="foo"/> title="bar"/>'

PS> $xml | Select-Xml -XPath '//book' | 
        Foreach {$_.Node.ParentNode.RemoveChild($_.Node)}

See more at.

Select-Xml '//User[contains(Friends/Friend/@Name, "Bar")]' $xml |%{$_.Node.Name}

With namespace:

$ns = @{ e = "http://avaya.com/OneXAgent/Settings" }
$xml = (gc lotusnotes.xml) 
Select-Xml -Xml $xml -XPath "/e:Settings/e:Directory/e:DirectorySetting/@name" -Namespace $ns | select -expand Node | select -expand '#text'

You can also search for all XML in a specific path or you can take advantage of poweshell pipeline and send a list of XML as input like this:

Get-ChildItem . -r -file | Select-Xml -XPath "//node[a='1']"

You can also use linq and all .Net XML features.

So be lazy - use XPATH!

Be lazy - implment your internal tools with python

From time to time you need to write small program for internal use. The program can be a client that check something , an internal tool for the developers, a small program that need to analysis an internal data ...

You can do it in your " main programming language " , but in case that your language is Java or C# (Or similar ) than you need to write a lot of code , download and add reference, compile , class-path ...
Too much headache even for small internal program.

These are the cases that scripting languages like python , ruby, groovy can help you.
Python is a very simple and powerful language , which has plenty of module that expand it. You can find great module for almost any purpose. You can find Facebook, Twitter,Google API written in Python.
You can find great Networking and Image processing module ... The main power of Python is the huge amount of available libraries. Unforgettably python doesn't shipped with package manager (In contrast to Ruby), so we need to download one of the python package manager available out there. We can use easy_install or PIP (there are more) for this purpose. If you are Linux citizens your life is simple (on this point). Once you install PIP than download package for python is very easy  - pip install ...

For windows citizens  download and use in python 3rd party libraries is more complicated and will require from you to spent more time and difficulty ,mainly because some that some of the packages required c compiler.
You can follow this link that will help you to install easy_install python package manager for windows or you can visit the following  http://www.lfd.uci.edu/~gohlke/pythonlibs/ to check if Christoph Gohlke provide a binary distribution for the package you are looking for.

The official one is good enough, so this isn't the target here .My goal is to show some simple and powerful things that you can do in python.


Sniff Network
Few lines and you have sniffer in python.You can add filter to the sniffer or sniff only limit amount of packets.

sniff(prn=addPacket)

def addPacket(packet)
    src = packet.sprintf("%IP.src%")
    dst = packet.sprintf("%IP.dst%")
    sport = packet.sprintf("%IP.sport%")
    dport = packet.sprintf("%IP.dport%")
    raw = packet.sprintf("%Raw.load%")

Pcap Analysis 
You can dump the network traffic by tcpdump or wireshark and than easly analysis the Netwrok traffic with pyton.

import dpkt
from dpkt.ip import IP
from dpkt.tcp import TCP

for ts, raw_pkt in pcap.pcap(file_path):
    ip = IP(raw_pkt[14:])
    if(type(ip) != IP):
        continue
    tcp = ip.data
    if(type(tcp) != TCP):
        continue
    do_something_with(tcp.data)

Tcp client server

    import socket
    HOST = 'google.com'    # The remote host
    PORT = 80              # The same port as used by the server
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((HOST, PORT))
    s.send('GET / HTTP/1.1\r\nHost: google.com\r\n\r\n')
    data = s.recv(1024)
    s.close()
    print 'Received', repr(data

Working with DB
The following demonstarte the easly of working with DB. You can see the executemany command that get a enumerator as a second parameter. You can easily create enumerator on a file or any other input.

con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.execute("create table characters(c)")
cur.executemany("insert into characters(c) values (?)", char_generator())

Working With File
You can easily read all lines , read them directly to formatted structured , filter lines or iterate line by line when dealing with big files.
with open("log.txt") as infile:

      for line in infile: 

            do_something_with(line)

Easily expose web service

import web

urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())

class hello:        
    def GET(self, name):
        if not name: 
            name = 'world'
        return 'Hello, ' + name + '!'

if __name__ == "__main__":
    app.run()

Be lazy - automate your web routines with Selenium

Few years ago i work on a project that once in a few days (sometimes even few times a day) all developers stop their work and do a sanity test on the web-product. I suggest few times to automate the procedure with Selenium but my suggestion wasn't accepted.
I looked on the short term and didn't invest my own time to create automate sanity tests.

Few years latter i got a mission to automate a browser tasks. I learn Selenium in order to do this mission and I was so angry about myself that I didn't did it few years ago and save months of development time.

Since i automate web-flow on complex web pages (Ajax, Generated Id...) i wasted lot of hours to overcome on some selenium pitfalls, like frames, Ajax,read only attributes ... The good news is that few weeks ago a new and great selenium book was published - "Selenium Testing Tools Cookbook".

This is not a Selenium tutorial. You will find lots of them in the web. The target is a quick introduction to show you how simple it is.

There ate 2 main ways to run Selenium automation either by using the Selenium IDE or by using Selenium WebDriver.

Selenium IDE:
Selenium IDE is an integrated development environment for performing Selenium tests.It can be added as an addon to firefox. 

You can open Selenium IDE from Firefox click the record button and record the steps that you are executing in Firefox.

Each action that executed in the Firefox will be recorded as a step in the recorded test case. While recording and after recording you can go to the IDE and edit the steps by add steps manually or change the step action.

A classic Selenium sequence is locating a WebElement and than take an action on the WebElement.

Selenium IDE in record mode record how to locate the WebElement and the action that was taken on the WebElement. 

Lets see a simple flow:

1. open Google

2. type " hello selenium " in the search box

3. click on search

I will focus on step 2. In step 2 we need to locate the search box input text and do a type action on it.

This is how it look on the IDE:

You can see that the command action is "type" and the target (WebElement) is element with id "gbqfq".

The IDE need to record how to locate the target. There are few ways to locate element on a webpage.

You can locate it by Id, name,css,dom,xpath ... 

The IDE will chose one of the way above to record how to locate the element. The IDE try first to record id locator than name locator ... Id locator is a good choice for element locator because element has unique id, however id locator will not always work. In some cases the element doesn't has ID or the ID is auto generated , so next time we will run the test we will get an exception that WebElement wasn't found. The IDE cannot guess it, so in some cases we need to go to the step and chose another locator. In the picture above you can see that the target drop-down contain different locator for the input text element.

After we record the test we can run it from the IDE or we can export it to one of the available WebDriver API (Java,C#,Pyton...)

In the IDE we go to File->Export Test Case As and chose one of the API that supported there.

A python code that execute the following will look like this:

        driver = self.driver

        driver.get(self.base_url + "/")

        driver.find_element_by_id("gbqfq").clear()

        driver.find_element_by_id("gbqfq").send_keys("hello selenium")

        driver.find_element_by_id("gbqfba").click()

The disadvantage of the IDE is the missing supported in other browser than Firefox. Most of the popular Browser have Selenium Driver, so you can run your test on most of the browsers.

Selenium API is very simple , few hours will make you familiar with the API and you will see that its simpler and faster to wrote the test by yourself using the webdriver.

There are some annoying things that you need to pay attention to them when you automate web flow.I will try to cover some of them in another post.

Be lazy - use clip.exe to send content to clipboard

Instead of mark and copy command (and most of the times you miss in the mark) you can use the clip command and send content by using pipe and clip to the clipboard.

gc c:/test/my.xml | clip

This will send the content of my.xml to clipboard.

Be lazy

From time to time I am amazed by how developers can be hard workers.
I can see developers create and change big xml files with notepad, replace value by value with CTRL +F or by open wide eyes and search for things with their eyes.
I can see developers do the same "browser routine" (open window, login, click here and their...) few times a day.
I can see developers do the same files routines (copy file, rename , delete) few times a day.
I can see developers see the same " key routine " few times a day.

We spent most of the day in front of the computer trying to create programs that will make human life easier , accurate and faster ,but when it come to us most of the time we work hard by doing the same and same other and other again.

In his book Programming Perl (O’Reilly), Larry Wall describes the three virtues of a
programmer as “laziness, impatience, and hubris.” Laziness, because you will expend effort to
reduce the amount of overall work necessary. I wish i was more lazy. I wish i will expand effort to
reduce the amount of overall work necessary. Most of the developers including me will do the same routine other and other before they will automate the process.

The main reason that we dont create shortcut to our routine is that we need to learn some tool in order to create a shortcut. Learning a new tool, language,framework required time and in most of the cases the time that will required to learn the replacement tool is more than the sum of the time that spent to do it several of time with the old routine.
Suppose that you need to replace few fields in a big xml file. You can open the xml in notepad look for the field and replace them. This will take you 2 minutes . As an alternative you can do it with powershell in single  line. In case that you are not familiar with powershell this will take you few hours (powershell is very simple, but still any new tool require some time to learn). From this reason most of the developers will stick with the old school attitude and will not learn a new tool in order to find a solution for these 2 minutes work.

As you can see in the diagram above any new tool has a learning curve and will take some time until we will get a better performance comparing to the old tool.

Eventually you do the same and the same routine and at the bottom line you pass the time that is needed in order to learn a new tool , so you do the routine more slowly and less accuracy and more boring than make them in an automating tool.

Neal Ford in his great book  - " The productive programmer " claim that one of the reason that some developers are seven time more productive than over developers is that they are using in productive tools that short tasks time. I don't know about this number , but i am sure that we all spent a lot of time by being too much " hard workers " and do things in the " old school way " and not looking for a tool that will do the job for us.

I am admire developers that don't touch their mouse and they environment is full with shortcut. You can learn a lot of quick and shortcut tip by watching how other developers organize their environment. Some time i am sitting with colleague on his computer in order to solve a bug and accidentally i pay attention to some clever organization or shortcut method. I wish the developer community has a show like Martha Stewart has that teach some clever trick and tip to our programming environment.

As i wrote above Neal Ford tried to do this in his book "The productive programmer" .The Productive Programmer offers critical timesaving and productivity tools that you can adopt right away, no matter what platform you use.

I will try to collect and add some of my own experience advice under "Be lazy" tag. If you have your own productive advice i will be great-full if you can send them to me (avihaimar@yahoo.com).

Be lazy - automate your windows tasks with powershell

From time to time we need to do some windows tasks - like backup files , change files , invoke windows commands,connect to process , track system event, maniplulate files, administartion tasks , connect to the web, invoke rest service, query database...... for these we can use windows powershell

Windows PowerShell is Microsoft's task automation framework, consisting of a command-line shell and associated scripting language built on top of .NET Framework. PowerShell provides full access to COM and WMI, enabling administrators to perform administrative tasks on both local and remote Windows systems.

Installation:
If you have windows 7 , so you already have powershell version2.
If you want to upgrade to powershell version 3 than make sure service pack1 and .Net4 are installed on your computer and than follow by this link for upgrade.
You can star powershell from the start menu or by just type powershell in cmd window (exit at the end will return to shell window).

This is not a powershell tutorial. You will find plenty of them in the web.The target of this post is to give a general overview of powershell capabilities through some practical examples.

Cmdlets
Cmdlets are specialized commands in the PowerShell environment that implement specific functions. These are the native commands in the PowerShell stack. Most of the windows tools like mssql,office
Most of the windows tools implements their own cmdlet and give you the ability to connect with them by using cmdlet. You can define and implement cmdlet by yourself by using any .Net language.
Since its all about .Net classes and object is very simple to define and interact with any .Net API. You can even create your own "little language".

If you need to do some administration operation on any tool or program check first if it the tool has powershell snapin for management their environment.

Cmdlets follow a - naming pattern, such as Get-ChildItem,but its a good practice to add alias , so you can use 'gci, dir, ls' as an alias for Get-ChildItem.

Pipeline
PowerShell, like Unix/Linux based shells, implements a pipeline. This pipeline enables the output of one cmdlet to be piped as input to another cmdlet. For example, the output of the Get-Process cmdlet can be piped to the Sort-Object cmdlet (e.g. to sort the objects by handle count) and then to the Where-Object to filter any process that has, say, less than 1 MB of paged memory, then finally to the Select-Object cmdlet to select just the first 10 (i.e. the 10 processes based on handle count).

Editors
There are a lot of free and commercial powershell editors tools that will help you to write and debug powershell script.windows powershell ISE that shipped with windows enable all the abvove and is a great place to start write your first powershell scripts.

Lets be practical
get-member
get-memeber will help you to listing the properties and the methods of object or command.

New-Object Net.Webclient | Get-Member | Out-GridView

Get-Content

Get-Content reads the contents of a file. The –Wait parameter waits for contents to be

appended to the file. The –Tail parameter indicates how many lines from the end of

the file are read.

Get-Content .\test.xml -Wait -Tail 1

Start-Process

The following will print a world file

Start-Process -FilePath "C:\Documents\Test.Docx" -Verb Print

The following will start notepad as an adminstartor

Start-Process notepad.exe -Verb RunAs

Format

${c:data.xml} | Format-Xml -AttributesOnNewLine > data.xml

This will format the data.xml file (the file should be in the current directory). The AttributesOnNewLine parameter will tell the formatter do write attributes in a new line.
Invoke-Item
The Invoke-Item cmdlet provides a way to run an executable file or to open a file (or set of files) from within Windows PowerShell. The following will open all txt files under scripts with your default text editor.

Invoke-Item c:\scripts\*.txt

ConvertTo-Html
Display the result as html. The following will produce a html report of the running process.

Get-Process | ConvertTo-Html name,path,fileversion -title "Process Information" -body "
 Information about the processes running on the computer." | Set-Content c:\scripts\test.htm

Send-Mail

The following will send an email with content , target and vendor as defined in the param map.

Send-MailMessage @param

Invoke-Command

You can invoke commands or run a complete powershell script on a group of remote computers.

Invoke-Command -ComputerName COMPUTER -ScriptBlock { COMMAND } -credential USERNAME

Working with XML

Select-Xml -Xml $x -XPath '//article[contains(body/text(),"ten")]'

Working with Regex

'jeho email je karel@novak.cz' -match ' \w+@[a-zA-Z_]+\.(?[a-zA-Z]{2,3})'

Out-GridView – OGV

get-Process |ogv

This will output a list ,so you can filter, sort, and copy the contents to your clipboard.

You have plenty of cmdlet that will help you to display output in an easy and beautiful way.(xml, html...)

String format
In some cases you need to generate program input based on a pattern this is the place that String.Foramt can help you.

[string]::Format("{0:C}", 1234567890)
$1,234,567,890.00

Stop-Process
You can stop process by name or any other process attribute. The advantage is that you can stop multiple process at the same time and you don't need to look for the process in the task-manager

Stop-Process -processname notepad

You can Stop all SQL services at once

Get-Service *SQL* | Stop-Service -Force

Start-Service

We can start and stop services.The following will open a grid with list of all stopped services. We can chose multiple services for start.

Get-Service | Where Status -eq Stopped |Out-GridView -PassThru |Start-Service

Monitor System Performance

You can do it by monitoring counters and use - Get-Counter

Get-counter -Counter "\Processor(_Total)\% Processor Time" -SampleInterval 2 -MaxSamples 100

Show-Command , Get-Command

Will help you figure out about available command and command parameters.Running the following without specify a specific command will show open a GUI with all available commands.

Show-Command

Get-History
one of the feature that i love in Unix shelll is the history command. You can run history and get a list of all last executed commands (up to some limit that can be configure).

powershell has similar feature.

You can run get-history or h (alias) and you will get commands history.Next to each command you have the command history id , so you can invoke this command by the history-id like this :

Invoke-History   (alias - ihy)

You can do the following in order to complete the command:

#history-id 

Uninstall list of programs
In some cases we want to unistall group of programs, where we group them by name or by vendor or date ... Powershell allow us to do this.

$app = Get-WmiObject -Class Win32_Product | Where-Object {  $_.Name -match "Microsoft sql" }

$app | % { $_.Uninstall() }




Additional resources

Under codeplex  you will find hundreds of powershel extension like powershell for selenium that port some of the webdriver capabilities to powershell...

Powershell magazine  contain hundreds of tips and tricks for powershell.

Windows powershell for developers - I read several books and this one is a must for developers. If you want more advanced example you can take a look at powershell in action