Powershell cmdlets with dynamic param AND $args don’t work

Over the weekend I tried to implement auto completion in Elixir’s  mix (in Windows). Unfortunately I didn’t make it without introducing some problems. So I didn’t committed my changes to upstream. Currently I try to reach some of the more renowned Elixir/Windows contributors, to discuss the changes .

Motivation

Under normal circumstances I don’t use more mix tasks then test, phoenix.server and release but sometimes you need this weird command, you just can’t remember. The command mix help  is your friend here as it shows you all available commands (project aware!). Yet I don’t like to look the documentation up, if I need just some information on spelling. For example in the beginning I often tried to start the phoenix project with mix phoenix.start (Hint: that does not work). I am used to auto completion in my development environments so I tried to extend mix  as well.

Background

As I am using the Powershell for all my command line related tasks and the default file extension of Powershell is ps1, my command mix  execute the mix.ps1  in the Elixir bin folder.

Approach

Powershell scripts can have auto completion of parameters with an so called [validateSet("Param1","Param2",...)] , which incorporate all valid parameters. Sadly this is of no help, if we have to hard code the possible values for the parameter. A possible solution to this problem is the usage of a DynamicParam with dynamic validateSet (good resource here). To test my various iterations I wrote down all test cases (sorry no automated testing yet).

Iteration 1

Changes

If you have a look at the original mix  script (here) you can see that the script locates the mix.bat , flattens possible array arguments (is this still needed?) and then execute the mix.bat  with the newly flattened arguments.

The first problem we see here is the usage of the $args array. As Keith Hill points out in this SO comment the $args array “… contain any argument that doesn’t map to a defined function parameter…”. Which introduces the first problem: The DynamicParam  ONLY works for defined function parameters.

I copied the linked resource (again, here) and moved the old script to the process block. Because we are creating a script and not a function the signature of function Test-DynamicValidateSet {...}  needs to be removed. To generate the validateSet I replaced the line $arrSet = ... with

This populates the $arrSet  with all valid task. I also changed the value of the variable $ParameterName  to 'Task'  and renamed the variable $Path  to $Task

Test

A short test shows, the command mix  does work, the command mix help  does not. Reason for that is, we assign the first value to the parameter $Task .

Iteration 2

Changes

The call to mix.bat in the last row now get the $Task parameter as well:

Test

mix  works, mixd help  works. Awesome! Lets try auto completion. mix [tab]  …

This is weird. The auto completion takes it times (this is actually the time mix help --names  takes to return all valid tasks) yet the auto completion fills in file names from that folder… To fix that we need to make it clear, that our dynamic parameter is actually the first parameter. So after we set the $ParameterAttribute.Position = 0  (it was 1) we repeat our test.

mix  works, mixd help  works, mix [tab]  works, mix he[tab]  works also. What about arguments to parameters? like mix help --names ?

Damn.

Iteration 3

Changes

OK, we need positional arguments. Lets add some.

I don’t like that approach, because this script will fail on having more than seven parameters (our dynamic and $p1  – $p6 ) with the aforementioned error message.

We also have to forward our new parameters to the mix.bat :

OK, besides the now unused “flatten possible array parameter” logic and our “it will fail on having eight or more parameters” problem, how good are we?

Test

All tests in the test cases pass. Yet we have some unfinished problems.

Problems with this solution

  1. We can have only a fixed amount of parameters. This is not a big problem (as we can add more parameters in the signature), but this is neither elegant nor good practice.
  2. We now completely omit the “flatten array logic”. I have to admit, I’m not sure if  this is still needed, so I asked the original contributor of this logic but still wait for response.
  3. Most of the code was copied from our resource. We clearly added some of our own logic, yet we probably shouldn’t use this code without asking for permissions. I asked the author if I could use this snippet and wait for a response.
  4. Even if I omit the “flatten array logic” I tripled the Lines Of Code. I don’t know if the auto complete feature is worth this much code (read about code as a liability here)

As soon as the problem 3 is clarified I will upload the file here. As soon as the other problems are clarified (and/or fixed) I create a pull request in GitHub to upstream the changes.

Elixir, Phoenix and Windows: Some insights on my 1000┬Ás problem

On Saturday I wrote about “Elixir, Phoenix and Windows: No faster responses than 1000 microseconds?“. I described two problems I had with Phoenix in Windows: My response times seem to be stuck at 1000 microseconds and Powershell couldn’t display the μ sign. I dug into some code and the mailing lists and found (with a lot of help) some answers.

1000 Microseconds on Windows?

The response times in Phoenix are often measured in Microseconds. Yet on Windows you won’t see any requests faster than 1000 Microseconds. That’s not because of a slower OS, but a not as precise as needed timer:

On Windows a developer has several options to get the system time. According to windowstimestamp.com there are (including the precision):

  • The time service (1 ms)
  • The performance counter (between 1 ms and 1 μs)
  • The GetSystemTimeAsFileTime API (100 ns)

The highest precision (100 nano seconds) can be achieved with the GetSystemTimeAsFileTime API (or the GetSystemTime API, which is the same data but differently formatted).

Actually this is the API  the Erlang VM (which provides the run time for Elixir and Phoenix) is using. So in theory we should be able to get more precise data out of these API. Yet the Erlang VM only returns millisecond as smallest unit. I’m pretty there are reasons for it, but I don’t understand them. If you are curious (and don’t fear a little bit C code) go ahead and look at the implementation of the timing in Erlang: sys_time.c in Erlang VM

1000┬Ás instead of 1000μs?

The second problem I had on saturday was the missing μ sign in my Powershell environment. I got hinted that I have to set the code page of Powershell to UTF8:

This fixed the problem for me. Unfortunately this introduced another, more grave bug to me: On putting out special characters in iex.bat, iex now fails to react completely. Until this bug is fixed, I strongly advise against this fix.

Elixir on Raspberry Pi 2 (using Windows)

Goal

Deploy an example Elixir application with the Nerves-Project to a Raspberry Pi 2, using an Ubuntu guest in VMware Player on a Windows host.

Synopsis

We are using the Nerves-project, buildroot and Ubuntu in a virtual machine to create a SD-Image, which contains the default application, blinky.

Motivation

After seeing this talk from Garth Hitchens on Embedded Elixir in Action I wanted to try the Nerves project to deploy an Elixir application to my Raspberry Pi 2. Yet, as I’ve blogged before I’m using a Windows machine and most of the build tools (for Raspberry Pi) are using Linux.

Prerequisites

We will need to install different software to our computer, therefore I assume you have the rights to do so. Also we need disk space (around 15 Gigabytes). Because we are using a virtual machine it does not hurt to have a beefy computer. More RAM+CPU = Better.

Steps

There are several steps needed to complete our goal, most of them require some Linux usage. Bear with me, we will make it! (Also have a look at Linux for Dummies)

Install VMware Player (10 Minutes)

On your Windows Computer do:

We will install a Hyper Visor on our computer. A Hyper Visor enables you to have an operating system in your original operating system (without the risk to destroy your computer).

There are several Hyper Visor solution available. On Windows 8 and 10 the Microsoft HyperV software is available (and installed by default, I think?). In general I would go with the defaults, but I deactivated HyperV for various reasons in the past (no 3d virtualization being one of the most pressing reasons) and choose VMware Player.

After installation of VMware player download the Ubuntu 14.04 (64bit )LTS server image (or the desktop image, I don’t care). So far, 32 WONT WORK!

Create a virtual machine (5 Minutes)

On your Windows Computer do:

Start VMware player and hit “create a new virtual machine”. In the wizard window choose the “Installer disc image file (iso)” option and select your previously downloaded Ubuntu server image.

Hit next and go along with the defaults* until it finishes.

* If you are constrained on the disk size, you can reduce it from 20 Gb to 10Gb. I tried it with 5Gb but failed on downloading all the needed tools.

Install Ubuntu 14.04 LTS (30 Minutes)

In the virtual machine do:

Start your virtual machine (green arrow) and go along with the installation defaults. I changed some aspects which I will highlight now:

  • Language: I’m using German as system language and as keyboard layout.
  • Encrypt File System: I politely declined. I couldn’t care less on this build machine. Also this is in theory a minor performance drain.
  • Packages to install: I checked the OpenSSH box

Let the installation finish. It will take some time and ask questions in between instead on the beginning. So have an eye on that. As a side note. Remember username and password. We need it. For convenience use KeePass 2 or similar software.

Get IP Address (1 Minute)

In the virtual machine do:

Finally we can log in. For later usage we need the IP address. After you logged in, enter

in the terminal. This will bring up the network interface configuration. We need the IP address of the interface eth0. In my case the line looks like

We only need the address: 102.168.62.129 in my case.

Install Nerves (10 Minutes + 45 Minutes of waiting…)

In the virtual machine do:

I installed everything in my home folder, but fell free to do so where ever you like (but remember it later on :-P)

Update your package sources:

The following code is provided by the Nerves-Project:

At this point the Nerves tutorial could be more on the point. They state “…  Change to the nerves-system-br directory…” yet there is no such folder. To fix this clone the Nerves-project/nerves-system-br project from Github!

Now we can change in the nerves-system-br folder:

Now we can follow the tutorial on the nerves-project Github page along again:

And now to kickstart the future “real” builds: Lets make a dry run to cache all files (this maywill take a while):

This takes around 45 Minutes.

Using Nerves (10 Minutes)

The tutorial states one has to source the environment. This sets some variables and is needed every time one brings up a new console

For testing I cloned the blinky example from Github

change in the new folder and in the blinky directory

Now the magic can happen:

This will create a file _images/blinky.fw

*.fw is the file format for fwup (don’t worry it is already installed). After some research (and lots of help by Frank Hunleth and Garth Hitchens I We figured out the correct command:

Using this command you create we create a Raspberry Pi image file (_images/blinky.img)

Get the image (5 Minutes)

On your Windows Computer do:

The resulting image is located in your virtual machine at ~/nerves-examples/blinky/_images/ .

To access that folder we need to download the folder from the virtual machine to your Windows computer. You can access the folder via WinSCP if you opted in for the OpenSSH package before:

Open WinSCP (if installed; otherwise: DownloadChoco Package) and create a new connection. Computer name is the IP address we saved before. Username and password are your credentials from the virtual machine (I told you to remember it!).

You will be presented with two explorer views. The left side is your computer, the right side is the virtual machine, already in the home folder of your user.

If you cloned the nerves-examples.git in the home folder as I suggested, you can easily follow the path. On the right side click through the folders ~/nerves-examples/blinky/_images/ until you find your file, which is called blinky.img. Copy the file to your computer, for example on the Desktop.

“Burn” the image (15 Minutes)

On your Windows Computer do:

To burn the image we need a tool which creates the file system from our downloaded image. Follow the tutorial on RaspberryPi.org.

Fun (Countless hours)

On your Raspberry Pi do:

Insert the SD-Card in your Raspberry Pi and power it on. Depending on your setup, something should happen! For example the blinky example lets you blink the LED, the demo example (should, didn’t work for me) let you connect via ethernet.

The fun you can have
The fun you can have

History:

  • Update 23.12.2015: Added hint for 64bit after @mdsebald mentioned it
  • Update 22.12.2015: Fixed the “Get the image” part
  • Update 22.12.2015: Fixed the tutorial
  • Update 21.12.2015: Marked the tutorial as broken

Elixir, Windows and Chocolatey: Updates

Recently I ran in a problem with Elixir on Windows, installed via Chocolatey:

After an update of the Elixir package:

(or a Windows update or something similar) I couldn’t use iex.bat anymore. Every time I tried to open iex.bat I got an crash_dump:

(That said, mix still worked…)

After several attempts on reinstalling Elixir (and Erlang, just to be sure) I found this comment on the Elixir package on Chocolatey.org from the package maintainer onor_io from July the 6th of 2015:

“That said, I have been meaning to modify the package to copy the *.bat files and the binaries to the Chocolatey packages directory”

If I have a look at the bin folder of Chocolatey (in my case c:\programData\Chocolatey\bin) I can see the elixir *.bat files. But they are available at the Elixir\bin folder as well (c:\programData\Chocolatey\lib\Elixir\bin). For a test I removed the Elixir related files from the Chocolatey\bin folder (elixir.bat, elixirc.bat, iex.bat, mix.bat) and added the Elixir\bin folder to the path.

After a restart of the Powershell, iex.bat produces the desired result:

Drawback

This method isn’t flawless: The uninstall (and probably upgrade) via Chocolatey does not work anymore… You can workaround this problem if you copy all *.bat files back to the Chocolatey\bin folder:

 

Elixir, postgres and chocolatey

You try to use the phoenix getting started guide on windows and the task “mix ecto.create” fails with an useless error? Chances are your postgresql database isn’t available with the ecto default credentials “postgres”:”postgres”. Try to logon to the database with the credentials “postgres”:”Postgres1234″ and change the password for the user to “postgres”. Also, don’t forget to change the password of the Windows user “postgres”.

Update 15.11.2015: You also have to change the logon information for the service, otherwise postgresql won’t start after an restart.

Long version

If you happen to start with elixir and phoenix you will probably install postgres at some point. If you also happen to use Windows AND are a user of chocolatey (which you should be!) it could happen that you run in a nasty, not very helpful error message when you try to use the phoenix getting started guide on the mix task:

The error states exactly nothing:

Which is not that helpful. Because of the error message you can’t exactly google for that particular error. (I mean, yeah you could look the codepoint up but… really?)

Because the ecto tasks failed and ecto is the database mapper, I tried to connect to my recently installed postgres database. Which immediately made my mistake clear: ecto expects an user “postgres” with the password “postgres” for the database connection. But these credentials didn’t work!

I tried to find the default password for postgres (“postgres” being the only answer I found) but failed after a quick googling. So I uninstalled the postgres package via chocolatey

from my computer (which unfortunately didn’t work… I deleted the folder afterwards :-/ ) and reinstalled it.

On the installation log I found the default password afterwards:

I logged on to the database and changed the password:

Afterwards I changed the password for the windows user, “postgres” created on the installation accordingly.

Now the mix task works as expected.

Flashing my Samsung Galaxy SIII

Today I decided to flash my Samsung Galaxy SIII

I couldn’t bear the badly written “improvements” which Samsung generously added to the Android anymore. My last smartphone, a now retired HTC Desire gained much out of the Cyanogen Custom Rom, therefore I gave it a try.

Setup:

I have a Samsung Galaxy SIII (Intl) – i9300 from O2, a german mobilephone provider. My Laptop runs Windows 7 and for guidiance I use the official cyanogen Wiki for my phone.

I’m feeling adventurous today, therefore I’ll try the nightly build

First step: Save your data!

Ok, you could do it the fancy way (Titanium), or just copy all files from the phone via explorer. Remember that some Apps doesn’t save their data to file but in the database. (The only one that mattered to me was Jiffy)

Second: Download stuff

As in the wiki advised, I downloaded the Heimdall Suite 1.4 and the linked ClockwordMod Recovery and because the cyanogen server aren’t the fastest ones, I started to download my CustomRom as well. (Step 1 from the Wiki)

Third: Do scary stuff

And here comes the fun: Shut down the phone (Step 2 from the Wiki) and boot in the download mode. I let the automatic device detection of windows do its work. Afterwards I startded the zadig.exe (Step 3.1). For the records, my phone was called “Gadget Serial”

Screenshot of zadig.exe

I copied the downloaded Recovery File as mentioned in Step 4 and proceeded with Step 5.

My working commandline was

It’s worth to mention, that the wiki points out NOT to use an USB-Hub, but to connect your phone directly. I didn’t read that part…

Other than that, no problems with the following steps.

Fourth:

Luckily I had the adb tool already on my pc, so my adb push wasn’t a pain in the ass.

I added the Google Apps as well.

I followed the “Installing CyanogenMod from recovery” manual and had no problems. But I can’t stress it enough, wipe data/factory reset is realy important!

It went well for me, and in a week or so I’ll write a post about living with cyanogen 10.1

Setting up Wifi on Ubuntu 12.04 Server via command line (yeah, finally some technical stuff )

Problem:

Setting up a wireless adapter on Ubuntu server 12.04 LTS via CLI isn’t THAT easy. Plug’n’Play doesn’t simply work. Good thing there is Google. And a lot of helpful sites.

Target:

Insert the stick (or boot with it) and connect automaticly to the preferred network

Continue reading Setting up Wifi on Ubuntu 12.04 Server via command line (yeah, finally some technical stuff )

Visualizing data with Gnuplot

On my pursuade for a tool to visualize simple list values I found gnuplot:

from Wikipedia:

gnuplot is a command-line program that can generate two- and three-dimensional plots of functions, data, and data fits.

I will focus on the capabilities of gnuplot on two-dimensional plots and in special howto visualize data without a corresponding second data value. But first here is the idea why I needed such a tool:

Continue reading Visualizing data with Gnuplot