Scala : easily load an parse configuration files

There are a lot of ways to handle configuration files in a Scala program; my favorite one is to use TypeSafe's Config project

This library scans the CLASSPATH for a predefined set of configuration files in different formats

Using it it's pretty straightforward so let's get to it :

1.- Add your dependency to sbt
libraryDependencies += "com.typesafe" % "config" % "1.2.1"
2.- Loading the configuration file

//load the configuration file from the classpath
val conf = ConfigFactory.load

The convenience method ConfigFactory.load() loads the following (first-listed are higher priority):

  • system properties
  • application.conf (all resources on classpath with this name)
  • application.json (all resources on classpath with this name)
  • application.properties (all resources on classpath with this name)
  • reference.conf (all resources on classpath with this name)

The idea is that libraries and frameworks should ship with a reference.conf in their jar. Applications should provide an application.conf, or if they want to create multiple configurations in a single JVM, they could use ConfigFactory.load("myapp") to load their own myapp.conf. (Applications can provide a reference.conf also if they want, but you may not find it necessary to separate it from application.conf.)

3.- Reading the configuration file values
//retrieve configuration files values
val remoteIp = conf.getString("server.ip")
val remotePort = conf.getInt("server.port")

That's it for today; have fun!

OS X ethernet self-assigned ip

Recently I had a problem with my mac, my wifi connection was working properly but my ethernet connection was not it was getting a self-assigned IP. After checking the obvious culprits (cable, router, switch) it turns out that the problem was elsewhere.

After searching the internet I realized that I wasn't the only one having this problem, and there are a lot of fixes such as :

  • Deleting the "/Machintosh HD/Library/Preferences/com.apple.alf.plist" and rebooting
  • Resetting the PRAM
  • Recreating a network configuration

By chance I decided to try and disable IPv6 on the network interfaces like so :

networksetup -setv6off Wi-Fi
networksetup -setv6off Ethernet

And after rebooting it worked !!

So there you go; I hope this works for you too!!

On a final note: Another thing I was about to try was uninstalling the virtual interfaces of my VMWARE but it didn't get to that

Git svn clear stored credentials

Recently a change was made in my company in the way the SVN repositories were handled; as a result we had to change the accounts we used to access the repos

Whenever I tried pulling or pushing info into those repositories I was getting errors like :


RA layer request failed: PROPFIND request failed on '/svn/2001/ufasoli/myproject/trunk': PROPFIND of '/svn/2001/ufasoli/myproject/trunk'
: 403 Forbidden (https://internal.svn.com) at /lib/Git/SVN.pm line 717

I realized that it was because the credential information was cached locally on the machine, so the system was sending old information , the tricky part was that the old accounts allowed authentication on the repositories, they were just no longer authorized to do anything on them, so I was getting an HTTP 403 error and the system was not prompting me for a new password (since the old accounts could still log in...)

So this is how I fixed it on my windows machine:

Under your user folder you should have a folder where subversion stores cached data C:\Users\ulf\.subversion (in my case)

This folder contains an auth folder were you will normally find another 3 folders :

  • svn.simple
  • svn.ssl.server
  • svn.username

To delete the cached credentials you can delete the content of the following folders :

  • svn.simple
  • svn.ssl.server

Once this folders deleted git should prompt you again for your username/password

Building and installing git 2.x on Windows

So Git 2.x has been out for a few months now and it came with some nice stuff, you can read more about it here

However if you are under windows, the git portage msysgit has yet to release a 2.X portage(as of this writing the latest version is 1.9.4)

If you want to install Git 2 on Windows you will need to build it from the source, and here is how it's done

1. Download the Git build environment

Head up to the Github page https://github.com/msysgit/msysgit and download the latest version from the master branch

Unzip it let's say to c:/tmp/git/msygit-env

2. Download the latest version of git

Head up to the Github page https://github.com/msysgit/git and download the latest version from the master branch

Unzip it let's say to c:/tmp/git/git

3. Copy the git folder

Now copy the entire contents of the c:/tmp/git/git under c:/tmp/git/msygit-env/git

4. Build Git

Now you can fire up a DOS console and head up to the msysgit and build Git


cd c:/tmp/git/msygit-env
msys.bat

Once the build starts you should see some output similar to

-------------------------------------------------------
Building and Installing Git
-------------------------------------------------------
GIT_VERSION = 2.1.0
    * new build flags
    CC credential-store.o
    * new link flags
    CC abspath.o
    CC advice.o
    CC alias.o
    CC alloc.o
    CC archive.o
    CC archive-tar.o

5. Moving the binaries and updating the path

Once the build is finished you will find the git binaires under the /bin folder

So let's say for example that I want to put my git binaries under c:/dev/bin/git2 go ahead and copy the following folders from the c:/tmp/git/msygit-env/git :

  • bin
  • cmd
  • etc
  • include
  • lib
  • libexec
  • mingw
  • share
  • ssl

Please note that I don't think you actually need all those folders but I copied them all just to be sure ..

Now for the final step the PATH; you will need to add the following folders to your PATH :

$PATH;C:\dev\bin\git2\bin;C:\dev\bin\git2\mingw\bin;C:\dev\bin\git2\cmd

Once the cmd prompt restarted you should be able to use your newly build Git

git --version
git version 2.1.0

And there you go if everything went as expected you should be running git 2.x

NodeJS, tune or disable HTTP agent pooling

When using NodeJS's HTTP module to query you might end-up having performance problems if you do some heavy HTTP querying.

By default when executing queries using the HTTP module NodeJS will use connection pooling with the maximum number of connection sockets being 5, so if you hit the bottleneck your HTTP queries will lag as they will be queued.

To my knowledge you have 2 options to "counter" this behavior :

  1. Increase the number of http sockets
  2. Disable the HTTP agent per request

1. Increase the number of http sockets

    var http = require('http');

    // increase the maximum number of http connections in the pool
    http.globalAgent.maxSockets = 100;

    //regular http request options
    var options = {
        port: 1337,
        hostname: '127.0.0.1',
        method: 'GET',
        path: '/test'
  };

  // make http query
  var req = http.request(options, function(res) {
       console.log('STATUS: ' + res.statusCode);
       console.log('HEADERS: ' + JSON.stringify(res.headers));
       res.setEncoding('utf8');
       res.on('data', function (chunk) {
          console.log('BODY: ' + chunk);
      });
   });



2.Disable the HTTP agent per request


    var http = require('http');
  
      //regular http request options
      var options = {
         port: 1337,
         hostname: '127.0.0.1',
         method: 'GET',
         path: '/test',
         agent:false //disable the nodejs agent (no connection pooling)
      };

      // make http query
      var req = http.request(options, function(res) {
       console.log('STATUS: ' + res.statusCode);
       console.log('HEADERS: ' + JSON.stringify(res.headers));
       res.setEncoding('utf8');
       res.on('data', function (chunk) {
          console.log('BODY: ' + chunk);
      });
   });

Please be aware that changing this options might have side effects on your host system such as higher memory and bandwidth use so use it wisely

Spring boot JPA broken encoding on database when loaded from data.sql file

As covered in a previous article (here and here) spring-boot is a very useful framework when working on full enabled Spring projects

Recently I wasted some time with yet another encoding problem.. where all my accentuated and special characters where corrupted.

I was using the data.sql convention that states that if a file named data.sql is found in the CLASSPATH spring will use this file to bootstrap the database, my SQL file was properly encoded in UTF-8 as well as my maven-resources-plugin however the data recovered from the database was corrupted (accents and special characters where broken)

After reading a bit the documentation I found out that the source of the problem was at the moment of reading the SQL file that the encoding was causing trouble.. thankfully you can fix this one pretty easlily by changing one property value :

spring.datasource.sqlScriptEncoding=UTF-8

spring-boot has a lot of useful properties that allow you to tune the framework which you can find here

Maven custom packaging with the assembly plugin

It's not uncommon to need a custom packaging or assembly for a given project for some reason or another, this can be accomplished in a certain number of ways

Recently I needed an application to be packaged as the following structure :

|- runnable-jar.jar
|- lib
    |-- axis.jar
    |-- commons.logging.jar 
|- certificates
   |-- cacerts.jks
|- config
   |-- config.properties
|- logs
   

The schema is pretty self-explanatory I needed a runnable JAR at the root of my dist folder with a few other folders such as config, logs, etc.

One way to accomplish this with maven is to use maven-assembly-plugin

The Assembly Plugin for Maven is primarily intended to allow users to aggregate the project output along with its dependencies, modules, site documentation, and other files into a single distributable archive.

Your project can build distribution assemblies easily, using one of the convenient, prefabricated assembly descriptors.

These descriptors handle many common operations, such as packaging a project's artifact along with generated documentation into a single ZIP archive. Alternatively, your project can provide its own descriptor and assume a much higher level of control over how dependencies, modules, file-sets, and individual files are packaged in the assembly.

Currently the plugin can create distributions in the following formats:

  • zip
  • tar
  • tar.gz
  • tar.bz2
  • jar
  • dir
  • war
  • format that the ArchiveManager has been configured

For this tutorial we will be using the dir format in order to output the project structure described above

Let's start with the easy part first, the maven assembly plugin configuration where we will tell the plugin where to find our assembly file and where to output the result of the assembly operation


    
...

    
        
            
                maven-assembly-plugin
                
                    false
                    
                        
                        src/main/resources/assembly.xml
                    
                    
                    ${project.build.directory}/dist/
                
            

            
                maven-resources-plugin
                
                    UTF-8
                
            
            
                maven-compiler-plugin
                
                    ${java.version}
                    ${java.version}
                    ${project.build.sourceEncoding}
                
            
        
    
...


Now since in our case we want all the files in the resource directory to be handled by the assembly plugin we will configure the maven resources plugin to ignore everything inside the resources folder (please note that this is an optional step and it might not apply to your project)


...  

    
       
          ${basedir}/src/main/resources
          
              **/**
           
        
     

...

Once the maven part is done we can get started with the core code

If you remember we will end up with a runnable JAR file so go ahead and create a Main class like so :


package com.ufasoli.maven.custom.assembly;

/**
 * User: Ulises Fasoli
 * Date: 23.06.2014
 * Time: 18:03
 * Project : maven-custom-assembly
 */
public class Main {

    public static void main(String[] args) {
        System.out.println("HELLO WORLD !");
    }
}

Since this is going to be a runnable JAR file we will need to configure the maven JAR plugin to generate the manifest file with our main class




   org.apache.maven.plugins
   maven-jar-plugin
   2.4
   
      
         
            com.ufasoli.maven.custom.assembly.Main
            lib/
            true
         
       
   

Now we get to the core of the subject, the assembly file :


    paynet-dir
    
        dir
    
    false
    
        
        
            /lib
            
                ${project.groupId}:${project.artifactId}
            

        
             
            
                /
                
                    ${project.groupId}:${project.artifactId}
                
            

    

    
        
            src/main/resources/keystore/Certificate.SSL
            /certificates
        
        
            src/main/resources/config
            /config
        

        
            src/main/resources/config
            /logs
            
                *
            
            false
        

        
            src/main/resources/docs
            /docs
            
                *
            
        
    

Now this file is pretty self-explanatory but below are the highlights :

  • format --> the output format for our assembly (directory in our case)
  • dependencySets--> which artifacts to include in the final structure and where to put them
  • fileSets--> which files to include in the final structure and where to put them

Now you can build your assembly by running the following maven goal :

    mvn clean package assembly:single

Once the goal completed under the /target folder you should find your custom project structure

As usual you can find the code source for this tutorial over at my github account here

OSX show used ports or listening applications with their PID

On OSX you can display applications listening on a given port using the lsof the commands described below will show listening application...