Wednesday, September 28, 2016

MKL and C# for dummies (Part 3)




Fast Fourier Transform

This time I want to talk about Fast Fourier Transform functions and how to use these in C# properly.

Intel MKL team work mainly with C++ and Fortran. They didn’t dedicate a lot of effort toward C# so sometimes if you work in C# and you want to use MKL you face some difficulties.

As I said in the previous article about MKL, the first difficulty is how to declare the functions in C# code. For this I suggest a document from Intel: 33774-Using_Intel_MKL_IPP_NET_Framework.pdf. You can find it on line. It is not only a problem of declaration, but also a question of how to interpret the error messages, sometimes you don’t know what’s going on.

For example if you wrong something, you receive messages like these:
  •          "Inconsistent configuration parameter"
  •          "Functionality is not implemented".

Where is a tutorial that explain what I have wronged? You have to contact the Intel Support and be patient.

With a little help you can understand that the first error is because you are assigning a wrong value to a variable (DftiSetValue). The second one is because you had configured a “Descriptor” with the wrong parameters (DftiCreateDescriptor).

Then after completed a declarative part you can test something, for example I used this web site (http://www.sccon.ca/sccon/fft/fft3.htm) with the FFT input and output certified, as reference.

For some example of FFT usage this is the page: https://software.intel.com/en-us/node/471390.

The problem is that, those example are written in Fortran or C++ and contains some little imprecisions that can confuse you.

Let consider the paragraph: “One-dimensional Out-of-place FFT”. Where they perform a real to complex conjugate-even transform.

They use a real vector for the input and a real vector for the output. But the output is a complex vector! Also the vector size was wrong. These things and other little imprecisions make me lost a lot of time. But not desperate because I will provide you what you need to use FFT in C#.

There are some possible versions of DftiComputeForward() function that compute the FF Transform so at a beginning I adopted this one:

status = DftiComputeForward ( desc_handle, xre_in, xim_in , yre_out, yim_out )

It allows 2 real vectors in input and two real vectors for the output. These vectors represents real and imaginary part.

By the way, although I didn’t receive any error from the method execution the result was always zero! Again problems!

But as I always say: “it ain't over til it's over” I have tried another version of the function, this time with a input a complex vector, and output a complex vector. And like a magic this time worked.

status = DftiComputeForward( desc_handle, x_in, y_out )


So for your joy here the code:


using System;
using System.Numerics;
using System.Runtime.InteropServices;
using System.Security;

namespace mkl
{
    [SuppressUnmanagedCodeSecurity]
    public unsafe class MKLWrapper
    {

    
        //==================================================
        #region FFT CONSTANTS
        enum DFTI_CONFIG_PARAM
        {
            /* Domain for forward transform. No default value */
            DFTI_FORWARD_DOMAIN = 0,

            /* Dimensionality, or rank. No default value */
            DFTI_DIMENSION = 1,

            /* Length(s) of transform. No default value */
            DFTI_LENGTHS = 2,

            /* Floating point precision. No default value */
            DFTI_PRECISION = 3,

            /* Scale factor for forward transform [1.0] */
            DFTI_FORWARD_SCALE = 4,

            /* Scale factor for backward transform [1.0] */
            DFTI_BACKWARD_SCALE = 5,

            /* Exponent sign for forward transform [DFTI_NEGATIVE]  */
            /* DFTI_FORWARD_SIGN = 6, ## NOT IMPLEMENTED */

            /* Number of data sets to be transformed [1] */
            DFTI_NUMBER_OF_TRANSFORMS = 7,

            /* Storage of finite complex-valued sequences in complex domain
               [DFTI_COMPLEX_COMPLEX] */
            DFTI_COMPLEX_STORAGE = 8,

            /* Storage of finite real-valued sequences in real domain
               [DFTI_REAL_REAL] */
            DFTI_REAL_STORAGE = 9,

            /* Storage of finite complex-valued sequences in conjugate-even
               domain [DFTI_COMPLEX_REAL] */
            DFTI_CONJUGATE_EVEN_STORAGE = 10,

            /* Placement of result [DFTI_INPLACE] */
            DFTI_PLACEMENT = 11,

            /* Generalized strides for input data layout [tigth, row-major for
               C] */
            DFTI_INPUT_STRIDES = 12,

            /* Generalized strides for output data layout [tight, row-major
               for C] */
            DFTI_OUTPUT_STRIDES = 13,

            /* Distance between first input elements for multiple transforms
               [0] */
            DFTI_INPUT_DISTANCE = 14,

            /* Distance between first output elements for multiple transforms
               [0] */
            DFTI_OUTPUT_DISTANCE = 15,

            /* Effort spent in initialization [DFTI_MEDIUM] */
            /* DFTI_INITIALIZATION_EFFORT = 16, ## NOT IMPLEMENTED */

            /* Use of workspace during computation [DFTI_ALLOW] */
            DFTI_WORKSPACE = 17,

            /* Ordering of the result [DFTI_ORDERED] */
            DFTI_ORDERING = 18,

            /* Possible transposition of result [DFTI_NONE] */
            DFTI_TRANSPOSE = 19,

            /* User-settable descriptor name [""] */
            DFTI_DESCRIPTOR_NAME = 20, /* DEPRECATED */

            /* Packing format for DFTI_COMPLEX_REAL storage of finite
               conjugate-even sequences [DFTI_CCS_FORMAT] */
            DFTI_PACKED_FORMAT = 21,

            /* Commit status of the descriptor - R/O parameter */
            DFTI_COMMIT_STATUS = 22,

            /* Version string for this DFTI implementation - R/O parameter */
            DFTI_VERSION = 23,

            /* Ordering of the forward transform - R/O parameter */
            /* DFTI_FORWARD_ORDERING  = 24, ## NOT IMPLEMENTED */

            /* Ordering of the backward transform - R/O parameter */
            /* DFTI_BACKWARD_ORDERING = 25, ## NOT IMPLEMENTED */

            /* Number of user threads that share the descriptor [1] */
            DFTI_NUMBER_OF_USER_THREADS = 26,

            /* Limit the number of threads used by this descriptor [0 = don't care] */
            DFTI_THREAD_LIMIT = 27,

            /* Possible input data destruction [DFTI_AVOID = prevent input data]*/
            DFTI_DESTROY_INPUT = 28
        };

        enum DFTI_CONFIG_VALUE
        {
            /* DFTI_COMMIT_STATUS */
            DFTI_COMMITTED = 30,
            DFTI_UNCOMMITTED = 31,

            /* DFTI_FORWARD_DOMAIN */
            DFTI_COMPLEX = 32,
            DFTI_REAL = 33,
            /* DFTI_CONJUGATE_EVEN = 34,   ## NOT IMPLEMENTED */

            /* DFTI_PRECISION */
            DFTI_SINGLE = 35,
            DFTI_DOUBLE = 36,

            /* DFTI_FORWARD_SIGN */
            /* DFTI_NEGATIVE = 37,         ## NOT IMPLEMENTED */
            /* DFTI_POSITIVE = 38,         ## NOT IMPLEMENTED */

            /* DFTI_COMPLEX_STORAGE and DFTI_CONJUGATE_EVEN_STORAGE */
            DFTI_COMPLEX_COMPLEX = 39,
            DFTI_COMPLEX_REAL = 40,

            /* DFTI_REAL_STORAGE */
            DFTI_REAL_COMPLEX = 41,
            DFTI_REAL_REAL = 42,

            /* DFTI_PLACEMENT */
            DFTI_INPLACE = 43,          /* Result overwrites input */
            DFTI_NOT_INPLACE = 44,      /* Have another place for result */

            /* DFTI_INITIALIZATION_EFFORT */
            /* DFTI_LOW = 45,              ## NOT IMPLEMENTED */
            /* DFTI_MEDIUM = 46,           ## NOT IMPLEMENTED */
            /* DFTI_HIGH = 47,             ## NOT IMPLEMENTED */

            /* DFTI_ORDERING */
            DFTI_ORDERED = 48,
            DFTI_BACKWARD_SCRAMBLED = 49,
            /* DFTI_FORWARD_SCRAMBLED = 50, ## NOT IMPLEMENTED */

            /* Allow/avoid certain usages */
            DFTI_ALLOW = 51,            /* Allow transposition or workspace */
            DFTI_AVOID = 52,
            DFTI_NONE = 53,

            /* DFTI_PACKED_FORMAT (for storing congugate-even finite sequence
               in real array) */
            DFTI_CCS_FORMAT = 54,       /* Complex conjugate-symmetric */
            DFTI_PACK_FORMAT = 55,      /* Pack format for real DFT */
            DFTI_PERM_FORMAT = 56,      /* Perm format for real DFT */
            DFTI_CCE_FORMAT = 57        /* Complex conjugate-even */
        };

        #endregion

        private MKLWrapper()
        {
        }
    
        //==================================================
        #region FFT (DECLARATIONS)

        //DFTI_EXTERN char* DftiErrorMessage(MKL_LONG);
        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern IntPtr DftiErrorMessage(
            long error_code
        );

        //DFTI_EXTERN MKL_LONG DftiCreateDescriptor(DFTI_DESCRIPTOR_HANDLE*,
        //                      enum DFTI_CONFIG_VALUE, /* precision */
        //                      enum DFTI_CONFIG_VALUE, /* domain */
        //                      MKL_LONG, ...);
        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int DftiCreateDescriptor(
            ref IntPtr handle_descriptor,
            int precision,
            int domain,
            int dime,
            int size
        );
        
        //DFTI_EXTERN MKL_LONG DftiCommitDescriptor(DFTI_DESCRIPTOR_HANDLE);
        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int DftiCommitDescriptor(
            IntPtr handle_descriptor
        );

        //DFTI_EXTERN MKL_LONG DftiSetValue(DFTI_DESCRIPTOR_HANDLE, enum DFTI_CONFIG_PARAM, ...);
        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int DftiSetValue(IntPtr handle_descriptor, int config_param, __arglist);

        //DFTI_EXTERN MKL_LONG DftiComputeBackward(DFTI_DESCRIPTOR_HANDLE, void*, ...);
        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int DftiComputeBackward(
            IntPtr handle_descriptor,
            [In] Complex[] x_in,
            [Out] Complex[] x_out
        );

        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int DftiComputeForward(
            IntPtr handle_descriptor,
            [In] Complex[] x_in,            
            [Out] Complex[] x_out
        );

        //DFTI_EXTERN MKL_LONG DftiFreeDescriptor(DFTI_DESCRIPTOR_HANDLE*);
        [DllImport("mkl_rt.dll", ExactSpelling = true, SetLastError = false, CallingConvention = CallingConvention.Cdecl)]
        internal static extern int DftiFreeDescriptor(
            ref IntPtr handle_descriptor
        );
        #endregion


        //==================================================
        #region FFT (METHODS)
        public static string MKL_FFT_Error(int error, string sText = "")
        {
            if (error == 0)
                return "";

            IntPtr s = DftiErrorMessage(error);
            string sMsg = Marshal.PtrToStringAnsi(s);            
            throw new Exception(sText + sMsg);
        }


        public static Complex[] MKL_FFT(Complex[] in_matrix, double fScalingFactor = 1)
        {
            int status;
            IntPtr des_handle = IntPtr.Zero;

            Complex[] out_matrix = new Complex[in_matrix.Length];

            try
            {

                status = DftiCreateDescriptor(ref des_handle, (int)DFTI_CONFIG_VALUE.DFTI_DOUBLE, (int)DFTI_CONFIG_VALUE.DFTI_COMPLEX, 1, in_matrix.Length);
                MKL_FFT_Error(status, "DftiCreateDescriptor() ");

                status = DftiSetValue(des_handle, (int)DFTI_CONFIG_PARAM.DFTI_FORWARD_SCALE, __arglist(fScalingFactor));
                MKL_FFT_Error(status, "DftiSetValue() ");

                status = DftiSetValue(des_handle, (int)DFTI_CONFIG_PARAM.DFTI_PLACEMENT, __arglist((int)DFTI_CONFIG_VALUE.DFTI_NOT_INPLACE));
                MKL_FFT_Error(status, "DftiSetValue() ");

                status = DftiCommitDescriptor(des_handle); //Finalize the descriptor
                MKL_FFT_Error(status, "DftiCommitDescriptor() ");

                status = DftiComputeForward(des_handle, in_matrix, out_matrix); //Compute the Backward FFT
                MKL_FFT_Error(status, "DftiComputeForward() ");

                status = DftiFreeDescriptor(ref des_handle); //Free the descriptor
                MKL_FFT_Error(status, "DftiFreeDescriptor() ");
                

                return out_matrix;
            }
            catch (Exception ex)
            {
                if (des_handle != IntPtr.Zero)
                    DftiFreeDescriptor(ref des_handle); //Free the descriptor
                throw ex;
            }

        }

        public static Complex[] MKL_FFT_Inv(Complex[] in_matrix, double fScalingFactor = 1)
        {
            int status;
            IntPtr des_handle = IntPtr.Zero;

            Complex[] out_matrix = new Complex[in_matrix.Length];
            
            
            try
            {

                status = DftiCreateDescriptor(ref des_handle, (int)DFTI_CONFIG_VALUE.DFTI_DOUBLE, (int)DFTI_CONFIG_VALUE.DFTI_COMPLEX, 1, in_matrix.Length); 
                MKL_FFT_Error(status, "DftiCreateDescriptor() ");

                status = DftiSetValue(des_handle, (int)DFTI_CONFIG_PARAM.DFTI_BACKWARD_SCALE, __arglist(fScalingFactor));
                MKL_FFT_Error(status, "DftiSetValue() ");
                
                status = DftiSetValue(des_handle, (int)DFTI_CONFIG_PARAM.DFTI_PLACEMENT, __arglist((int)DFTI_CONFIG_VALUE.DFTI_NOT_INPLACE));
                MKL_FFT_Error(status, "DftiSetValue() ");

                status = DftiCommitDescriptor(des_handle); //Finalize the descriptor
                MKL_FFT_Error(status, "DftiCommitDescriptor() ");

                status = DftiComputeBackward(des_handle, in_matrix, out_matrix); //Compute the Backward FFT
                MKL_FFT_Error(status, "DftiComputeBackward() ");

                status = DftiFreeDescriptor(ref des_handle); //Free the descriptor


                return out_matrix;
            }
            catch (Exception ex)
            {
                if (des_handle != IntPtr.Zero)
                    DftiFreeDescriptor(ref des_handle); //Free the descriptor
                throw ex;
            }
        }
        #endregion

    }
}

Here the example:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using mkl;
using System.Numerics;

namespace FFTTest
{
    class Program
    {

        #region  TEST INPUT MATRIX
        //http://www.sccon.ca/sccon/fft/fft3.htm
        //INPUT
        //REAL[0] = 1.000,   IMAG[0] = 0.000
        //REAL[1] = 0.000,   IMAG[1] = 0.000
        //REAL[2] = 0.000,   IMAG[2] = 0.000
        //REAL[3] = 0.000,   IMAG[3] = 0.000
        //REAL[4] = 0.000,   IMAG[4] = 0.000
        //REAL[5] = 0.000,   IMAG[5] = 0.000
        //REAL[6] = 0.000,   IMAG[6] = 0.000
        //REAL[7] = 0.000,   IMAG[7] = 0.000

        //RESULTS
        //REAL[0] = 0.125,   IMAG[0] = 0.000
        //REAL[1] = 0.125,   IMAG[1] = 0.000
        //REAL[2] = 0.125,   IMAG[2] = 0.000
        //REAL[3] = 0.125,   IMAG[3] = 0.000
        //REAL[4] = 0.125,   IMAG[4] = 0.000
        //REAL[5] = 0.125,   IMAG[5] = 0.000
        //REAL[6] = 0.125,   IMAG[6] = 0.000
        //REAL[7] = 0.125,   IMAG[7] = 0.000
        #endregion


        static void Main(string[] args)
        {

            Complex[] A = new Complex[8];
            Complex[] B = null;
            Complex[] C = null;


            A[0] = new Complex(0, 0);
            A[1] = new Complex(1, 0);
            A[2] = new Complex(0, 0);
            A[3] = new Complex(0, 0);
            A[4] = new Complex(0, 0);
            A[5] = new Complex(0, 0);
            A[6] = new Complex(0, 0);
            A[7] = new Complex(0, 0);


            try
            {
                DebugMatrix(A, "A:");

                B = MKLWrapper.MKL_FFT(A, 1.0f/ A.Length);
                DebugMatrix(B, "FFT() B:");

                C = MKLWrapper.MKL_FFT_Inv(B); //FFT Backward
                DebugMatrix(C, "iFFT() C:");

                //C = MKLWrapper.MKL_FFT(B);
                //DebugMatrix(lstResults, C, "A:");


                //MessageBox.Show("Operation completed succesfully!");
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error: " + ex.Message);
            }

            Console.ReadLine();
        }



        public static void DebugMatrix(Complex[] a, string sCaption, int iRowLimit = 10)
        {
            if (a == null) return;

            string sLine = "";
            Console.WriteLine("");
            Console.WriteLine(sCaption);
            int iDime1 = a.GetUpperBound(0);
            int iStart1 = a.GetLowerBound(0);

            Console.WriteLine("Size: " + a.GetLength(0).ToString());
            for (int i = iStart1; i <= iDime1; i++)
            {
                sLine = i.ToString() + "> ";
                sLine += a[i].ToString("0.###E+000") + "\t";
                //sLine += a[i, j].ToString("") + "\t";
                Console.WriteLine(sLine);
                if (i > iRowLimit)
                    break;
            }
            Console.WriteLine("");
            Console.WriteLine("");
        }
    }
}


Part 1 | Part 2 | Part 3 | Part 4

Friday, May 27, 2016

Memory Mapped Files Study


In this Post I face a new argument: Memory Mapped Files.

Why this choice?


Simply because I need a shared memory between processes where to store data, and MMF is a really good way to do this. I have a software that needs to read and write data to and from external devices, and the devices registries are global variables in my programs.

This solution guaranties a fast and easy storage, no database mechanisms, between data and applications (logic layer). I don’t pretend to be understood but simply share my studies about MMF.
Just to be clear, I’m not an expert of MMF but only a new user.


Window offers three groups of functions for managing memory in applications: memory-mapped file functions, heap memory functions, and virtual-memory functions.



Memory Mapped Files


A memory-mapped file contains the contents of a file in virtual memory. This mapping between a file and memory space enables an application, including multiple processes, to modify the file by reading and writing directly to the memory. Starting with the .NET Framework 4, you can use managed code to access memory-mapped files in the same way that native Windows functions access memory-mapped files, as described in Managing Memory-Mapped Files in Win32 in the MSDN Library. 

A good article is also: Managing Memory-Mapped Fileds (https://msdn.microsoft.com/en-us/library/ms810613.aspx).

There are two types of memory-mapped files:
  • Persisted memory-mapped files
Persisted files are memory-mapped files that are associated with a source file on a disk. When the last process has finished working with the file, the data is saved to the source file on the disk. These memory-mapped files are suitable for working with extremely large source files.
  • Non-persisted memory-mapped files
Non-persisted files are memory-mapped files that are not associated with a file on a disk. When the last process has finished working with the file, the data is lost and the file is reclaimed by garbage collection. These files are suitable for creating shared memory for inter-process communications (IPC).




This and more can be read in MSDN (https://msdn.microsoft.com/en-us/library/dd997372.aspxhttps://msdn.microsoft.com/en-us/library/dd997372.aspx). 


There are two types of views: 

Stream access view 
Random access view. 

Use stream access views for sequential access to a file; this is recommended for non-persisted files and IPC. Random access views are preferred for working with persisted files.

Functions


Memory-mapped file functions can be thought of as second cousins to the virtual-memory management functions in Windows. Like the virtual-memory functions, these functions directly affect a process's address space and pages of physical memory. No overhead is required to manage the file views, other than the basic virtual-memory management that exists for all processes. These functions deal in reserved pages of memory and committed addresses in a process. The entire set of memory-mapped file functions are not here discussed but only what I need for my example.


Example Project


In my example I will use a Random access view because I need to move up and down the variables list. I defined a class named MemoryManager which encapsulate all the code usefull for the MMF management.



public class MemoryManager
    {
        string msMutexName = "map_mutex";
        string msName = ""; //Memory Area Name
        MemoryMappedFile moMem = null;
        MemoryMappedViewAccessor moAncestor = null;
        
        public string Name { get { return msName;  } set { msName = value; } }

        int miObjSize = 0;
        int miCount = 0; //number of elements\objects

        public MemoryManager()
        {


I image to have a list of objects to map in a MMF, and a list of methods to query and writes values in this list.

No Generics or Classes


In general my list could be a generic list of object, but it is not possible to create standard method that manage Generics like this:

public void Create(List lListOfObjects)

because T or Objects can be nullable! So a Struct is the right choice. In My example I called it MyStruct.


/// /// In order to use MMF it is neccessary using not nullable types (16Byte)
/// 
public struct MyStruct
{
    public int Key; //Primary Key
    public int ValueInt; //Current In Value
    public float ValueFloat; //Current Float Value
    public int EditInt; //New Int value to write
    public float EditFloat; //New Float value to write
    public int Edit; //There is at least one value to write: Int or Float or both

    public MyStruct(int key, int value_int = 0, float value_float = 0, int edit_int = 0, float edit_float = 0, int edit = 0)
    {
        Key = key;
        ValueInt = value_int;
        ValueFloat = value_float;
        EditInt = edit_int;
        EditFloat = edit_float;
        Edit = edit;
    }
}}


public void Create(List lListOfObjects)
{
if (lListOfObjects == null) return;
            
bool mutexCreated;

miObjSize = Marshal.SizeOf(typeof(MyStruct));
miCount = lListOfObjects.Count;

int iBufferSize = miObjSize * miCount;

moMem = MemoryMappedFile.CreateNew(msName, miObjSize * miCount);
moAncestor = moMem.CreateViewAccessor(0, iBufferSize);

Mutex mutex = new Mutex(true, msMutexName, out mutexCreated);

int iIndex = 0;
for (int i = 0; i < miCount; i++)
{
        MyStruct o = lListOfObjects[i];
moAncestor.Write(iIndex, ref o);
iIndex += miObjSize;
       }               

       mutex.ReleaseMutex();
}


The method Create allocate a memory map with CreateNew(), assigning a name and a size to a map. While CreateViewAnchestor() create a random access view. The for section initialize the memory with the list.

From this moment ahead this memory space is available from other process that wants read and write into it.


public string Name { get { return msName;  } set { msName = value; }
public bool Open(int iCount)
{
try
{
miCount = iCount;
moMem = MemoryMappedFile.OpenExisting(msName);
return true;
}
catch (FileNotFoundException)
{
return false;
}            
}
public MyStruct Read(int iIndex)
{
    if (moMem == null)
        return new MyStruct(-1);

    miObjSize = Marshal.SizeOf(typeof(MyStruct));

    if (moAncestor == null)
    {
        int iBufferSize = miObjSize * miCount;
        moAncestor = moMem.CreateViewAccessor(0, iBufferSize);
    }
                

    int iBufferIndex = iIndex * miObjSize;

    MyStruct oRet;
    moAncestor.Read(iBufferIndex, out oRet);

    return oRet;
}

public void Write(int iIndex, MyStruct oValue)
{
    if (moMem == null)
        return;

    bool mutexCreated;
    Mutex mutex = new Mutex(true, msMutexName, out mutexCreated);
    mutex.WaitOne();

    miObjSize = Marshal.SizeOf(typeof(MyStruct));

    if (moAncestor == null)
    {
        int iBufferSize = miObjSize * miCount;
        moAncestor = moMem.CreateViewAccessor(0, iBufferSize);
    }

    int iBufferIndex = iIndex * miObjSize;
            
    if (moAncestor.CanWrite)
    {
        moAncestor.Write(iBufferIndex, ref oValue);
        Debug.Print(string.Format("Write index: {0} -> {1}, {2}, {3}", oValue.Key, oValue.ValueInt, oValue.ValueFloat, oValue.Edit));
    }
    mutex.ReleaseMutex();
}

Shortcut methods

public int ReadInt(int iIndex)
{
    MyStruct oValue = Read(iIndex);
    return oValue.ValueInt;
}

public void WriteInt(int iIndex, int iValue)
{
    MyStruct oValue = Read(iIndex);
    oValue.ValueInt = iValue;
    Write(iIndex, oValue);
}


download the example

Monday, April 11, 2016

How to query a remote database, located in a hosted server.


How can I get information, located in a remote database, which is hosted by a Service Provider Company?

You cannot have any direct access to it, because the Service Provider does not allow this access level.
However, if you have a dynamic web site hosted in there that uses this database, it can query the database tables for you. Is this a strange goal?

I mean, I have not any Server availability but a hosted one. I maintain some dynamic sites available on internet, and the Service Provider hosts these sites.
I can use this service for other purpose, like to provide news to a program running on windows or any other operating system, Or to maintain license information in order to manage a network program key.

Many other ideas can arise of course.

The solution I suggest: is to query the database via php so that it will return results in JSON format.

JSON 

JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language.

JSON example
{"news":[
    {"title":"ver 1.0", "text":"this version …"},
    {"title":"ver 1.1", "text":"this version …"},
    {"title":"ver 1.2", "text":"this version …"}
]}


MySQL 

MySQL is the RDBMS where are stored the information used for this example.

CREATE TABLE news (
`news_id` int AUTO_INCREMENT PRIMARY KEY,
`product_id` int,
`published` datetime,
`title` nvarchar(100),
`text` TEXT
);

Image to have n records, where each of this reflects a new edition availability of your program.

PHP 

Image to have a Php file that queries database using information passed by user, this is product_id. The query extracts the last published news and return a JSON file with a string that contains that information.

query_news.php:

$array = array();

$pid = $_POST["pid"];
$hostname = "localhost";
$database = "helpdesk";
$username="root";
$password="root";

$cnn = new PDO("mysql:host=$hostname;dbname=$database", $username, $password); 
$sql = "SELECT published, title, text FROM news WHERE product_id = $pid ORDER BY news_id DESC LIMIT 1;";
$q = $cnn->query($sql);

$record = $q->fetch(PDO::FETCH_BOTH); //there is only one record returned!
$output = $record[0]."|".$record[1]."|".$record[2];
array_push($array, $output);
echo json_encode($array);
$cnn = null;

Many other improvements could be taken of course.

C# Here we are! 

What we have to do now, is to call Php file in order to get some information about last news published.

Console example:

static void Main(string[] args)
{
    string URI = "http://mydomain//query_news.php";

    string myParameters = "pid=1";

    using (WebClient wc = new WebClient())
    {
        wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
        string sHtmlResult = wc.UploadString(URI, myParameters);
        sHtmlResult = sHtmlResult.Substring(4);
        sHtmlResult = sHtmlResult.Substring(0, sHtmlResult.Length - 2);
        Console.WriteLine("" + sHtmlResult);
    }

    Console.ReadLine();
}

This simple console program uses a webclient service to call the Php file and shows the results.

Security 

Once we have understand how the previous code works, it is possible to go ahead. For example using security precautions like integrated browser specifications TLS, SSL or simply a cryptographic functions implemented directly in the code. In this example I use The Advanced Encryption Standard (AES), also known as Rijndael (wiki).

Php Side:

class CryptoClass 
{
    public $ky = "lkirwf897+22#bbtrm8814z5qq=498j5"; // 32 * 8 = 256 bit key
    public $iv = "741952hheeyy66#cs!9hjv887mxx7@8y"; // 32 * 8 = 256 bit iv
    
    public function __construct() {
      
    }

    public function encrypt($string_to_encrypt)
    { 
$rtn = $this->encryptRJ256($this->ky, $this->iv, $string_to_encrypt);
return($rtn);
    }

    public function decrypt($string_to_decrypt)
    {
$rtn = $this->decryptRJ256($this->ky, $this->iv, $string_to_decrypt);
return($rtn);
    }


    function decryptRJ256($key, $iv, $string_to_decrypt)
    {
$string_to_decrypt = base64_decode($string_to_decrypt);        
$rtn = mcrypt_decrypt(MCRYPT_RIJNDAEL_256, $key, $string_to_decrypt, MCRYPT_MODE_CBC, $iv);
return($rtn);
    }

    function encryptRJ256($key,$iv,$string_to_encrypt)
    {
$rtn = mcrypt_encrypt(MCRYPT_RIJNDAEL_256, $key, $string_to_encrypt, MCRYPT_MODE_CBC, $iv);
$rtn = base64_encode($rtn);
return($rtn);
    }        
}

C# Side:

public class CryptoClass
{
    string sKy = "lkirwf897+22#bbtrm8814z5qq=498j5"; //32 chr shared ascii string (32 * 8 = 256 bit)
    string sIV = "741952hheeyy66#cs!9hjv887mxx7@8y"; //32 chr shared ascii string (32 * 8 = 256 bit)
        
    public CryptoClass()
    { }
        
    public string Decrypt(string prm_text_to_decrypt)
    {
        return DecryptRJ256(sKy, sIV, prm_text_to_decrypt);
    }
        
    public string Encrypt(string prm_text_to_encrypt)
    {
        return EncryptRJ256(sKy, sIV, prm_text_to_encrypt);
    }

    private string DecryptRJ256(string prm_key, string prm_iv, string prm_text_to_decrypt)
    {

        try
        {
            var sEncryptedString = prm_text_to_decrypt;

            var myRijndael = new RijndaelManaged()
            {
                Padding = PaddingMode.Zeros,
                Mode = CipherMode.CBC,
                KeySize = 256,
                BlockSize = 256
            };

            UTF8Encoding encoding = new UTF8Encoding();

            var key = encoding.GetBytes(prm_key); //Encoding.ASCII.GetBytes
            var IV = encoding.GetBytes(prm_iv);//Encoding.ASCII.GetBytes

            var decryptor = myRijndael.CreateDecryptor(key, IV);

            var sEncrypted = Convert.FromBase64String(sEncryptedString);

            var fromEncrypt = new byte[sEncrypted.Length];

            var msDecrypt = new MemoryStream(sEncrypted);
            var csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read);

            csDecrypt.Read(fromEncrypt, 0, fromEncrypt.Length);

            return (encoding.GetString(fromEncrypt));//Encoding.ASCII.GetBytes
        }
        catch (Exception ex)
        {
            return "error: " + ex.Message + Environment.NewLine + prm_text_to_decrypt;
        }
    }

    private string EncryptRJ256(string prm_key, string prm_iv, string prm_text_to_encrypt)
    {

        var sToEncrypt = prm_text_to_encrypt;

        var myRijndael = new RijndaelManaged()
        {
            Padding = PaddingMode.Zeros,
            Mode = CipherMode.CBC,
            KeySize = 256,
            BlockSize = 256
        };

        var key = Encoding.ASCII.GetBytes(prm_key);
        var IV = Encoding.ASCII.GetBytes(prm_iv);

        var encryptor = myRijndael.CreateEncryptor(key, IV);

        var msEncrypt = new MemoryStream();
        var csEncrypt = new CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write);

        var toEncrypt = Encoding.ASCII.GetBytes(sToEncrypt);

        csEncrypt.Write(toEncrypt, 0, toEncrypt.Length);
        csEncrypt.FlushFinalBlock();

        var encrypted = msEncrypt.ToArray();

        return (Convert.ToBase64String(encrypted));
    }
}

query_news.php:

$cry = new CryptoClass();
$array = array();

$input = $_POST["pid"]; 
if (!isset($input)) { 
    echo "  Post Error parameter!  ";
    return;
}

$pid = chop($cry->decrypt($input), "\0");
$hostname = "localhost";
$database = "helpdesk";
$username="root";
$password="root";

$cnn = new PDO("mysql:host=$hostname;dbname=$database", $username, $password);

if (!$cnn) {
    die("Server connection $hostname failed!");
}

try {

    $sql = "SELECT published, title, text FROM news WHERE product_id = $pid ORDER BY news_id DESC LIMIT 1";
    $q = $cnn->query($sql);
    if ($q == false)
    {
        echo "  SQL query error ([input=$input]  [pid=$pid])!  ";
        return;
    }
    $q->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

    $record = $q->fetch(PDO::FETCH_BOTH);
    $output = $record[0]."|".$record[1]."|".$record[2];
    $output = $cry->encrypt($output);
    array_push($array, $output);
    echo json_encode($array);
}
catch (PDOException $e) {
    $cnn = null;
    array_push($array, 'Error: ' . $e->getMessage());
    echo json_encode($array);
}

C# Console:

static void Main(string[] args)
{

    CryptoClass cc = new CryptoClass();
    string sURI = "http://localhost:33137//helpdesk//helpdesk//query_news.php";

    string sKey = cc.Encrypt("1"); 
    string myParameters = "pid=" + sKey;

    using (WebClient wc = new WebClient())
    {
        wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
               
        NameValueCollection reqparm = new System.Collections.Specialized.NameValueCollection();
        reqparm.Add("pid", sKey);
        byte[] responsebytes = wc.UploadValues(sURI, "POST", reqparm);
        string sHtmlResult = Encoding.UTF8.GetString(responsebytes);

        sHtmlResult = sHtmlResult.Substring(4);
        sHtmlResult = sHtmlResult.Substring(0, sHtmlResult.Length - 2);
        string sRes = cc.Decrypt(sHtmlResult);
        Console.WriteLine("" + sRes);
    }

    Console.ReadLine();
}


Now you have enough elements to try yourself.