Friday, July 31, 2009

Why is XML so Difficult for SAS Programmers? There must be a better way than PROC CDISC. http://bit.ly/ZPJKD

Why is XML so Difficult for SAS Programmers?

The FDA has adopted the use of XML as a standard file format for everything from a standard product labeling file format to electronic submission. This standard file format is widely used on the Internet and is useful for storing data along with metadata that explains the context of the data values. I am a team member of the SDS, part of CDISC establishing the way we use XML in the DEFINE.XML used to document clinical data in a submission. The XML is very well suited for the storage of the metadata describing how each variable is derived and detailing its code lists (controlled terms).

One challenge I find is that most users that work with clinical data are either data managers or SAS analysts who are responsible on generating the DEFINE.XML, do not understand XML. Syntactically, XML is a mark up language and the schema it uses is not rocket science. However, its requires a very different skill set since most users are familiar with SAS or SQL and does not really normally work with XML or the XSL that accompanies it for display. SAS’s attempt at implementing PROC CDISC in my opinion has come up short in closing this gap. Many users are therefore faced with the challenge of looking through the XSD files to understand the structure of the XML file. This is like trying to pick up Japanese when you have been using English for many years.

XML is useful for both storing data and describing data. Its accompanying tools such as XSL can also make use out of the data stored to have very useful functionality. For example, a XML file coupled with the right XSL can result in an HTML or even an Excel XLS looking result. I tried using some of these techniques in resolving the problem I see between SAS analysts and the need for generating DEFINE.XML. You can log in and sign up for free to try out Definedoc and how it can capture the metadata from SAS dataset to then generate a DEFINE.XML. This can be done without having to understand the XSD and XSL that accompanies it. I prefer to jump into a car and use it to get me from point A to B rather than learning all the details of the internal combustion engine. I hope this solution can help to close the gap and allow users to easily work with XML to gain the value it gives you as a powerful file format.

Thursday, July 30, 2009

invited to teach a course on Clinical Data Management Software in Amman, Jordan. Before I sign my life away, any suggestions?

Wednesday, July 29, 2009

Teaching a two day course on CDISC SDTM and DEFINE.XML for U.S. Military HIV Research http://bit.ly/6holA

Tuesday, July 28, 2009

A useful free directory of CDISC software and services Wiki. http://bit.ly/8gZdf

Monday, July 27, 2009

Generating DEFINE.XML for CDISC SDS team with latest draft guidelines. Still in draft but getting close. http://bit.ly/dao0C

Saturday, July 25, 2009

Facebook Identity Crisis with SAS

Facebook has become the most popular social networking site and have users spending more time on its site than any other leaving Yahoo and Google behind. It is becoming the entry point for many users as they log on. It is therefore a natural integration point for applications such as SAS any my humble little free data viewer Syview :). In the past, it has been challenging to share and view SAS data since it requires the user receiving the data to either have SAS or the right version of a viewer and on specific OS or platform.

I have been experimenting with a method where a Facebook account can be synchronized with Syview so that users do not have to use separate accounts. In this case, the user can use their Facebook account and also log into Syview to view SAS datasets.


I have an identify crisis sometimes with my personal use of social network and my professional online persona. Wouldn’t it be nice if I can reconcile this identity crisis. At the very least, with the user ID.

Thursday, July 23, 2009

helping answer CDISC and SAS questions at new forum http://bit.ly/YXf9l ... needs a new logo :)

Monday, July 20, 2009

I have been invited to host and moderate a CDISC Panel Discussion in September. How cool is that? http://bit.ly/6n6Ar

CDISC Panel Discussion

I have been invited to host and moderate a panel discussion at the next Western Users of SAS Software (WUSS) conference in San Jose. This will be in the CDISC section so we will be discussing meta data. I will share with you some general information about the discussion below.

Title
DEFINE.XML - A Practical Hands on Real World Experiences on Data Definition

Date Wednesday, September 2, 2009
Time 5:30-6:30pm
Where Fairmont San Jose in San Jose, California

Abstract
As CDISC guidelines nears becoming a regulation, it becomes apparent for the need of an effective method for documenting and communicating metadata to facilitate the review of clinical data during a submission. DEFINE.XML has proven in initial implementations and FDA pilot project to be an essential part in helping reviewers navigate data and related reports while also pointing out the needed improvements and challenges from implementing this in XML. This panel discussion will highlight real world examples of various approaches and shares lesson learned and innovative unique approaches. This includes technical techniques using SAS and processes and methodologies that spans any specific technologies.

Sunday, July 19, 2009

Is the long tail of software killing software as we know it? How niche apps will displace the block buster apps. http://bit.ly/KY3Ra

The Long Tail of Software

In the traditional software model where we purchase it on CD and it comes in a shrink wrap box with a user manual is becoming extinct in a similar way that memos became extinct when email came along. The way software is delivered has dramatically changed in a “flat world” as Thomas Friedman describes our world is flattening as high speed internet connection becomes more ubiquitous. Some software is delivered directly as a service across the web (or the cloud) as Google and companies like Salesforce.com is incorporating it into browsers. What is more interesting is the proliferation of smaller applications developed for niche usage delivered to mobile devices such as the iPhone available through the AppStore. In Chris Anderson’s book, The Long Tail, his thesis was how the efficiency gains in distribution have created a proliferation of products on the tail of a distribution curve which becomes very long compared to its head. Although he uses examples of pop culture products such as music CDs, DVDs and books and how they are delivered through Amazon and Netflix. This has expanded the tail as consumers explore the recommendations purchase items they did not before. This is possible since it is more cost effective of these products to become available compared to its bricks and mortar stores which can not affordably deliver the same physical products. This long tail phenomenon is occurring in all industry and is not limited to just books on Amazon. In the world of software, the AppStore represents a democratizing affect where an independent software developer can compete directly with the Microsoft. Not only does this affect the head of the curve as block buster software is delivered, but the more and more smaller niche applications are on the rise delivering useful apps to their specific audience.

Software in every industry is no longer going to be only made available through the 500 pound guerilla within their own sectors as delivered in a box. There is a shift was more interesting, innovative and more niche applications are coming out and are finding their own audience. Similar to how Amazon pave the way for less known books or iTunes pave the way for music, the AppStore will streamline software delivery to bring about a whole new set of niche applications that will augment and displace the older model of software as we know it.

Tuesday, July 14, 2009

Setting Up SAS Application Server for iPhone

An application “server” in this context is essentially a SAS session waiting to receive requests from the user. Once a request is received, it would execute the specified SAS scripts and then deliver its output back to the user. One application server can serve many users at once. The server is able to do this when queuing up multiple requests and then executes it one at a time.


Each application server is assigned a set of SAS libraries to easily access predefined data and related programs.The SAS program that is requested by the user also resides on the server as defined to a library. This can be pre-assigned similar to how you can assign a LIBNAME or FILENAME in an AUTOEXEC file used during an execution of a SAS program. The assignment of these libraries is established when the application server is setup.
Each application server communicates to its corresponding iPhone application through its own assigned TCP Socket port. This ensures that there is no collision between multiple applications for optimal performance. If needed, it is optional to increase the number of ports to handle many requests. In this case, a separate SAS session is started on the server running simultaneously to enable greater bandwidth.

Secure Users
There are two distinct roles that users play to enable the delivery of SAS data and programs to the iPhone including an administrator and the iPhone user. The administrator is usually the SAS analyst or statistician that developed and manages the data and SAS program on the server. The iPhone user receives the reports and data onto their iPhone. Before a user can access SAS programs and data, the following steps are taken to ensure proper security.
Step 1 – Account Setup
An account is set up on the server with the proper credentials in order to identify and authorize the user. The key attributes needed to authenticate the user include a unique user ID and user defined password.
Step 2 – User Privileges
By default, the user only has access to a set of sample SAS macro programs and datasets that come with the system. In order to deliver real information, additional SAS macros and data need to be registered and have read and execute permissions granted.



.



The permissions model is rather simplified compared to an operating system since the only two types of privileges needed include “read” permission for SAS data and “execute” permissions for SAS macros. It is implied that if the user can execute a macro that they also have read permission to the macro and its related output.

Step 3 – Login and View
Once the administrator applies the correct permissions, the iPhone user can see the programs appear on their selection lists. They can then view the data in a viewer or execute the macro and view its results.

It is useful to have users access the data to which they have been granted access. The management of each program and data however adds administrative overhead. In this example, the effort is kept to a minimal so that both the administrator and iPhone user can get to the information that is needed efficiently.

Variable

Type

Label

Objid

Num

Object ID

libname

Char

Library Name

Items

Char

Items

Type

Char

Object Type

userinter

Char

User Interaction Name

datetime

Num

Date Time of User Interaction

privileges

Char

Privileges



All the user permission is handled on the server and stored centrally within a SAS dataset. Each dataset and SAS macro metadata is managed centrally on the server within a dataset with each object assigned a unique identifier. The dataset structure that stores this is shown here.
In this case the privileges variable stores a simply value “read” or “write”. This dataset and set of functions to manage privileges have been intentionally simplified for this specific purpose to optimally manage the access control list for the iPhone application.

Sunday, July 12, 2009

How do you execute a SAS macro from an iPhone? http://bit.ly/9lkEX

Saturday, July 11, 2009

SAS Macro on the iPhone

In the traditional batch environment, users specify the options for a macro by typing the selected values in a SAS program script when invoking a macro. The entered values would then be processed by the macro by inserting them into the specified parameter in the code. An example macro call is:

*** Generate Report of data by specific subset ***;
%dataview (indata=mylib.demog,
sortby = subjid startdt,
reptitle = Demographic data sorted by subject ID and start date);

In this example, there are three parameters including: INDATA, SORTBY, and REPTITLE. The iPhone application makes this more user friendly as compared to having the user write a SAS macro call. In this case, the iPhone user selects the values from a macro parameter through the multi touch interface of the iPhone user interface. The user would therefore access and execute the %dataview macro by performing the following steps.

Step 1 – Selecting Macro

The user would navigate to the macro section by tapping on the macro button on the navigational bar at the bottom. This brings the user to a list of all the SAS program macros that the user has privileges to in the current library.


The user can navigate through different libraries by taping on the library choice as shown in the current “PGMDAT” example. In this example, they would tap on the “%DATAVIEW Macro” to drill down to its parameters.

Step 2 – Selecting Macro Parameters
Upon the selection of the %datataview macro, all the macro parameters will be presented with standard iPhone user interface elements. Each parameter will be listed as in the order in which it is defined.


In the traditional macro approach, the user would input the values of the parameter by typing the text inside a SAS script according to the named parameters. This is difficult for users when they are not familiar with the macro or associated data. This commonly leads to erroneous entries and errors in the macro execution. The iPhone interface is much more user friendly. In this example, three distinct entry types are displayed.

  1. Input Data – The input data parameter uses a standard SAS two level dot notation which is LIBNAME.DATASET. In this example, it is “mylib.demog” which refers to a library “MYLIB” and the dataset DEMOG. The user can edit this as open text or in addition, there is the option of drilling down to a list for the selection of libraries and datasets in selection.

  2. Sort Variables – The sort variables is a standard multiple option selection list. There are several different type of selection list. In this case, a simplified multiple selection list is presented as an example.

  3. Report Title – The report title requires a standard text entry field which the user can type any text value with the aid of an onscreen keyboard.

There are many other types of controllers which parameters can be configured to make the user entry more intuitive. Keep in mind that the user is using their index finger or thumb on a multi-touch screen. This is very different than a pointer device such as a mouse on a large desktop monitor. These other example selection options illustrates how the iPhone interface can optimize to allow users to select their parameters effectively from a mobile device.


  1. Check List

  2. Spin Controller

Some of the graphical user interface elements are similar to those found on desktop windows applications. The text entry and check list is similar to a text entry or list box on desktop applications. They only differ in their layout so it is easy for users to tap with their fingers. However, some user interface elements diverge from the desktop interface to fully take advantage of the mutli-touch smaller screen of an iPhone. This is particularly apparent in the spinning control which looks very different compared to any controller on a desktop application. The author of the macro would configure their macros and select which controller is best suited for each parameter. Default list of values will be displayed based on values of a dataset or the SAS system views such as SASHELP.VSLIB for list of available libraries. Once the macro authors configures this on the server, the user can benefit from having a user friendly method of selecting and executing the macros.

Wednesday, July 8, 2009

Viewing SAS Datasets on iPhone

SAS macros are very effective at allowing users to perform analysis by dynamically requesting for reports or views of the data. However, there are times when all the user needs is to view the actual raw data. On the Windows desktop environment, you may use the SAS data viewer. The iPhone has different capabilities such as multi touch pinch to zoom and swipe to slide the views. This can be used to view the data directly and easily. The following illustrates the steps taken to view SAS data on the iPhone.

Step 1 – Selecting Library
SAS datasets are stored in libraries which points to physical folders on the server. This is also refereed to as SAS LIBNAMEs. The main datasets screen on the iPhone contains the latest list of datasets. The first item on this list is the library that is currently being viewed.




By taping on the library right arrow labeled “datlib”, the user drills down to a list of libraries that the user has permissions to. The current selected library is identified with a check mark and colored with standard selected blue tint. The user can then select any other library can by taping on the desired library.


Step 2 – Select Data
The main dataset list displays all dataset names along with the associated label available within the selected library. This can be efficiently scrolled through by using the vertical swipe action which is standard for the iPhone interface. The user can then select the data to be viewed by taping on the right arrow or anywhere in the row of the dataset. The text size and layout of the dataset list is optimized for the user selection within the iPhone interface.

Step 3 – View Data
Upon the specific data selection, a data viewer is presented with all the rows of the data. Chunks of the data of delivered to the iPhone which can be configured. In this example, 20 rows at a time are displayed. The user can then jump to specific blocks of data using the next and previous buttons or the horizontal slider control to navigate to specific chunks of the data.

The interface has been modeled after other existing standard iPhone applications such as iPod player which has standard buttons and layout. Apple has devoted extensive research into developing a user interface that is user friendly for the mobile device optimizing object size and placement for optimal functionality. Many of the same user elements are implemented in this data viewer leveraging upon Apple’s standard to enhance user’s experience. Since users who own an iPhone are most likely familiar with the interface of an iPod to select music, they do not have to learn a new interface for selecting and viewing datasets. In the list view, the selection and viewing of datasets is analogous to playing and selecting options of each song being played on the iPod.

The code and algorithm logic that goes into developing the SAS application server and iPhone client application is extensive. This section will select one section in step 1 on libraries and describe how this is accomplished. The following steps are applied to present the list of library screen.

Step 1 – Capture Libraries
A SAS program is initiated on the server to capture all libraries available. It is then compared to the user permissions before the final list is sent to the client iPhone. The following example code is used to capture this is:

   do while(fetchobs(dsid,cnt) = 0);







libname1 = upcase(getvarc(dsid,varnum(dsid,'libname')));







if libname1 not in ('WORK','SASUSER','SASHELP','SASMSG','MAPS',







'PGMLIB','_MXI','SERLIB','DATLIB','USERS') then do;







if searchc(liblst, libname1, 1, 1, 'Y') = 0 then do;







rc = insertc(liblst,libname1,-1);







path = lowcase(getvarc(dsid, varnum(dsid, 'libpath')));







idx = index(path, '\');







path = substr(path, idx);







path = tranwrd(path, '\', '/');







rc = insertc(pathlst, path, -1);







end;







end;







cnt = cnt + 1;







end;







dsid = close(dsid);











In this case, it captures libraries from the SASHELP.VSLIB view and inserts its findings into a list to be used later.

Step 2 – Subset user Privileges
A SAS program is initiated on the server to capture all libraries available. It is compared to the user permissions access control list. BI Flash stores all the user permissions as described in the permission section above.


Step 3 – XML Library List
Once the final list of library is captured, an XML file is created capturing all the libraries as shown here.


   <Libraries>







<Library id="1">







<shortTitle>pgmlib</shortTitle>







<longTitle></longTitle>







<descriptionTitle>/myserv/lib1</descriptionTitle>







</Library>







<Library id="2">







<shortTitle>datlib</shortTitle>







<longTitle></longTitle>







<descriptionTitle>/myserv/testlib</descriptionTitle>







</Library>







<Library id="3">







<shortTitle>study</shortTitle>







<longTitle></longTitle>







<descriptionTitle>/myserv/testlib</descriptionTitle>







</Library>







...







</Libraries>





This is information is then delivered to the iPhone via a web server used to for the user selection.

Tuesday, July 7, 2009

Distributing SAS Applications on IPhone, things will never be the same... http://bit.ly/19EleG

Distributing SAS Applications on IPhone

The distribution of products and services has radically changed in recent years as it becomes more efficient for companies like Google, eBay and Amazon to take advantage of efficiencies of Internet distribution. Rather than concentrating on the top few block buster products, these companies are finding that the many niche products such as specialized advertisement on Google and esoteric books on Amazon add up to be just as significant in total sales as their supper sellers. The cost for them to distribute the large array of solutions has become profitable when efficiently delivered across the internet as compared to shrink wrap boxes sold in stores. This shift in the market is also taking shape in how software is being delivered. Apple has been at the forefront with this type of distribution with smaller unknown musical artist on its ITunes store. It is now extending that making the AppStore a new way in distributing more specialized software within this hyper efficient distribution model. This is apparent when over 50,000 applications have been made available and growing. The mobile computing platform such as the iPhone has become a potent platform to deliver useful and specialized analytics software to users. At the current moment, however, there is very little in this space for business intelligence applications on the Appstore and none for SAS. It is unquestionable that SAS can deliver sophisticated business intelligence information by delivering SAS macros coupled with its programming language to deliver tremendous analytical power. In the past, the access to these tools has been limited to power users who can write SAS code or utilize BI tools on SAS servers. BI Flash software presents examples where the power of traditional SAS programs can be delivered to users on the iPhone. This bridges the gap by delivering endless possible aggregate views of the data to users without requiring them to understand SAS programming or the details of the SAS system. This will truly liberate business intelligence to be accessed by any user allowing them to get the latest information anytime directly from a device in their pocket.

Monday, July 6, 2009

SAS iPhone App Architecture


The delivery of SAS data and reports to an iPhone application is unique and requires a different architecture as compared to traditional client server systems. This is similar to web applications in how it is delivered to a browser but in this case, the application replaces the browser. The diagram below illustrates the components of the iPhone SAS application architecture including: the iPhone application, a web server and the BI Flash server.


In this example, the iPhone application on the iPhone communicates through standard TCP/IP protocol to a web server. The web server then communicates to an application server which is actually a SAS session processing SAS programs and data. The output resulting from the SAS program is then delivered back to the iPhone in a similar way a web browser would access a web page the serer. The distinction however is that the iPhone application is not a web browser and the SAS session running on the server is more dynamic compare to a static page. The SAS data and macro program it executes may be simple and standard but facilitating the communication takes a little more effort. The request from the iPhone application and the delivery of information from the server is handled by BI Flash. This makes the experience more dynamic and delivers the full power of SAS on the server. The following steps are taken in order to facilitate the access of SAS data from an iPhone App.


Step 1 – Download iPHone Application


One of the most unique and successful aspects of the iPhone is how users can easily download an iPhone App directly from the iPhone App Store. There is a user friendly way of searching for and downloading the application to your iPhone. In this case, you can download BI Flash App which is the client component which enables the communication between the iPhone and the SAS server.


Step 2 – BI Flash Application Server

More details on this step will be explained in the “Application Server” section but an administrator would execute the BI Flash server. This server functions as a listener waiting for a request from the iPhone. Upon receipt, it would process the request similar to how you would submit SAS programs from display manager. The server would generate a SAS log and output results in XML which is then sent back to the iPhone to be viewed.


Step 3 – Connect iPhone to Application Server


Most iPhone applications have configuration settings. This allows for users to easily configure settings during setup and then use the application without further changes unless future configuration changes are needed. In order for the BI Flash example iPhone app to access SAS, the user would need to configure the following:


  1. Host Name – This the name of the server or an IP address of the SAS server.



  2. User Name – A valid user name that has been defined on the server needed during authentication



  3. Password – A user defined password to secure access


There may be other configuration options which will set the default behavior of the application but the parameters above show the minimum requirement in order to connect to a SAS server.



Step 4 – Run Application
The final step taken by the user to access SAS data is to execute the SAS macros from the iPhone. This request is initiated from the iPhone app and sent directly to the server with user selected options. The results are then returned to the iPhone displaying the most updated information on the server.

The system architecture in this example is rather simple compared to other systems that require multiple layers of middleware. This is similar to the SAS/IntrNet where users are on a web browser accessing SAS data and programs on the server through the broker and SAS application server. The difference however is that the client is not a browser, but rather a dynamic iPhone application.

The current iPhone application BI Flash is being reviewed by Apple AppStore and should be made available soon upon approval. You can view a sample video of the application which run SAS macros directly on the iPhone below.


Thursday, July 2, 2009

just received my WUSS registration mailer, ready to wussout in San Jose in September. http://bit.ly/10VDjm

Wednesday, July 1, 2009

Interesting new video on verifying controlled terminology against CDISC http://ping.fm/onUCQ