thinking-out-loud

Slightly updated xmlserializer section handler

I have been looking at the configuration for TexMex, so the XmlSerializerSectionHandler came back to mind. Shortly after I posted my original mod to Craig Andera's XmlSerializerSectionHandler, I made a minor tweak to it to allow the configuration data to be stored in a file external (and relative) to the config file itself. I hadn't posted the diff, but you might find it useful, so it follows here. The  discovery bit for me was the [it seems, somewhat oddly-placed]

ConfigurationException.GetXmlNodeFilename(sectionNode);

This allows you to get the FileName from the underlying, undocumented IConfigXmlNode (RIYF). Here is how the configuration file looks, and the contents of the referenced file looks just like it did in the older version:

<configuration>
  <configSections>
    <section name='NodeConfiguration' type='Improving.Configuration.XmlSerializerSectionHandler, Improving.Utilities' />
    <section name='TestData' type='Improving.Configuration.XmlSerializerSectionHandler, Improving.Utilities'/>
  </configSections>

    <NodeConfiguration file='test.config'/>
    <TestData file="TestData\Tests.xml"/>
    ...
</configuration>

The code follows:

using System; 
using System.Configuration; 
using System.IO;
using System.Text;
using System.Xml; 
using System.Xml.Serialization; 
using System.Xml.XPath; 

namespace Improving.Configuration
{
  public class XmlSerializerSectionHandler : IConfigurationSectionHandler 
  { 
    public XmlSerializerSectionHandler()
    {
    }

    public static string QualifyPath(XmlNode sectionNode, string file)
    {
      if(Path.IsPathRooted(file))
      {
        return file;
      }
      string configPath=ConfigurationException.GetXmlNodeFilename(sectionNode);
      string path = Path.GetDirectoryName(configPath);
      return Path.Combine(path, file);
    }

    public XmlNode LoadFile(string fileName)
    {
      try
      {
        XmlDocument document = new XmlDocument();
        document.Load(fileName);
        return document.DocumentElement;
      }
      catch(Exception e)
      {
        throw new Exception("Unable to load configuration from file " + fileName, e);
      }
    }


    public object Create( object parent, object configContext, System.Xml.XmlNode section) 
    { 
      if(section == null)
        throw new Exception("Could not configure section because the node was null");

      XmlNode sectionNode = section;
      if(section.Attributes["file"] != null)
      {
        sectionNode = LoadFile(QualifyPath(section, section.Attributes["file"].Value));
      }
      return CreateFromSection(sectionNode);
    }

    public object CreateFromSection(XmlNode section)
    {
      string typename = section.Attributes["type"].Value;
      if(typename == null)
        throw new Exception("You must specify a type name for the configuration to load");
      
      Type t = Type.GetType ( typename, true ); 
      
      XmlNodeList list  = section.SelectNodes("./IncludedTypes/Type/text()");
      XmlNode configNode = section.SelectSingleNode("child::*[not(local-name() = 'IncludedTypes')]");
      Type[] types = new Type[list.Count];

      for(int i = 0; i < list.Count; i++)
      {
        types[i] = Type.GetType(list[i].Value, true);
      }
      XmlSerializer ser = new XmlSerializer (t, new XmlAttributeOverrides(), types, new XmlRootAttribute(configNode.LocalName), null);
      return ser.Deserialize(new XmlNodeReader(configNode));
    }
  }
}

* Reflector is your friend

What's the difference with regression testing?

Many winters ago I was a tester, and then a maintenance developer doing regression testing.  While maintaining a printer driver which produced binary printer data, we used a crafty technique (at least it seemed so then) to test when some fix or enhancement we made to the driver caused adverse effects to the output.  That technique I’ll call difference testing.  Here’s how it worked:

• Run a bunch of documents through the printer driver and capture the output
• Make a change to the driver
• Run the same bunch of documents through the printer driver and test each output against the previous output from the same document
• When a difference is detected, visually compare the printed output to determine if it’s a good change or a bad change
• And, (this is important) update the baseline output when the difference is good

I’ve recently had the opportunity to use this technique again, regaining an appreciation for its effectiveness.  I’m just wrapping up the development of a pricing engine that can accept product orders in an XML format and output the same format with prices attached to each line item and sub-item.    The testing challenge here is to check the dazzling number of combinations of options and the interplay of business rules applied to those combinations via the prices they produce.  So, here’s what we did:

• Order a bunch of products with specific combinations of options designed to test edge cases of pricing rules, capturing the XML containing product orders and prices as baseline data
• Make a change to the pricing engine
• Iterate over the previously captured XML product orders and generate pricing again, comparing each file output to the corresponding baseline output
• When a difference is detected, visually or programatically compare the XML elements
• And, (this is important) update the baseline output when the difference is good
• Also, as with any regression suite, add new test cases to the suite when a previously untested error condition is discovered

I originally planned to parse XML elements and spit out some specific error statements about the differences.  But, my client suggested just comparing the files with a visual diff, thus tossing the data in question up in front of the user with all the context required to determine the real error.  Way to leverage the readability of XML!

Feel free to use this technique anytime – no charge.  Oh, and make sure the XML is generated with <new lines> or your diff will just show one long line in error.

SQL Server 2005

Last week I attended the Microsoft Architect Council where several speakers presented new/updated MS products built on SQL Server 2005.  Here are the highlights.

SQL Server Mobile – Kevin Collins

• Syncs w/ SQL Server  using Merge Replication w/ column tracking (e.g. pictures don’t sync every time a row changes) – actually a feature of SQL Svr 2005 and usable by SQL Express
• Syncs over HTTP/S with compression
• Multi-user (more than one app at a time, background sync)
• Based on .Net Compact Framework 2.0, ADO.Net 2.0
• Histograms for indexed columns – improved performance over SQL Server CE
• SQLCEResultSet – direct cursor mechanism – don’t use DataSet b/c causes double/triple buffering
• Create large DBs on desktop & dist to devices via file rather than long sync for each client

SQL Server Integration Server (www.sqlis.com) – Donald Farmer

• Collect/Analyze/Report
• Motivations: 24/7 business, greater volumes of data (e-commerce, RFID, call-center)
• Before Integration Services – doesn’t scale
  • separate, staged operations for different sources
  • different applications for each type of data
  • then warehousing, then data mining
• With Integration Service
  • data transformation, cleansing, merging, mining all done in single auditable process
  • warehouse separately for auditing
  • gen reports and access via mobile out of merged data, parallel w/ warehousing
• Components: sources, transformations, destinations
  • Source Adapters: .Net code writes app data into internal buffer format (effectively a row set)
  • Transformations: row set operations (union, intersection, sorting, etc.) and *fuzzy grouping*
  • Destination Adapters: translate buffer format to external apps
• Performance: proven to scale to 117 GB/Hr, 64bit allows operations on enormous datasets (e.g. MSN click-tracking)
• Case Study – Barnes & Noble: 1.6TB raw data => 50% less dev time vs. Oracle, 400% improved perf, more meaningful, up-to-date info for stock levels
• Graphical description of data processing flow, transformations, output, visualization w/ drill-down to composite flow charts
• Interactive selection/data profiling
• Note: isn’t designed for streaming information such as live click-stream.  Can use a polling mode and get small batches.  Also can pull data from SQL server using a built-in SQL Server adapter w/o .Net coding.

SQL Server 2005 Reporting Services - Brian Walker, Program Manager

• Uses ADO.Net driver and any DB supported by that
• Nice drag-n-drop interface to building a queries and setting up filtering, ditto for visual design of reports
• New data-picker control in next beta
• Sorting can be done with nested groups – re-sorts with a click of the sort control appearing on the report
• Headers stay on page with scrolling through the report
• Improved expression editor with syntax checking using a .Net expression language (defaults to VB.Net)
• SP2 will contain a set of WebParts for use with Sharepoint (ReportExplorer and ReportViewer, which can be linked together automatically)
• Visual Studio 2005 will have a report viewer control so apps can connect to Reporting Server or local databases (doesn’t require SQL Server)
• Direct-print (via ActiveX control) where server gens report and browser routes to printer (adjustable margins, etc. during run-time)
• ReportBuilder: design tool for business user
  • click-once app running as low-privileged user
  • template-based with drag-n-drop table/field assignment
  • auto-totaling and sub-totaling is easy
  • the “model schema” is sent to server and translated into a query by the server
  • Model Designer tool can public models to the Report Server
  • Part of SQL Server license but require Report Server running

TexMex Configuration Thoughts

I cannot tell you how many times I have been using typepad to enter a blog entry, and have accidentally hit a link in some other random application that took over the existing instance of IE and caused me to lose my post because I hadn't saved it as a draft. This is one of those posts. I had a nice explanation of where I was heading with TexMex, its configuration, and it's configuration's use of XInclude.Net. You, however, do not get to see that post just now, because I have to complete redo it. Serves me right for not using a different blogging client. Expect to see this message title again soon.

D'oh.

Print Formatting with CSS and XML

I enjoyed this article on print formatting using CSS+XML.

TexMex 1.7 -- link doesn't 404 anymore

In my previous, sleepy state, I apparently failed either to upload the zip file to the server or to wait for it to complete. In my new, sleepier state, I can't remember which. Regardless, it is now uploaded.

Have a nice day!

TexMex 1.7, WS-MetadataExchange, three-alarm spicy!

My poor, sweet daughter is teething--four molars. Ouch. She had a hard time sleeping last night. I sleep very lightly as it is, so I also had a hard time sleeping. One the old ticker starts to ticking, I can't stop it, and I usually have to get up and do something. Now, if I Iie there, I tend to have a million amazingly wonderful ideas escape from my brain. As soon as the body goes vertical, though, they all go away. A shame.

So anyway, at least I had some time for hacking. I didn't get to the configuration, but I did a lot of clean up that I wanted to get done. I do remember one thing from when I was lying down and thinking. I thought a lot about how I would want to code around the fact that we as an industry choose to conflate semantic chunking (i.e. namespaces) and versioning into one thing, namespaces. Sometimes that makes sense, because the new version is semantically distinct enough, but for nominal uses of Soap and WS-Addressing, the namespace differences are just annoying.

System.Xml.Serialization will let you get away with a lot. System.Xml.XPath... not so much. What I would really like to be able to do is to say "Please consider these two namespaces, urn:a and urn:b to be synonomous, and pretend that they are the latter," and I think that could be relatively easy to do with a custom xml reader, but I haven't done anything about it yet. Maybe soon.

WS-Addressing+Soap=Soap

So the question is this: Now that w3 owns ws-addressing and soap, shouldn't soap 1.3 be soap 1.2 + ws-addressing internalized under the soap xmlns umbrella?

WS-Addressing, refprops-b-gone

Via Mark Baker. The WS-Addressing group agrees to remove the ReferenceProperties. Nice. I was also happy when they closed the following, even if the motivation is just a standardization issue. It is helpful in bootstrapping, I suppose, but it just didn't feel right.

The WS-A Submission normatively references WS-Policy, which is not a recognized standard.

Resolution 2004-10-26 - We will not reference the WS-Policy spec.

I wonder whether this particular bit of the design is going to sneak its way back in as a part of the ws-policy spec? I also wonder what impact it has on the vendors that are implementing these specs - indigo, anyone?

Note also, that Mark got a 98 on the Nerd test.

Mine:

I think that one of the questions should have been:

The url of your blog:

1. Contains your name.
2. Is intended to be funny.
3. Has a science, science fiction, or technology reference.
4. Has a math reference.
5. Contains the date it was created, because you are religious about cool uris.

The latter of which would, of course, push Mark up into the 99th percentile, while the fourth would have given me a little boost.

TexMex, WS-MetadataExchange

A new cut. I have started to refactor the serialization into a object model for data transfer and to address the client side a bit. The client side is currently coupled to WSE 2.0 sp2 for convenience, but as the aforementioned object model gets fleshed out more, I will remove the dependency to allow for soap 1.1/1.2 and WS-Addressing 0304/0804. I am going to look into Dilip's requests now.