Is there a template for working with mainframe data?

Question

NOTE. Refined my question below.

I am wondering if there could be a (normal) pattern for processing request / response from older mainframe systems? In the examples below, IQ is the request and RSIQ is the response . In the first example, I request a list of all account codes, and in the second request, I request a closed date for each account code. Since they are associated only with ordinal position, it is quite easy to display data in a structured data class. Each answer in this case represents several entries.

In the second example, I request several bits of information for one record. In this case, each response is a single record and a small number of data points.

This is a message that the client sends to the server to request specific information from the database.

The inquiry message has this general format: IQ~<msg id>~A<unit#>~B<device type>~D<acct#>~F<password>~G<file>~H<hierarchicrecordpath>~J<field> **One field from many records**: Beginning with first share (ordinal zero) on Account 101 return all the Share ID fields in first message then get all Close Dates in second message. IDs and Close Dates correspond positionally within the two responses. IQ~1~A0~BVENDOR~D101~F7777~HSHARE=0~JID=ALL RSIQ~1~K0~JID=0000~JID=0003~JID=0004~JID=0005~JID=0025~JID=0050 IQ~1~A0~BVENDOR~D101~F7777~HSHARE=0~JCLOSEDATE=ALL RSIQ~1~K0~JCLOSEDATE=00000000~JCLOSEDATE=20030601~JCLOSEDATE=00000000~JCLOSEDATE=00000000~JCLOSEDATE=00000000~JCLOSEDATE=00000000 **Many fields from one record**: Using the previous requests get additional information from open shares (two examples). IQ~1~A0~BVENDOR~D101~F7777~HSHARE#0005~JCLOSEDATE~JSHARECODE~JDIVTYPE~JBALANCE~JAVAILABLEBALANCE RSIQ~1~K0~JCLOSEDATE=00000000~JSHARECODE=0~JDIVTYPE=2~JBALANCE=234567~JAVAILABLEBALANCE=234567 IQ~1~A0~BVENDOR~D101~F7777~HSHARE#0025~JCLOSEDATE~JSHARECODE~JDIVTYPE~JBALANCE~JAVAILABLEBALANCE RSIQ~1~K0~JCLOSEDATE=00000000~JSHARECODE=1~JDIVTYPE=5~JBALANCE=654321~JAVAILABLEBALANCE=654321 

REFERENCE INFORMATION . I already use the work / repository unit template in my applications. Each application deals with several data stores (SQL DB, files, web services, sockets, etc.). The idea is that each repository provides (part of the complete) data model.

My initial thinking is to create the specific calls I need in the repository, such as GetAccounts(acctId) , and send the method to the correct requests , then create an object graph from all the reponses , finally returning the object graph.

Now I'm looking for a design template to handle the internal elements of each of these methods without having to execute a ton of string commands. Replace () or StringBuilder calls. Since the maximum size of any request is 8000 characters, you can see where the ~ J fields can become quite complex. (And I'm still looking for all the possible codes that may be in the ~ J fields.)

A small example:

 public List<SymitarAccount> GetAccounts(string accountId) { var retAccounts = new List<SymitarAccount>(); // Is there a pattern to do this repetitve but ever changing task? // // Example: Mock response then handle... // // NOTE: There will be many request/response calls here, not just one! // var rsp = @"RSIQ~1~K0~JCLOSEDATE=00000000~JSHARECODE=1~JDIVTYPE=5~JBALANCE=654321~JAVAILABLEBALANCE=654321"; var response = rsp.Split(new[] {'~'}); foreach (var q in response) { if (q.StartsWith("J") && q.Contains("=")) { // get Key Value Pair // // map KVP to SymitarAccount data point (big ugly switch(){}??) // sa.Id = // KVP for ID // sa.Balanace = // KVP for BALANCE // } retAccounts.Add(sa); } return retAccounts; } 

Any thoughts or ideas?

NOTE. I am using C # (last one).

---

APPENDIX No. 1:

 public List<SymitarAccount> GetAccounts(string accountId) { var retAccounts = new List<SymitarAccount>(); // Get all account IDs... var response = UnitOfWork.SendMessage("IQ~1~A0~BVENDOR~D101~F7777~HSHARE=0~JID=ALL"); ParseResponse(response, ref retAccounts); // Get all account close dates (00000000 means it is open)... response = UnitOfWork.SendMessage("IQ~1~A0~BVENDOR~D101~F7777~HSHARE=0~JCLOSEDATE=ALL"); ParseResponse(response, ref retAccounts); // Get extra info for all OPEN accounts... foreach (var account in retAccounts.Where(a => !a.IsClosed)) { var request = "IQ~1~A0~BVENDOR~D101~F7777~HSHARE#[acct]~JCLOSEDATE~JSHARECODE~JDIVTYPE~JBALANCE~JAVAILABLEBALANCE"; request = request.Replace("[acct]", account.Id.ToString("0000")); response = UnitOfWork.SendMessage(request); ParseResponse(response, ref retAccounts, account.Id); } return retAccounts; } private void ParseResponse(string response, ref List<SymitarAccount> accountList, int? id = null) { var list = response.Split(new[] {'~'}); var index = 0; var chain = new ChainInquiryAccountInfo(); var parser = chain.Parser; foreach (var q in list.Where(q => q.StartsWith("J"))) // && q.Contains("="))) { if (accountList.Count < index || accountList[index] == null) accountList.Add(new SymitarAccount {PositionalIndex = index}); var val = q.Split(new[] {'='}); if ((id.HasValue && accountList[index].Id == id.Value) || !id.HasValue) accountList[index] = parser.Parse(val, accountList[index]); index++; } } 

Answer 1

An example is, they say, deserialization, not XML or JSON, but from some kind of custom text format. You can go in the direction of other serializers, then when you create classes and bind their fields to help serialize / deserialize. This can be called the Attributed Serializer pattern, which I consider ...

Let us create some custom attribute to annotate serialized classes:

 [AttributeUsage(AttributeTargets.All, Inherited = false, AllowMultiple = true)] sealed class SomeDataFormatAttribute : Attribute { readonly string name; // This is a positional argument public SomeDataFormatAttribute(string positionalString) { this.name = positionalString; } public string Name { get { return name; } } } 

and then you can describe your data objects as:

 class SymitarAccount { [SomeDataFormat("CLOSEDATE")] public string CloseDate; [SomeDataFormat("SHARECODE")] public int ShareCode; } 

Now you need a Reflection-based serializer / deserializer that will match the attribute fields with the string. Here I use regular expressions (and without error checking for simplicity):

 public class SomeDataFormatDeserializer { public static T Deserlize<T>(string str) where T : new() { var result = new T(); var pattern = @"RSIQ~1~K0(?:~J(\w+=\d+))*"; var match = Regex.Match(str, pattern); // Get fields of type T var fields = typeof(T).GetFields(BindingFlags.Public | BindingFlags.Instance); foreach (var field in fields) { // Get out custom attribute of this field (might return null) var attr = field.GetCustomAttribute(typeof(SomeDataFormatAttribute)) as SomeDataFormatAttribute; // Find regex capture that starts with attributed name (might return null) var capture = match.Groups[1].Captures .Cast<Capture>() .FirstOrDefault(c => c.Value.StartsWith(attr.Name)); if (capture != null) { var stringValue = capture.Value.Split('=').Last(); // Convert string to the proper type (like int) var value = Convert.ChangeType(stringValue, field.FieldType); field.SetValue(result, value); } } return result; } } 

And then you can use it as simple as:

 public static List<SymitarAccount> GetAccounts(string accountId) { var retAccounts = new List<SymitarAccount>(); var responses = new List<string>() { @"RSIQ~1~K0~JCLOSEDATE=00000000~JSHARECODE=1" }; foreach (var response in responses) { var account = SomeDataFormatDeserializer.Deserlize<SymitarAccount>(response); retAccounts.Add(account); } return retAccounts; } 

Note. SomeDataFormatDeserializer is written for clarity, not performance. Of course, it can be optimized (for example, caching GetFields , etc.)

Answer 2

MY DECISION:

Attribute Definition:

 [AttributeUsage(AttributeTargets.All, Inherited = false, AllowMultiple = true)] internal sealed class SymitarInquiryDataFormatAttribute : Attribute { private readonly string _name; // This is a positional argument public SymitarInquiryDataFormatAttribute(string positionalString) { this._name = positionalString; } public string Name { get { return _name; } } } 

Data class:

 [Serializable] public class SymitarAccount { public int PositionalIndex; public bool IsClosed{get { return CloseDate.HasValue; }} [SymitarInquiryDataFormatAttribute("ID")] public int Id; [SymitarInquiryDataFormatAttribute("CLOSEDATE")] public DateTime? CloseDate; [SymitarInquiryDataFormatAttribute("DIVTYPE")] public int DivType; [SymitarInquiryDataFormatAttribute("BALANCE")] public decimal Balance; [SymitarInquiryDataFormatAttribute("AVAILABLEBALANCE")] public decimal AvailableBalance; } 

Extensions:

 public static class ExtensionSymitar { public static List<string> ValueList(this string source, string fieldType) { var list = source.Split('~').ToList(); return list.Where(a => a.StartsWith(fieldType)).ToList(); } public static string KeyValuePairs(this string source, string fieldType) { return source.ValueList(fieldType).Aggregate(string.Empty, (current, j) => string.Format("{0}~{1}", current, j)); } public static bool IsMultiRecord(this string source, string fieldType) { return source.ValueList(fieldType) .Select(q => new Regex(Regex.Escape(q.Split('=').First())).Matches(source).Count > 1).First(); } public static int ParseInt(this string val, string keyName) { int newValue; if (!int.TryParse(val, out newValue)) throw new Exception("Could not parse " + keyName + " as an integer!"); return newValue; } public static decimal ParseMoney(this string val, string keyName) { decimal newValue; if (!decimal.TryParse(val, out newValue)) throw new Exception("Could not parse " + keyName + " as a money amount!"); return newValue; } public static DateTime? ParseDate(this string val, string keyName) { if (val.Equals("00000000")) return null; var year = val.Substring(0, 4).ToInt(); var mon = val.Substring(4, 2).ToInt(); var day = val.Substring(6, 2).ToInt(); if (year <= 1800 || year >= 2200 || mon < 1 || mon > 12 || day < 1 || day > 31) throw new Exception("Could not parse " + keyName + " as a date!"); return new DateTime(year, mon, day); } } 

deserializer:

 public class SymitarInquiryDeserializer { /// <summary> /// Deserializes a string of J field key value pairs /// </summary> /// <param name="str">The request or response string</param> /// <param name="source">Optional: Use this if you are adding data to the source object</param> /// <param name="fieldName">Optional: Use this if you are only populating a single property and know what it is</param> /// <typeparam name="T">The target class type to populate</typeparam> /// <returns>New T Object or optional Source Object</returns> public static T DeserializeFieldJ<T>(string str, T source = null, string fieldName = null) where T : class, new() { var result = source ?? new T(); const string pattern = @"(?:~J(\w+=\d+))*"; var match = Regex.Match(str, pattern); // Get fields of type T var fields = typeof(T).GetFields(BindingFlags.Public | BindingFlags.Instance).ToList(); if (fieldName != null && fieldName.StartsWith("J")) fieldName = fieldName.Replace("J", ""); if (!fieldName.IsNullOrEmpty()) { var field = fields.FirstOrDefault(a => a.Name.Equals(fieldName, StringComparison.CurrentCultureIgnoreCase)); var stringValue = GetValue(field, match); if (!stringValue.IsNullOrEmpty()) SetProperty(field, stringValue, result); } else { foreach (var field in fields) { var stringValue = GetValue(field, match); if(!stringValue.IsNullOrEmpty()) SetProperty(field, stringValue, result); } } return result; } private static string GetValue(FieldInfo field, Match match) { // Get out custom attribute of this field (might return null) var attr = field.GetCustomAttribute(typeof(SymitarInquiryDataFormatAttribute)) as SymitarInquiryDataFormatAttribute; if (attr == null) return null; // Find regex capture that starts with attributed name (might return null) var capture = match.Groups[1] .Captures .Cast<Capture>() .FirstOrDefault(c => c.Value.StartsWith(attr.Name, StringComparison.CurrentCultureIgnoreCase)); return capture == null ? null : capture.Value.Split('=').Last(); } private static void SetProperty<T>(FieldInfo field, string stringValue, T result) { // Convert string to the proper type (like int) if (field.FieldType.FullName.Contains("Int32")) field.SetValue(result, stringValue.ParseInt(field.Name)); else if (field.FieldType.FullName.Contains("Decimal")) field.SetValue(result, stringValue.ParseMoney(field.Name)); else if (field.FieldType.FullName.Contains("DateTime")) field.SetValue(result, stringValue.ParseDate(field.Name)); else { var value = Convert.ChangeType(stringValue, field.FieldType); field.SetValue(result, value); } } } 

Finally, in my repository:

 public List<SymitarAccount> GetAccounts(string accountId) { var accountList = new List<SymitarAccount>(); // build request, get response, parse it... var request = "IQ~1~A20424~BAUTOPAY~D101~F7777~HSHARE=0~JID=ALL"; var response = UnitOfWork.SendMessage(request); ParseResponse(response, ref accountList); foreach (var account in accountList.Where(a => a.IsClosed == false)) { request = "IQ~1~A20424~BAUTOPAY~D101~F7777~HSHARE#" + account.Id.ToString("0000") + "~JCLOSEDATE~JSHARECODE~JDIVTYPE~JBALANCE~JAVAILABLEBALANCE"; response = UnitOfWork.SendMessage(request); ParseResponse(response, ref accountList, account.Id); } return accountList; } private void ParseResponse(string response, ref List<SymitarAccount> accountList, int? id = null) { var index = 0; var list = response.ValueList(fieldType: "J"); var jString = response.KeyValuePairs(fieldType: "J"); var isMultiRecord = response.IsMultiRecord(fieldType: "J"); SymitarAccount account; if (isMultiRecord && !id.HasValue) foreach (var q in list.Where(a => a.StartsWith("J"))) { // Add object if we don't yet have it in the collection... if (accountList.Count <= index) accountList.Add(new SymitarAccount { PositionalIndex = index }); account = accountList.FirstOrDefault(a => a.PositionalIndex == index); SymitarInquiryDeserializer.DeserializeFieldJ("~" + q, account, q.Split('=').First()); index++; } else if(id.HasValue) { account = accountList.FirstOrDefault(a => a.Id == id.Value); SymitarInquiryDeserializer.DeserializeFieldJ(jString, account); } } 

The difference between the two calls to ParseResponse is that in the first case I ask you to return several records (only one data property!), And in the second case I ask for additional data properties for one record to be sent back.